Documentation only: no algorithm, argument or fitted result changes.
sblp complexity, excess levels in ivb and gmb, the
fast_mdlp, mdlp and cm criteria, PAVA cost, sketch memory, ldb
bandwidth) and fixed the WoE convention: events are the numerator.DESCRIPTION and ?obwoe: 28 algorithms (37 implementations); no
genetic algorithms or MILP solver. The algorithm tables say what each engine
does (bb, mdlp, milp, mblp, mrblp and oslp merge greedily), and
algorithm = "auto" is a fixed rule by target type, not a search for the
highest IV.ob_numerical_cm(): chi_merge_threshold does not change the result,
because bins are merged down to max_bins before the significance test is
reached. Documented and covered by a test; ob_categorical_cm() does use it
as its stopping rule.ldb lets min_bins override bin_cutoff; jedi
leaves the WoE of an empty cell unclamped (about -20 to -22); ir caps its
pre-bins at max_bins; lpdb ignores polynomial_degree.control.obwoe() and vignette("introduction"): the default
bin_cutoff = 0.05 can cost IV on categorical features with many levels,
because the rare levels share one bin. The vignette shows how to compare
lower cutoffs on held-out data. The default is unchanged.?obwoe, ?summary.obwoe, vignette("introduction")):
described as a rule of thumb; the 0.50 band is no longer credited to
Siddiqi or called "almost always leakage".inst/references.bib corrected, and ?obwoe cites the
optimal binning method as the arXiv preprint it is. The DOI given before
belongs to an unrelated paper.Crashes, hangs and out-of-bounds reads fixed; one NA/Inf rule for all numerical binners; several algorithms now do what they document. Many fitted bins change: in every such case the old result broke the documented algorithm or contract. No argument was added or removed and every returned list keeps its names, types and order.
ob_numerical_cm() segfaulted on a range that
overflows (−1e308, 1e308); ob_numerical_fetb() and
ob_categorical_dp() crashed on max_n_prebins = 0; ob_numerical_ir()
aborted on empty input; ob_apply_woe_num() segfaulted on a one-bin fit
(no cut points); ob_cutpoints_cat() segfaulted on character(0).ob_categorical_jedi_mwoe() with max_bins = 1,
ob_numerical_udt() when 3 <= distinct values < min_bins, and
ob_categorical_swb() when min_bins > max_bins.ob_numerical_fast_mdlp() (monotonic merge
indexed merged vectors with pre-merge positions; sum(count) could exceed
n threefold), ob_categorical_ivb() (feature shorter than target),
ob_cutpoints_num() (short target), ob_preprocess() (IQR with n <= 2),
and undefined behavior in ob_apply_woe_cat() (trim("")), in
std::sort over NaN (fetb, mdlp), in the KDE grid on an infinite span
(ldb, lpdb) and in int overflow of count products (cm numerical and
categorical, shared chi-square helper) once bins exceed ~46k rows.ob_preprocess() overwrote the caller's feature vector in place and
ob_cutpoints_num() sorted the caller's cutpoints in place.The 21 numerical binners disagreed: six stopped on an NA feature (including
the default jedi, so obwoe() failed on any numeric feature with a missing
value), eleven dropped it, fetb sorted NaN into a bin and udt gave it a
bin of its own. They now all follow one rule, implemented in C++ so that the
ob_numerical_*() wrappers and obwoe() agree:
ob_numerical_udt() keeps its documented
trailing "NA" bin, which now holds NaN only.)jedi, jedi_mwoe, lpdb, ewb, ir, kmb,
fast_mdlp, mdlp and mob are gone.Targets are now validated instead of truncated: every wrapper used
as.integer(target), which turned 0.7 into class 0 and "yes" into NA. Integer,
whole-valued double, logical, and factor/character holding whole numbers are
accepted; anything else is an error.
cm, both types): the critical value came from a table keyed
by confidence instead of significance, so the default
chi_merge_threshold = 0.05 gave 0.004 instead of 3.841 and the
significance rule almost never merged; it is now qchisq(1 - alpha, 1)
(Kerber, 1992). The Yates correction is clamped at zero as in
chisq.test(). The numerical chi-square cache returned stale values
after merges (its invalidation was a no-op); it is replaced by an
incrementally updated vector of adjacent-pair statistics, so merges now
match a chisq.test()-based ChiMerge. The categorical default now merges
more.fetb, both types): the merge criterion was the
point probability of the observed table, not a p-value. It is now the
two-sided Fisher p-value, matching fisher.test(). The numerical loop
could stop on IV convergence with more than max_bins bins.fast_mdlp's acceptance test omitted the
k1 E(S1) + k2 E(S2) term of Fayyad & Irani (1993); it is now included.
When max_bins binds, splits are kept best-first by gain instead of
dropping the rightmost ones. mdlp's documentation of its merge cost and
bin format was corrected.ivb): a banded search excluded valid partitions, so
the result was not optimal; the monotonic repair could drop the last bin's
categories. DP (dp, categorical): recovered counts by splitting
labels on the separator and rejected valid partitions with a bogus
monotonicity test.jedi, jedi_mwoe, mba and gmb (categorical).bb, dmiv, dp,
ldb, lpdb, mblp, mdlp, mob, mrblp, oslp (numerical) and
milp, sblp (categorical), mostly by refreshing WoE after merges and
re-checking after the max_bins reduction.max_bins / min_bins not honored: ldb, lpdb, dmiv
(categorical: 2000 bins for max_bins = 5), fetb, sketch, sblp,
mba, sab, swb, udt and fetb/gmb/ivb on high-cardinality
features, which collapsed to a single bin with IV 0.ob_numerical_udt() returned Gini as
2·AUC instead of 2·AUC − 1 and merged on an IV that was always 0;
ob_numerical_kmb() reported grid centroids instead of bin means;
ob_categorical_mob() gave every initial bin a count of 1;
ob_categorical_sketch() returned count-min estimates as bin counts;
ldb/lpdb WoE used the pre-bin count in its Laplace denominator.oslp and ubsd labeled bins [a;b) while assigning
(a;b]; ewb, numerical sketch and ubsd produced empty bins or
labels that excluded the minimum.ob_categorical_sab() stopped after one iteration whenever the first
move was rejected, and now draws from R's RNG only, so results are
reproducible across platforms under set.seed().ob_categorical_milp() merged the two WoE extremes together; its merge
phase now combines WoE-adjacent bins with the least IV loss.min_bins rarest categories in jedi,
jedi_mwoe and categorical udt; it keeps the most frequent ones.dmiv pooled bin was labeled "PREBIN_OTHER", so its
categories could not be scored; it now lists them.obwoe_apply() is vectorized (match() instead of two closures per
row): 25 to 50 times faster on wide or long data. The scorecard applies
the binning once, to the model variables only (predict() about 40 times
faster)."" was lost when splitting labels, and was scored
as missing by obwoe_apply(), obwoe_sql() and bake().step_obwoe() ignored control = list(bin_separator = ...);
obwoe_sql() and obwoe_select() now default to the fitted separator.obwoe_apply(), as in obwoe_sql() and ob_apply_woe_cat().obwoe_gains() sorted a numeric grouping column as text (KS 25.7%
instead of 50%); obwoe_gains_variable() pooled distinct WoE values that
agree to six decimals.ob_apply_woe_cat() now scores NA with the fitted missing-value bin, as
obwoe_apply() and the SQL do, and matches categories in O(n).udt feature go to its "NA" bin in obwoe_apply()
and in the SQL (every row used to get the mean WoE).obwoe_apply() on a 0-row frame, obwoe_select() on an infinite WoE and
obwoe_psi() with NA or repeated levels no longer fail.obcorr(): the distance correlation was wrong (a column against itself
gave 1.69); it is now Székely's V-statistic in O(n log n). Ties are exact
(results were scale-dependent), Pearson is two-pass, Spearman uses
pairwise-complete observations. Kendall and Hoeffding are O(n log n):
about 500 and 300 times faster at n = 10^4.ob_check_distincts() compared numbers through six-decimal strings.obcorr(threads = n) no longer warns on a build without OpenMP (the
macOS default); threads is ignored there, as documented."obwoe" scorecard engine's logistic regression is now a plain C++
Newton-IRLS (glm's convergence rule); it no longer drops standard errors
of well-conditioned small-scale designs and reports the real iteration
count.Median speed-ups at n = 10^6 unless stated: mdlp 11x, mob 24x, oslp
28x, ubsd 75x, dmiv 7x, ewb 4x, kmb 4x, jedi (the default) 2.5x,
fast_mdlp 2.6x (24x on alternating classes); categorical with 2000 levels:
milp 190x, mob 21x, jedi 19x, jedi_mwoe 14x; dmiv with 3000 levels
160x; swb/sketch/udt with 5000 levels 175-600x; ob_apply_woe_cat()
270x; ob_preprocess(method = "grubbs") 65x.
RcppEigen and RcppNumerical are no longer needed (LinkingTo: Rcpp
only), so the package keeps building when RcppEigen moves to Eigen 5
(1.13.x uses MappedSparseMatrix, which Eigen 5 removed). Their headers
emitted several hundred -Wignored-attributes warnings on every source
install; the package now compiles without a single warning under gcc
(-Wall -Wextra -Wpedantic -Wshadow -Wconversion) and clang (-Wall -Wextra -Wpedantic -Wshadow). The unused BLAS/LAPACK link flags are gone
too.dmiv, bb).Both were long known and patched downstream by fisiv; they now live where
the defect is.
.ob_score_metrics() ignored the scaling direction. The AUC was
always mirrored (1 - AUC_event), which is right for a
higher_is_safer score and reports the complement for
higher_is_riskier -- AUC 0.05 next to a KS of 0.78 in the
01_Model_Summary sheet. obwoe_scorecard() now passes
scaling$direction and the mirror applies only when the score is
high-for-safe. Gini follows. KS was never affected.
07_Score_Gains$mean_score was hard-coded NA. obwoe_report()
now fills it from the frozen training bands (new band_breaks element
of the obwoe_scorecard object; recomputed from the training score for
objects saved by earlier versions). Rows are matched by band label, so a
label that does not reproduce yields NA rather than a shifted number.
Single-defect release, and the defect is in the test suite rather than in the
package. The CRAN checks for 1.13.4 were OK on all thirteen regular flavours;
the additional noLD check -- x86_64 Linux with R-devel configured
--disable-long-double -- reported two test failures.
tests/testthat/test-obwoe-sql.R failed on noLD at the boundary
check "boundaries hold for cut points that need full double precision":
the observation sitting exactly on the first cut point was reported in
bin 2 rather than bin 1, twice.
The generated SQL was correct. obwoe_sql() wrote the cut point
-1.3003598122031599 with all seventeen significant digits it needs, and
any conforming reader -- every database this SQL targets included -- parses
that string back to exactly the fitted double.
The fault was in the miniature CASE evaluator the tests use to score the
generated SQL. It read each literal back with as.numeric(), and
as.numeric() is not a conforming reader on a build without long
double. R accumulates the decimal digits of a string in LDOUBLE; where
that is no wider than a double, a literal carrying more than fifteen
digits can come back a few ULP from the double it names. There
-1.3003598122031599 was read one ULP low, so the observation equal to the
cut point failed x <= -1.3003598122031599 and fell through to the next
branch. The evaluator, not the SQL, put it in the wrong bin.
The evaluator now decodes a literal without trusting as.numeric() with
anything it cannot be shown to parse exactly. Digits below 2^53 with at
most 22 decimals are read as m / 10^nd, one correctly rounded division of
two exactly representable doubles. Otherwise sprintf() -- which hands the
rendering to the C library, correctly rounded on every platform -- confirms
as.numeric()'s answer by printing it back, and where it does not print
back, the neighbouring doubles are bracketed around the literal by exact
decimal comparison and the nearer of the two wins.
This also removes a smaller inaccuracy in the other direction: where
LDOUBLE is 80 bits, as on x86_64, as.numeric() can double-round a
sixteen-digit literal and disagree with the database over a literal the
package considers valid.
A regression test was added for the evaluator's reader itself, over
3,600 values spanning 1e-9 to 1e9: the literal obwoe_sql() writes must
decode back to the identical double. The boundary tests are only as
trustworthy as that reader, and on noLD this is what fails first if it
regresses.
No package code changed, so no generated SQL, no fitted binning and no API changed with this release.
Single-defect release. The CRAN checks for 1.13.3 failed on
r-release-macos-arm64 and r-oldrel-macos-arm64 only; every other flavour,
macOS on x86_64 included, was clean.
obwoe_sql() could write a cut point one bit away from the fitted
value on Apple silicon, moving any observation that sits exactly on
that boundary into the next bin. The generated SQL was internally
consistent and gave no warning; it simply scored those rows with the
wrong bin's Weight of Evidence.
The literal writer searched for the shortest decimal string that parses
back to the identical double, checking each candidate with as.numeric()
before emitting it. That check is not sound on aarch64. R accumulates
the decimal digits of a string in LDOUBLE, which on aarch64 macOS is no
wider than a double, so beyond about fifteen digits as.numeric() can
land one bit from the nearest double -- rejecting a literal that does round
trip and accepting one that does not. With every candidate rejected, the
search fell through to a fallback that wrote fewer digits than the value
needs: the cut point -0.13964785691628961 came out as
-0.13964785691629, which is the smaller number, so an observation equal
to the cut point failed x <= -0.13964785691629 and fell one bin up.
The search now decides for itself. A candidate with nd decimals is
checked as m / 10^nd, where m is its digits read as an integer: while
m is below 2^53 and nd is at most 22, both operands are exact, so IEEE
754 gives the correctly rounded quotient -- the nearest double to the
candidate -- on every platform R runs on. Candidates outside those bounds
are not judged at all; the value falls back to seventeen significant
digits, the width that identifies a double uniquely. as.numeric() still
has to agree before a candidate is accepted, not to decide the question but
so that a literal R itself reads back as a different double is never
written into an audit artifact.
Cut points that are exact in binary -- integers, halves, quarters -- still read short. About five per cent of values now carry one more digit than they did in 1.13.3, being those the new check declines to judge.
digits now means decimal places, as documented. It rounded to the
requested number of places and then formatted the result with R's default
seven significant digits, so digits = 8 on 1234.5678901234 emitted
1234.568 rather than 1234.56789012. Values needing at most seven
significant digits -- the common case, and the one the test suite covered
-- were unaffected.
macos-latest is back in the GitHub Actions check matrix. It had been
disabled while infer had no ARM64 binary; that binary is back, and the
architecture is the one this release repairs.
Three regression tests were added to tests/testthat/test-obwoe-sql.R: a bulk
round trip over 3,500 values, a boundary check on cut points taken from
continuous data, and a diagnostic that spells out a one-bit disagreement
between the SQL and obwoe_apply(), which waldo reports only as "don't know
how to show the difference". The existing boundary test used integer-valued cut
points, which any literal writer renders exactly, and so could not catch this.
Bug-fix release from a third internal audit, covering the WoE/IV return
contract, the converged flag and the max_bins constraint across all 37
algorithm/type combinations. Every item below was reproduced before the fix
and re-verified after it. The regression suite gained
tests/testthat/test-audit-regressions.R, which fails on 1.13.2.
mdlp (numerical), gmb and fetb (categorical) now honour
max_bins. All three stopped on their own criterion and never re-checked
the cap, so max_bins = 5 returned 18, 11 and 10 bins respectively --
silently, with no warning. All three roxygen blocks already documented
max_bins as a hard constraint, so the code was wrong, not the
documentation.
These three algorithms now produce different bins. For mdlp the
returned partition is no longer the unconstrained MDL optimum when the cap
binds: merging continues past the MDL stopping point, each step taking the
pair with the smallest increase in MDL cost. min_bins is never violated
to satisfy max_bins.
obwoe_apply() now refuses multiclass models instead of scoring them
wrongly. A multinomial fit carries a bins x classes WoE matrix; the
lookup linear-indexed it column-major and returned class 1's WoE for
every row, discarding the other classes with no error and no warning. It
now stops with an actionable message. The per-class matrix is still in
$results for callers that want to handle it themselves.
dmiv now reports a numeric total_iv where summary$total_iv was
previously NA for every feature. Code that tested for that NA will see
a number.
The categorical sketch engine computed every WoE against the wrong
marginal. It passed (total_neg, total_pos) to helpers whose signature
is (total_pos, total_neg). On a random 8-category feature with no real
signal, the reported IV was 10.4021 against a true value of 0.0043 --
wrong by three orders of magnitude, and wrong in the direction that makes
a useless variable look like the strongest predictor in the model. WoE
deviated from log((pos_i/TP)/(neg_i/TN)) by up to 2.7721; it is now
within 0.0002.
The identical defect was fixed in the numerical twin some releases ago and never replicated here. A numerical-vs-categorical parity test now covers every algorithm that has both variants, so this class of divergence cannot recur silently.
dmiv returns iv and total_iv alongside the divergence measures it
already reported. IV is computed from the smoothed class distributions
rather than from woe, because the default bin_method = "woe1" is
Zeng's log-odds ln((pos + 0.5)/(neg + 0.5)), which differs from standard
WoE by the constant ln(TP/TN); deriving IV from it would be wrong.
cm (categorical) exposes total_iv at top level, like the other 15
categorical engines, instead of only inside metadata. metadata is
unchanged, so existing callers keep working.
sab (categorical) no longer aborts with unordered_map::at when a
category contains zero events -- a routine situation in credit data. The
positive-count map only gained a key for categories with at least one
event, but was read with .at() in seven places.
obwoe_scorecard() and the cutoff table now reject a missing target with
a clear message naming the sample and the count, instead of failing deep
inside with missing value where TRUE/FALSE needed. The development frame
was already checked; samples passed through validation = were not. The
NA is rejected at the boundary rather than swept up with
na.rm = TRUE, which would have silently charged every unknown-outcome
row to the non-events and corrupted KS and AUC.
converged now means the same thing everywhereThe flag was effectively inverted in several engines: initialised false and
set true only on a degenerate shortcut or an in-loop tolerance test, while
the normal successful exit -- reaching the bin-count target -- set nothing. So
ordinary well-binned features reported FALSE and only degenerate ones
reported TRUE.
dmiv and bb (numerical) report converged on reaching the bin-count
target, matching the categorical dmiv, which already did.ir (numerical) reports converged for the exact binning it produces when
a feature has two or fewer distinct values. Every binary 0/1 feature
previously reported FALSE despite a correct result.dp, gmb, mba, sketch, cm and dmiv (categorical) report
converged on every successful termination path, including the fast paths
that bypassed the flag entirely.The intended contract is now documented in src/common/bin_structures.h:
converged == true means the algorithm reached a valid stopping state
(tolerance met, monotonicity achieved, or the bin-count target reached);
false means it exhausted max_iterations.
lpdb, ldb and numerical udt scaled as n^2.00, n^2.00 and n^2.30. A single
variable with 10^6 rows would have taken lpdb roughly 72 minutes; measured
against jedi at the same size they were 2,772x the median algorithm's cost.
Nothing warned about it.
Two distinct causes, neither of them inherent to the methods:
ldb and lpdb estimated the density with a naive double loop,
evaluating the Gaussian kernel of every observation against every other
one. The same defect had been written twice, in two files, which is how it
survived. It is replaced by the standard linear-binning estimator -- the
one R's own density() uses -- which now lives once in
src/common/optimal_binning_common.h so the two cannot diverge again.
udt rescanned every observation once per candidate split, allocating
two vectors and recomputing the parent entropy each time, giving O(u x n).
Information gain depends only on integer counts, so a single sweep carrying
running totals produces the identical value.
Measured at n = 50,000, against a build of the previous revision:
| algorithm | before | after | speedup |
|---|---|---|---|
| ldb | 10.740s | 0.010s | 1074x |
| lpdb | 10.725s | 0.011s | 975x |
| udt | 7.354s | 0.019s | 387x |
All three now scale linearly and land within a factor of two of jedi, the
package default. At n = 400,000 they take 0.085s, 0.087s and 0.170s against
jedi's 0.103s -- sizes the previous code could not reach at all.
Results. udt and ldb are bit-identical to the previous revision: udt
by construction, and ldb's local-minimum search resolves the grid estimate to
the same cut points. Both are pinned by a new regression test.
lpdb changes. It differentiates the density twice to find inflection
points, and finite differences taken between adjacent observations are not the
same thing as finite differences on a properly sampled curve. Its critical
points are now located on the estimation grid. On German Credit the partitions
generally improve -- duration goes from 2 bins and IV 0.0923 to 5 bins and IV
0.2635, age from 2 bins and 0.0628 to 4 bins and 0.0781 -- and no variable
tested got materially worse. Anyone with a fitted lpdb model should expect
different cut points.
Also removed OBN_LPDB::local_polynomial_density(), which no longer had a
caller and never did local polynomial regression despite its name.
dmiv ignored max_binsob_categorical_dmiv() returned L - 1 bins for an L-category
feature, whatever max_bins was set to. The merge loop compared the cost
of the best available merge against convergence_threshold and broke out
when it had barely moved. With many similarly sized categories the
second-cheapest merge costs exactly what the cheapest one did, so the test
fired on the second iteration, after a single merge, and nothing
re-imposed the cap afterwards. Reported as converged = TRUE, with no
warning.
Reproduced at n = 60,000 with max_bins of 3, 5 and 8 alike: 40 levels
returned 31 bins, 60 returned 50, 120 returned 119 and 300 returned 299.
All nine divergence_method choices and both bin_method choices were
affected. The roxygen already documented max_bins as a hard constraint,
so the code was wrong, not the documentation.
The loop now records the convergence and keeps merging by the same
criterion — the pair with the lowest divergence, i.e. the most similar
pair — until the cap is met (src/OBC_DMIV_v5.cpp). This is the remedy
already applied to fetb, whose loop had the same shape. The merge
ordering is unchanged, and min_bins is never violated to satisfy
max_bins.
dmiv produces different bins wherever the cap used to be abandoned;
on inputs where it was reached anyway, results are byte-identical to
the previous release — verified across all 16 categorical engines and 13
variables of the bundled German Credit data, 208 combinations, with the
RNG stream fixed so the stochastic engines are comparable. The numerical
dmiv never had the defect and is untouched.
A new regression test asserts that every categorical engine honours
max_bins on a high-cardinality feature, at two values of bin_cutoff
and two of max_bins.
?ob_categorical_dmiv described convergence_threshold as stopping the
merging. It records convergence; max_bins is a hard constraint and
merging continues until the bin count meets it.
find_most_similar_bins() now seeds its best pair with the first mergeable
one instead of {0, 0}. The search only replaces that seed on a strictly
smaller divergence, so a distance matrix that was entirely double::max --
or that held a NaN, against which every comparison is false -- would have
returned a pair naming the same bin twice, and merging a bin with itself
then erasing the duplicate would have dropped its observations. No input
reaching that state was found, so this closes a defensive gap rather than a
demonstrated defect.
mba read past the end of its bin vectorob_categorical_mba() performed an out-of-range read while reducing the
pre-bins. The list of candidate bins to merge is built before any merging
starts, and every merge erases a bin, so an index taken later in that list
could point past the end of the shrunken vector. The existing guard only
clamped the indices passed to the merge itself, not the read that selects a
merge partner.
This is undefined behaviour, not a wrong number: under a checked standard
library it aborts the R session, and without one it reads foreign memory and
carries on. Reproduced with 60 categories, max_n_prebins = 20 and
n = 60,000, at either the default bin_cutoff or a smaller one.
Stale indices are now skipped. The guard fires only where the previous code
was already out of range, so results are byte-identical on every input
that worked before — verified across all 16 categorical engines and 13
variables of the bundled German Credit data. mba rejoins the regression
test that asserts every categorical engine's bins account for every
observation.
ivb and gmb dropped observationsob_categorical_ivb() and ob_categorical_gmb() discarded the categories
that did not fit within max_n_prebins instead of pooling them, so those
observations left the binning entirely. The vector of bins was resized, and
everything past the cap was destroyed along with its counts. Nothing
signalled it: no warning, error = FALSE, converged = TRUE, and a
total_iv reported as if it described the whole sample.
Default settings hide the defect — bin_cutoff = 0.05 lets at most 20
categories survive the rare-category merge, which is exactly the default
max_n_prebins — but a smaller cutoff reaches it. With 60 levels and
bin_cutoff = 0.005, both engines lost 39,306 of 60,000 rows (65.5%);
dp and jedi accounted for every row on the same input.
Both now fold the excess into the smallest retained bin
(src/OBC_IVB_v5.cpp, src/OBC_GMB_v5.cpp). Which categories are kept as
separate identities is unchanged — still the max_n_prebins most frequent —
so results on the path where the cap never bound are byte-identical to
the previous release: verified across all 16 categorical engines and 13
variables of the bundled German Credit data, 208 combinations, with the RNG
stream fixed so the stochastic engines are comparable.
A new regression test asserts that every categorical engine's bins account for every observation.
New vignette, Algorithm Reference: the 37 Binning Engines. Reference
documentation for all 28 algorithms across their 37 algorithm/feature-type
combinations, written from a reading of the C++ implementation of every
engine and a check of that reading against the literature each one invokes.
Entries are organised by the mechanism the code actually uses — exact
dynamic programming, recursive entropy partitioning, statistical merging,
isotonic regression, density estimation, divergence, streaming,
metaheuristic, and greedy IV merging — which groups the engines differently
from their names.
Two rules governed it: a claim appears only if it traces to a specific line of the shipped source or to a publication verified to exist, and the name is never taken as evidence of the mechanism. Where neither could be established, the vignette says so — it closes with a section listing what could not be verified, including the provenance of the Information Value interpretation bands and whether Kerber's ChiMerge uses the continuity correction this package implements.
Substantive findings it documents: obwoe() does not forward
algorithm-specific parameters and drops them silently, demonstrated live;
mdlp does not implement the Fayyad–Irani criterion it cites while
fast_mdlp does; fetb computes a hypergeometric point probability, not
a Fisher exact-test p-value; bb neither branches nor bounds and
numerical dp builds no dynamic-programming table, while the package's only
real DP sits inside sketch behind an n <= 50 threshold; the numerical
sketch departs from the KLL construction it cites in both compactor
capacity and compaction rule, so the quoted error bound does not transfer;
and dmiv's default bin_method = "woe1" is a per-bin log-odds rather
than standard WoE.
It also records parameters that are accepted and never read
(polynomial_degree in lpdb; max_n_prebins in fast_mdlp, numerical
sketch, categorical fetb and sab), convergence_threshold being inert
in five numerical engines, and three citation errors in the shipped
documentation.
max_n_prebins is documented as the modelling decision it is. For
numerical features, pre-binning runs before any algorithm sees the data, so
the default of 20 quantile cells can smear a heavy tail and lose the signal
in it before optimization begins -- silently, with converged = TRUE.
Benchmarks on two open datasets (76,020 x 369 and 590,540 x 454, five-fold
held-out IV) move the median held-out IV of heavy-tailed numerical
predictors by +129% and +39% when the parameter is raised from 20 to 200,
with the bin count essentially unchanged.
The default is deliberately not changed. The same experiment shows the
effect runs both ways: on the larger benchmark, raising it cost 15% of
held-out IV on the twenty-five strongest predictors while inflating IV on
weak ones, and no cheap rule separated the two cases. ?control.obwoe now
says so, and recommends per-variable tuning against held-out data.
min_bins is documented as frequently binding. It reads as a safety
floor but often sets the partition, because several algorithms stop merging
on their own criterion well before max_bins. On German Credit amount
with max_bins = 5, the default returns two bins and an IV of 0.0016 while
min_bins = 4 returns 0.0824 -- same data, same algorithm.
Corrected the 1.13.0 entry on binary size. It described -Os,
-fvisibility=hidden, -ffunction-sections, -Wl,--gc-sections and a
cleanup script, none of which are in the package: the visibility and
section flags hid Rcpp symbols and broke the build on every platform and
were reverted in b74e95b. The entry is corrected rather than deleted so
the flags are not reinstated by someone reading the old claim.
An unmeasured IV is no longer reported as an IV of zero. With no finite
IV anywhere, summary() printed Total IV: 0.0000 and
IV Range: [Inf, -Inf], asserting "no predictive power" about features
that were never measured; print() on a step_obwoe recipe printed
total IV=0.0000. Both report NA with a note. obwoe_gains() failed
with attempt to select less than one element in get1index and
plot(type = "iv") with need finite 'xlim' values; both now say what is
actually wrong.
sketch (numerical) returns a bin label for a constant feature, so
summary$n_bins is 1 instead of NA.
udt (numerical) keeps WoE and IV finite with laplace_smoothing = 0.
A bin holding no events gave woe = -Inf and iv = Inf; both
distributions are now floored with the package-wide epsilon, so the
parameter stays usable. Reachable only through ob_numerical_udt()
directly -- obwoe() never forwards the argument.
obwoe() returns the control it used, alongside the effective
min_bins, max_bins and algorithm. call was never a substitute: it
records only the arguments the caller typed, never the defaults that
actually applied, so a saved model could not say what produced it.
A custom bin_separator now works end to end. Everything that has to
split grouped categories out of a bin label reads the separator from the
model instead of assuming the package default: obwoe_apply(), the points
table, the points SQL and the WoE SQL emitted into the workbook.
This was a silent corruption, not a cosmetic gap. Fitting with
control.obwoe(bin_separator = "||") and applying the model back to its
own training data put 754 of 900 rows on the na_woe fallback,
because the labels were split on "%;%" and never matched. The generated
SQL was worse: a||e||c came out as
g IN ('a', '|', '|', 'e', '|', '|', 'c') -- broken SQL, no warning.
Both are exact now.
Models saved by earlier versions keep working. They carry no control
element, so the separator falls back to the package default rather than
erroring on a missing field.
Nothing changes for the default configuration: "%;%" remains the default
everywhere, so only the custom-separator path -- which was broken -- behaves
differently.
Bug-fix release addressing an internal code audit of 1.13.0. Every item below either changes a computed value that was previously wrong, or removes an API surface the author never intended to publish. See the pull request for the full item-by-item breakdown.
fit_logistic_regression() is no longer exported. It is still used
internally (renamed to .ob_fit_logistic_regression()) to implement
engine = "obwoe" in obwoe_scorecard(), which is unaffected.
ob_gains_table() and ob_gains_table_feature() are renamed to
obwoe_gains_score() and obwoe_gains_variable() respectively, to sit
alongside the rest of the obwoe_* family. Behavior is unchanged; only
the names moved. obwoe_gains() (the higher-level, plot-producing
function) is a different function and was not renamed.
obwoe_apply() now routes a missing categorical value to the fitted
"missing" bin's WoE when the binning built one, instead of always using
na_woe. Previously obwoe_apply() ignored any missing-value bin
learned during training and always returned na_woe for NA, while the
generated deployment SQL (obwoe_sql(), null_to_na_bin = TRUE by
default) routed IS NULL to that bin's WoE — so R and the SQL scored the
same missing value differently. na_woe is now only a fallback for
variables where no missing-value bin exists. If you fit a model with real
NAs in a categorical predictor, the WoE obwoe_apply()/predict()
return for NA may change in this version; it now matches the SQL and
the bin actually fitted. See ?obwoe_apply and ?obwoe_sql.
obwoe_scorecard(..., drop_negative = TRUE) now actually removes a
single negative-coefficient variable instead of silently keeping it.
Previously the removal loop stopped one variable too early whenever
exactly one variable had a negative coefficient, so that case fell
through to a warning instead of either fixing the model or raising
stop(). It now keeps removing negative-coefficient variables until
either none remain or only one variable is left (in which case, if it is
still negative, obwoe_scorecard() stops with an error, as documented).
ob_cutpoints_num()'s bins are now right-closed (a, b], and both
ob_cutpoints_num() and ob_cutpoints_cat() return the pieces
(id, and cutpoints for the numerical version) ob_apply_woe_num()/
ob_apply_woe_cat() require. Previously ob_cutpoints_num() built
left-closed [a, b) bins — the opposite of ob_apply_woe_num()'s
include_upper_bound = TRUE default — so a value sitting exactly on a
cutpoint could get a different, often sign-flipped, WoE depending on
whether it went through the fit or the apply side; and neither manual
cutpoint function's result could be handed to its matching apply
function at all, because both lacked the id element (and
ob_cutpoints_num() the top-level cutpoints) the apply side requires.
ob_cutpoints_cat()'s emitted bin labels also now use "%;%" (matching
ob_apply_woe_cat()'s default separator and the main pipeline) instead
of echoing the "+"-joined input verbatim; the "+" input format is
unchanged. If you use ob_cutpoints_num()/ob_cutpoints_cat() and parse
their bin labels yourself, check the new format. See
?ob_cutpoints_num and ?ob_cutpoints_cat.
sort_by = "bin" (including the score bands in
obwoe_scorecard()) were ordered lexicographically instead of by the
bins' natural (level) order, silently understating KS by as much as a
third on real data.step_obwoe(algorithm = "auto") could resolve to the multiclass
algorithm for a genuinely binary outcome whenever the outcome factor
declared an unobserved third level.print() on an unprepped step_obwoe recipe step errored when
algorithm = tune::tune().obwoe_gains(use_column = "woe", n_groups = k) returned NA WoE and
zero total IV for every bin.plot.obwoe_gains(type = "cumulative") assumed every bin held an equal
share of the population; the x-axis now reflects each bin's real size.ir algorithm was implemented, documented and tested, but was
never counted.obwoe_scorecard() — the pipeline as one artefactRuns the whole origination workflow in a single call — stratified split,
binning, screening by Information Value and correlation, model fitting, PDO
scaling — and returns an object that also writes itself out as an .xlsx model
document. The point is not convenience: each stage records why it did what it
did, so the workbook is reviewable evidence rather than a set of numbers.
The binning sees the training rows only. Binning is supervised: cut points and WoE are both chosen against the target, so fitting them on the full base before splitting leaks the hold-out into the transformation. The split therefore happens first, and the binning is fitted once, on the training rows, and applied everywhere else. Measured on German Credit, the difference is a hold-out AUC of 0.768 fitted the leaky way against 0.709 fitted correctly — the leak buys six points of AUC that do not exist.
A negative WoE coefficient is a fault, not a result. The WoE already carries the direction of risk, so a negative slope means the model is reversing a variable to compensate for another. Such variables are dropped one at a time, worst first, and the model refitted, with each removal recorded.
The points table is fixed, not per applicant. Points follow the Siddiqi
allocation, points_ij = Offset/k - Factor(beta_j WoE_ij + alpha/k), so a
bin is worth the same number of points to everyone. Rounding each bin once
costs at most k/2 points against the unrounded model score; that drift is
measured on every sample and reported rather than hidden by re-apportioning
the rounding per row, which would make the same bin worth different points
to different applicants.
The deployment SQL reproduces the R score exactly. Both the WoE form and
the integer-points form are generated, the total computed in an outer
SELECT over a subquery — referring to a select-list alias within the same
SELECT is a MySQL extension that ANSI SQL, SQLite, PostgreSQL, SQL Server
and Oracle all reject. Verified against live SQLite on all 1000 German
Credit rows: maximum difference 0, unseen categories included.
A value in no fitted bin scores a defined fallback. It is counted and
warned about, per sample and per variable, but it still produces a score —
both in R and in SQL, from one and the same na_woe figure. A single unseen
category cannot void an application.
Thirteen sheets. Model summary, scorecard, coefficients with standard errors, bin statistics, screening funnel with a reason per rejected variable, correlations before and after pruning, score gains, PSI between samples, cut-off strategy, SQL in both forms, and a reproducibility record.
Engines are a three-function contract — fit, link, coef — with
glm, the package's own C++ L-BFGS logistic regression, and glmnet
registered, and custom engines accepted. coef() returning NULL declares
the model non-additive, and the pipeline then produces no points table
rather than fabricating one from a model that has no per-variable
decomposition. A missing engine package is an error by default, not a
silent substitution: a workbook documenting a model the analyst did not ask
for is worse than a call that fails.
obwoe_scale() and obwoe_score() implement PDO scaling,
Factor = PDO/ln 2 and Offset = Score0 - Factor ln(Odds0), in both score
directions. Doubling the good:bad odds moves the score by exactly one PDO
at every point of the range.obwoe_prune() removes redundant variables iteratively, dropping the
worse-ranked member of the strongest surviving pair and recomputing, rather
than resolving all pairs against the original matrix at once.obwoe_psi() computes the Population Stability Index between two samples,
for both binned and continuous inputs, using interior quantiles so that a
merely shifted distribution does not produce a degenerate band.obwoe_report() writes the workbook for an already-fitted scorecard.predict() methods for "score", "card", "link", "prob" and
"woe", reading only the binning, the coefficients and the scaling.The screening funnel named the wrong stage. Every variable that left
the pipeline after Information Value screening was labelled
corr_pruned, including those dropped by the negative-coefficient check
and those that turned out constant after the WoE transform. The funnel is
the sheet a reviewer reads to learn why a variable is absent, so a
mislabelled rejection is a false statement in a model document. stage
now distinguishes in_model, sign_rejected, corr_pruned,
constant_woe and screened_out, each assigned by the step that actually
rejected the variable.
The cut-off strategy sheet ignored the score direction. It always
approved applicants scoring at or above the cut, which is right under the
default scale but exactly inverted under
direction = "higher_is_riskier" — the sheet then approved the worst
applicants and reported an approved bad rate above the rejected one, at
every one of its twenty cut-offs. The approval rule now follows the
direction of the scale.
file is validated before the pipeline runs. A missing openxlsx, a
non-existent directory or an unwritable one used to surface only at the
final write, after the binning, the screening and the fit had all been
computed and were about to be discarded. The check now happens up front:
the failure arrives in about 0.04 s instead of after the whole run.
openxlsx and glmnet are added to Suggests. Neither is needed unless a
workbook is written or engine = "glmnet" is requested.
Two additions close the gap between a fitted binning and a deployed scorecard: deciding which variables deserve to enter the model, and shipping the accepted transformation to the database where the data lives.
obwoe_select() — automated variable screeningScreens every binned variable of an obwoe model against the two criteria that
govern variable admission in credit risk practice — predictive strength,
graded with the Siddiqi (2006) Information Value bands, and guaranteed rank
ordering, measured by monotonicity of the bin event rate — and returns a
verdict for each.
Nothing is ever dropped. A base with 500 candidate variables yields 500
rows. Each carries a selected flag, a machine-readable reason listing
every rule it violated, and a reason_desc in plain language, so the
automatic verdict can be reviewed, overridden, or replaced by the analyst's
own triage.
Two levels of detail. detail = "summary" gives one row per variable
with its headline metrics; detail = "full" expands to one row per variable
and optimised bin, carrying the complete gains table of
ob_gains_table() (31 metrics: WoE, IV, lift, cumulative KS, precision,
recall, F1, KL and JS divergence, ...) together with the interval bounds of
numerical bins and the merged category lists of categorical ones.
Metrics that describe the deployed score. ks, auc and gini are
computed on the WoE actually applied to the data, with bins ranked by that
WoE and merged when tied. auc uses the tie-corrected Mann-Whitney form,
which reproduces the rank-based AUC of the WoE-transformed observations to
machine precision. ks is therefore the true KS of the transformed
variable even when the binning is not monotonic.
An honest treatment of ordering. For numerical variables the bin
sequence is intrinsic, so monotonicity is a genuine constraint. For nominal
categories the sequence is a free relabelling, so the default
require_monotonic = "numeric" applies the constraint only where it means
something; "all" and "none" are available.
Rejects suspicious variables by default. iv_max = 0.50 excludes the
Siddiqi Suspicious band, where a single-variable IV is far more often a
symptom of target leakage than of a dominant predictor. On the Statlog
German Credit benchmark this is exactly what happens to the
checking-account status (IV = 0.67).
Further gates for min_bins, max_bins, minimum bin population share,
bins with no events or no non-events, and a top_n cut by IV, KS, Gini or
AUC. An Excellent/Good/Fair/Rejected quality tier summarises how
cleanly the algorithm categorised each variable.
Returns a data.table when that package is installed and an identical
data.frame otherwise; the table is assembled column-wise in a single
pass, so 500 variables are screened in about a second.
obwoe_sql() — SQL code generationTranslates a fitted binning into executable SQL, so the WoE transformation runs inside the database with no round trip through R.
Exact interval semantics. Numerical bins are half-open on the right,
(lower, upper], exactly as obwoe_apply() assigns them. Boundaries come
from the fitted cutpoints vector, never parsed back from bin labels,
whose formatting varies between algorithms. By default each branch states
both of its bounds (WHEN x > 7 AND x <= 10 THEN ...) so it is correct in
isolation; explicit_bounds = FALSE emits the shorter cascading form.
Cut points that survive the round trip. Literals are written as the
shortest fixed-notation decimal that parses back to the identical IEEE 754
double, and never in scientific notation. Rounding a boundary such as
4049.5 would silently move observations between bins, so exactness is the
default; digits makes rounding an explicit choice.
NULL cannot leak. In SQL, NULL <= 5 is NULL, not FALSE, so a
missing value matches no comparison and would fall through to ELSE. Every
generated expression opens with an explicit WHEN <col> IS NULL branch,
and when the binner folded training missings into a category bin,
null_to_na_bin = TRUE routes database NULLs to that same bin.
Escaping that holds. Single quotes are doubled per ANSI SQL; on MySQL, MariaDB and the Hive family backslashes are doubled too. Category names are matched byte for byte, whitespace included. Identifiers are quoted with the dialect's own delimiters and, by default, only when the name needs it or collides with a reserved word — which keeps generated code readable on case-folding engines such as Oracle and Snowflake.
14 dialects (ansi, postgres, mysql, mariadb, sqlserver, oracle,
spark, hive, databricks, bigquery, snowflake, redshift,
duckdb, sqlite), four assembly styles (select, case, cte,
view), four output modes (woe, bin, index, both), an audit header
recording package version and algorithm, and direct file output.
Accepts an obwoe object, a prepped step_obwoe() step, or a prepped
recipe containing one, and composes with obwoe_select() through
features = sel$feature[sel$selected].
Real benchmark data is bundled. inst/extdata/germancredit.csv.gz
holds the Statlog (German Credit) dataset from the UCI Machine Learning
Repository (1000 applications, 7 numerical and 13 categorical attributes,
CC BY 4.0), in its labelled form — category names carrying spaces, slashes
and colons make it a realistic test bed for SQL escaping.
obwoe_select() reproduces the published Information Values of that
benchmark to four decimals for the 14 attributes the optimiser keeps
ungrouped, and its IV, KS, AUC and Gini match an independent computation
made from the raw observations to 1e-10.
The generated SQL is validated by parsing and executing it: the test
suite carries a small CASE interpreter that reads the emitted text the
way an engine would, and the resulting bin assignment is checked against
the counts the binning algorithm itself reported — across five algorithms
and all 20 German Credit variables. During development the same statements
were additionally run against a live SQLite engine, whose results were
identical. Edge cases covered include observations sitting exactly on a cut
point, cut points with no exact binary representation, magnitudes from
1e-9 to 1e9, single-bin variables, degenerate bins, reserved-word
column names, and categories containing quotes, backslashes, tabs,
accented characters and significant whitespace.
Both defects below were uncovered while validating the two new functions
against real data, and both are pinned by tests in
tests/testthat/test-regression-audit.R that fail on 1.11.0 — 31 assertions
in total.
obwoe_apply() and bake.step_obwoe() silently failed to score any
category carrying leading or trailing whitespace. Both rebuilt their
category-to-bin lookup with
trimws(strsplit(bin_label, "%;%", fixed = TRUE)[[1]]). The binning
engines join the original category strings with the separator and add no
padding — verified here for all 15 categorical algorithms — so the split
pieces are already the categories byte for byte, and the trimming turned a
category such as " N/A " or "PENDING " into a key no observation
could match. Those rows were assigned bin = NA and woe = na_woe, i.e.
scored as unseen, with no warning: on a base whose codes come from a
CHAR(n) column or a hand-maintained code table, an entire segment could
be dropped from the model without a trace. The trimming is gone; categories
are now matched exactly, as obwoe_sql() already did.
obwoe_gains() reported the same lift for every bin. The column was
built with ifelse(overall_rate > 0, df$pos_rate / overall_rate, 0);
ifelse() returns a result shaped like its test, and that test is a
scalar, so the expression collapsed to a length-one vector that R then
recycled across the table. Every gains table with more than one bin
reported the first bin's lift throughout, and plot(type = "lift") drew a
flat line. The R gains table now agrees with the independent C++ engine
(ob_gains_table()) to machine precision on lift, woe, iv and ks.
ob_numerical_ir() reported counts that did not describe its own bins,
and bins that were not monotonic. applyIsotonicRegression() ran the
Pool Adjacent Violators algorithm over the bin event rates and then
overwrote each bin with count_pos <- round(fitted_rate * count). Pooling
adjacent violators means those bins form one block; keeping them separate
and back-solving synthetic counts produced two defects at once. First,
count_pos and count_neg no longer described the observations falling
between the reported cutpoints, so WoE, IV, KS and every gains table
derived from them referred to a distribution that does not exist — on the
German Credit amount attribute the reported and observed event rates
differed in four of six bins. Second, because of the rounding the reported
bins were not even monotonic, which is the one property an isotonic binner
exists to guarantee.
PAVA blocks are now merged into single bins. This reproduces the isotonic
fit exactly — with the bin counts as weights, a block's pooled rate is
sum(count_pos) / sum(count) over that block — while count, count_pos
and count_neg stay equal to what was observed. Features whose event rate
is already monotone are unaffected and return bit-identical results; where
PAVA had violators to pool, the binning is now coarser and genuinely
rank-ordering. min_bins becomes a target rather than a guarantee for this
algorithm, since monotonicity cannot always be attained at that resolution.
The README is a third of its former length. It now covers what the package is, the four steps of a run, how to choose an algorithm and where to read more; the extended worked examples moved into the vignettes, where an analyst can study them properly. Algorithm selection is presented as a decision graph (rendered by GitHub) with the family reference table kept below it.
Two vignettes replace the single kitchen-sink one.
Optimal Binning and Weight of Evidence: A Practical Guide is the working
reference: what WoE and IV measure and why monotonicity of the event rate
and of the WoE are the same statement, reading bin and gains tables,
screening a base with obwoe_select(), how the algorithm families differ,
applying the transformation, exporting SQL, and preprocessing.
An Industrial Scorecard Pipeline is new and runs an origination scorecard
end to end: a wide synthetic base with the pathologies that matter (missing
fields, rare dealer codes, near-duplicate vendor variables, pure noise and
a leaky post-booking field), screening at scale, redundancy pruning with
obcorr() in the WoE space, a recipes pipeline around step_obwoe(),
logistic regression with the coefficient sign check, PDO scorecard points,
out-of-time validation with gains and PSI, deployment through
obwoe_sql(), tuning with tidymodels, and a governance checklist.
Both vignettes run on the bundled German Credit benchmark or on data they
generate, so they build from recipes alone. The previous vignette
required the Suggested scorecard package and failed to build without it;
scorecard and pROC are no longer used anywhere and have been dropped
from Suggests.
data.table is used when installed, for rbindlist() and for the returned
table type, and is declared in Suggests only: the package has no new hard
dependency and behaves identically without it, as the test suite verifies.Follow-up to the 1.10.0 static audit, this time driven by instrumented builds
(-fsanitize=address,undefined), a degenerate-input stress harness, and a
golden-output regression suite covering all 37 exported algorithms
(~3,200 result comparisons). All fixes below are covered by new tests in
tests/testthat/test-regression-audit.R, each of which fails on 1.10.0.
ob_categorical_ivb crashed the R session (segmentation fault, not a
catchable error) for any feature with no more categories than
max_bins — with the default max_bins = 5 this meant every 2-, 3-, 4-
or 5-level predictor, i.e. sex, marital status, region, education. In
perform_binning() the ncat <= max_bins fast path skipped
initialize_dp_structures(), the only place stats_cache was created,
while result assembly dereferenced it unconditionally. The cache is now
built before the branch.
ob_numerical_ir, ob_numerical_jedi, ob_numerical_jedi_mwoe hung
forever whenever min_bins exceeded the number of distinct feature
values — a routine situation when one min_bins is applied across a whole
feature set. The "ensure at least min_bins" loops called a split routine
that silently declines to split unbounded intervals (and, in JEDI, ran
before counts existed, so it always targeted the unsplittable (-Inf, e1]
bin), making no progress and never terminating. Since these loops contained
no R_CheckUserInterrupt(), the hang could not even be interrupted with
Ctrl-C. The loops now consider only splittable bins and stop as soon as no
progress is possible.
ob_categorical_sab is now reproducible under set.seed(). It was
seeded from std::random_device, so identical input returned a different
binning on every call with no way to control it — unusable for auditable
or regulated models, and unreliable on some MinGW toolchains. The
simulated-annealing search is now seeded from R's own RNG stream.
This changes ob_categorical_sab results, which were previously
random and therefore had no stable baseline to preserve.ob_numerical_sketch now returns the bin label field (plus
total_iv) like every other numerical algorithm. Its absence broke
generic consumers, including this package's own test helper. bin_lower
and bin_upper are retained, so the change is purely additive; no numeric
output changed.
The ob_numerical_sketch test disabled since 1.0.7 for a segfault has been
re-enabled: the underlying MergeCache defect was removed in an earlier
round, and the case was re-verified clean under
-fsanitize=address,undefined at n = 500/1000/2000/5000.
obcorr) — CRAN policy and determinismobcorr() no longer seizes every core. With the default threads = 0
it called omp_set_num_threads(std::thread::hardware_concurrency()),
taking all available cores. CRAN Repository Policy requires a package never
to use more than two cores simultaneously by default, since the check farm
is a shared resource; this was an archival risk. The default is now at most
2 threads, honouring any lower limit already set through OMP_NUM_THREADS,
and capped by omp_get_num_procs(). An explicit positive threads is
still respected.
obcorr() returned its rows in a non-deterministic order. Per-thread
result buffers were spliced together inside an omp critical block, so the
row order depended on thread scheduling — two runs on the same data with
the same thread count could return the pairs in different orders. Any
caller using head(), positional indexing, or a positional join saw
different numbers between runs. Each iteration now writes to its own slot
in a pre-sized vector, which is deterministic, independent of the thread
count, and removes the critical section. Correlation values are
unchanged; only the row order is now stable.
*Cópia em conflito*.cpp/.h)
from src/. R CMD build would have shipped and compiled them, producing
duplicate symbols. Guards added to .gitignore and .Rbuildignore.(a, b] (changes numeric output)The package previously disagreed with itself about what a bin is. Bin labels
advertised (a;b], but many algorithms assigned observations as [a;b);
ob_apply_woe_num() had its two binary searches swapped, so both settings of
include_upper_bound did the opposite of what the argument documents; and
several equal-frequency pre-binners split runs of tied values, so their reported
counts could not be derived from their own reported cutpoints under any
convention. A value landing exactly on a cutpoint — routine for integer, rounded
or currency features — could be scored into a different bin than the one it was
trained in.
Measured across 63 algorithm/dataset combinations, 31 violated the documented
(a, b] convention before this release; 0 do now.
ob_apply_woe_num(): include_upper_bound = TRUE now really means (a, b]
(lower_bound search) and FALSE really means [a, b) (upper_bound).
The two were previously exchanged.ob_numerical_dp, _fetb, _ldb, _oslp: bin lookup switched from
upper_bound to lower_bound, so boundary values stay in the bin below.ob_numerical_bb, _dmiv: dropped an + EPSILON added to the search key,
which silently turned the documented upper >= value test into a strict one.ob_numerical_sketch: interval test changed from [lower, upper) to
(lower, upper] (first bin remains closed on the left, so the minimum is
included).ob_numerical_mob, _mdlp, _mrblp: equal-frequency pre-binning is now
tie-aware — a run of identical values is never split across two pre-bins —
and boundaries are the last value in the bin rather than the first value
of the next. Their labels changed from [a;b) to (a;b] to match.ob_numerical_fast_mdlp: the force_min_bins() fallback picked a split at
a raw index midpoint, which could land inside a run of tied observations.
It now moves the split to a genuine value boundary, as the MDL recursion
already did.ob_numerical_cm silently discarded observations. Its equal-frequency
pre-binner, on hitting a tie that straddled a bin boundary, advanced past the
tied records without ever assigning them to a bin. On tied or discrete features
this dropped a large share of the data — 26% on an integer feature and 40% on a
coarse one in testing — and the reported WoE/IV were computed from the surviving
subset. The tied records are now absorbed into the preceding bin.
ob_categorical_dp and ob_categorical_fetb report WoE = IV = 0 for
perfectly separating bins instead of applying smoothing.event_rate is present in
only 13 of 37, total_iv in 28 of 37, and iv is missing from both
_dmiv variants.ob_apply_woe_num() does not support the multinomial *_jedi_mwoe
variants, whose output carries per-class counts rather than count_pos /
count_neg.This release is the result of a full static audit of the C++ engine covering all 36 binning algorithms. No public R API was changed.
OB_LogisticRegression — Replaced exact det != 0 singularity guard
with a threshold-based check (|det| > 1e-10 × ‖H‖); replaced
hessian.inverse() with Eigen::LDLT decomposition for numerical
stability; added .cwiseMax(0.0) before sqrt() to prevent NaN
standard errors from near-zero diagonal entries.OBN_MDLP — monotonicity direction bug — is_monotonic() and
enforce_monotonicity() previously hardcoded ascending direction, causing
unnecessary merges on negatively-correlated features. Both now auto-detect
the dominant trend via Welford's slope algorithm before
checking/enforcing monotonicity.OBC_DP — DP backtracking out-of-bounds — Added guard before
static_cast<size_t>(prev_j) in backtrack_optimal_bins(); invalid
predecessor index now raises a descriptive runtime error instead of
silent undefined behaviour.OBN_DP — push before validate — Target value validation in
optimal_binning_numerical_dp() now fires before the value is appended to
target_vec, preventing insertion of invalid data.OBN_MDLP — log2(0) in MDL cost — Guard added for the single-bin
case where log2(k-1) would evaluate to log2(0) = -Inf.NumericalBin constructor invariant — The 7-arg constructor now
derives count = count_pos + count_neg regardless of the c argument,
enforcing the count == total() invariant at construction time.OBC_DP — DP outer loop removed — The deterministic DP in
perform_dynamic_programming() was wrapped in a redundant
max_iterations outer loop (default 1000). Removing it yields up to
1000× speedup for the categorical DP algorithm.OBC_DP::ensure_max_prebins() — O(m² log m) full re-sort per merge
step replaced with O(m log m + m²) std::lower_bound + insert.OBN_MDLP::apply_mdl_merging() — O(k³) full-vector copy per
candidate merge eliminated; MDL delta is now computed analytically from
bin statistics in O(k²) per outer step.OBN_BB::quantile() — Per-call sort-copy O(n_prebins × n log n)
eliminated; prebinning() sorts once and passes the sorted vector to a
stateless quantile().monotonicity_utils.h — Welford index allocation — Removed
unnecessary std::vector<double> indices(n) heap allocation; loop index
cast directly to double.OBN_DP — Pearson correlation instability — Replaced naive
two-pass Pearson formula (catastrophic cancellation risk) with
detect_trend_from_correlation() using Welford's online algorithm.safe_math.h — All 6 functions changed from constexpr to inline;
std::log, std::exp, std::abs and std::isfinite are not guaranteed
constexpr in C++11/14, risking compilation failure on SOLARIS/Studio.chi_square_utils.h — CHI_SQUARE_CRITICAL_VALUES namespace-scope
const replaced with a function returning a static local instance
(one shared copy per process, C++11 thread-safe init).entropy_utils.h — ENTROPY_LUT (~81 KB) replaced with
entropy_lut_instance() returning a static local; eliminates one copy
per translation unit.OBC_CM_v5 — Duplicate ChiSquareCache class (global namespace)
removed; file now uses OptimalBinning::ChiSquareCache from
chi_square_utils.h..cpp files — Duplicate using namespace Rcpp appearing before
#include "common/" headers removed, preventing potential name-resolution
ordering issues.OBC_DP — Dead commented-out code blocks (// struct CategoryStats,
// Local CategoricalBin definition removed) deleted.OBN_DP — Local variable total_count renamed to rare_total to
fix shadowing of the class member with the same name.OBN_IR — [[Rcpp::plugins(cpp17)]] standardised to cpp11 for
consistency with the rest of the package.OBC_DP — Auto-detection of monotonicity direction (monotonic_trend = "auto") implemented in compute_and_sort_event_rates() via
detect_trend_welford_woe().Rf_error with Rcpp::stop:
::Rf_error inside catch(...) blocks. Updated all instances to use Rcpp::stop to ensure proper C++ stack unwinding and avoid memory leaks.src/OBN_LPDB_v5.cpp, src/OBN_EWB_v5.cpp, src/OBN_KMB_v5.cpp, src/OBN_LDB_v5.cpp, src/OBN_MBLP_v5.cpp.IVCache and CumulativeStatsCache in anonymous namespaces within OBC_GMB_v5.cpp, OBC_IVB_v5.cpp, and OBC_JEDI_v5.cpp. This resolves Link-Time Optimization (LTO) warnings/errors on CRAN checks.UBSAN Investigation Fix (2026-01-27) - Addressing persistent memory safety errors:
Temporarily disabled ob_categorical_sketch tests: The sketch-based categorical binning
algorithm is under investigation for persistent UBSAN memory errors that appear to be
related to cache invalidation timing in GitHub Actions CI environment.
Removed MergeCache class from OBC_Sketch_v5.cpp: Completely removed the caching
mechanism and implemented on-the-fly divergence calculation to eliminate potential
memory corruption sources.
Affected Files:
src/OBC_Sketch_v5.cpp: MergeCache class removed, divergence calculated on-the-flytests/testthat/test-categorical-all.R: Sketch tests temporarily commented outNo API Changes: Fully backward compatible with v1.0.6.
CRAN Fix (2026-01-26) - Resolving AddressSanitizer memory safety errors:
Fixed heap-buffer-overflow in OBN_CM_v5.cpp: The calculate_inconsistency_rate() function
was accessing bins[j-1] when j=0 and bins.size()==1, causing invalid memory access.
Restructured bin-finding loop to avoid negative index access.
Fixed uninitialized bool in OBC_MBA_v5.cpp: The MergeCache::enabled member was not
explicitly initialized, causing "load of value 128, which is not a valid value for type 'bool'"
runtime error. Added explicit bool enabled = false initialization.
Affected Files:
src/OBN_CM_v5.cpp (lines 863-887): Safe bin-finding logicsrc/OBC_MBA_v5.cpp (line 26): Explicit bool initializationNo API Changes: Fully backward compatible with v1.0.5.
CRAN Fix (2026-01-25) - Resolving ERROR on macOS platforms during vignette re-build:
Fixed obwoe_apply() "breaks are not unique" error: Enhanced cutpoint deduplication logic to properly handle cases where sort(unique(cutpoints)) reduces the number of intervals. When the deduplicated cutpoint count doesn't match the original bin count, the function now uses a fallback mapping with dynamically generated interval labels and mean WoE values, avoiding the cut.default() error.
This addresses the vignette build failure reported on r-release-macos-arm64, r-release-macos-x86_64, r-oldrel-macos-arm64, and r-oldrel-macos-x86_64 platforms.
Internal Changes:
CRITICAL CRAN Fixes (2026-01-24) - Addressing ERROR and NOTE on macOS platforms:
Fixed macOS vignette ERROR: Added comprehensive validation for duplicate cutpoints in obwoe_apply() and bake.step_obwoe(). The R base cut() function now receives guaranteed unique, sorted breaks, preventing the "'breaks' are not unique" error that was causing vignette build failures on macOS platforms.
Attempted to reduce the package binary size with size optimization
flags (-Os, -fvisibility=hidden, -ffunction-sections,
-fdata-sections), the -Wl,--gc-sections linker flag and a cleanup
script for symbol stripping.
This was reverted and no longer describes the package.
-fvisibility=hidden and -Wl,--gc-sections hid Rcpp symbols and broke
the build on every platform, so they were removed in commit b74e95b;
-Os and the cleanup script did not survive either. src/Makevars
now sets only the OpenMP and BLAS/LAPACK flags. The entry is corrected
here rather than deleted so that nobody reinstates flags that are
already known to break the build.
The size itself is not a problem: the shared object is large only
because it carries debug symbols. Measured on 1.13.3, .debug_info
alone is 33Mb against 2.2Mb of .text, and stripping takes the library
from 70.4Mb to 2.6Mb. R CMD check --as-cran reports the installed size
as INFO, not as a NOTE.
Internal Changes:
Added src/common/cutpoints_validator.h - new C++ utility header with validate_cutpoints() function to ensure cutpoint uniqueness across all numerical binning algorithms. Uses floating-point tolerance (1e-10) for safe duplicate detection.
Modified get_cutpoints() in src/OBN_MOB_v5.cpp (line 180) to apply validation before returning cutpoints.
Modified update_cutpoints() in src/OBN_UBSD_v5.cpp (line 874) to apply validation before storing cutpoints.
Added R-level validation in obwoe_apply() (R/obwoe.R, line 1550): cutpoints are now sorted and deduplicated using sort(unique(cutpoints)) before constructing breaks vector.
Added R-level validation in bake.step_obwoe() (R/step_obwoe.R, line 789): same deduplication logic for recipes integration.
Enhanced vignette robustness (vignettes/introduction.Rmd): Added try-catch error handling in scorecard workflow to prevent build failures on edge-case data distributions.
Affected Algorithms: All 21 numerical binning algorithms now validate cutpoints to prevent duplicate breaks:
No API Changes: Fully backward compatible with v1.0.3. All existing code will continue to work without modification.
Critical Bug Fixes - KLL Sketch Algorithm (2026-01-20):
KLLSketch::compact_level() - the compactors.push_back() call was invalidating references to vector elements, causing crashes with datasets larger than ~200 observations.calculate_metrics() calls - swapped (total_good, total_bad) to correct order (total_pos, total_neg), fixing incorrect WoE calculations.[lower, upper] check for the last bin to ensure boundary values are correctly assigned.enforce_bin_cutoff() - corrected iterator invalidation when merging bins by always erasing the higher-indexed bin.k >= 2 and k < n to prevent undefined behavior with edge cases.compactor.size() < 2 before iteration.ob_numerical_sketch() with clearer parameter descriptions and simplified examples.special_codes parameter with max_n_prebins for consistency with other algorithms.CRAN Reviewer Feedback (2026-01-17):
Siddiqi, Navas-Palencia) in DESCRIPTION.obwoe_apply.\dontrun{} with \donttest{} in 12 function examples.par() restoration in examples and vignettes.inst/WORDLIST to include technical terms and author names (MILP, Navas, Palencia) to resolve spelling notes.README.md links for CONTRIBUTING.md and CODE_OF_CONDUCT.md to use absolute GitHub URLs, ensuring compliance with CRAN URI checks for ignored files.Language: en-US to DESCRIPTION metadata.README.Rmd with detailed algorithm descriptions, tidymodels integration examples, and performance metrics.CODE_OF_CONDUCT.md (Contributor Covenant v2.1) and CONTRIBUTING.md guidelines.inst/WORDLIST for spell checking.DESCRIPTION with corrected fields (Authors, BugReports, Depends, References).cran-comments.md for submission notes.OptimalBinningWoE is a high-performance R package for optimal binning and Weight of Evidence (WoE) transformation, designed for credit scoring and predictive modeling.
Rcpp and RcppEigen for maximum efficiency and scalability.obwoe(): Master function for optimal binning with automatic type detection and algorithm selection.ob_apply_woe_num() / ob_apply_woe_cat(): Functions to apply learned binning mappings to new data.step_obwoe(): A complete recipes step for integrating optimal binning into machine learning pipelines.tune() for hyperparameter optimization of binning parameters (algorithm, min_bins, etc.).JEDI-MWoE for handling multi-class target variables.ob_preprocess(): Utilities for missing value handling and outlier detection/treatment (IQR, Z-score, Grubbs).ob_gains_table(): Computation of detailed gains tables including IV, WoE, KS, Gini, Lift, Precision, Recall, KL Divergence, and Jensen-Shannon Divergence.plot() methods for visualizing binning results and WoE patterns.vignette("introduction", package = "OptimalBinningWoE")) for detailed examples and theoretical background.