Skip to content

Mirror the R fixes to dy.dx, NNS.diff, dy.d_, NNS.stack and NNS.boost - #161

Closed
gitRasheed wants to merge 17 commits into
rasheed/r-fixes-dominance-anovafrom
rasheed/r-fixes-regression-stack
Closed

gitRasheed wants to merge 17 commits into
rasheed/r-fixes-dominance-anovafrom
rasheed/r-fixes-regression-stack

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

NNS-python copied five regression-side R bugs. dy.dx depended on row order, NNS.diff reported convergence with errors up to 5.7e-4, dy.d_ differentiated a row-mean index instead of the fit, the stack's ALL candidate was scored as a constant mean but deployed as a different model, and NNS.boost returned class codes for a two-valued numeric response. We mirror each, and make nns_stack reproduce R's cor and X* rounding so partitions match R.

Stacked on #160; merge with a merge commit.

Changes

  • nns_stack's dim.red cor follows R's C_cor long-double arithmetic, the Method 1 fold design uses a column-sequential mat-vec like R's reference dgemv, and Method 2 uses a port of NNS_xstar_path_cpp. cor is bit-identical to R on 180 of 180 coefficients (74 before), and on Boston with folds 3 and 5 the stack differed from R by up to 10.8 and now matches at 1e-10.
  • dy_dx takes the step from the spacing of sorted unique x. For y = x^3 on 500 unsorted points, the derivatives at (-1.5, 0, 0.5, 1.5) go from 3.731 3.881 3.748 3.726 to 6.892 0.004 0.754 6.900, the sorted-input values.
  • nns_diff solves for h at tol / 10 and stops when the bracket on B is narrower than tol * max(1, |B|). The (x^4 - 3x)'(-2) error goes from 5.7e-4 to 5.1e-8.
  • dy_d differentiates the multivariate nns_reg fit on the actual regressors. For y = 2 x1 at three points, dy/dx2 (truth 0) goes from 0.644-0.753 to between -0.067 and 0.065, and dy/dx1 (truth 2) from 0.524-0.694 to 1.079-1.407. Magnitudes stay attenuated by the wide bandwidths. dy_d_best, which always returned what dy_d returns, is a deprecated alias.
  • Method 1 scores ALL in every fold with the estimator it deploys. On pure noise with one column, a local k = 6 now wins with OBJfn.reg 154.14, where ALL won on a constant-mean SSE of 126.865. This also fixes the nns_var(objective="max") multivariate stage.
  • nns_boost maps the final stack's class codes back to the response values: a -5/5 response came back as 1 1 1 2 2 and now as -5 -5 -5 5 5.
  • The R test helpers pass the seed to NNS.stack/NNS.boost, which re-seeded with 123 and so ignored seeds 7, 9 and 42. New cases pin R's single-member-class balancing fix and multivariate noise.reduction="mode", which needed no Python change.

Tests

New parity cases for each fix against R 13.3 with the audit's R patches; the NNS.stack.numeric, NNS.boost.numeric, NNS.diff.custom, dy.dx.numeric and dy.d.scalar shards are regenerated from it. The dy_dx row-order inputs move to a 1/1024 grid so the cache key is portable. The univariate n.best labels test compares n.best and predictions with R instead of asserting NA. The stack property strategy now skips nonzero values below 1e-100: on data such as {0, 2.2e-308, 1}, R's NNS.stack raises "No candidate produced a finite out-of-fold objective" or a nonfinite point.est, and the port now raises the same errors.

Two xfail marks are dropped. The Boston factor-path practical example, xfailed since #155, passes with the cor and X* rounding fix, and test_public_nns_var_objective_max_matches_r passes with the ALL fix.

Mirrors OVVO-Financial/NNS#80, OVVO-Financial/NNS#83 and OVVO-Financial/NNS#84
Part of OVVO-Financial/NNS#65

Review

GPT-6.1 Sol reviewed this slice and raised one finding, which we rejected: the intercept-scaled stop in nns_diff claims convergence with a 2% error once a constant is added to f. The R branch this slice follows has the same rule, and #168 mirrors R's fix. At the top of the stack the reviewer's case returns 0.001999999995 and does not claim convergence.

NNS partitions are discontinuous in X*: a last-ulp change can move a
regression point. Three computations in the dimension-reduction path
rounded differently from R, so folds partitioned differently, and the OOF
objectives, n.best, weights and predictions drifted from R:

- dim.red.method = "cor" used float64 two-pass Pearson on ranks. R's
  C_cor (cov.c) takes double deviations from long-double means
  (with a refinement pass), sums products sequentially in long double,
  and takes long-double square roots. On Boston, 6 of 13 coefficients
  were 1 ulp off.
- The per-fold Method 1 design used `design @ coef` (OpenBLAS). R's
  `%*%` is reference-BLAS dgemv, which accumulates column by column.
- The Method 2 fold candidates and the final projection used `@` too.
  R builds them with NNS_xstar_path_cpp, which accumulates in
  |coef|-descending order and divides by the running count of non-zero
  coefficients.

The port now uses an R-exact Spearman cor, a column-sequential
mat-vec, and a port of NNS_xstar_path_cpp. The path also builds all
cumulative projections of a fold in one pass instead of one product per
dimension count. Only the numerics change.

Evidence:
- cor: bit-identical to R on 180 of 180 coefficients (30 random designs
  with ties); the old code matched 74.
- Audit A/B on 47 nns_stack cases vs NNS 13.3: no case moved away from
  R (changes elsewhere are at most 1e-13). OBJfn.reg goes from mismatched
  to 1e-15 on Boston default/pi/L1/folds3/seed7 and the CHAS factor
  cases. fac3_m12 stack goes from 4.4e-3 to 5e-15 and its weights from
  .01 to 3e-17 off. iris optimize.threshold=FALSE n.best goes from 3
  (R 1) to 1, with weights matching.
- New parity test (Boston, cv_size 0.25, folds 3 and 5): before, stack
  differs from R by up to 10.8; after, all outputs match at 1e-10.

Remaining stack mismatches come from separate findings (NNS.dep ties,
class coding, the "ALL" candidate) and are unchanged. nns_var still uses
the float64 helper: its cor(matrix) call goes through a different R
code path, and the coefficients are rounded to 2 digits.
With chas treated as a real two-level factor (two dummy columns, as in R) and
nns_stack reproducing R's cor and X* rounding, the factor path matches the R
reference again (stack RMSE, n.best, threshold and stack head).
… fix)

The finite-difference step was gravity(|diff(x)|) * h with diff() taken
on x in the order supplied. For unsorted x that is the typical gap
between consecutive rows (about sd(x)), so the 2..sqrt(n) step
multiples spanned the whole range and the derivative depended on row
order. R now uses gravity(diff(sort(unique(x)))); do the same.

Before -> after (y = x^3, 500 unsorted U(-2, 2) x (R seed 1), points
-1.5 0 0.5 1.5, truth 6.75 0 0.75 6.75; = patched R):
  unsorted  3.731 3.881 3.748 3.726 -> 6.892 0.004 0.754 6.900
  sorted    6.892 0.004 0.754 6.900 (unchanged)
Sorted inputs without ties are unchanged. R's other dy.dx fixes need no
Python change: one deleted dead code (no result change) and the other
stops early without eval.point, which dy_dx already did. The dy.dx.numeric
and dy.dx.overall shards are regenerated from the patched R.
…rror of the R fix)

Each iteration solved for the inferred h with uniroot at its default
tolerance (about 1.2e-4) and declared convergence when |h| < tol
(1e-10), so uniroot's stopping point decided both the derivative
accuracy and the "converged" flag: log'(5), tanh'(0.5) and
(x^4 - 3x)'(-2) were reported converged with errors of 4e-7 to 6e-4.
R now solves with uniroot(tol = tol / 10) and declares
convergence when the bisection bracket on B is narrower than
tol * max(1, |B|); nns_diff mirrors uniroot step for step, so do the same.

Before -> after (DERIVATIVE error, iterations, converged; = patched R):
  sin'(1)          -8.4e-08 29 1 ->  5.0e-10 30 1
  (x^3)'(2)         8.6e-06 29 1 ->  4.4e-08 32 1
  log'(5)          -3.8e-07 28 1 ->  2.6e-10 30 1
  tanh'(0.5)       -1.0e-05 31 1 -> -8.8e-10 28 1
  (x^4-3x)'(-2)     5.7e-04 30 1 ->  5.1e-08 32 1
Immediate exits (linear f, |x| at -1, point = 0) are unchanged. The
NNS.diff.custom shard is regenerated from the patched R.
Follow-up to the mirror of the R dy.dx fix: docs/conventions.md still
described the step as gravity(|diff(x)|) * h.
…e R fix)

The nns_stack_numeric and nns_boost_numeric R helpers called set.seed(seed)
and then NNS.stack / NNS.boost, which re-seed with their own default
seed = 123L, so every R reference requested with seed 7, 9 or 42
actually ran with seed 123 (the seeded parity tests only compare
structure, which hid it). Pass seed = as.integer(seed) to the R
functions instead (123L when no seed is given, R's default). Cache keys
do not include the script, so the NNS.stack.numeric and
NNS.boost.numeric shards are regenerated from the patched R.

Also pin the R fix (balancing samples within a one-member class
instead of from 1:g): NNS Python's R sample() emulation already indexed
the group, and with a single-member class the balanced Method 1 stack
now matches the patched R exactly.
… (mirror of the R fix)

In the Method 1 n.best search the ALL candidate was scored out of fold
as a constant prediction, mean(training y), but when ALL won the final
fit was nns_reg(n_best="all") (multivariate) or the univariate nns_reg
fit, both non-constant estimators; model selection compared the local
candidates against a model that was never deployed, and OBJfn.reg
reported the SSE of a constant mean. Score ALL in every fold
with the estimator used for the final fit, as R now does.

Before -> after (pure-noise y, 160 training rows, 5 folds, seed 5;
= patched R):
  one column:  ALL wins, OBJfn.reg 126.865 (constant-mean SSE)
               -> local k = 6 wins, OBJfn.reg 154.14
  two columns: ALL wins, OBJfn.reg 126.865 -> ALL wins, OBJfn.reg 127.398
               (out-of-fold SSE of nns_reg(n_best="all"))
The selected n.best and OBJfn.reg change wherever ALL was involved in
the decision; this also fixes the nns_var(objective="max") multivariate
stage, which now matches the patched R. The NNS.stack.*, NNS.boost.* and
NNS.VAR* shards are regenerated from the patched R (no further entry
changes).
With the stack "all" fix (the univariate "all" candidate is scored with
the model it deploys) a k candidate wins on the n.best test data, so the
test now compares n.best and the predictions with R instead of asserting
NA. Re-record that entry from R 13.3 + the patch series.
…or of the R fix)

nns_boost treats a numeric response as a regression unless
type="CLASS", but its final nns_stack call auto-classifies a numeric
response with exactly two unique values and returns class codes 1/2,
which were passed through as regression estimates, so a {0, 1} or
{-5, 5} response came back as 1s and 2s. When the final stack
reports class levels for a regression boost, map its codes (results
and pred.int) back to the response values, as R now does.

Before -> after (40 points, y = -5/5 by x > 0.3, 5 test rows,
learner_trials=5, epochs=3, seed 1; = patched R):
  results  1 1 1 2 2 -> -5 -5 -5 5 5
  (y = 0/1 likewise 1/2 -> 0/1)
Continuous and factor responses are unchanged. The NNS.boost.* shards
are regenerated from the patched R (no existing entry changes).
…he R fix)

R's native multivariate setup reduced noise.reduction = "mode" cells
with the exact most-frequent value (smallest on ties), while its pure-R
reference (and the order = "max" path) uses NNS.mode(discrete = FALSE).
NNS Python follows the reference, so it disagreed with R's
default path; R now calls NNS.mode in the native reducer too.

No Python change. New parity case (300 rounded 2-D points):
  nns_reg(x, y, noise_reduction="mode") R2 0.81303
  R 13.3 default 0.78043 -> patched R 0.81303 (= Python, = pure-R path)
… R fix)

dy_d estimated f(x +/- h e_wrt) with nns_stack on the equal-weight
synthetic regressor X* = rowMeans(x), for both the training design and
the evaluation points, so every estimate depended only on mean(x) and
each regressor received about the index slope / p whatever its true
effect: for y = 2*x1 the derivative wrt the irrelevant x2 was about 1.
R now uses the multivariate NNS.reg fit on the actual
regressors (NNS.reg(x, y, point.est = pts, order = NULL)); do the same.

Before -> after (y = 2*x1, 500 R-seed-1 normal pairs, points (-1,0),
(0,1), (1,-1); = patched R):
  dy/dx1 (truth 2):  0.694 0.524 0.666 -> 1.184 1.407 1.079
  dy/dx2 (truth 0):  0.644 0.753 0.723 -> 0.065 0.037 -0.067
Signs and relative effects are now right; magnitudes stay attenuated by
the wide v0.5.7 bandwidths, left for a follow-up R change.
Runtime is unchanged or lower (100 x 2, "mean" 0.23 -> 0.20 s,
"obs" 0.14 -> 0.09 s).

dy_d_best, a Python-only function that always returned exactly what
dy_d returns, becomes a deprecated alias of dy_d that emits a
DeprecationWarning. dy_d keeps messages=True, R's default. The
dy.d.scalar shard is regenerated from the patched R and new parity
cases pin the irrelevant-regressor behaviour.
…key is portable

x**3 is not correctly rounded on every numpy build, so the y values, and
with them the hashed R-cache key, differed in the last bits between
machines and the row-order entry missed. x is now drawn from
integers / 1024 in [-2, 2], where x * x * x is exact, as the bandwidth
test already does. Unpatched NNS 13.3 gives 3.696 3.877 3.790 3.707 on
these inputs, so the comment's range changes.

The new entry comes from the patched R; the old entry is removed.
With the stack fixes mirrored, the case matches R again.
The new B-bracket stopping rule moves the quintic case from Inferred h
8.5e-11 after 32 iterations to 1.187e-09 after 31, in R and Python alike.
They point at notes that don't live in this repo.
Data such as {0, 2.2e-308, 1} make R's NNS.stack fail with "No candidate
produced a finite out-of-fold objective" or a nonfinite point.est, and the
port now raises the same errors, so the finite-output property can't hold
there.
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head 898260bcbc9fd1b91bf5db989e231cd29ae91f00 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/r-fixes-regression-stack branch November 7, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant