Skip to content

[R14 integration] Preserve full function-port stack and align Python with NNS R 14.0 - #192

Merged
OVVO-Financial merged 285 commits into
mainfrom
automation/r14-stack-cache-20261106
Nov 7, 2026
Merged

OVVO-Financial merged 285 commits into
mainfrom
automation/r14-stack-cache-20261106

Conversation

@OVVO-Financial

@OVVO-Financial OVVO-Financial commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Verified NNS R 14.0 integration

This is the single integration landing for all 41 original function-port PRs (#148-#189 excluding #177), the #191 audit layer, and the complete R14 alignment repair. Original commits and authorship are preserved. Merge commit only; no squash or rebase.

Source of truth and exact publication

  • R oracle: OVVO-Financial/NNS@7b07a837c130ba7220dbe16c584221508757d5ae, NNS 14.0.
  • R source archive SHA256: 5060f6ec426c889501740bf8a585d41f37a746554cb6fdf81f301ca41232cd5b.
  • Delivered patch SHA256: cf1f01084e7edfe43db69bc4a7fd068e86476a6c4c6317219cf7bc3f8b3158b1.
  • Repair commit: d2fb4b3de179980348108fd602ba1672ee12c193. All 56 file postimages matched the delivered manifest in publication verification.
  • Final head: f2bda92aa006503d5f1ee26c8f9085e2bae0ee7c. The only additional change is an explicit float64-array annotation for the CI-pinned mypy 2.3.1. No numerical change, ignore directive, or gate relaxation.
  • Temporary transfer files and their commit history are excluded from the integration.

Final GitHub verification against main: PASS

Native backend CI, run 42402380593 completed successfully on the final head for Python 3.11, 3.12, 3.13 and 3.14. The jobs include native builds, invariant tests, cached R14 parity, pure-Python fallbacks, property tests, plotting tests, vignettes, Ruff, mypy, distribution builds, and strict final parity enforcement.

Additional checks passed:

Post-merge retirement

After observing the successful merge, verify all recorded heads against updated main before closing or deleting anything. Retire any still-open included function PRs and #191 with attribution comments. Archive #190's independent initial-cache snapshot and close it as superseded, not merged unchanged. Delete only exact preserved branch heads that no remaining open PR depends on; retain any advanced or unincorporated work.

No tolerance relaxation, fabricated R output, force-push, protection bypass, or premature PR closure was used. This integration updates main; it does not publish a PyPI release.

Below 75 columns, sd_efficient_set and nns_sd_cluster evaluated every
column's curve on one global grid (np.unique of all columns) and only
switched to per-pair evaluation at 75 columns. For degree 3 (piecewise
quadratic LPM2) and continuous FSD (linear-fractional ratio), extra knots
from unrelated columns can expose a crossing between a pair's own knots,
so the set depended on which other columns were present:

  x = (-0.85, -1.52, 0.68), y = (-0.21, -0.33, -1.67), z = (-3, 0.16, 0.2)
  sd_efficient_set([x, y], 3)          -> [0]     (R: X_1)
  sd_efficient_set([x, y, z], 3)       -> [0, 1]  (R: X_1)
  [x, y, z] + 71 dominated junk cols   -> [0, 1]  (R: X_1)
  [x, y, z] + 72 dominated junk cols   -> [0]     (R: X_1)

docs/conventions.md already promised per-pair grids. All sizes now use
the paths previously reserved for >= 75 columns (order-statistic matrix
for degree-1 discrete, lazy kept-only prefix scan otherwise), and the
now-unused global-grid code and the column switch are removed. For
degree-1 discrete and degree 2 the two grids were already equivalent, so
those results do not change; degree 3 and continuous FSD now match R in
the cases above. (Checking TSD and continuous FSD only at the knots is
shared with R and is left as is.)

Matches R on the 11-asset finance sets and clusters (degrees 1-3,
continuous) and on 159 of the audit's 160 random matrices (the remaining
one is R's mutual-dominance bug). Faster at small sizes too:
11 assets x 4232 days, degree 1/2/3/continuous: 23/25/31/51 ms ->
0.9/4.9/4.9/11 ms; nns_sd_cluster degree 2: 30 -> 7.4 ms.

The tests that compared the removed global-grid path or toggled the
column switch now compare against the per-pair bivariate test, plus an
invariance test with unrelated columns and a parity test against R.
sd_efficient_set only checked 1 <= degree <= 3, so degree=1.5 fell
through _prefix/curve dispatch to the LPM2 (third-order) curve with the
degree > 1 mean gate on: on the audit's 11 assets it returned the 7-asset
set [5, 0, 4, 6, 7, 1, 8], where R (as.integer(1.5) = 1, FSD) returns all
11. degree="2" raised a TypeError from the comparison.

Both entry points (sd_efficient_set and nns_sd_cluster, which already
used int()) now share one helper that truncates toward zero like R's
as.integer and then requires 1, 2 or 3, raising ValueError otherwise
(0.5, 4, NaN, None, "abc"). Integer degrees behave as before. Parity
tests against R for 1.5, 2.9 and "2".
When the last peel empties the pool, nns_sd_cluster assigned
clusters["Cluster_<n>"] = [], creating an empty final cluster. R's
NNS.SD.cluster assigns colnames() of a 0-column matrix, which is NULL,
and assigning NULL to a list element adds nothing. With min_cluster >= 1
the empty cluster was later merged away, so only min_cluster=0 was
affected: five shifted copies gave {..., 'Cluster_5': ['X_5'],
'Cluster_6': []} where R stops at Cluster_5.

The final cluster is now only added when columns remain. Parity test
against R for min_cluster 0, 1 and 2.
API gaps against R found in the audit:

- fsd(x, y, type="continuous") raised a TypeError; R's NNS.FSD has
  type=. fsd now takes type and runs the same bivariate test on the
  degree-1 ratio LPM1/(LPM1 + UPM1). Where both partial moments vanish
  (a constant sample) the ratio is 0, R's C++ convention (R's
  R-level NNS.FSD returns NA there). On the audit's 798-case corpus it
  matches R's NNS.FSD(type="continuous") everywhere R is not NA, except
  one permuted-sample case where R's curve-identity check fails on
  summation order and R reports dominance between identical samples.
- A misspelt type (e.g. "contin") was silently treated as discrete. Like
  R it now warns "type needs to be either 'discrete' or 'continuous'"
  and follows R's branch: continuous for NNS.FSD, discrete for
  NNS.FSD.uni and the degree-1 efficient set / cluster.
- fsd/ssd/tsd and the *_uni wrappers rejected single-column 2-D inputs
  ("x must be 1D"); R accepts single-column matrices and data frames.
  They are squeezed now, and wider inputs get R's message.
- nns_sd_cluster(DataFrame) labelled columns X_1.. instead of the
  frame's column names (R uses colnames). An explicit names= still wins.

Default-argument results do not change. sd_efficient_set still returns
positional indices for DataFrames (R returns names) to keep its
documented list[int] return type.
_dominates_from_curves, _prefix_directional_candidates_to_target and
_sd_efficient_set_names had no references anywhere. The all-pairs
_dominance_matrix_from_prefix_pairs (with _prefix_pair_candidate_matrix,
_prefix_directional_candidate_matrix and its block-size constant) was
only called from tests, as a second vectorised evaluator to check the
lazy kept-only scan against.

The tests now check the production evaluators (_dominates_from_prefix_pair,
the order-statistic matrix and the kept-only scan) against a per-pair
oracle built from the bivariate fsd/ssd/tsd test, which is independent of
the prefix code. No production behaviour changes.
nns_anova had its own _lpm_var/_upm_var that located the 25% and 12.5%
partial-moment targets with scipy's bounded minimize_scalar, while R's
LPM.VaR/UPM.VaR (and nns.var.lpm_var/upm_var, already ported) invert the
integer-degree partial moments exactly. For SPY at 25% the optimiser gave
-3.556843063e-03 vs -3.556845568e-03 in R, which moved Certainty by up to
2.7e-7 (SPY vs TLT 1.3e-7, finance pairwise matrix 2.7e-7, iris multi-group
3e-11).

The targets now come from nns.var. The degree-0 bounds (effect-size and
robust-certainty quantiles) use the same functions behind a small wrapper
that keeps returning NaN when every certainty is NaN (degenerate groups).
Also drops _lower_area_share, which computed r / (r + (1 - r)) for
r = lpm_ratio(1, ...), i.e. lpm_ratio(1, ...) itself, as R does.

Differences against R after the change (audit script anova_py.py): every
binary, multi-group and pairwise case on iris and the 11-asset matrix is
within 6.2e-15 (was 2.7e-7). means.only results are unchanged.
nns_anova drew its effect-size bootstrap (and robust resamples) from
numpy's default_rng, so a seeded Python run could only agree with R in
distribution. R draws sample(x, n_boot * length(x), replace = TRUE) per
group from its Mersenne-Twister stream, which the package already
reproduces in nns._rrng for nns_stack/nns_boost.

- RRNG gains sample_int_replace(n, size): R's sample.int(n, size,
  replace = TRUE) with the R >= 3.6 rejection sampler, vectorised by
  loading R's MT state into numpy's MT19937 (the same generator) and
  cutting the stream into fixed-width attempts. It is draw-for-draw and
  state-for-state identical to the scalar sample_int (tested), and about
  40x faster (5M draws: ~0.12 s).
- nns_anova(random_seed=s) now replays set.seed(s): the 1000-replicate
  mean/median bootstrap is laid out as R's column-major matrix, and the
  effect-size bounds match R to rounding (atol 1e-12 in the new parity
  tests; e.g. seed 5: LB -0.3320447563778621 vs R -0.33204475637786213).
  Without a seed the stream starts from fresh entropy, as before.

Seeded effect-size bounds therefore change value (same distribution).
Certainty and the other keys do not change. The CI path costs a little
more than with numpy's choice (n = 5000 per group: ~95 -> ~290 ms; R 582
ms). The robust index draws move to the same stream; their pairing is
fixed separately. Test helper _call_r_anova_custom accepts optional
seed/ci/tails keys (old payloads unchanged).
With tails="left" or "right" and a confidence_interval, Python seeded
control_upper = inf / control_lower = -inf and returned
Effect_Size_LB = -inf and Effect_Size_UB = +inf for both tails, i.e. no
information (x ~ N(0,1), y ~ N(1,1), n = 50: left (-inf, inf), right
(-inf, inf), both (0.44, 1.41)). R's NNS.ANOVA.bin crashes on the same
input ("object 'control_upper' not found" / "'control_lower' not
found"), including robust = TRUE.

The correct one-sided bounds are coordinated with the R patch (R has
the same bug), so for now nns_anova raises a ValueError explaining the
limitation, before drawing anything, whenever tails is one-sided and a
confidence_interval is requested. tails="both" and confidence_interval=None
(any tails) are unchanged. Parity test pins the R failure alongside.
Robust nns_anova resampled control and treatment with the same index
vector, i.e. a paired bootstrap. R builds treatment_matrix as
cbind(resample 1, resamples 100..1), so control resample g is paired
with treatment resample 1 (g = 1) or 102 - g: 98 of the 100 replicates
resample the two groups independently, which is also what the book
describes. For correlated series the paired scheme kept the day-by-day
co-movement: SPY vs QQQ (first 300 days, mean of 30 seeds) gave
0.927 [0.890, 0.956] in Python vs 0.862 [0.674, 0.962] in R.

Now, on the R Mersenne-Twister stream from the previous commit:
- the 100 index vectors are drawn first and the effect-size bootstrap
  after, in R's order, and paired exactly as R does;
- the bounds are keyed like R, "Lower <level>% CI"/"Upper <level>% CI"
  (one-sided: "... bound", honouring tails), instead of
  "Lower/Upper Bound Robust Certainty";
- confidence_interval=None returns only "Robust Certainty Estimate",
  as R does, instead of 95% bounds.

This is exact stream parity, not just statistical: for SPY/QQQ, seeds
1..30, estimate and bounds agree with R to 2.2e-15 per seed (means
0.86152428 / 0.67369344 / 0.96238441 on both sides); the new parity
test covers means/medians, CI 0.95/0.9/None. R's robust truncation to
min(n1, n2) observations is shared with R and kept. The
effect-size keys stay "Effect_Size_LB"/"Effect_Size_UB" (R's
"Effect_Size_LB.2.5%" is an unlist() artefact). Public robust results
and key names change.
_as_group filtered values[np.isfinite(values)], so +-Inf was dropped as
silently as NaN: appending inf to a group left Certainty unchanged
(0.768). An infinite observation has no partial-moment meaning, and R's
binary NNS.ANOVA fails on it ("invalid 'xmax' value" inside LPM.VaR).

Infinite values now raise a ValueError naming the group; NaN is still
dropped (R's multi-group path does the same with na.omit; its binary
path propagates NaN, which is less useful). Both rules are now written
down in docs/conventions.md. Finite inputs are unaffected.
nns_norm treated any non-ndarray as a sequence of series, and iterating
a DataFrame yields its column labels, so nns_norm(prices_df) failed with
"could not convert string to float: 'SPY'". A data.frame is R's most
common NNS.norm input.

A DataFrame-like input (has columns and to_numpy) is now normalised
through the matrix path and returned as a DataFrame with the original
index and R's "<column> Normalized" column names. The numbers are
bitwise identical to passing the same values as a C-ordered ndarray
(and match R to 5e-15 on the audit's 11-asset prices). ndarray and list
inputs are unchanged.
nns_ss checked isinstance(confidence_interval, bool), so a flag computed
with numpy (np.True_ from a comparison or array element) was rejected
with "confidence_interval must be a single TRUE/FALSE value", although R
accepts any logical scalar. np.bool_ is now accepted; other types (e.g.
the integer 1) still raise. Results do not change.
The vignette still looked up "Lower/Upper Bound Robust Certainty" and printed
[nan, nan]; it now shows e.g. robust = 0.0113 [0.0000, 0.1288].
They point at notes that don't live in this repo.
The bivariate nns_copula(x, y) path used co_moments._as_pair, which checks
neither size nor finiteness, and handed the data to the native copula_nd
kernel, which assumes n >= 1:

- nns_copula(np.array([]), np.array([])) crashed the interpreter (SIGSEGV)
- one NaN in x returned 1.0 (R: "You have some missing values")
- one Inf in x returned 0.728 (R: "`data` must be finite")
- NaN/Inf target_x/target_y were accepted

The matrix form already rejected these. The bivariate form now uses
nns._helpers._as_pair (non-empty, equal length, finite) and validates the
targets, and _copula refuses an empty matrix before calling the native
kernel. Results for valid input are unchanged.
NNS.copula takes target <- colMeans(X), which R computes as a sequential
long-double column sum divided by n. Python used np.mean (pairwise double
summation), which can differ in the last ulp. The copula then compares
sum(cupm + clpm) of the degree-0 PM.matrix to 1 exactly, and the ulp decides
whether observations sitting on the mean count as lower or upper.

Example (new parity test): x has np.mean 0.4999999999999999 and R colMeans
0.5, with four observations equal to 0.5. R NNS.copula(cbind(x, y)) = 1;
Python returned 0.8819; it now returns 1. On R's dy.dx sin grid used by the
regression default-order heuristic, nns_copula(m) goes from 1.0 to
0.5759122837209066, which is R's value to the last digit.

nns_copula (bivariate and matrix form) now uses _r_col_means for the default
targets. Explicit targets are unchanged; on continuous data the copula value
is unchanged. The internal callers _reg_engine._dependence and
regression._nns_copula_matrix still build their own np.mean targets; they
can switch to nns.copula._r_col_means to close the np.mean vs colMeans
gap fully.
Uni.caus needs NNS.dep(x, y, asym = TRUE) twice per direction. _asym_dep
rebuilt the X-only partition in pure Python (_xonly_partition appends to
lists and concatenates label strings per element) and then evaluated the
Python copula per cell; at n = 2e4 61% of nns_causation time was in
_xonly_partition. The native nns_dep_pair kernel computes the same value.

_asym_dep now calls nns_dep(x, y, asym=True) for finite input and keeps
the pure-Python version as _asym_dep_python for non-finite input.

A/B against the previous implementation on the 41 audit cases plus random,
small-n and integer data (tau 0/1/3, 135 calls, and causal_matrix with tau
0 and 2): every value identical. Timing (loaded 4-core box, one process):

  nns_causation n=1000          0.0176 s -> 0.0070 s
  causal_matrix 10 cols n=1000  1.109 s  -> 0.399 s   (R 0.39 s)
  causal_matrix 10 cols n=5000  3.578 s  -> 1.780 s   (R 1.59 s)

Results do not change.
nns_part rejected order=np.int64(3) and order=3.0 with "TypeError: order
must be an integer" (R accepts any integral numeric), silently treated
order=0 as 1 (R: "[order] must be NULL, 'max', or a positive integer."),
and accepted obs_req=2.5 (R: "[obs.req] must be a nonnegative integer.").

order and obs_req now accept numbers.Integral (Python and NumPy integers)
and integral, finite reals; order must be >= 1 and obs_req >= 0, otherwise
ValueError. Booleans and non-numeric values raise TypeError. R's NNS.reg
also errors for order = 0.

The hypothesis property test drew order from 0..5; it now draws 1..5.
Results for valid arguments are unchanged.
For noise_reduction other than "off" nns_part ran a Python loop that
called np.unique on growing string-label arrays at every depth, built a
boolean mask per group (O(n * groups)) to compute centres, assign digits
and aggregate regression points, and allocated the label arrays as
dtype "<U{order + 1}" although the depth never exceeds floor(log2(n)).

- Groups are dense integer ids whose numeric order is the lexicographic
  order of their labels (current labels are the leaves of the partition
  tree, so children sorted by (parent id, digit) keep label order); label
  strings are built once per group, not per point.
- Digits are assigned for all split points at once.
- Per-group reductions read a stable argsort-by-group slice, which holds
  exactly the values x[group == g] in the same order, so np.mean / the
  mode / gravity give bit-identical results; medians are computed for
  all groups from one lexsort (odd: middle value, even: mean of the two
  middle values, as np.median).
- order is capped at floor(log2(n)) before anything is allocated (and
  before it is passed to the native kernel, so a huge order cannot
  overflow its C int).

A/B against the previous implementation: 26,624 calls (41 audit cases plus
random n = 1..333, Inf/NaN, integer and outlier data x type None/XONLY x
order None/1/2/3/6/40 x obs_req 0/1/8/50 x min_obs_stop x all five noise
reductions, plus the pure-Python "off" path against the native kernel):
order, quadrant, prior.quadrant and regression points identical.

Timing, one core on a loaded box, before -> after (R 13.3 in parentheses):

  n = 1e4  mean 0.090 -> 0.005 s, median 0.172 -> 0.018 s,
           mode 0.090 -> 0.011 s, XONLY median 0.631 -> 0.033 s
  n = 1e5  mean 1.41 -> 0.038 s (R 0.057), median 3.21 -> 0.26 s (R 0.18),
           mode 1.78 -> 0.16 s (R 0.125), mode_class 3.21 -> 0.18 s,
           XONLY median 30.5 -> 0.55 s, XONLY mean 4.78 -> 0.074 s

Memory for n = 1000, noise_reduction="mean": order=20000 took
2.34 s and 328 MB peak, now 0.003 s and 0.1 MB (same result as order=9).
Label arrays are now as wide as the longest label rather than order + 1.
Results do not change.
R's NNS.dep(X) on a matrix or data frame returns Correlation and Dependence
matrices (NNS.dep.matrix); nns_dep(M) raised "missing 1 required
positional argument: 'y'" and nns_dep(M, None) "x and y must be 1D".
causal_matrix returned a bare ndarray even for a labelled DataFrame, while
R keeps the dimnames.

- nns_dep(x, y=None, asym=False): a 2-D array or DataFrame x with y=None
  returns {"Correlation": M, "Dependence": M} built pairwise like R:
  symmetric with a unit diagonal for asym=False; for asym=True entry
  [i, j] = nns_dep(X[:, i], X[:, j], asym=True) (row -> column, the
  orientation R actually returns; its man page says the reverse),
  with R's NaN -> 0 on the lower triangle. A DataFrame input returns
  DataFrames labelled with its columns. 1-D x without y, or a single
  column, raises ValueError.
- The nns_dep docstring now says what asym=True measures (y as a function
  of x) and that p.value is not implemented.
- causal_matrix returns a DataFrame labelled with the input columns for
  DataFrame input; ndarray input still returns an ndarray.

Verified against R NNS.dep(X) and NNS.dep(X, asym = TRUE) on a 4-column
matrix (new parity tests, exact). The bivariate API and all existing
results are unchanged.
The internal gravity (nns.dependence._gravity and the native gravity_exact
kernel behind nns_part "off", nns_dep and nns_causation) built a dense
histogram of width IQR / sqrt(n) and clipped it at 1e6 bins
(_MAX_DENSE_BINS / the C++ constant). R does not cap, so with an outlier
every bulk value beyond bin 1e6 was moved into the last bin and the mode
term landed on the outlier. Audit case (n = 1e4 normals, x[0] = -1e5):
_gravity(x) = -21558.3 vs R NNS.gravity -2.510; nns_part(x, y, order = 1)
regression point x = -21558 vs -2.51; nns_dep 0.3633 vs 0.3592.

- _gravity_python counts only the occupied bins (runs of floor((v - x1) /
  width) over the sorted values), so no cap and no allocation are needed.
  The bin-count rule, including R's int overflow to one bin (kept as in
  R), is unchanged. Wherever the old dense code fit, the arithmetic is
  the same (bit-identical to gravity_exact on 600 randomised arrays in the
  new test); beyond 1e6 bins with tied maxima it uses R's long-double sum.
- The native kernels cannot be rebuilt here, so they are guarded:
  _gravity uses gravity_exact only when the bin count of that call is below
  the cap (O(n) partition for the quartiles). For native partitions a cheap
  a-priori bound covers every subset of at least obs_req + 1 points
  (q3 - q1 >= min_j (s[j+k] - s[j]) over the full sample); if it does not
  prove safety, every gravity call of the returned partition is recomputed
  from the final labels and the prior.quadrant groups. If any would hit the
  cap, nns_part falls back to the exact Python partitioner and nns_dep to
  its Python path.
- docs/conventions.md described a 4 * len(input) cap that the code never
  had; it now describes this behaviour.

Checks: the guard never accepted a capped partition (3200 native vs Python
comparisons on outlier/Cauchy/rounded data, 0 violations; no plain-normal
case was flagged); the part A/B harness (26,624 calls) differs from the
previous code only on its outlier dataset with noise "off". New parity
tests match R on the outlier case for NNS.gravity, NNS.part (XY and XONLY)
and NNS.dep (sym and asym; n = 1e4 Dependence 0.5995 -> R's 0.5733).

Cost of the guards (one core): nns_dep unchanged; nns_part default
n = 1e3 1.1 -> 1.3 ms, 1e4 17 -> 24 ms, 1e5 0.33 -> 0.42 s; XONLY
obs_req = 0 (NNS.reg's call) 1.7 -> 2.3 ms at 1e3, 0.21 -> 0.25 s at 1e5;
nns_reg n = 1e3 5.4 -> 6.0 ms; _gravity of 10 values 0.9 -> ~11 us.
Removing the 1e6 clip in gravity_exact_impl (_nnscore_bindings.cpp) would
make the guards unnecessary.

Results change only where the cap was hit (they now match R).
- conventions.md "Partition" said R only distinguishes NULL from non-NULL
  type and that order="max" raises TypeError. Both R 13.3 and the code
  validate type (ValueError for anything but None/"XONLY") and support
  order="max"; it now also states the order/obs_req rules and that the
  returned order is the depth reached (<= floor(log2(n))).
- conventions.md "Copula" and api_reference.md described only a bivariate
  scalar form; the matrix form with any number of columns is documented,
  with the colMeans default targets and input validation.
- api_reference.md named the R API "Causal.matrix", which does not exist;
  it is NNS.caus on a matrix (internally NNS.caus.matrix). The
  orientation of causal_matrix and its DataFrame labels are documented.
- nns_dep: document the matrix form, what asym=True measures and the
  asymmetric matrix orientation (row -> column, as R returns it), and that
  p.value is not implemented. The nns_part docstring gains the argument
  rules.

(The stale "4 * len(input)" gravity cap text was replaced with the
gravity fix.) Documentation only.
For x = linspace(-2, 2, 100), y = x^2, C(x--->y) is +18.44 while
causal_matrix[0, 1] is -18.44, so the docs had the direction reversed.
They point at notes that don't live in this repo.
_mreg_distances built an (m, n, p) broadcast tensor plus two more
temporaries of the same size for every query chunk, which made it the
dominant cost of multivariate NNS.reg and of NNS.boost/NNS.stack:
8.1 s of 14 s in the n=1e4, p=5 default fit.

Accumulate into (m, n) buffers one active predictor column at a time,
with the same per-element arithmetic ((x - r) / range, |z| + z^2).
The old reduction ran over a strided predictor axis, so NumPy already
summed column by column in column order (also R's kernel order); the
new code is bit-for-bit identical, including p >= 8 where NumPy would
otherwise switch to pairwise sums. FACTOR counts mismatches exactly.
The query chunk is reduced from 4e6 to 1e6 distance cells, which keeps
the buffers cache-resident; predictions are row-independent, so the
chunk size does not affect results.

Results: bit-identical to the previous code on 56 nns_reg / nns_m_reg /
nns_stack / nns_boost configurations (all dists, noise reductions,
orders, class mode, Boston p=13). A new invariant test compares against
the broadcast formulation for p in 1..40 and all four metrics.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_reg mv n=1e4 p=5 default: 17.5 s -> 5.1 s
- n.best = 1: 19.1 s -> 8.1 s; order = "max": 10.3 s -> 3.2 s
- binary type = "CLASS": 11.3 s -> 4.8 s
- 5 equity-return regressors (n=4200): 4.6 s -> 1.8 s
- n=3e4, p=2, order=2, n.best=1: 0.60 s -> 0.61 s (p=2, unchanged)
- nns_boost n=2000 p=5 (50/50): 30.2 s -> 24.1 s
- nns_stack n=2000 p=5: 0.79 s -> 0.77 s
_mreg_build_rpm called _reduce_value (and through it nns_gravity and
np.median) once per cell and per column, singleton cells included:
58k calls and 2.3 s in the n=1e4, p=5 default fit (9711 cells, almost
all singletons), and 377k calls / 15.7 s cumulative in NNS.boost at
n=2000, p=5.

Reduce whole columns at once with closed forms that reproduce the
reducers exactly:
- a lone finite value is the value (NaN when non-finite, as before);
- the gravity/median of two or three finite values is (lo + hi) / 2 or
  the middle value;
- "+ 0.0" reproduces the -0.0 -> +0.0 normalisation of NumPy's
  mean/median (which accumulate from +0.0); the mode and class reducers
  keep the sign, as before.
Larger cells, cells with non-finite members and the mode/class reducers
of multi-member cells still call _reduce_value.

Results: bit-identical (values and signed zeros) on the 56-configuration
harness and on 4800 randomised cells (all noise reductions, class mode,
ties, +/-0, NaN). An invariant test compares against the per-cell
reducer.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_reg mv n=1e4 p=5 default: 7.2 s -> 5.2 s
- nns_boost n=2000 p=5 (50/50): 18.9 s -> 13.7 s
With order = "max" every distinct training row is its own regression
point (duplicate rows share one), so the default k = 1 rule maps each
row to its own RPM y.hat. R short-circuits this case in NNS.M.reg
(Multivariate_Regression.R, "Maximum order with the default k = 1 fit
is an exact limit condition"); Python still ran the full
n x nrow(RPM) distance search to rediscover the identity (126x
slower than R at n=1e4, p=5).

Map each row to its RPM row through the sorted unique IDs, as R does
with match(ids, row.ids). "+ 0.0" keeps the regression-mode result
bit-identical to the gravity of a single tied value. Point estimates
still use the distance rule.

Results: bit-identical to the previous code on the harness (order="max"
with and without duplicates, all noise reductions, class mode, Boston);
vs R 13.3 on a 400-row duplicate-heavy design: max |py - R| = 0 for
noise.reduction off/median, 1.8e-15 for mean (np.mean rounding, unchanged).

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_reg mv n=1e4 p=5 order = "max": 3.37 s -> 0.20 s (R 0.08 s)
Three per-query-row Python loops remained around the distance matrix:
- _stable_topk: argsort of the candidates of each row (0.7 s in the
  n=1e4 default fit);
- the k = 1 rule in _mreg_predict and _mreg_predict_path: one
  nns_gravity / weighted-mode call per query row (10k calls);
- the class-mode weighted mode for k > 1: one np.unique + np.add.at
  per query row (10k calls, 0.5 s in the binary class fit).

Replace them with whole-chunk operations that keep the exact
semantics:
- top-k: one argpartition per chunk; rows with exactly k candidates
  (distance <= k-th smallest, i.e. no tie at the boundary) sort their k
  indices and then stable-sort by distance; rows with boundary ties take
  every candidate from nonzero() and one stable lexsort by (row,
  distance), so equal distances stay in index order; rows with NaN
  distances keep the old per-row code;
- k = 1: a unique nearest row is resolved directly (value, NaN when
  non-finite, -0.0 -> +0.0 like gravity's median); only true ties call
  the reducer;
- class mode: one np.add.at over (row, class) in row-major order, which
  accumulates each total in the same order from the same 0.0 as the
  per-row code, so class ties break identically (smallest class).

Results: bit-identical on the 56-configuration harness; invariant
tests compare each helper against the previous per-row code on
tie-heavy inputs (full ties, NaN, -0.0, zero/negative weights).

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- end-to-end the gain is within the noise of this machine: mv n=1e4
  p=5 default 5.8 s -> 6.0 s, n.best = 1 3.6 s -> 3.5 s, binary class
  3.9 s -> 4.0 s, nns_boost n=2000 p=5 9.9 s -> 11.1 s;
- the replaced pieces, min of 5 in-process: top-k of a 103 x 9711
  chunk with k = 92 7.7 ms -> 4.4 ms (625 x 1600, k = 200: 33 ms ->
  12 ms); k = 1 ties per chunk 2.3 ms -> 1.2 ms; class weighted mode
  of 10000 x 77 neighbours 276 ms -> 69 ms.
With point_only = TRUE the multivariate engine returns R2 = NULL and
Fitted.xy = NULL, and without confidence_interval the fitted values
are not used for the interval residual quantiles either; they were
still computed with a full n x nrow(RPM) distance search. NNS.boost
fits every learner with point_only = TRUE and no interval, so ~80% of
its prediction work (1280 training rows vs 320 holdout rows per
learner at n=2000) was thrown away. R's NNS.M.reg does the
same wasted search.

Compute the fitted values only when they are reported or needed for
the interval; the metric is computed only when reported.

Results: Point.est, pred.int, RPM, n.best and every returned field are
unchanged (bit-identical on the harness, which now includes point_only
cases with and without intervals, class mode, order="max", n.best=1,
and on nns_stack / nns_boost). An invariant test compares point_only
fits with the full fit.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_boost n=2000 p=5 (50/50): 18.9 s -> 11.1 s
_mreg_ensemble_weights rebuilt the exponential, lognormal and
inverse-square rank components (two scipy pdf calls plus three
(m, k) broadcast copies and normalisations) on every call, although
they depend only on k. _mreg_predict calls it once per query chunk and
_mreg_predict_path (NNS.stack, the NNS.boost final stage) once per
chunk and per k = 2..kmax: 1.9 s of 5.8 s in nns_stack
at n=10000, p=5.

Cache the three normalised rows per k (lru_cache, read-only arrays) and
add them by broadcasting. Each row is normalised with the same
row-sum and division as a row of the former broadcast matrix, and the
components are added in the same order, so the weights are
bit-identical. The data-dependent components (t, inverse distance,
normal, RBF) are unchanged.

Results: bit-identical on the harness; invariant tests compare the
weights with the former implementation for k in {2, 3, 7, 40, 129} and
check that every column of the k-path equals the single-k prediction.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_stack n=10000 p=5: 5.7 s -> 4.9 s
- nns_stack n=2000 p=5: 1.43 s -> 1.29 s
This was referenced Nov 7, 2026
@github-actions
github-actions Bot deleted the automation/r14-stack-cache-20261106 branch November 7, 2026 18:51

Copy link
Copy Markdown
Owner Author

Integration and cleanup complete

Landed in main as merge commit bd19c3303222af46da79d8c97f258248eb5674d2. The merged tree a545629e397ff80bcdc6b6a572bb0c8d3bd84b5f is identical to the tested final head f2bda92aa006503d5f1ee26c8f9085e2bae0ee7c.

Both final CI matrices passed: pre-merge run 42402380593 and the post-merge main run 42402976802, on Python 3.11, 3.12, 3.13 and 3.14. Each pre-merge JUnit artifact independently reports 2,118 passing parity tests and 721 passing invariant tests, zero failures/errors, and the 13 existing skips. Native/fallback, properties, plotting, vignettes, Ruff, mypy and distribution builds passed. Docs also built and deployed successfully in 42402976751.

The original 44-open-PR queue is now zero. Cleanup run 42403007916 verified every recorded head before acting and completed successfully. All 41 function-port heads and #191 are preserved in main's ancestry. PRs still open were closed as incorporated with attribution comments; a Closed badge rather than Merged on an older stacked PR does not indicate discarded work. #148 and #192 have merged status; the other incorporated reviews were retired, not separately merged.

45 obsolete branches removed: 44 PR branches plus the temporary publication branch. No changed/unincorporated heads were deleted, and there were no cleanup exceptions. Three unrelated pre-existing branches were left untouched.

Independent snapshots remain archived:

  • archive/r14/pr-190-20261107 -> bdd4708246e73fb4d3816be516928964f26aa3f2. Regenerate R parity cache for NNS 14.0 #190 was closed as superseded; its older cache was not merged unchanged.
  • archive/r14/publication-evidence-20261107 -> 5e9c1422aeb707386c10f3ab25e60103e919331f.

The cleanup artifact r14-pr-retirement-evidence includes every PR head, deletion, archive and final zero-open-PR result. ZIP SHA256: c9b5035eeed10e000421eafd2e00766cafbc098da3e55df54d72fbbacc492290.

No new PyPI version was published by this operation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants