Skip to content

Multivariate nns_reg, nns_stack and nns_boost: bit-identical speedups, R's means - #154

Closed
gitRasheed wants to merge 13 commits into
rasheed/dependence-partitionfrom
rasheed/mreg-performance
Closed

gitRasheed wants to merge 13 commits into
rasheed/dependence-partitionfrom
rasheed/mreg-performance

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Multivariate nns_reg was 5-126x slower than R, and nns_boost 3.5-4.5x. Most of the time went to a 3-D broadcast distance tensor, one reducer call per RPM cell, per-row Python loops around the distance matrix, and fitted values that point_only throws away. We replace each with whole-array code that gives bit-identical results. The slice also carries two parity fixes, for noise_reduction="mean" and for signed zeros.

Stacked on #153; merge with a merge commit.

Changes

  • Distances accumulate one predictor column at a time instead of an (m, n, p) tensor, in the same order as R's kernel. The n = 1e4, p = 5 default fit goes from 17.5 s to 5.1 s.
  • RPM cells with one to three members are reduced in closed form instead of one _reduce_value call each. The same fit goes from 7.2 s to 5.2 s, and nns_boost at n = 2000, p = 5 from 18.9 s to 13.7 s.
  • order="max" with k = 1 maps each row to its own RPM row, as R short-cuts it: 3.37 s to 0.20 s at n = 1e4, p = 5 (R 0.08 s).
  • Top-k selection, the k = 1 rule and the class weighted mode run per chunk instead of per row. End to end the gain is within this machine's noise, but the replaced pieces are faster on their own: the class weighted mode of 10000 x 77 neighbours goes from 276 ms to 69 ms.
  • With point_only and no interval, the fitted-value search is skipped. nns_boost at n = 2000, p = 5 goes from 18.9 s to 11.1 s.
  • The rank-only ensemble weights are cached per k (nns_stack at n = 10000, p = 5 from 5.7 s to 4.9 s), and _rank_average is vectorised (5.4 s to 4.3 s).
  • Univariate order="max" groups are reduced without a per-group call: n = 1e5 goes from 2.77 s to 0.28 s (R 0.99 s).
  • noise_reduction="mean" uses R's long-double mean() and NNS_part_cpp's mean_no_na. On Boston, 30 NNS.ID mismatches with R go to 0, and nns_m_reg on Boston[, 1:4] goes from max |py - R| 0.456 to 1.4e-14. Other noise reductions are bit-identical.
  • -0.0 and 0.0 share one split() key, as R's as.character(-0) is "0". On 600 points with 7 zeros, regression points go from 134 to R's 129.

All timings are medians of interleaved runs on a shared 4-core machine and are noisy in absolute terms.

Tests

Every speedup has an invariant in tests/invariants/test_mreg_fast_paths.py against the previous formulation, and a 56-configuration harness was bit-identical before and after. New parity tests compare NNS.ID, partitions and quadrants for noise_reduction="mean" and signed-zero regression points with R. Both files import the R helpers from _r, so uv run pytest collects them. The parity cache encodes unpatched R 13.3.

Update

This push makes the default LPM/UPM and co-moment target use r_mean, R's mean. With np.mean, a constant 0.7 sample sits one rounding error away from its own mean, so the lower and upper ratios came out 0/1. R gives 1/0, and so does this branch now.

Review

GPT-6.1 Sol reviewed this slice and found nothing. It checked 208 base-equivalence configurations and live-R edge cases on both paths.

_mreg_distances built an (m, n, p) broadcast tensor plus two more
temporaries of the same size for every query chunk, which made it the
dominant cost of multivariate NNS.reg and of NNS.boost/NNS.stack:
8.1 s of 14 s in the n=1e4, p=5 default fit.

Accumulate into (m, n) buffers one active predictor column at a time,
with the same per-element arithmetic ((x - r) / range, |z| + z^2).
The old reduction ran over a strided predictor axis, so NumPy already
summed column by column in column order (also R's kernel order); the
new code is bit-for-bit identical, including p >= 8 where NumPy would
otherwise switch to pairwise sums. FACTOR counts mismatches exactly.
The query chunk is reduced from 4e6 to 1e6 distance cells, which keeps
the buffers cache-resident; predictions are row-independent, so the
chunk size does not affect results.

Results: bit-identical to the previous code on 56 nns_reg / nns_m_reg /
nns_stack / nns_boost configurations (all dists, noise reductions,
orders, class mode, Boston p=13). A new invariant test compares against
the broadcast formulation for p in 1..40 and all four metrics.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_reg mv n=1e4 p=5 default: 17.5 s -> 5.1 s
- n.best = 1: 19.1 s -> 8.1 s; order = "max": 10.3 s -> 3.2 s
- binary type = "CLASS": 11.3 s -> 4.8 s
- 5 equity-return regressors (n=4200): 4.6 s -> 1.8 s
- n=3e4, p=2, order=2, n.best=1: 0.60 s -> 0.61 s (p=2, unchanged)
- nns_boost n=2000 p=5 (50/50): 30.2 s -> 24.1 s
- nns_stack n=2000 p=5: 0.79 s -> 0.77 s
_mreg_build_rpm called _reduce_value (and through it nns_gravity and
np.median) once per cell and per column, singleton cells included:
58k calls and 2.3 s in the n=1e4, p=5 default fit (9711 cells, almost
all singletons), and 377k calls / 15.7 s cumulative in NNS.boost at
n=2000, p=5.

Reduce whole columns at once with closed forms that reproduce the
reducers exactly:
- a lone finite value is the value (NaN when non-finite, as before);
- the gravity/median of two or three finite values is (lo + hi) / 2 or
  the middle value;
- "+ 0.0" reproduces the -0.0 -> +0.0 normalisation of NumPy's
  mean/median (which accumulate from +0.0); the mode and class reducers
  keep the sign, as before.
Larger cells, cells with non-finite members and the mode/class reducers
of multi-member cells still call _reduce_value.

Results: bit-identical (values and signed zeros) on the 56-configuration
harness and on 4800 randomised cells (all noise reductions, class mode,
ties, +/-0, NaN). An invariant test compares against the per-cell
reducer.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_reg mv n=1e4 p=5 default: 7.2 s -> 5.2 s
- nns_boost n=2000 p=5 (50/50): 18.9 s -> 13.7 s
With order = "max" every distinct training row is its own regression
point (duplicate rows share one), so the default k = 1 rule maps each
row to its own RPM y.hat. R short-circuits this case in NNS.M.reg
(Multivariate_Regression.R, "Maximum order with the default k = 1 fit
is an exact limit condition"); Python still ran the full
n x nrow(RPM) distance search to rediscover the identity (126x
slower than R at n=1e4, p=5).

Map each row to its RPM row through the sorted unique IDs, as R does
with match(ids, row.ids). "+ 0.0" keeps the regression-mode result
bit-identical to the gravity of a single tied value. Point estimates
still use the distance rule.

Results: bit-identical to the previous code on the harness (order="max"
with and without duplicates, all noise reductions, class mode, Boston);
vs R 13.3 on a 400-row duplicate-heavy design: max |py - R| = 0 for
noise.reduction off/median, 1.8e-15 for mean (np.mean rounding, unchanged).

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_reg mv n=1e4 p=5 order = "max": 3.37 s -> 0.20 s (R 0.08 s)
Three per-query-row Python loops remained around the distance matrix:
- _stable_topk: argsort of the candidates of each row (0.7 s in the
  n=1e4 default fit);
- the k = 1 rule in _mreg_predict and _mreg_predict_path: one
  nns_gravity / weighted-mode call per query row (10k calls);
- the class-mode weighted mode for k > 1: one np.unique + np.add.at
  per query row (10k calls, 0.5 s in the binary class fit).

Replace them with whole-chunk operations that keep the exact
semantics:
- top-k: one argpartition per chunk; rows with exactly k candidates
  (distance <= k-th smallest, i.e. no tie at the boundary) sort their k
  indices and then stable-sort by distance; rows with boundary ties take
  every candidate from nonzero() and one stable lexsort by (row,
  distance), so equal distances stay in index order; rows with NaN
  distances keep the old per-row code;
- k = 1: a unique nearest row is resolved directly (value, NaN when
  non-finite, -0.0 -> +0.0 like gravity's median); only true ties call
  the reducer;
- class mode: one np.add.at over (row, class) in row-major order, which
  accumulates each total in the same order from the same 0.0 as the
  per-row code, so class ties break identically (smallest class).

Results: bit-identical on the 56-configuration harness; invariant
tests compare each helper against the previous per-row code on
tie-heavy inputs (full ties, NaN, -0.0, zero/negative weights).

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- end-to-end the gain is within the noise of this machine: mv n=1e4
  p=5 default 5.8 s -> 6.0 s, n.best = 1 3.6 s -> 3.5 s, binary class
  3.9 s -> 4.0 s, nns_boost n=2000 p=5 9.9 s -> 11.1 s;
- the replaced pieces, min of 5 in-process: top-k of a 103 x 9711
  chunk with k = 92 7.7 ms -> 4.4 ms (625 x 1600, k = 200: 33 ms ->
  12 ms); k = 1 ties per chunk 2.3 ms -> 1.2 ms; class weighted mode
  of 10000 x 77 neighbours 276 ms -> 69 ms.
With point_only = TRUE the multivariate engine returns R2 = NULL and
Fitted.xy = NULL, and without confidence_interval the fitted values
are not used for the interval residual quantiles either; they were
still computed with a full n x nrow(RPM) distance search. NNS.boost
fits every learner with point_only = TRUE and no interval, so ~80% of
its prediction work (1280 training rows vs 320 holdout rows per
learner at n=2000) was thrown away. R's NNS.M.reg does the
same wasted search.

Compute the fitted values only when they are reported or needed for
the interval; the metric is computed only when reported.

Results: Point.est, pred.int, RPM, n.best and every returned field are
unchanged (bit-identical on the harness, which now includes point_only
cases with and without intervals, class mode, order="max", n.best=1,
and on nns_stack / nns_boost). An invariant test compares point_only
fits with the full fit.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_boost n=2000 p=5 (50/50): 18.9 s -> 11.1 s
_mreg_ensemble_weights rebuilt the exponential, lognormal and
inverse-square rank components (two scipy pdf calls plus three
(m, k) broadcast copies and normalisations) on every call, although
they depend only on k. _mreg_predict calls it once per query chunk and
_mreg_predict_path (NNS.stack, the NNS.boost final stage) once per
chunk and per k = 2..kmax: 1.9 s of 5.8 s in nns_stack
at n=10000, p=5.

Cache the three normalised rows per k (lru_cache, read-only arrays) and
add them by broadcasting. Each row is normalised with the same
row-sum and division as a row of the former broadcast matrix, and the
components are added in the same order, so the weights are
bit-identical. The data-dependent components (t, inverse distance,
normal, RBF) are unchanged.

Results: bit-identical on the harness; invariant tests compare the
weights with the former implementation for k in {2, 3, 7, 40, 129} and
check that every column of the k-path equals the single-k prediction.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_stack n=10000 p=5: 5.7 s -> 4.9 s
- nns_stack n=2000 p=5: 1.43 s -> 1.29 s
The Spearman scores behind NNS.stack's dim.red.method = "cor" ranked
each column with a Python while-loop over the sorted values (one
iteration per distinct value): 0.74 s of 5.8 s in nns_stack at
n=10000, p=5.

Compute the runs of equal sorted values with one comparison and assign
(start + 1 + end) / 2 per run with np.repeat. The rank values are the
same exact half-integers; NaNs still form singleton runs and -0.0 ties
with 0.0, as before.

Results: identical to the former loop on 3000 randomised vectors (ties,
NaN, signed zeros, empty) and on the harness; an invariant test checks
against scipy.stats.rankdata(method="average").

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_stack n=10000 p=5: 5.4 s -> 4.3 s
- _rank_average on 8000 values: 3.1 ms -> 1.0 ms (in-process)
_group_reduce_sorted (R's split(y, as.character(x)) for univariate
order = "max" and for duplicate regression points) walked the sorted
keys in a Python loop and called _reduce_value -> nns_gravity ->
np.median once per group, singletons included: 100k calls and 3.8 s of
4.2 s for nns_reg(order = "max") at n=1e5 (R is 2.3x faster at
n=1e4 and 1e5).

Keep the 15-significant-digit key strings (they define R's groups and
the returned keys), but find the group boundaries with one array
comparison and reduce the groups with _reduce_cells, which reproduces
_reduce_value exactly (lone values, two/three-value medians, larger
groups through the reducer). The function now takes noise_reduction
and is_class instead of a reducer callable (private, three call sites).

Results: bit-identical (values and signed zeros) on 12000 randomised
groupings, including keys equal only to 15 significant digits, and on
the harness's univariate cases; an invariant test compares against the
former loop.

Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_reg univariate n=1e4, order = "max": 0.283 s -> 0.035 s (R 0.071 s)
- n=1e5, order = "max": 2.77 s -> 0.28 s (R 0.99 s)
np.mean (pairwise double sums) does not reproduce the tied value of a
constant group: np.mean(np.full(30, 19.58)) is 19.580000000000002,
while R's mean() and NNS_part_cpp's mean_no_na accumulate in long
double and return 19.58. On Boston's indus that ulp moved a
partition boundary above 30 tied observations, so 30 multivariate
NNS.ID values (and the RPM row order) differed from R.

Add nns._rmean with exact ports:
- r_mean: R's mean() (summary.c): sequential long-double sum / n plus
  the second pass s += sum(x - s) / n; used by _reduce_value("mean")
  (multivariate RPM cells, univariate split groups) and NNS.part's
  order = "max" reducer, which call mean() in R;
- r_mean_no_na / r_group_means_no_na: NNS_part_cpp's mean_no_na
  (long-double sum of non-NaN values / count, in row order); used for
  NNS.part's split centres and regression points.
np.longdouble is the 80-bit x87 type R uses on x86-64.

Evidence (R 13.3, 17-digit output):
- r_mean equals R mean() bit for bit on 400 random vectors (np.mean
  differed on 247);
- Boston nns_reg(noise_reduction = "mean"): NNS.ID mismatches 30 -> 0,
  rhs.partitions and RPM now bit-identical to R (were up to 9.7e-13
  off and row-permuted); fitted/point estimates stay within 2.1e-14 /
  1.1e-14 of R (unchanged; distance/weight arithmetic);
- nns_part(noise_reduction = "mean"), XONLY order 5, NULL order 4 and
  order "max": regression points bit-identical to R (were 2e-16 to
  1.8e-15 off);
- nns_m_reg(noise_reduction = "mean") on Boston[, 1:4]: max |py - R|
  0.456 -> 1.4e-14, R2 0.39766 -> 0.40153 (R 0.4015256).

Public results change only for noise_reduction = "mean" (towards R);
all other noise reductions are bit-identical. The cost is negligible.
Parity tests compare NNS.ID, partitions and NNS.part quadrants with R
(new R cache entries); unit tests cover tied values, NaN, inf, signed
zeros and long-double overflow headroom.
target=None used np.mean, which can miss a tied decimal value by an ulp:
np.full(3, 0.7) gives 0.6999999999999998, so LPM.ratio/UPM.ratio and
Co.LPM/Co.UPM at degree 0 returned 0 / 1 where R gives 1 / 0. Use r_mean,
R's two-pass long-double mean.
_group_reduce_sorted reproduces R's split(y, x) by grouping on 15-digit
key strings, but f"{-0.0:.15g}" is "-0" while R's as.character(-0) is
"0". A stable sort leaves -0.0 and 0.0 interleaved in input order, so
every alternation started a new group: univariate order = "max" (and
the duplicate-regression-point merge) produced several regression
points at x = 0 where R has one. Signed zeros are common in practice
(np.round(-0.04, 1) is -0.0).

Normalise the key with "+ 0.0" before formatting, so -0.0 and 0.0 share
the key "0" and the returned regression point is 0.0, like R.

Evidence (R 13.3), 600 points with 7 zeros (4 of them -0.0):
regression points 134 -> 129 (R 129); max |fitted - R| 1.70 -> 0
(bit-identical); point estimate at 0: 0.95 -> -0.749 (R -0.749). On a
1500-point series rounded to 1 decimal (31 -0.0 values) order = "max"
had 82 regression points instead of 58.

Only inputs with signed zero keys change. A parity test checks
regression points, fitted values and point estimates against R
(off/median), plus a unit test of the grouping; the reference loop in
the group-reduction invariant test gets the same key normalisation.
… tests

Every other test imports `_r` through pytest's pythonpath. `tests._r`
only resolves when the repository root is on sys.path, so `uv run pytest`
failed to collect these two files, and under `python -m pytest` it loaded
a second copy of the module with its own cache.
They point at notes that don't live in this repo.
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head cda10a6cef77d74d353826cd3bd39a69d6207572 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/mreg-performance branch November 7, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant