Repository navigation
Multivariate nns_reg, nns_stack and nns_boost: bit-identical speedups, R's means - #154
Closed
gitRasheed wants to merge 13 commits into
Closed
gitRasheed wants to merge 13 commits into
gitRasheed wants to merge 13 commits into
Conversation
_mreg_distances built an (m, n, p) broadcast tensor plus two more temporaries of the same size for every query chunk, which made it the dominant cost of multivariate NNS.reg and of NNS.boost/NNS.stack: 8.1 s of 14 s in the n=1e4, p=5 default fit. Accumulate into (m, n) buffers one active predictor column at a time, with the same per-element arithmetic ((x - r) / range, |z| + z^2). The old reduction ran over a strided predictor axis, so NumPy already summed column by column in column order (also R's kernel order); the new code is bit-for-bit identical, including p >= 8 where NumPy would otherwise switch to pairwise sums. FACTOR counts mismatches exactly. The query chunk is reduced from 4e6 to 1e6 distance cells, which keeps the buffers cache-resident; predictions are row-independent, so the chunk size does not affect results. Results: bit-identical to the previous code on 56 nns_reg / nns_m_reg / nns_stack / nns_boost configurations (all dists, noise reductions, orders, class mode, Boston p=13). A new invariant test compares against the broadcast formulation for p in 1..40 and all four metrics. Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_reg mv n=1e4 p=5 default: 17.5 s -> 5.1 s - n.best = 1: 19.1 s -> 8.1 s; order = "max": 10.3 s -> 3.2 s - binary type = "CLASS": 11.3 s -> 4.8 s - 5 equity-return regressors (n=4200): 4.6 s -> 1.8 s - n=3e4, p=2, order=2, n.best=1: 0.60 s -> 0.61 s (p=2, unchanged) - nns_boost n=2000 p=5 (50/50): 30.2 s -> 24.1 s - nns_stack n=2000 p=5: 0.79 s -> 0.77 s
_mreg_build_rpm called _reduce_value (and through it nns_gravity and np.median) once per cell and per column, singleton cells included: 58k calls and 2.3 s in the n=1e4, p=5 default fit (9711 cells, almost all singletons), and 377k calls / 15.7 s cumulative in NNS.boost at n=2000, p=5. Reduce whole columns at once with closed forms that reproduce the reducers exactly: - a lone finite value is the value (NaN when non-finite, as before); - the gravity/median of two or three finite values is (lo + hi) / 2 or the middle value; - "+ 0.0" reproduces the -0.0 -> +0.0 normalisation of NumPy's mean/median (which accumulate from +0.0); the mode and class reducers keep the sign, as before. Larger cells, cells with non-finite members and the mode/class reducers of multi-member cells still call _reduce_value. Results: bit-identical (values and signed zeros) on the 56-configuration harness and on 4800 randomised cells (all noise reductions, class mode, ties, +/-0, NaN). An invariant test compares against the per-cell reducer. Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_reg mv n=1e4 p=5 default: 7.2 s -> 5.2 s - nns_boost n=2000 p=5 (50/50): 18.9 s -> 13.7 s
With order = "max" every distinct training row is its own regression point (duplicate rows share one), so the default k = 1 rule maps each row to its own RPM y.hat. R short-circuits this case in NNS.M.reg (Multivariate_Regression.R, "Maximum order with the default k = 1 fit is an exact limit condition"); Python still ran the full n x nrow(RPM) distance search to rediscover the identity (126x slower than R at n=1e4, p=5). Map each row to its RPM row through the sorted unique IDs, as R does with match(ids, row.ids). "+ 0.0" keeps the regression-mode result bit-identical to the gravity of a single tied value. Point estimates still use the distance rule. Results: bit-identical to the previous code on the harness (order="max" with and without duplicates, all noise reductions, class mode, Boston); vs R 13.3 on a 400-row duplicate-heavy design: max |py - R| = 0 for noise.reduction off/median, 1.8e-15 for mean (np.mean rounding, unchanged). Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_reg mv n=1e4 p=5 order = "max": 3.37 s -> 0.20 s (R 0.08 s)
Three per-query-row Python loops remained around the distance matrix: - _stable_topk: argsort of the candidates of each row (0.7 s in the n=1e4 default fit); - the k = 1 rule in _mreg_predict and _mreg_predict_path: one nns_gravity / weighted-mode call per query row (10k calls); - the class-mode weighted mode for k > 1: one np.unique + np.add.at per query row (10k calls, 0.5 s in the binary class fit). Replace them with whole-chunk operations that keep the exact semantics: - top-k: one argpartition per chunk; rows with exactly k candidates (distance <= k-th smallest, i.e. no tie at the boundary) sort their k indices and then stable-sort by distance; rows with boundary ties take every candidate from nonzero() and one stable lexsort by (row, distance), so equal distances stay in index order; rows with NaN distances keep the old per-row code; - k = 1: a unique nearest row is resolved directly (value, NaN when non-finite, -0.0 -> +0.0 like gravity's median); only true ties call the reducer; - class mode: one np.add.at over (row, class) in row-major order, which accumulates each total in the same order from the same 0.0 as the per-row code, so class ties break identically (smallest class). Results: bit-identical on the 56-configuration harness; invariant tests compare each helper against the previous per-row code on tie-heavy inputs (full ties, NaN, -0.0, zero/negative weights). Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - end-to-end the gain is within the noise of this machine: mv n=1e4 p=5 default 5.8 s -> 6.0 s, n.best = 1 3.6 s -> 3.5 s, binary class 3.9 s -> 4.0 s, nns_boost n=2000 p=5 9.9 s -> 11.1 s; - the replaced pieces, min of 5 in-process: top-k of a 103 x 9711 chunk with k = 92 7.7 ms -> 4.4 ms (625 x 1600, k = 200: 33 ms -> 12 ms); k = 1 ties per chunk 2.3 ms -> 1.2 ms; class weighted mode of 10000 x 77 neighbours 276 ms -> 69 ms.
With point_only = TRUE the multivariate engine returns R2 = NULL and Fitted.xy = NULL, and without confidence_interval the fitted values are not used for the interval residual quantiles either; they were still computed with a full n x nrow(RPM) distance search. NNS.boost fits every learner with point_only = TRUE and no interval, so ~80% of its prediction work (1280 training rows vs 320 holdout rows per learner at n=2000) was thrown away. R's NNS.M.reg does the same wasted search. Compute the fitted values only when they are reported or needed for the interval; the metric is computed only when reported. Results: Point.est, pred.int, RPM, n.best and every returned field are unchanged (bit-identical on the harness, which now includes point_only cases with and without intervals, class mode, order="max", n.best=1, and on nns_stack / nns_boost). An invariant test compares point_only fits with the full fit. Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_boost n=2000 p=5 (50/50): 18.9 s -> 11.1 s
_mreg_ensemble_weights rebuilt the exponential, lognormal and
inverse-square rank components (two scipy pdf calls plus three
(m, k) broadcast copies and normalisations) on every call, although
they depend only on k. _mreg_predict calls it once per query chunk and
_mreg_predict_path (NNS.stack, the NNS.boost final stage) once per
chunk and per k = 2..kmax: 1.9 s of 5.8 s in nns_stack
at n=10000, p=5.
Cache the three normalised rows per k (lru_cache, read-only arrays) and
add them by broadcasting. Each row is normalised with the same
row-sum and division as a row of the former broadcast matrix, and the
components are added in the same order, so the weights are
bit-identical. The data-dependent components (t, inverse distance,
normal, RBF) are unchanged.
Results: bit-identical on the harness; invariant tests compare the
weights with the former implementation for k in {2, 3, 7, 40, 129} and
check that every column of the k-path equals the single-k prediction.
Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_stack n=10000 p=5: 5.7 s -> 4.9 s
- nns_stack n=2000 p=5: 1.43 s -> 1.29 s
The Spearman scores behind NNS.stack's dim.red.method = "cor" ranked each column with a Python while-loop over the sorted values (one iteration per distinct value): 0.74 s of 5.8 s in nns_stack at n=10000, p=5. Compute the runs of equal sorted values with one comparison and assign (start + 1 + end) / 2 per run with np.repeat. The rank values are the same exact half-integers; NaNs still form singleton runs and -0.0 ties with 0.0, as before. Results: identical to the former loop on 3000 randomised vectors (ties, NaN, signed zeros, empty) and on the harness; an invariant test checks against scipy.stats.rankdata(method="average"). Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_stack n=10000 p=5: 5.4 s -> 4.3 s - _rank_average on 8000 values: 3.1 ms -> 1.0 ms (in-process)
_group_reduce_sorted (R's split(y, as.character(x)) for univariate order = "max" and for duplicate regression points) walked the sorted keys in a Python loop and called _reduce_value -> nns_gravity -> np.median once per group, singletons included: 100k calls and 3.8 s of 4.2 s for nns_reg(order = "max") at n=1e5 (R is 2.3x faster at n=1e4 and 1e5). Keep the 15-significant-digit key strings (they define R's groups and the returned keys), but find the group boundaries with one array comparison and reduce the groups with _reduce_cells, which reproduces _reduce_value exactly (lone values, two/three-value medians, larger groups through the reducer). The function now takes noise_reduction and is_class instead of a reducer callable (private, three call sites). Results: bit-identical (values and signed zeros) on 12000 randomised groupings, including keys equal only to 15 significant digits, and on the harness's univariate cases; an invariant test compares against the former loop. Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_reg univariate n=1e4, order = "max": 0.283 s -> 0.035 s (R 0.071 s) - n=1e5, order = "max": 2.77 s -> 0.28 s (R 0.99 s)
np.mean (pairwise double sums) does not reproduce the tied value of a
constant group: np.mean(np.full(30, 19.58)) is 19.580000000000002,
while R's mean() and NNS_part_cpp's mean_no_na accumulate in long
double and return 19.58. On Boston's indus that ulp moved a
partition boundary above 30 tied observations, so 30 multivariate
NNS.ID values (and the RPM row order) differed from R.
Add nns._rmean with exact ports:
- r_mean: R's mean() (summary.c): sequential long-double sum / n plus
the second pass s += sum(x - s) / n; used by _reduce_value("mean")
(multivariate RPM cells, univariate split groups) and NNS.part's
order = "max" reducer, which call mean() in R;
- r_mean_no_na / r_group_means_no_na: NNS_part_cpp's mean_no_na
(long-double sum of non-NaN values / count, in row order); used for
NNS.part's split centres and regression points.
np.longdouble is the 80-bit x87 type R uses on x86-64.
Evidence (R 13.3, 17-digit output):
- r_mean equals R mean() bit for bit on 400 random vectors (np.mean
differed on 247);
- Boston nns_reg(noise_reduction = "mean"): NNS.ID mismatches 30 -> 0,
rhs.partitions and RPM now bit-identical to R (were up to 9.7e-13
off and row-permuted); fitted/point estimates stay within 2.1e-14 /
1.1e-14 of R (unchanged; distance/weight arithmetic);
- nns_part(noise_reduction = "mean"), XONLY order 5, NULL order 4 and
order "max": regression points bit-identical to R (were 2e-16 to
1.8e-15 off);
- nns_m_reg(noise_reduction = "mean") on Boston[, 1:4]: max |py - R|
0.456 -> 1.4e-14, R2 0.39766 -> 0.40153 (R 0.4015256).
Public results change only for noise_reduction = "mean" (towards R);
all other noise reductions are bit-identical. The cost is negligible.
Parity tests compare NNS.ID, partitions and NNS.part quadrants with R
(new R cache entries); unit tests cover tied values, NaN, inf, signed
zeros and long-double overflow headroom.
target=None used np.mean, which can miss a tied decimal value by an ulp: np.full(3, 0.7) gives 0.6999999999999998, so LPM.ratio/UPM.ratio and Co.LPM/Co.UPM at degree 0 returned 0 / 1 where R gives 1 / 0. Use r_mean, R's two-pass long-double mean.
_group_reduce_sorted reproduces R's split(y, x) by grouping on 15-digit
key strings, but f"{-0.0:.15g}" is "-0" while R's as.character(-0) is
"0". A stable sort leaves -0.0 and 0.0 interleaved in input order, so
every alternation started a new group: univariate order = "max" (and
the duplicate-regression-point merge) produced several regression
points at x = 0 where R has one. Signed zeros are common in practice
(np.round(-0.04, 1) is -0.0).
Normalise the key with "+ 0.0" before formatting, so -0.0 and 0.0 share
the key "0" and the returned regression point is 0.0, like R.
Evidence (R 13.3), 600 points with 7 zeros (4 of them -0.0):
regression points 134 -> 129 (R 129); max |fitted - R| 1.70 -> 0
(bit-identical); point estimate at 0: 0.95 -> -0.749 (R -0.749). On a
1500-point series rounded to 1 decimal (31 -0.0 values) order = "max"
had 82 regression points instead of 58.
Only inputs with signed zero keys change. A parity test checks
regression points, fitted values and point estimates against R
(off/median), plus a unit test of the grouping; the reference loop in
the group-reduction invariant test gets the same key normalisation.
… tests Every other test imports `_r` through pytest's pythonpath. `tests._r` only resolves when the repository root is on sys.path, so `uv run pytest` failed to collect these two files, and under `python -m pytest` it loaded a second copy of the module with its own cache.
They point at notes that don't live in this repo.
gitRasheed
force-pushed
the
rasheed/mreg-performance
branch
from
November 2, 2026 23:09
6eccffe to
cda10a6
Compare
gitRasheed
force-pushed
the
rasheed/dependence-partition
branch
from
November 2, 2026 23:09
7ac7c3b to
dd188de
Compare
Contributor
|
Incorporated through NNS R 14.0 integration #192, merged as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multivariate
nns_regwas 5-126x slower than R, andnns_boost3.5-4.5x. Most of the time went to a 3-D broadcast distance tensor, one reducer call per RPM cell, per-row Python loops around the distance matrix, and fitted values thatpoint_onlythrows away. We replace each with whole-array code that gives bit-identical results. The slice also carries two parity fixes, fornoise_reduction="mean"and for signed zeros.Stacked on #153; merge with a merge commit.
Changes
_reduce_valuecall each. The same fit goes from 7.2 s to 5.2 s, andnns_boostat n = 2000, p = 5 from 18.9 s to 13.7 s.order="max"with k = 1 maps each row to its own RPM row, as R short-cuts it: 3.37 s to 0.20 s at n = 1e4, p = 5 (R 0.08 s).point_onlyand no interval, the fitted-value search is skipped.nns_boostat n = 2000, p = 5 goes from 18.9 s to 11.1 s.nns_stackat n = 10000, p = 5 from 5.7 s to 4.9 s), and_rank_averageis vectorised (5.4 s to 4.3 s).order="max"groups are reduced without a per-group call: n = 1e5 goes from 2.77 s to 0.28 s (R 0.99 s).noise_reduction="mean"uses R's long-doublemean()andNNS_part_cpp'smean_no_na. On Boston, 30 NNS.ID mismatches with R go to 0, andnns_m_regonBoston[, 1:4]goes from max |py - R| 0.456 to 1.4e-14. Other noise reductions are bit-identical.split()key, as R'sas.character(-0)is "0". On 600 points with 7 zeros, regression points go from 134 to R's 129.All timings are medians of interleaved runs on a shared 4-core machine and are noisy in absolute terms.
Tests
Every speedup has an invariant in
tests/invariants/test_mreg_fast_paths.pyagainst the previous formulation, and a 56-configuration harness was bit-identical before and after. New parity tests compare NNS.ID, partitions and quadrants fornoise_reduction="mean"and signed-zero regression points with R. Both files import the R helpers from_r, souv run pytestcollects them. The parity cache encodes unpatched R 13.3.Update
This push makes the default LPM/UPM and co-moment target use
r_mean, R's mean. Withnp.mean, a constant 0.7 sample sits one rounding error away from its own mean, so the lower and upper ratios came out 0/1. R gives 1/0, and so does this branch now.Review
GPT-6.1 Sol reviewed this slice and found nothing. It checked 208 base-equivalence configurations and live-R edge cases on both paths.