Repository navigation
[R14 integration] Preserve full function-port stack and align Python with NNS R 14.0 - #192
Conversation
Below 75 columns, sd_efficient_set and nns_sd_cluster evaluated every column's curve on one global grid (np.unique of all columns) and only switched to per-pair evaluation at 75 columns. For degree 3 (piecewise quadratic LPM2) and continuous FSD (linear-fractional ratio), extra knots from unrelated columns can expose a crossing between a pair's own knots, so the set depended on which other columns were present: x = (-0.85, -1.52, 0.68), y = (-0.21, -0.33, -1.67), z = (-3, 0.16, 0.2) sd_efficient_set([x, y], 3) -> [0] (R: X_1) sd_efficient_set([x, y, z], 3) -> [0, 1] (R: X_1) [x, y, z] + 71 dominated junk cols -> [0, 1] (R: X_1) [x, y, z] + 72 dominated junk cols -> [0] (R: X_1) docs/conventions.md already promised per-pair grids. All sizes now use the paths previously reserved for >= 75 columns (order-statistic matrix for degree-1 discrete, lazy kept-only prefix scan otherwise), and the now-unused global-grid code and the column switch are removed. For degree-1 discrete and degree 2 the two grids were already equivalent, so those results do not change; degree 3 and continuous FSD now match R in the cases above. (Checking TSD and continuous FSD only at the knots is shared with R and is left as is.) Matches R on the 11-asset finance sets and clusters (degrees 1-3, continuous) and on 159 of the audit's 160 random matrices (the remaining one is R's mutual-dominance bug). Faster at small sizes too: 11 assets x 4232 days, degree 1/2/3/continuous: 23/25/31/51 ms -> 0.9/4.9/4.9/11 ms; nns_sd_cluster degree 2: 30 -> 7.4 ms. The tests that compared the removed global-grid path or toggled the column switch now compare against the per-pair bivariate test, plus an invariance test with unrelated columns and a parity test against R.
sd_efficient_set only checked 1 <= degree <= 3, so degree=1.5 fell through _prefix/curve dispatch to the LPM2 (third-order) curve with the degree > 1 mean gate on: on the audit's 11 assets it returned the 7-asset set [5, 0, 4, 6, 7, 1, 8], where R (as.integer(1.5) = 1, FSD) returns all 11. degree="2" raised a TypeError from the comparison. Both entry points (sd_efficient_set and nns_sd_cluster, which already used int()) now share one helper that truncates toward zero like R's as.integer and then requires 1, 2 or 3, raising ValueError otherwise (0.5, 4, NaN, None, "abc"). Integer degrees behave as before. Parity tests against R for 1.5, 2.9 and "2".
When the last peel empties the pool, nns_sd_cluster assigned
clusters["Cluster_<n>"] = [], creating an empty final cluster. R's
NNS.SD.cluster assigns colnames() of a 0-column matrix, which is NULL,
and assigning NULL to a list element adds nothing. With min_cluster >= 1
the empty cluster was later merged away, so only min_cluster=0 was
affected: five shifted copies gave {..., 'Cluster_5': ['X_5'],
'Cluster_6': []} where R stops at Cluster_5.
The final cluster is now only added when columns remain. Parity test
against R for min_cluster 0, 1 and 2.
API gaps against R found in the audit:
- fsd(x, y, type="continuous") raised a TypeError; R's NNS.FSD has
type=. fsd now takes type and runs the same bivariate test on the
degree-1 ratio LPM1/(LPM1 + UPM1). Where both partial moments vanish
(a constant sample) the ratio is 0, R's C++ convention (R's
R-level NNS.FSD returns NA there). On the audit's 798-case corpus it
matches R's NNS.FSD(type="continuous") everywhere R is not NA, except
one permuted-sample case where R's curve-identity check fails on
summation order and R reports dominance between identical samples.
- A misspelt type (e.g. "contin") was silently treated as discrete. Like
R it now warns "type needs to be either 'discrete' or 'continuous'"
and follows R's branch: continuous for NNS.FSD, discrete for
NNS.FSD.uni and the degree-1 efficient set / cluster.
- fsd/ssd/tsd and the *_uni wrappers rejected single-column 2-D inputs
("x must be 1D"); R accepts single-column matrices and data frames.
They are squeezed now, and wider inputs get R's message.
- nns_sd_cluster(DataFrame) labelled columns X_1.. instead of the
frame's column names (R uses colnames). An explicit names= still wins.
Default-argument results do not change. sd_efficient_set still returns
positional indices for DataFrames (R returns names) to keep its
documented list[int] return type.
_dominates_from_curves, _prefix_directional_candidates_to_target and _sd_efficient_set_names had no references anywhere. The all-pairs _dominance_matrix_from_prefix_pairs (with _prefix_pair_candidate_matrix, _prefix_directional_candidate_matrix and its block-size constant) was only called from tests, as a second vectorised evaluator to check the lazy kept-only scan against. The tests now check the production evaluators (_dominates_from_prefix_pair, the order-statistic matrix and the kept-only scan) against a per-pair oracle built from the bivariate fsd/ssd/tsd test, which is independent of the prefix code. No production behaviour changes.
nns_anova had its own _lpm_var/_upm_var that located the 25% and 12.5% partial-moment targets with scipy's bounded minimize_scalar, while R's LPM.VaR/UPM.VaR (and nns.var.lpm_var/upm_var, already ported) invert the integer-degree partial moments exactly. For SPY at 25% the optimiser gave -3.556843063e-03 vs -3.556845568e-03 in R, which moved Certainty by up to 2.7e-7 (SPY vs TLT 1.3e-7, finance pairwise matrix 2.7e-7, iris multi-group 3e-11). The targets now come from nns.var. The degree-0 bounds (effect-size and robust-certainty quantiles) use the same functions behind a small wrapper that keeps returning NaN when every certainty is NaN (degenerate groups). Also drops _lower_area_share, which computed r / (r + (1 - r)) for r = lpm_ratio(1, ...), i.e. lpm_ratio(1, ...) itself, as R does. Differences against R after the change (audit script anova_py.py): every binary, multi-group and pairwise case on iris and the 11-asset matrix is within 6.2e-15 (was 2.7e-7). means.only results are unchanged.
nns_anova drew its effect-size bootstrap (and robust resamples) from numpy's default_rng, so a seeded Python run could only agree with R in distribution. R draws sample(x, n_boot * length(x), replace = TRUE) per group from its Mersenne-Twister stream, which the package already reproduces in nns._rrng for nns_stack/nns_boost. - RRNG gains sample_int_replace(n, size): R's sample.int(n, size, replace = TRUE) with the R >= 3.6 rejection sampler, vectorised by loading R's MT state into numpy's MT19937 (the same generator) and cutting the stream into fixed-width attempts. It is draw-for-draw and state-for-state identical to the scalar sample_int (tested), and about 40x faster (5M draws: ~0.12 s). - nns_anova(random_seed=s) now replays set.seed(s): the 1000-replicate mean/median bootstrap is laid out as R's column-major matrix, and the effect-size bounds match R to rounding (atol 1e-12 in the new parity tests; e.g. seed 5: LB -0.3320447563778621 vs R -0.33204475637786213). Without a seed the stream starts from fresh entropy, as before. Seeded effect-size bounds therefore change value (same distribution). Certainty and the other keys do not change. The CI path costs a little more than with numpy's choice (n = 5000 per group: ~95 -> ~290 ms; R 582 ms). The robust index draws move to the same stream; their pairing is fixed separately. Test helper _call_r_anova_custom accepts optional seed/ci/tails keys (old payloads unchanged).
With tails="left" or "right" and a confidence_interval, Python seeded
control_upper = inf / control_lower = -inf and returned
Effect_Size_LB = -inf and Effect_Size_UB = +inf for both tails, i.e. no
information (x ~ N(0,1), y ~ N(1,1), n = 50: left (-inf, inf), right
(-inf, inf), both (0.44, 1.41)). R's NNS.ANOVA.bin crashes on the same
input ("object 'control_upper' not found" / "'control_lower' not
found"), including robust = TRUE.
The correct one-sided bounds are coordinated with the R patch (R has
the same bug), so for now nns_anova raises a ValueError explaining the
limitation, before drawing anything, whenever tails is one-sided and a
confidence_interval is requested. tails="both" and confidence_interval=None
(any tails) are unchanged. Parity test pins the R failure alongside.
Robust nns_anova resampled control and treatment with the same index vector, i.e. a paired bootstrap. R builds treatment_matrix as cbind(resample 1, resamples 100..1), so control resample g is paired with treatment resample 1 (g = 1) or 102 - g: 98 of the 100 replicates resample the two groups independently, which is also what the book describes. For correlated series the paired scheme kept the day-by-day co-movement: SPY vs QQQ (first 300 days, mean of 30 seeds) gave 0.927 [0.890, 0.956] in Python vs 0.862 [0.674, 0.962] in R. Now, on the R Mersenne-Twister stream from the previous commit: - the 100 index vectors are drawn first and the effect-size bootstrap after, in R's order, and paired exactly as R does; - the bounds are keyed like R, "Lower <level>% CI"/"Upper <level>% CI" (one-sided: "... bound", honouring tails), instead of "Lower/Upper Bound Robust Certainty"; - confidence_interval=None returns only "Robust Certainty Estimate", as R does, instead of 95% bounds. This is exact stream parity, not just statistical: for SPY/QQQ, seeds 1..30, estimate and bounds agree with R to 2.2e-15 per seed (means 0.86152428 / 0.67369344 / 0.96238441 on both sides); the new parity test covers means/medians, CI 0.95/0.9/None. R's robust truncation to min(n1, n2) observations is shared with R and kept. The effect-size keys stay "Effect_Size_LB"/"Effect_Size_UB" (R's "Effect_Size_LB.2.5%" is an unlist() artefact). Public robust results and key names change.
_as_group filtered values[np.isfinite(values)], so +-Inf was dropped as
silently as NaN: appending inf to a group left Certainty unchanged
(0.768). An infinite observation has no partial-moment meaning, and R's
binary NNS.ANOVA fails on it ("invalid 'xmax' value" inside LPM.VaR).
Infinite values now raise a ValueError naming the group; NaN is still
dropped (R's multi-group path does the same with na.omit; its binary
path propagates NaN, which is less useful). Both rules are now written
down in docs/conventions.md. Finite inputs are unaffected.
nns_norm treated any non-ndarray as a sequence of series, and iterating a DataFrame yields its column labels, so nns_norm(prices_df) failed with "could not convert string to float: 'SPY'". A data.frame is R's most common NNS.norm input. A DataFrame-like input (has columns and to_numpy) is now normalised through the matrix path and returned as a DataFrame with the original index and R's "<column> Normalized" column names. The numbers are bitwise identical to passing the same values as a C-ordered ndarray (and match R to 5e-15 on the audit's 11-asset prices). ndarray and list inputs are unchanged.
nns_ss checked isinstance(confidence_interval, bool), so a flag computed with numpy (np.True_ from a comparison or array element) was rejected with "confidence_interval must be a single TRUE/FALSE value", although R accepts any logical scalar. np.bool_ is now accepted; other types (e.g. the integer 1) still raise. Results do not change.
The vignette still looked up "Lower/Upper Bound Robust Certainty" and printed [nan, nan]; it now shows e.g. robust = 0.0113 [0.0000, 0.1288].
They point at notes that don't live in this repo.
The bivariate nns_copula(x, y) path used co_moments._as_pair, which checks neither size nor finiteness, and handed the data to the native copula_nd kernel, which assumes n >= 1: - nns_copula(np.array([]), np.array([])) crashed the interpreter (SIGSEGV) - one NaN in x returned 1.0 (R: "You have some missing values") - one Inf in x returned 0.728 (R: "`data` must be finite") - NaN/Inf target_x/target_y were accepted The matrix form already rejected these. The bivariate form now uses nns._helpers._as_pair (non-empty, equal length, finite) and validates the targets, and _copula refuses an empty matrix before calling the native kernel. Results for valid input are unchanged.
NNS.copula takes target <- colMeans(X), which R computes as a sequential long-double column sum divided by n. Python used np.mean (pairwise double summation), which can differ in the last ulp. The copula then compares sum(cupm + clpm) of the degree-0 PM.matrix to 1 exactly, and the ulp decides whether observations sitting on the mean count as lower or upper. Example (new parity test): x has np.mean 0.4999999999999999 and R colMeans 0.5, with four observations equal to 0.5. R NNS.copula(cbind(x, y)) = 1; Python returned 0.8819; it now returns 1. On R's dy.dx sin grid used by the regression default-order heuristic, nns_copula(m) goes from 1.0 to 0.5759122837209066, which is R's value to the last digit. nns_copula (bivariate and matrix form) now uses _r_col_means for the default targets. Explicit targets are unchanged; on continuous data the copula value is unchanged. The internal callers _reg_engine._dependence and regression._nns_copula_matrix still build their own np.mean targets; they can switch to nns.copula._r_col_means to close the np.mean vs colMeans gap fully.
Uni.caus needs NNS.dep(x, y, asym = TRUE) twice per direction. _asym_dep rebuilt the X-only partition in pure Python (_xonly_partition appends to lists and concatenates label strings per element) and then evaluated the Python copula per cell; at n = 2e4 61% of nns_causation time was in _xonly_partition. The native nns_dep_pair kernel computes the same value. _asym_dep now calls nns_dep(x, y, asym=True) for finite input and keeps the pure-Python version as _asym_dep_python for non-finite input. A/B against the previous implementation on the 41 audit cases plus random, small-n and integer data (tau 0/1/3, 135 calls, and causal_matrix with tau 0 and 2): every value identical. Timing (loaded 4-core box, one process): nns_causation n=1000 0.0176 s -> 0.0070 s causal_matrix 10 cols n=1000 1.109 s -> 0.399 s (R 0.39 s) causal_matrix 10 cols n=5000 3.578 s -> 1.780 s (R 1.59 s) Results do not change.
nns_part rejected order=np.int64(3) and order=3.0 with "TypeError: order must be an integer" (R accepts any integral numeric), silently treated order=0 as 1 (R: "[order] must be NULL, 'max', or a positive integer."), and accepted obs_req=2.5 (R: "[obs.req] must be a nonnegative integer."). order and obs_req now accept numbers.Integral (Python and NumPy integers) and integral, finite reals; order must be >= 1 and obs_req >= 0, otherwise ValueError. Booleans and non-numeric values raise TypeError. R's NNS.reg also errors for order = 0. The hypothesis property test drew order from 0..5; it now draws 1..5. Results for valid arguments are unchanged.
For noise_reduction other than "off" nns_part ran a Python loop that
called np.unique on growing string-label arrays at every depth, built a
boolean mask per group (O(n * groups)) to compute centres, assign digits
and aggregate regression points, and allocated the label arrays as
dtype "<U{order + 1}" although the depth never exceeds floor(log2(n)).
- Groups are dense integer ids whose numeric order is the lexicographic
order of their labels (current labels are the leaves of the partition
tree, so children sorted by (parent id, digit) keep label order); label
strings are built once per group, not per point.
- Digits are assigned for all split points at once.
- Per-group reductions read a stable argsort-by-group slice, which holds
exactly the values x[group == g] in the same order, so np.mean / the
mode / gravity give bit-identical results; medians are computed for
all groups from one lexsort (odd: middle value, even: mean of the two
middle values, as np.median).
- order is capped at floor(log2(n)) before anything is allocated (and
before it is passed to the native kernel, so a huge order cannot
overflow its C int).
A/B against the previous implementation: 26,624 calls (41 audit cases plus
random n = 1..333, Inf/NaN, integer and outlier data x type None/XONLY x
order None/1/2/3/6/40 x obs_req 0/1/8/50 x min_obs_stop x all five noise
reductions, plus the pure-Python "off" path against the native kernel):
order, quadrant, prior.quadrant and regression points identical.
Timing, one core on a loaded box, before -> after (R 13.3 in parentheses):
n = 1e4 mean 0.090 -> 0.005 s, median 0.172 -> 0.018 s,
mode 0.090 -> 0.011 s, XONLY median 0.631 -> 0.033 s
n = 1e5 mean 1.41 -> 0.038 s (R 0.057), median 3.21 -> 0.26 s (R 0.18),
mode 1.78 -> 0.16 s (R 0.125), mode_class 3.21 -> 0.18 s,
XONLY median 30.5 -> 0.55 s, XONLY mean 4.78 -> 0.074 s
Memory for n = 1000, noise_reduction="mean": order=20000 took
2.34 s and 328 MB peak, now 0.003 s and 0.1 MB (same result as order=9).
Label arrays are now as wide as the longest label rather than order + 1.
Results do not change.
R's NNS.dep(X) on a matrix or data frame returns Correlation and Dependence
matrices (NNS.dep.matrix); nns_dep(M) raised "missing 1 required
positional argument: 'y'" and nns_dep(M, None) "x and y must be 1D".
causal_matrix returned a bare ndarray even for a labelled DataFrame, while
R keeps the dimnames.
- nns_dep(x, y=None, asym=False): a 2-D array or DataFrame x with y=None
returns {"Correlation": M, "Dependence": M} built pairwise like R:
symmetric with a unit diagonal for asym=False; for asym=True entry
[i, j] = nns_dep(X[:, i], X[:, j], asym=True) (row -> column, the
orientation R actually returns; its man page says the reverse),
with R's NaN -> 0 on the lower triangle. A DataFrame input returns
DataFrames labelled with its columns. 1-D x without y, or a single
column, raises ValueError.
- The nns_dep docstring now says what asym=True measures (y as a function
of x) and that p.value is not implemented.
- causal_matrix returns a DataFrame labelled with the input columns for
DataFrame input; ndarray input still returns an ndarray.
Verified against R NNS.dep(X) and NNS.dep(X, asym = TRUE) on a 4-column
matrix (new parity tests, exact). The bivariate API and all existing
results are unchanged.
The internal gravity (nns.dependence._gravity and the native gravity_exact kernel behind nns_part "off", nns_dep and nns_causation) built a dense histogram of width IQR / sqrt(n) and clipped it at 1e6 bins (_MAX_DENSE_BINS / the C++ constant). R does not cap, so with an outlier every bulk value beyond bin 1e6 was moved into the last bin and the mode term landed on the outlier. Audit case (n = 1e4 normals, x[0] = -1e5): _gravity(x) = -21558.3 vs R NNS.gravity -2.510; nns_part(x, y, order = 1) regression point x = -21558 vs -2.51; nns_dep 0.3633 vs 0.3592. - _gravity_python counts only the occupied bins (runs of floor((v - x1) / width) over the sorted values), so no cap and no allocation are needed. The bin-count rule, including R's int overflow to one bin (kept as in R), is unchanged. Wherever the old dense code fit, the arithmetic is the same (bit-identical to gravity_exact on 600 randomised arrays in the new test); beyond 1e6 bins with tied maxima it uses R's long-double sum. - The native kernels cannot be rebuilt here, so they are guarded: _gravity uses gravity_exact only when the bin count of that call is below the cap (O(n) partition for the quartiles). For native partitions a cheap a-priori bound covers every subset of at least obs_req + 1 points (q3 - q1 >= min_j (s[j+k] - s[j]) over the full sample); if it does not prove safety, every gravity call of the returned partition is recomputed from the final labels and the prior.quadrant groups. If any would hit the cap, nns_part falls back to the exact Python partitioner and nns_dep to its Python path. - docs/conventions.md described a 4 * len(input) cap that the code never had; it now describes this behaviour. Checks: the guard never accepted a capped partition (3200 native vs Python comparisons on outlier/Cauchy/rounded data, 0 violations; no plain-normal case was flagged); the part A/B harness (26,624 calls) differs from the previous code only on its outlier dataset with noise "off". New parity tests match R on the outlier case for NNS.gravity, NNS.part (XY and XONLY) and NNS.dep (sym and asym; n = 1e4 Dependence 0.5995 -> R's 0.5733). Cost of the guards (one core): nns_dep unchanged; nns_part default n = 1e3 1.1 -> 1.3 ms, 1e4 17 -> 24 ms, 1e5 0.33 -> 0.42 s; XONLY obs_req = 0 (NNS.reg's call) 1.7 -> 2.3 ms at 1e3, 0.21 -> 0.25 s at 1e5; nns_reg n = 1e3 5.4 -> 6.0 ms; _gravity of 10 values 0.9 -> ~11 us. Removing the 1e6 clip in gravity_exact_impl (_nnscore_bindings.cpp) would make the guards unnecessary. Results change only where the cap was hit (they now match R).
- conventions.md "Partition" said R only distinguishes NULL from non-NULL type and that order="max" raises TypeError. Both R 13.3 and the code validate type (ValueError for anything but None/"XONLY") and support order="max"; it now also states the order/obs_req rules and that the returned order is the depth reached (<= floor(log2(n))). - conventions.md "Copula" and api_reference.md described only a bivariate scalar form; the matrix form with any number of columns is documented, with the colMeans default targets and input validation. - api_reference.md named the R API "Causal.matrix", which does not exist; it is NNS.caus on a matrix (internally NNS.caus.matrix). The orientation of causal_matrix and its DataFrame labels are documented. - nns_dep: document the matrix form, what asym=True measures and the asymmetric matrix orientation (row -> column, as R returns it), and that p.value is not implemented. The nns_part docstring gains the argument rules. (The stale "4 * len(input)" gravity cap text was replaced with the gravity fix.) Documentation only.
For x = linspace(-2, 2, 100), y = x^2, C(x--->y) is +18.44 while causal_matrix[0, 1] is -18.44, so the docs had the direction reversed.
They point at notes that don't live in this repo.
_mreg_distances built an (m, n, p) broadcast tensor plus two more temporaries of the same size for every query chunk, which made it the dominant cost of multivariate NNS.reg and of NNS.boost/NNS.stack: 8.1 s of 14 s in the n=1e4, p=5 default fit. Accumulate into (m, n) buffers one active predictor column at a time, with the same per-element arithmetic ((x - r) / range, |z| + z^2). The old reduction ran over a strided predictor axis, so NumPy already summed column by column in column order (also R's kernel order); the new code is bit-for-bit identical, including p >= 8 where NumPy would otherwise switch to pairwise sums. FACTOR counts mismatches exactly. The query chunk is reduced from 4e6 to 1e6 distance cells, which keeps the buffers cache-resident; predictions are row-independent, so the chunk size does not affect results. Results: bit-identical to the previous code on 56 nns_reg / nns_m_reg / nns_stack / nns_boost configurations (all dists, noise reductions, orders, class mode, Boston p=13). A new invariant test compares against the broadcast formulation for p in 1..40 and all four metrics. Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_reg mv n=1e4 p=5 default: 17.5 s -> 5.1 s - n.best = 1: 19.1 s -> 8.1 s; order = "max": 10.3 s -> 3.2 s - binary type = "CLASS": 11.3 s -> 4.8 s - 5 equity-return regressors (n=4200): 4.6 s -> 1.8 s - n=3e4, p=2, order=2, n.best=1: 0.60 s -> 0.61 s (p=2, unchanged) - nns_boost n=2000 p=5 (50/50): 30.2 s -> 24.1 s - nns_stack n=2000 p=5: 0.79 s -> 0.77 s
_mreg_build_rpm called _reduce_value (and through it nns_gravity and np.median) once per cell and per column, singleton cells included: 58k calls and 2.3 s in the n=1e4, p=5 default fit (9711 cells, almost all singletons), and 377k calls / 15.7 s cumulative in NNS.boost at n=2000, p=5. Reduce whole columns at once with closed forms that reproduce the reducers exactly: - a lone finite value is the value (NaN when non-finite, as before); - the gravity/median of two or three finite values is (lo + hi) / 2 or the middle value; - "+ 0.0" reproduces the -0.0 -> +0.0 normalisation of NumPy's mean/median (which accumulate from +0.0); the mode and class reducers keep the sign, as before. Larger cells, cells with non-finite members and the mode/class reducers of multi-member cells still call _reduce_value. Results: bit-identical (values and signed zeros) on the 56-configuration harness and on 4800 randomised cells (all noise reductions, class mode, ties, +/-0, NaN). An invariant test compares against the per-cell reducer. Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_reg mv n=1e4 p=5 default: 7.2 s -> 5.2 s - nns_boost n=2000 p=5 (50/50): 18.9 s -> 13.7 s
With order = "max" every distinct training row is its own regression point (duplicate rows share one), so the default k = 1 rule maps each row to its own RPM y.hat. R short-circuits this case in NNS.M.reg (Multivariate_Regression.R, "Maximum order with the default k = 1 fit is an exact limit condition"); Python still ran the full n x nrow(RPM) distance search to rediscover the identity (126x slower than R at n=1e4, p=5). Map each row to its RPM row through the sorted unique IDs, as R does with match(ids, row.ids). "+ 0.0" keeps the regression-mode result bit-identical to the gravity of a single tied value. Point estimates still use the distance rule. Results: bit-identical to the previous code on the harness (order="max" with and without duplicates, all noise reductions, class mode, Boston); vs R 13.3 on a 400-row duplicate-heavy design: max |py - R| = 0 for noise.reduction off/median, 1.8e-15 for mean (np.mean rounding, unchanged). Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_reg mv n=1e4 p=5 order = "max": 3.37 s -> 0.20 s (R 0.08 s)
Three per-query-row Python loops remained around the distance matrix: - _stable_topk: argsort of the candidates of each row (0.7 s in the n=1e4 default fit); - the k = 1 rule in _mreg_predict and _mreg_predict_path: one nns_gravity / weighted-mode call per query row (10k calls); - the class-mode weighted mode for k > 1: one np.unique + np.add.at per query row (10k calls, 0.5 s in the binary class fit). Replace them with whole-chunk operations that keep the exact semantics: - top-k: one argpartition per chunk; rows with exactly k candidates (distance <= k-th smallest, i.e. no tie at the boundary) sort their k indices and then stable-sort by distance; rows with boundary ties take every candidate from nonzero() and one stable lexsort by (row, distance), so equal distances stay in index order; rows with NaN distances keep the old per-row code; - k = 1: a unique nearest row is resolved directly (value, NaN when non-finite, -0.0 -> +0.0 like gravity's median); only true ties call the reducer; - class mode: one np.add.at over (row, class) in row-major order, which accumulates each total in the same order from the same 0.0 as the per-row code, so class ties break identically (smallest class). Results: bit-identical on the 56-configuration harness; invariant tests compare each helper against the previous per-row code on tie-heavy inputs (full ties, NaN, -0.0, zero/negative weights). Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - end-to-end the gain is within the noise of this machine: mv n=1e4 p=5 default 5.8 s -> 6.0 s, n.best = 1 3.6 s -> 3.5 s, binary class 3.9 s -> 4.0 s, nns_boost n=2000 p=5 9.9 s -> 11.1 s; - the replaced pieces, min of 5 in-process: top-k of a 103 x 9711 chunk with k = 92 7.7 ms -> 4.4 ms (625 x 1600, k = 200: 33 ms -> 12 ms); k = 1 ties per chunk 2.3 ms -> 1.2 ms; class weighted mode of 10000 x 77 neighbours 276 ms -> 69 ms.
With point_only = TRUE the multivariate engine returns R2 = NULL and Fitted.xy = NULL, and without confidence_interval the fitted values are not used for the interval residual quantiles either; they were still computed with a full n x nrow(RPM) distance search. NNS.boost fits every learner with point_only = TRUE and no interval, so ~80% of its prediction work (1280 training rows vs 320 holdout rows per learner at n=2000) was thrown away. R's NNS.M.reg does the same wasted search. Compute the fitted values only when they are reported or needed for the interval; the metric is computed only when reported. Results: Point.est, pred.int, RPM, n.best and every returned field are unchanged (bit-identical on the harness, which now includes point_only cases with and without intervals, class mode, order="max", n.best=1, and on nns_stack / nns_boost). An invariant test compares point_only fits with the full fit. Timing: median of 3 runs, before/after runs interleaved in separate processes, OMP/BLAS threads = 1, on a shared 4-core machine with load average 8-12 from other jobs (absolute times are inflated and noisy): - nns_boost n=2000 p=5 (50/50): 18.9 s -> 11.1 s
_mreg_ensemble_weights rebuilt the exponential, lognormal and
inverse-square rank components (two scipy pdf calls plus three
(m, k) broadcast copies and normalisations) on every call, although
they depend only on k. _mreg_predict calls it once per query chunk and
_mreg_predict_path (NNS.stack, the NNS.boost final stage) once per
chunk and per k = 2..kmax: 1.9 s of 5.8 s in nns_stack
at n=10000, p=5.
Cache the three normalised rows per k (lru_cache, read-only arrays) and
add them by broadcasting. Each row is normalised with the same
row-sum and division as a row of the former broadcast matrix, and the
components are added in the same order, so the weights are
bit-identical. The data-dependent components (t, inverse distance,
normal, RBF) are unchanged.
Results: bit-identical on the harness; invariant tests compare the
weights with the former implementation for k in {2, 3, 7, 40, 129} and
check that every column of the k-path equals the single-k prediction.
Timing: median of 3 runs, before/after runs interleaved in separate
processes, OMP/BLAS threads = 1, on a shared 4-core machine with load
average 8-12 from other jobs (absolute times are inflated and noisy):
- nns_stack n=10000 p=5: 5.7 s -> 4.9 s
- nns_stack n=2000 p=5: 1.43 s -> 1.29 s
Integration and cleanup completeLanded in Both final CI matrices passed: pre-merge run 42402380593 and the post-merge The original 44-open-PR queue is now zero. Cleanup run 42403007916 verified every recorded head before acting and completed successfully. All 41 function-port heads and #191 are preserved in main's ancestry. PRs still open were closed as incorporated with attribution comments; a Closed badge rather than Merged on an older stacked PR does not indicate discarded work. #148 and #192 have merged status; the other incorporated reviews were retired, not separately merged. 45 obsolete branches removed: 44 PR branches plus the temporary publication branch. No changed/unincorporated heads were deleted, and there were no cleanup exceptions. Three unrelated pre-existing branches were left untouched. Independent snapshots remain archived:
The cleanup artifact No new PyPI version was published by this operation. |
Verified NNS R 14.0 integration
This is the single integration landing for all 41 original function-port PRs (#148-#189 excluding #177), the #191 audit layer, and the complete R14 alignment repair. Original commits and authorship are preserved. Merge commit only; no squash or rebase.
Source of truth and exact publication
OVVO-Financial/NNS@7b07a837c130ba7220dbe16c584221508757d5ae, NNS 14.0.5060f6ec426c889501740bf8a585d41f37a746554cb6fdf81f301ca41232cd5b.cf1f01084e7edfe43db69bc4a7fd068e86476a6c4c6317219cf7bc3f8b3158b1.d2fb4b3de179980348108fd602ba1672ee12c193. All 56 file postimages matched the delivered manifest in publication verification.f2bda92aa006503d5f1ee26c8f9085e2bae0ee7c. The only additional change is an explicit float64-array annotation for the CI-pinned mypy 2.3.1. No numerical change, ignore directive, or gate relaxation.Final GitHub verification against main: PASS
Native backend CI, run 42402380593 completed successfully on the final head for Python 3.11, 3.12, 3.13 and 3.14. The jobs include native builds, invariant tests, cached R14 parity, pure-Python fallbacks, property tests, plotting tests, vignettes, Ruff, mypy, distribution builds, and strict final parity enforcement.
Additional checks passed:
Post-merge retirement
After observing the successful merge, verify all recorded heads against updated main before closing or deleting anything. Retire any still-open included function PRs and #191 with attribution comments. Archive #190's independent initial-cache snapshot and close it as superseded, not merged unchanged. Delete only exact preserved branch heads that no remaining open PR depends on; retain any advanced or unincorporated work.
No tolerance relaxation, fabricated R output, force-push, protection bypass, or premature PR closure was used. This integration updates main; it does not publish a PyPI release.