Repository navigation
Re-vendor NNS-core, bind its M.reg kernels and honour ncores - #157
Closed
gitRasheed wants to merge 14 commits into
Closed
gitRasheed wants to merge 14 commits into
gitRasheed wants to merge 14 commits into
Conversation
This was referenced Nov 2, 2026
gitRasheed
force-pushed
the
rasheed/native-core
branch
from
November 2, 2026 04:12
e8c62df to
a0536f1
Compare
extern/NNS-core was a port of the R 13.0 src tree. Re-vendor NNS-core with scripts/sync_nns_core_snapshot.py. That snapshot ports R 13.3 and fixes, ahead of R: - The vectorised (>32 targets) LPM/UPM backend cancelled catastrophically for degree >= 3. UPM(8, 100 targets, n=500) had a max relative error of 2.3e4 against the scalar path (R 13.3 gives the same wrong values); it is now 1.5e-15, and the native and NumPy results agree. - Discrete NNS.mode beyond the int range returned -2147483648 (now 3e9). Gravity/mode histograms have O(n) bins: nns_gravity on 1000 points at 5 +- 1e-7 plus one 6 took 8.1 s, now <1 ms, with the same value. smooth_counts_tri7 no longer reads past the end of the counts. - generate_vectors bounds checks, negative/NaN degrees rejected. - fast_lm means are long double, two pass, like Rcpp. - New kernels: mreg_setup/predict/predict_path and xstar_path (R 13.1-13.3 M.reg and NNS.stack), and a global thread setting with a serial threshold. None of these are bound yet. Public results change only where R 13.3 is wrong: >32-target LPM/UPM at degree >= 3, discrete nns_mode beyond the int range, and nns_gravity/nns_mode when the histogram would need more than INT_MAX bins. In that last case R's int cast overflows and the histogram collapses to one bin: for 1e4 normals with one value at -1e9, nns_gravity was -250024999.98 (= R 13.3). It is now -49999.98, the value the patched R (gravity bin fix) gives. The bindings build unchanged. The full suite gives the same results as before. The 9 vignette subprocess tests fail in this environment because nns is not installed there (PYTHONPATH is the repo root); that is unrelated. No parity cache shard depended on the old >32-target kernel: the vector-target parity tests use 20 targets. So no shard needed regenerating. A new invariant test checks >32 targets at degrees 3-12 against the scalar kernel and the NumPy fallback; it fails on the old snapshot at degrees 8 and 12. The sync script now vendors only the nnscore build inputs (CMakeLists.txt, include/, src/), the same layout as before, and each entry still matches the upstream git object verbatim as verify_release_provenance.py requires. It also updates sync/nns_source.json in place instead of rewriting it. A rewrite would reset vendored_r_tarball and the notes, and would need R fields that describe a different snapshot, so the R fields are now optional.
nns_dep's per-cell copula took the cell means with np_mean, an emulation of NumPy's pairwise sum, in the native kernel and np.mean in the fallback. R's NNS_dep.cpp copula_signed (and NNS-core's dependence.cpp) uses a sequential double sum / n. On evenly spaced or tied data a point sits exactly on the cell mean, so a last-ulp difference in the mean moves it between the concordant and discordant counts. R also scales the counts by 1/n and takes the degree-1 shares in one pass, so `d0_Co == 1` could hold in one implementation and not the other. The native kernel now hashes the X-only partition labels with std::hash<std::string> and calls nns::dep_pair, NNS-core's port of R's NNS_dep_pair_core. It is R's code, and it also adds the cell terms in R's unordered_map order. The fallback (_copula_signed, _copula_degree0_unsigned) repeats R's sums in R's order. Its only difference is that it adds the cell terms in first-seen order, which can move the totals by a few ulp. Evidence (R NNS 13.3): - x = R seq(0, 4*pi, pi/100), y = sin(x): R Correlation 0.22716789893355843, Dependence 0.92465837278014773. Before: native 0.22781038989761246 / 0.9240158818160937. After: native equal to R bit for bit; fallback 0.2271678989335584 / 0.9246583727801478. - DEP A/B (work/DEP cases, 32 data sets, tolerance 1e-8): 55 mismatches before, 14 after. All 14 dep/asym/copula mismatches are gone, e.g. binary_eq Correlation 1.0 -> 0.588 = R, discrete_dep Dependence 0.5477 -> 0.4962 = R. 12 NNS.caus mismatches remain (mono_rank, step3, y_neg_x), plus the const_y error text. - NNS.dep(x_j, medv, asym=TRUE) on Boston now matches R for all 13 columns (ptratio 0.38679 -> 0.46074, black 0.26503 -> 0.30305 = R), and Uni.caus for 12 of 13. In the REG A/B (87 nns_reg cases), boston_dr_NNS.dep now matches R; 5 cases still differ. boston_dr_NNS.caus and _all differ because Uni.caus(medv, chas) is 0 vs R 0.0137. nns_norm's np.corrcoef is 2e-15 off R's cor(), which shifts the scaled binary column by an ulp and flips the cell copula. - The iris balance vignette call now matches R (OBJfn.dim.red 0.9643, weights .85/.15; before 0.9571, .88/.12), as do binary iris (OBJfn.reg, threshold), mtcars am (weights .44/.56, threshold .83; before .26/.74, .78), Boston method 1:2 dim.red "NNS.dep" (weights, stack) and the boost threshold=1.5 feature frequencies (crim/nox/dis 9, rm/lstat 8, ptratio 7). The "NNS.caus"/"all" stacks keep the Uni.caus difference above. Public results change on tied or evenly spaced data, toward R. The new parity test covers 5 tie-sensitive data sets, symmetric and asym. All 10 cases fail before this change and pass after, with the native kernel and with the fallback. The causation invariant now compares native and fallback to 1e-14 instead of bit for bit.
The bindings had their own gravity port (gravity_exact) for nns_part, nns_dep and nns_reg. It clipped its dense histogram at 1e6 bins, which put the mode term on the outlier, and it summed in pairwise double. R's NNS_gravity_cpp uses long-double sums, so gravity_exact matched R NNS.gravity in only 2030 of 3000 random vectors. To stay correct, dependence.py and part.py checked every gravity call of a native partition against the cap (_gravity_subsets_are_safe, _native_partition_is_exact, _dep_native_is_exact) and recomputed in Python when it could bind. gravity_exact is now nns::gravity from the re-vendored NNS-core. That is R's algorithm with an O(n) histogram that only shortens runs of empty bins, so no cap is needed and the guards are removed. The NumPy fallback (_gravity_python, now also behind nns_gravity) follows the same arithmetic: a long-double sequential mean and weighted bin average, bin names computed in double, occupied bins only, and the 2**52 width widening. Evidence: - 3000 vectors (normal, integer, rounded t2, offset exponential, -1e5 outlier, tight IQR + one far value): native gravity_exact == R NNS.gravity 3000/3000, fallback == native 3000/3000. - outlier.csv: _gravity -2.5102461601040074 -> -2.5102461601040078 (= R; it went through the Python path before). nns_dep and the nns_part(order=1) regression point are unchanged and equal R. - The DEP A/B (work/DEP cases) and the full suite are unchanged. - nns_dep(n=1e4) takes about 11 ms, about 17 ms with the guards. nns_part takes 16 ms (was 22) and 30 ms for XONLY (was 40). Public results change only by last-ulp gravity centres, toward R (a centre equal to a data point can move a split). The one exception is a histogram with more than INT_MAX bins: R 13.3 collapses it to one bin, while NNS-core and the patched R keep the real one (see the re-vendor commit). The gravity outlier tests now check the fallback against the native kernel bit for bit without a cap precondition, including >1e6-bin, >INT_MAX-bin and tight-IQR inputs.
Up to now every native kernel used all hardware threads. ncores was accepted and ignored with a warning (nns_reg, nns_m_reg, nns_arma_optim, nns_var) or dropped silently (nns_stack, nns_boost). No kernel released the GIL. - nns.set_num_threads(n) / nns.get_num_threads() expose NNS-core's global thread setting. n=None restores the automatic choice: the NNS_NUM_THREADS environment variable when it is a positive integer, else the hardware thread count. - ncores caps the native threads for one call of nns_reg, nns_m_reg, nns_stack, nns_boost, nns_arma_optim and nns_var. None keeps the default, 1 runs serially, and anything other than None or a positive integer raises ValueError, as R's NNS.M.reg does. The cap lives in a thread_local in the bindings, set by nns._native.native_threads, so concurrent Python threads with different ncores do not interfere. The "ncores ... ignored" warnings are gone; nowcast still warns. - The vectorised partial moments, the co-moments, the n-d moments and their batch form, pm_matrix, copula_nd, gravity/mode, nns_part_off and nns_dep_pair release the GIL around their computation. Results never depend on the thread count, because the kernels write disjoint slots. A new invariant test checks this for lpm/upm with 200 targets, pm_matrix and co_lpm_nd at 1/2/4 threads, and for nns_reg with ncores=1 vs 4. It also checks the scoping and validation, and four Python threads running lpm concurrently. Timing (4-core host): the per-call thread spawn that made small calls slow was removed by NNS-core's serial work threshold in the re-vendor. co_lpm_v with 2 targets at n=200 takes 3.9 us, down from 49 us (2 scalar calls take 2.8 us). pm_matrix at n=200, d=3 takes 16 us, down from 182 us. pm_matrix at n=2e4, d=12 takes 6.3 ms on 1 thread and 5.4 ms on 2 threads on this shared machine.
_mreg_predict and _mreg_predict_path (NNS.M.reg, nns_reg on 2-D x, nns_stack method 1, nns_boost, dy.d_) ran R's NNS_mreg_predict_v2 kernels as NumPy distance matrices, partial sorts and a SciPy density ensemble per k. Bind NNS-core's ports (mreg_predict, mreg_predict_path) and use them, with the GIL released and the ncores cap. The NumPy code stays as the fallback. The ports evaluate the Student-t ensemble weight as a closed form, while R uses Rmath's dt. The two differ only where that density underflows, i.e. for queries far outside the training ranges with many neighbours. _native_mreg_is_exact bounds every distance from the queries' normalised deviations and keeps those calls on the NumPy (SciPy) path. Evidence: 5400 kernel calls on random designs (normal, integer ties, duplicated columns, constant columns, mixed scales; NNS/L2/L1/FACTOR; regression and class; k = 1, 2, 5, n and the path) compared with R's NNS:::NNS_mreg_predict_v2_cpp / _path_v2_cpp: - native: max relative difference 2.8e-11, above 1e-12 in 1 call. The NumPy path has the same 2.8e-11 there. - NumPy path: above 1e-12 in 31 calls, up to relative 1, on k-neighbour sets with exactly equal distances. These are new mismatches of the fallback, left for a separate fix. On tie-free in-range data native and NumPy agree to 1e-12 (new invariant test). The M.reg/stack/boost parity tests are unchanged. In the STK A/B, Boston stacks, boost and mtcars match R as before, now to 5e-14 instead of 6e-14. Timings (4-core host, best of 2-3): - nns_m_reg, n = 1e4, 5 regressors, 200 point_est: 3.16 s -> 0.59 s (fitted values to 1.6e-15 of before). - nns_stack Boston (400/106) method 1: folds = 1 0.05 s -> 0.04 s, folds = 5 0.16 s -> 0.10 s, identical predictions. - vignette 07 main(): 3.1-4.5 s before and after. It is dominated by matplotlib rendering. NNS-core's mreg_setup and xstar_path are not bound. The NumPy partition, RPM reduction and X* path already take 0.12 s and a few ms at n = 1e4, and mreg_setup's "mode" reducer is the exact-frequency mode, not NNS.mode. The NumPy chunking test now forces the NumPy path and checks the native path against it.
honours_ncores wraps nns_reg, nns_m_reg, nns_stack, nns_boost, nns_arma_optim and nns_var, so the fixed stacklevel=2/3 in their warnings (balance -> CLASS, removed method 2, reduced folds, fewer feature subsets, no re-passing combination, unsupported R arguments) pointed at nns/_native.py. Several also pointed into nns internals before. The ones in nested helpers and the nns_boost -> nns_stack calls did. warnings now use external_stacklevel(), which walks to the first frame outside the nns package. Messages and categories are unchanged; a test checks that the nns_var nowcast warning names the calling file.
The pure-Python fallbacks (used when nns._nnscore is missing, e.g. an sdist installed without a compiler) were never tested, and nothing could force them. - NNS_DISABLE_NATIVE=1 (true/yes/on) makes nns._native report no extension, so every public function takes its NumPy path. native-backend-ci.yml runs the invariant and cached parity tests that way as an extra step. Running them that way showed three fallback differences, all fixed here: - nns_mode: the fallback was a separate implementation with a dense histogram capped at 1e6 bins, pairwise sums and a reflect-until-inside edge rule. Against native nns::mode on 2000 random vectors it disagreed in 1453 of 8000 calls (multi/discrete combinations), plus 722 sign-of-zero differences. It now ports NNS-core's algorithm: the same shortened bin list, clamped mirror smoothing, long-double sums, keys on the rounded double. Over 12000 calls it is bit-identical to native, including -0.0. The tight-IQR input (1000 points at 5 +- 1e-6 plus a 6) takes <1 ms, down from 2.0 s. With multi=True it now also returns an array for n <= 3, as the native path does. - Signed zeros in the gravity fallback: C++ sums start at +0.0 and the n <= 3 median is t[n/2] or 0.5*(a+b), while np.median maps -0.0 to +0.0. The all-equal shortcut returns the first value in input order, not after sorting. The copula fallback's sequential sums also start at +0.0. _nearest_int_half_up keeps ceil(-0.4) = -0.0. - That fixes the three test_mreg_fast_paths failures (mode reducer signbit) seen with NNS_DISABLE_NATIVE=1. With the switch set, tests/invariants and the cached tests/parity pass (2232 passed). test_co_moments is excluded locally: its multivariate_normal inputs depend on the LAPACK build, so the cache keys miss under numpy 2.5.4 with or without the extension. CI pins numpy<2.5. Known remaining fallback difference: on k-neighbour sets whose distances are exactly equal, the NumPy M.reg ensemble can take the sd of the distances as exactly 0, where R's kernel gets a tiny positive sd from its inverse-range distances. The weights then differ (31 of 5400 random kernel calls; see the M.reg commit). The native kernel matches R there.
NNS-core's notify-python.yml dispatches nns-core-updated to
.github/workflows/sync-from-nns-core.yml. That file did not exist, and
workflow-baseline.yml would delete it from any PR, so core updates never
reached NNS-python. NNS-core now checks for the receiver
and skips the dispatch with a warning when it is missing.
The receiver (repository_dispatch nns-core-updated, or workflow_dispatch with
a core_commit):
- validates client_payload.{core_commit, r_repo, r_commit, r_version,
r_src_tree_hash}. Values reach the shell only through env:.
- fetches that NNS-core commit and re-vendors it with
scripts/sync_nns_core_snapshot.py. Only core_commit is recorded in the
manifest; the core's R provenance goes into the PR body, because r_commit
in sync/nns_source.json describes the R package behind the parity cache.
- builds the extension and runs invariants, cached parity, the same with
NNS_DISABLE_NATIVE=1, and the tools tests.
- opens automation/nns-core-<sha>, as a draft unless every gate passed.
workflow-baseline.yml now protects the receiver. Its restore loop removes a
protected workflow that the default branch does not have yet, instead of
failing on `git show`. docs/sync_contract.md describes the receiver and adds
the fallback gate.
tests/parity/test_labels.py (the pandas name/level parity tests) imports pandas at module level, but pandas was in neither the dev dependency group nor the CI install lines, so CI would fail at collection; other pandas tests were silently skipped. Add pandas to the dev group and to the three workflows that run the suite. Also return an explicit float64 array from _smooth_counts_tri7 (mypy no-any-return under numpy 2.5).
A build or native-import failure stopped the job before the PR step, so no diagnostic draft was opened. create-pull-request also leaves an existing PR's draft state alone, so a failed rerun kept a ready PR ready. Record the build outcome, skip the gates after a failed build, upload both logs, and set an updated PR's draft state from the gate outcome.
Thread.join() swallowed them, so the test passed (1 passed) when every worker raised; with futures the injected RuntimeError now fails it.
They point at notes that don't live in this repo.
…of the R fix) The single mode took the first kept peak of maximal height, so with equal peaks the answer depended on the sort: nns_mode(np.repeat(np.arange(20.), 400)) was 0.894 in the fallback. As R now does, equally high peaks are averaged (9.441176, the mean of the multi modes). The native kernel gets the same rule with the next NNS-core vendor; until then its bit-identity check on such a sample is a strict xfail.
gitRasheed
force-pushed
the
rasheed/remove-dead-code
branch
from
November 2, 2026 23:09
1b8b8ed to
e51a646
Compare
gitRasheed
force-pushed
the
rasheed/native-core
branch
from
November 2, 2026 23:09
a0536f1 to
31f85d0
Compare
Contributor
|
Incorporated through NNS R 14.0 integration #192, merged as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
extern/NNS-corewas a port of the R 13.0 sources. Its >32-target LPM/UPM backend cancelled at degree >= 3, its gravity and mode histograms had no bin cap, and it lacked the R 13.1-13.3 M.reg and stack kernels, so Python ran those in NumPy. Every native call used all hardware threads and held the GIL, the pure-Python fallbacks were never tested, and core updates could not reach this repo because the receiver workflow did not exist. We re-vendor NNS-core with the code in OVVO-Financial/NNS-core#30, OVVO-Financial/NNS-core#31 and OVVO-Financial/NNS-core#32 and build on it.Stacked on #156; merge with a merge commit.
Changes
UPM(8)with 100 targets at n = 500 had a max relative error of 2.3e4 against the scalar path, the same wrong value as R 13.3, and now has 1.5e-15. Discretenns_modebeyond the int range returned -2147483648 and now returns 3e9.nns_gravityon 1000 points at 5 +- 1e-7 plus one 6 took 8.1 s and now takes under 1 ms.nns_gravitywas -250024999.98 (R 13.3) and is now -49999.98, the patched R value. The re-vendor moves results away from R 13.3 only in these cases and the two above, where R 13.3 is wrong.nns_dep's per-cell copula calls NNS-core's port ofNNS_dep_pair_core, with R's sequential sums. On R's sin grid, Correlation goes from 0.22781038989761246 to R's 0.22716789893355843, bit for bit. In the dependence A/B over 32 data sets, mismatches with R go from 55 to 14; the ones left areNNS.causcases and one error message. The iris balance stack, binary iris, mtcars am and the Boston "NNS.dep" stack now match R.NNS.gravityon 3000 of 3000 vectors where the old binding port matched 2030. We remove the cap guards from Copula, dependence and partitions: validation, R's arithmetic, matrix nns_dep #153._mreg_predictand_mreg_predict_pathcall NNS-core's ports of R'sNNS_mreg_predict_v2kernels.nns_m_regat n = 1e4 with 5 regressors and 200 points goes from 3.16 s to 0.59 s. Far outside the training range, the core's closed-form t density underflows differently from Rmath'sdt, so those queries stay on the NumPy path until Vendor NNS-core from the R-fixes follow-up and drop the M.reg underflow guard #169.nns.set_num_threads(n)andnns.get_num_threads()expose the core's thread setting, andncorescaps the native threads for one call ofnns_reg,nns_m_reg,nns_stack,nns_boost,nns_arma_optimandnns_var. The kernels release the GIL. Results do not depend on the thread count. A 2-targetco_lpm_vat n = 200 goes from 49 us to 3.9 us, because the core no longer spawns threads for small calls.NNS_DISABLE_NATIVE=1forces every NumPy fallback, and native-backend-ci.yml runs the invariant and cached parity tests that way. That found three fallback differences, now fixed. Thenns_modefallback disagreed with native in 1453 of 8000 calls and is now bit-identical over 12000.core_commitrecords the vendored snapshot; Vendor NNS-core from the R-fixes follow-up and drop the M.reg underflow guard #169 repins it to the tip of Port the R stochastic dominance, NNS.seas and ARMA weighting fixes NNS-core#34..github/workflows/sync-from-nns-core.ymlreceives NNS-core'snns-core-updateddispatch, re-vendors, runs the gates and opens a PR. workflow-baseline.yml protects it.tests/parity/test_labels.pyand the other pandas tests run. Warnings from thencoreswrapper point at the caller.Tests
New invariants for >32 targets at degrees 3-12 (they fail on the old snapshot at degrees 8 and 12), thread-count independence and scoping, the fallbacks against native, and the M.reg kernels against the NumPy path. A new parity test covers 5 tie-sensitive
nns_depdata sets; all 10 cases fail before and pass after, native and fallback. The parity cache still encodes unpatched R 13.3; no shard depended on the old >32-target kernel.The
test_nns_stack_ts_test_method1_shapeOOM from #148 is gone from here up.Vendors OVVO-Financial/NNS-core#30, OVVO-Financial/NNS-core#31 and OVVO-Financial/NNS-core#32
Part of OVVO-Financial/NNS#66
See #147
Update
This push changes three things.
isDraftand changes it only when it disagrees with the gates. Before, an existing PR kept whatever draft state it had.nns_modefallback averages the centres of equally high peaks whenmulti=False, as NNS.mode: the same answer on every platform when peaks tie NNS#102 does in R.rep(0:19, each=400)gives 9.441176, where the fallback gave 0.894.Review
GPT-6.1 Sol reviewed this slice and raised two findings. We took one and deferred one.
join()swallows exceptions. It now collects worker results through futures, and the reviewer's injected-failure case fails as it should.mreg_predict_pathkeeps a weight table for every k up to kmax, son_best="all"uses memory quadratic in the number of regression points (about 64 MB at 2000). That code lives in NNS-core, so it is mreg_predict_path keeps a weight table for every k up to kmax NNS-core#35. Results are unaffected.GPT-6.1 Sol reviewed the round-3 fixes. Taken: on an unchanged rerun create-pull-request reports no operation, so the draft-state step was skipped and a failed build could leave a ready PR; the step now runs whenever a PR exists and reconciles isDraft. A second Sol round found nothing.