Skip to content

Re-vendor NNS-core, bind its M.reg kernels and honour ncores - #157

Closed
gitRasheed wants to merge 14 commits into
rasheed/remove-dead-codefrom
rasheed/native-core
Closed

gitRasheed wants to merge 14 commits into
rasheed/remove-dead-codefrom
rasheed/native-core

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

extern/NNS-core was a port of the R 13.0 sources. Its >32-target LPM/UPM backend cancelled at degree >= 3, its gravity and mode histograms had no bin cap, and it lacked the R 13.1-13.3 M.reg and stack kernels, so Python ran those in NumPy. Every native call used all hardware threads and held the GIL, the pure-Python fallbacks were never tested, and core updates could not reach this repo because the receiver workflow did not exist. We re-vendor NNS-core with the code in OVVO-Financial/NNS-core#30, OVVO-Financial/NNS-core#31 and OVVO-Financial/NNS-core#32 and build on it.

Stacked on #156; merge with a merge commit.

Changes

  • The vectorised partial moments no longer cancel. UPM(8) with 100 targets at n = 500 had a max relative error of 2.3e4 against the scalar path, the same wrong value as R 13.3, and now has 1.5e-15. Discrete nns_mode beyond the int range returned -2147483648 and now returns 3e9. nns_gravity on 1000 points at 5 +- 1e-7 plus one 6 took 8.1 s and now takes under 1 ms.
  • When a histogram would need more than INT_MAX bins, R 13.3's int cast collapses it to one bin. For 1e4 normals with one value at -1e9, nns_gravity was -250024999.98 (R 13.3) and is now -49999.98, the patched R value. The re-vendor moves results away from R 13.3 only in these cases and the two above, where R 13.3 is wrong.
  • nns_dep's per-cell copula calls NNS-core's port of NNS_dep_pair_core, with R's sequential sums. On R's sin grid, Correlation goes from 0.22781038989761246 to R's 0.22716789893355843, bit for bit. In the dependence A/B over 32 data sets, mismatches with R go from 55 to 14; the ones left are NNS.caus cases and one error message. The iris balance stack, binary iris, mtcars am and the Boston "NNS.dep" stack now match R.
  • Partitions use NNS-core's gravity, which matched R NNS.gravity on 3000 of 3000 vectors where the old binding port matched 2030. We remove the cap guards from Copula, dependence and partitions: validation, R's arithmetic, matrix nns_dep #153.
  • _mreg_predict and _mreg_predict_path call NNS-core's ports of R's NNS_mreg_predict_v2 kernels. nns_m_reg at n = 1e4 with 5 regressors and 200 points goes from 3.16 s to 0.59 s. Far outside the training range, the core's closed-form t density underflows differently from Rmath's dt, so those queries stay on the NumPy path until Vendor NNS-core from the R-fixes follow-up and drop the M.reg underflow guard #169.
  • nns.set_num_threads(n) and nns.get_num_threads() expose the core's thread setting, and ncores caps the native threads for one call of nns_reg, nns_m_reg, nns_stack, nns_boost, nns_arma_optim and nns_var. The kernels release the GIL. Results do not depend on the thread count. A 2-target co_lpm_v at n = 200 goes from 49 us to 3.9 us, because the core no longer spawns threads for small calls.
  • NNS_DISABLE_NATIVE=1 forces every NumPy fallback, and native-backend-ci.yml runs the invariant and cached parity tests that way. That found three fallback differences, now fixed. The nns_mode fallback disagreed with native in 1453 of 8000 calls and is now bit-identical over 12000.
  • core_commit records the vendored snapshot; Vendor NNS-core from the R-fixes follow-up and drop the M.reg underflow guard #169 repins it to the tip of Port the R stochastic dominance, NNS.seas and ARMA weighting fixes NNS-core#34.
  • .github/workflows/sync-from-nns-core.yml receives NNS-core's nns-core-updated dispatch, re-vendors, runs the gates and opens a PR. workflow-baseline.yml protects it.
  • CI installs pandas, so tests/parity/test_labels.py and the other pandas tests run. Warnings from the ncores wrapper point at the caller.

Tests

New invariants for >32 targets at degrees 3-12 (they fail on the old snapshot at degrees 8 and 12), thread-count independence and scoping, the fallbacks against native, and the M.reg kernels against the NumPy path. A new parity test covers 5 tie-sensitive nns_dep data sets; all 10 cases fail before and pass after, native and fallback. The parity cache still encodes unpatched R 13.3; no shard depended on the old >32-target kernel.

The test_nns_stack_ts_test_method1_shape OOM from #148 is gone from here up.

Vendors OVVO-Financial/NNS-core#30, OVVO-Financial/NNS-core#31 and OVVO-Financial/NNS-core#32
Part of OVVO-Financial/NNS#66
See #147

Update

This push changes three things.

  • sync-from-nns-core.yml: a failed build still opens a draft PR, with both logs uploaded and the gates skipped. Whenever a PR already exists, including on an unchanged rerun, a last step reads its isDraft and changes it only when it disagrees with the gates. Before, an existing PR kept whatever draft state it had.
  • The nns_mode fallback averages the centres of equally high peaks when multi=False, as NNS.mode: the same answer on every platform when peaks tie NNS#102 does in R. rep(0:19, each=400) gives 9.441176, where the fallback gave 0.894.
  • New native-vs-fallback tests for M.reg distance ties and for mode ties. Both are strict xfail here, because the vendored core still breaks those ties differently. Vendor NNS-core from the R-fixes follow-up and drop the M.reg underflow guard #169 re-vendors and removes both xfails.

Review

GPT-6.1 Sol reviewed this slice and raised two findings. We took one and deferred one.

  • Taken: the threading test passed even when every worker raised, because join() swallows exceptions. It now collects worker results through futures, and the reviewer's injected-failure case fails as it should.
  • Deferred: mreg_predict_path keeps a weight table for every k up to kmax, so n_best="all" uses memory quadratic in the number of regression points (about 64 MB at 2000). That code lives in NNS-core, so it is mreg_predict_path keeps a weight table for every k up to kmax NNS-core#35. Results are unaffected.

GPT-6.1 Sol reviewed the round-3 fixes. Taken: on an unchanged rerun create-pull-request reports no operation, so the draft-state step was skipped and a failed build could leave a ready PR; the step now runs whenever a PR exists and reconciles isDraft. A second Sol round found nothing.

extern/NNS-core was a port of the R 13.0 src tree. Re-vendor NNS-core
with scripts/sync_nns_core_snapshot.py. That snapshot ports R 13.3 and
fixes, ahead of R:

- The vectorised (>32 targets) LPM/UPM backend cancelled
  catastrophically for degree >= 3. UPM(8, 100 targets, n=500) had a max
  relative error of 2.3e4 against the scalar path (R 13.3 gives the same
  wrong values); it is now 1.5e-15, and the native and NumPy results agree.
- Discrete NNS.mode beyond the int range returned
  -2147483648 (now 3e9). Gravity/mode histograms have O(n) bins: nns_gravity
  on 1000 points at 5 +- 1e-7 plus one 6 took 8.1 s, now <1 ms, with the
  same value. smooth_counts_tri7 no longer reads past the end of the counts.
- generate_vectors bounds checks, negative/NaN degrees rejected.
- fast_lm means are long double, two pass, like Rcpp.
- New kernels: mreg_setup/predict/predict_path and xstar_path (R 13.1-13.3
  M.reg and NNS.stack), and a global thread setting with a serial
  threshold. None of these are bound yet.

Public results change only where R 13.3 is wrong: >32-target LPM/UPM at
degree >= 3, discrete nns_mode beyond the int range, and nns_gravity/nns_mode
when the histogram would need more than INT_MAX bins. In that last case R's
int cast overflows and the histogram collapses to one bin: for 1e4
normals with one value at -1e9, nns_gravity was -250024999.98 (= R 13.3). It
is now -49999.98, the value the patched R (gravity bin fix) gives.

The bindings build unchanged. The full suite gives the same results as
before. The 9 vignette subprocess tests fail in this environment because nns
is not installed there (PYTHONPATH is the repo root); that is unrelated. No
parity cache shard depended on the old >32-target kernel: the vector-target
parity tests use 20 targets. So no shard needed regenerating. A new invariant
test checks >32 targets at degrees 3-12 against the scalar kernel and the
NumPy fallback; it fails on the old snapshot at degrees 8 and 12.

The sync script now vendors only the nnscore build inputs (CMakeLists.txt,
include/, src/), the same layout as before, and each entry still matches the
upstream git object verbatim as verify_release_provenance.py requires. It
also updates sync/nns_source.json in place instead of rewriting it. A rewrite
would reset vendored_r_tarball and the notes, and would need R fields that
describe a different snapshot, so the R fields are now optional.
nns_dep's per-cell copula took the cell means with np_mean, an emulation of
NumPy's pairwise sum, in the native kernel and np.mean in the fallback. R's
NNS_dep.cpp copula_signed (and NNS-core's dependence.cpp) uses a
sequential double sum / n. On evenly spaced or tied data a point sits exactly
on the cell mean, so a last-ulp difference in the mean moves it between the
concordant and discordant counts. R also scales the counts by 1/n and takes
the degree-1 shares in one pass, so `d0_Co == 1` could hold in one
implementation and not the other.

The native kernel now hashes the X-only partition labels with
std::hash<std::string> and calls nns::dep_pair, NNS-core's port of R's
NNS_dep_pair_core. It is R's code, and it also adds the cell terms in R's
unordered_map order. The fallback (_copula_signed,
_copula_degree0_unsigned) repeats R's sums in R's order. Its only
difference is that it adds the cell terms in first-seen order, which can
move the totals by a few ulp.

Evidence (R NNS 13.3):
- x = R seq(0, 4*pi, pi/100), y = sin(x): R Correlation 0.22716789893355843,
  Dependence 0.92465837278014773. Before: native 0.22781038989761246 /
  0.9240158818160937. After: native equal to R bit for bit; fallback
  0.2271678989335584 / 0.9246583727801478.
- DEP A/B (work/DEP cases, 32 data sets, tolerance 1e-8): 55 mismatches
  before, 14 after. All 14 dep/asym/copula mismatches are gone, e.g.
  binary_eq Correlation 1.0 -> 0.588 = R, discrete_dep Dependence 0.5477 ->
  0.4962 = R. 12 NNS.caus mismatches remain (mono_rank, step3, y_neg_x),
  plus the const_y error text.
- NNS.dep(x_j, medv, asym=TRUE) on Boston now matches R for all 13
  columns (ptratio 0.38679 -> 0.46074, black 0.26503 -> 0.30305 = R), and
  Uni.caus for 12 of 13. In the REG A/B (87 nns_reg cases),
  boston_dr_NNS.dep now matches R; 5 cases still differ. boston_dr_NNS.caus
  and _all differ because Uni.caus(medv, chas) is 0 vs R 0.0137.
  nns_norm's np.corrcoef is 2e-15 off R's cor(), which shifts the scaled
  binary column by an ulp and flips the cell copula.
- The iris balance vignette call now matches R (OBJfn.dim.red
  0.9643, weights .85/.15; before 0.9571, .88/.12), as do binary iris
  (OBJfn.reg, threshold), mtcars am (weights .44/.56, threshold .83;
  before .26/.74, .78), Boston method 1:2 dim.red "NNS.dep" (weights,
  stack) and the boost threshold=1.5 feature frequencies (crim/nox/dis 9,
  rm/lstat 8, ptratio 7). The "NNS.caus"/"all" stacks keep the Uni.caus
  difference above.

Public results change on tied or evenly spaced data, toward R. The new parity
test covers 5 tie-sensitive data sets, symmetric and asym. All 10 cases fail
before this change and pass after, with the native kernel and with the
fallback. The causation invariant now compares native and fallback to 1e-14
instead of bit for bit.
The bindings had their own gravity port (gravity_exact) for nns_part,
nns_dep and nns_reg. It clipped its dense histogram at 1e6 bins, which put
the mode term on the outlier, and it summed in pairwise double.
R's NNS_gravity_cpp uses long-double sums, so gravity_exact matched R
NNS.gravity in only 2030 of 3000 random vectors. To stay correct,
dependence.py and part.py checked every gravity call of a native partition
against the cap (_gravity_subsets_are_safe, _native_partition_is_exact,
_dep_native_is_exact) and recomputed in Python when it could bind.

gravity_exact is now nns::gravity from the re-vendored NNS-core. That is R's
algorithm with an O(n) histogram that only shortens runs of empty bins,
so no cap is needed and the guards are removed. The NumPy
fallback (_gravity_python, now also behind nns_gravity) follows the same
arithmetic: a long-double sequential mean and weighted bin average, bin
names computed in double, occupied bins only, and the 2**52 width widening.

Evidence:
- 3000 vectors (normal, integer, rounded t2, offset exponential, -1e5
  outlier, tight IQR + one far value): native gravity_exact == R
  NNS.gravity 3000/3000, fallback == native 3000/3000.
- outlier.csv: _gravity -2.5102461601040074 -> -2.5102461601040078
  (= R; it went through the Python path before). nns_dep and the
  nns_part(order=1) regression point are unchanged and equal R.
- The DEP A/B (work/DEP cases) and the full suite are unchanged.
- nns_dep(n=1e4) takes about 11 ms, about 17 ms with the guards. nns_part
  takes 16 ms (was 22) and 30 ms for XONLY (was 40).

Public results change only by last-ulp gravity centres, toward R (a centre
equal to a data point can move a split). The one exception is a histogram
with more than INT_MAX bins: R 13.3 collapses it to one bin, while NNS-core
and the patched R keep the real one (see the re-vendor commit). The gravity
outlier tests now check the fallback against the native kernel bit for bit
without a cap precondition, including >1e6-bin, >INT_MAX-bin and tight-IQR
inputs.
Up to now every native kernel used all hardware threads. ncores was accepted
and ignored with a warning (nns_reg, nns_m_reg, nns_arma_optim, nns_var) or
dropped silently (nns_stack, nns_boost). No kernel released the GIL.

- nns.set_num_threads(n) / nns.get_num_threads() expose NNS-core's global
  thread setting. n=None restores the automatic choice: the NNS_NUM_THREADS
  environment variable when it is a positive integer, else the hardware
  thread count.
- ncores caps the native threads for one call of nns_reg, nns_m_reg,
  nns_stack, nns_boost, nns_arma_optim and nns_var. None keeps the
  default, 1 runs serially, and anything other than None or a positive
  integer raises ValueError, as R's NNS.M.reg does. The cap lives in a
  thread_local in the bindings, set by nns._native.native_threads, so
  concurrent Python threads with different ncores do not interfere. The
  "ncores ... ignored" warnings are gone; nowcast still warns.
- The vectorised partial moments, the co-moments, the n-d moments and their
  batch form, pm_matrix, copula_nd, gravity/mode, nns_part_off and
  nns_dep_pair release the GIL around their computation.

Results never depend on the thread count, because the kernels write disjoint
slots. A new invariant test checks this for lpm/upm with 200 targets,
pm_matrix and co_lpm_nd at 1/2/4 threads, and for nns_reg with ncores=1 vs
4. It also checks the scoping and validation, and four Python threads
running lpm concurrently.

Timing (4-core host): the per-call thread spawn that made small calls slow
was removed by NNS-core's serial work threshold in the re-vendor.
co_lpm_v with 2 targets at n=200 takes 3.9 us, down from 49 us (2 scalar
calls take 2.8 us). pm_matrix at n=200, d=3 takes 16 us, down from 182 us.
pm_matrix at n=2e4, d=12 takes 6.3 ms on 1 thread and 5.4 ms on 2 threads
on this shared machine.
_mreg_predict and _mreg_predict_path (NNS.M.reg, nns_reg on 2-D x,
nns_stack method 1, nns_boost, dy.d_) ran R's NNS_mreg_predict_v2 kernels
as NumPy distance matrices, partial sorts and a SciPy density ensemble per
k. Bind NNS-core's ports (mreg_predict,
mreg_predict_path) and use them, with the GIL released and the ncores cap.
The NumPy code stays as the fallback.

The ports evaluate the Student-t ensemble weight as a closed form, while R
uses Rmath's dt. The two differ only where that density underflows, i.e.
for queries far outside the training ranges with many neighbours.
_native_mreg_is_exact bounds every distance from the queries' normalised
deviations and keeps those calls on the NumPy (SciPy) path.

Evidence: 5400 kernel calls on random designs (normal, integer ties,
duplicated columns, constant columns, mixed scales; NNS/L2/L1/FACTOR;
regression and class; k = 1, 2, 5, n and the path) compared with R's
NNS:::NNS_mreg_predict_v2_cpp / _path_v2_cpp:
- native: max relative difference 2.8e-11, above 1e-12 in 1 call. The
  NumPy path has the same 2.8e-11 there.
- NumPy path: above 1e-12 in 31 calls, up to relative 1, on k-neighbour
  sets with exactly equal distances. These are new mismatches of the
  fallback, left for a separate fix. On tie-free in-range data native and
  NumPy agree to 1e-12 (new invariant test).
The M.reg/stack/boost parity tests are unchanged. In the STK A/B, Boston
stacks, boost and mtcars match R as before, now to 5e-14 instead of 6e-14.

Timings (4-core host, best of 2-3):
- nns_m_reg, n = 1e4, 5 regressors, 200 point_est: 3.16 s -> 0.59 s
  (fitted values to 1.6e-15 of before).
- nns_stack Boston (400/106) method 1: folds = 1 0.05 s -> 0.04 s,
  folds = 5 0.16 s -> 0.10 s, identical predictions.
- vignette 07 main(): 3.1-4.5 s before and after. It is dominated by
  matplotlib rendering.

NNS-core's mreg_setup and xstar_path are not bound. The NumPy partition, RPM
reduction and X* path already take 0.12 s and a few ms at n = 1e4, and
mreg_setup's "mode" reducer is the exact-frequency mode, not NNS.mode.
The NumPy chunking test now forces the NumPy path and checks the
native path against it.
honours_ncores wraps nns_reg, nns_m_reg, nns_stack, nns_boost,
nns_arma_optim and nns_var, so the fixed stacklevel=2/3 in their warnings
(balance -> CLASS, removed method 2, reduced folds, fewer feature subsets,
no re-passing combination, unsupported R arguments) pointed at
nns/_native.py. Several also pointed into nns internals before. The ones in
nested helpers and the nns_boost -> nns_stack calls did. warnings now use
external_stacklevel(), which walks to the first frame outside the nns
package. Messages and categories are unchanged; a test checks that the
nns_var nowcast warning names the calling file.
The pure-Python fallbacks (used when nns._nnscore is missing, e.g. an sdist
installed without a compiler) were never tested, and nothing could force
them.

- NNS_DISABLE_NATIVE=1 (true/yes/on) makes nns._native report no extension,
  so every public function takes its NumPy path. native-backend-ci.yml runs
  the invariant and cached parity tests that way as an extra step.

Running them that way showed three fallback differences, all fixed here:
- nns_mode: the fallback was a separate implementation with a dense
  histogram capped at 1e6 bins, pairwise sums and a reflect-until-inside
  edge rule. Against native nns::mode on 2000 random vectors it disagreed
  in 1453 of 8000 calls (multi/discrete combinations), plus 722
  sign-of-zero differences. It now ports NNS-core's algorithm: the same
  shortened bin list, clamped mirror smoothing, long-double sums, keys on
  the rounded double. Over 12000 calls it is bit-identical to native,
  including -0.0. The tight-IQR input (1000 points at 5 +- 1e-6 plus a 6)
  takes <1 ms, down from 2.0 s. With multi=True it now also returns an
  array for n <= 3, as the native path does.
- Signed zeros in the gravity fallback: C++ sums start at +0.0 and the
  n <= 3 median is t[n/2] or 0.5*(a+b), while np.median maps -0.0 to +0.0.
  The all-equal shortcut returns the first value in input order, not after
  sorting. The copula fallback's sequential sums also start at +0.0.
  _nearest_int_half_up keeps ceil(-0.4) = -0.0.
- That fixes the three test_mreg_fast_paths failures (mode reducer
  signbit) seen with NNS_DISABLE_NATIVE=1.

With the switch set, tests/invariants and the cached tests/parity pass
(2232 passed). test_co_moments is excluded locally: its multivariate_normal
inputs depend on the LAPACK build, so the cache keys miss under numpy 2.5.4
with or without the extension. CI pins numpy<2.5.

Known remaining fallback difference: on k-neighbour sets whose distances
are exactly equal, the NumPy M.reg ensemble can take the sd of the
distances as exactly 0, where R's kernel gets a tiny positive sd from its
inverse-range distances. The weights then differ (31 of 5400 random kernel
calls; see the M.reg commit). The native kernel matches R there.
NNS-core's notify-python.yml dispatches nns-core-updated to
.github/workflows/sync-from-nns-core.yml. That file did not exist, and
workflow-baseline.yml would delete it from any PR, so core updates never
reached NNS-python. NNS-core now checks for the receiver
and skips the dispatch with a warning when it is missing.

The receiver (repository_dispatch nns-core-updated, or workflow_dispatch with
a core_commit):
- validates client_payload.{core_commit, r_repo, r_commit, r_version,
  r_src_tree_hash}. Values reach the shell only through env:.
- fetches that NNS-core commit and re-vendors it with
  scripts/sync_nns_core_snapshot.py. Only core_commit is recorded in the
  manifest; the core's R provenance goes into the PR body, because r_commit
  in sync/nns_source.json describes the R package behind the parity cache.
- builds the extension and runs invariants, cached parity, the same with
  NNS_DISABLE_NATIVE=1, and the tools tests.
- opens automation/nns-core-<sha>, as a draft unless every gate passed.

workflow-baseline.yml now protects the receiver. Its restore loop removes a
protected workflow that the default branch does not have yet, instead of
failing on `git show`. docs/sync_contract.md describes the receiver and adds
the fallback gate.
tests/parity/test_labels.py (the pandas name/level parity tests) imports
pandas at module level, but pandas was in neither the dev dependency group nor
the CI install lines, so CI would fail at collection; other pandas tests were
silently skipped. Add pandas to the dev group and to the three workflows that
run the suite.

Also return an explicit float64 array from _smooth_counts_tri7 (mypy
no-any-return under numpy 2.5).
A build or native-import failure stopped the job before the PR step, so no
diagnostic draft was opened. create-pull-request also leaves an existing PR's
draft state alone, so a failed rerun kept a ready PR ready. Record the build
outcome, skip the gates after a failed build, upload both logs, and set an
updated PR's draft state from the gate outcome.
Thread.join() swallowed them, so the test passed (1 passed) when every worker
raised; with futures the injected RuntimeError now fails it.
They point at notes that don't live in this repo.
…of the R fix)

The single mode took the first kept peak of maximal height, so with equal
peaks the answer depended on the sort: nns_mode(np.repeat(np.arange(20.), 400))
was 0.894 in the fallback. As R now does, equally high peaks are averaged
(9.441176, the mean of the multi modes). The native kernel gets the same rule
with the next NNS-core vendor; until then its bit-identity check on such a
sample is a strict xfail.
@gitRasheed
gitRasheed force-pushed the rasheed/remove-dead-code branch from 1b8b8ed to e51a646 Compare November 2, 2026 23:09
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head 31f85d0524d6e92d7eb834d73b8d1711064a2a4b is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/native-core branch November 7, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant