Skip to content

Copula, dependence and partitions: validation, R's arithmetic, matrix nns_dep - #153

Closed
gitRasheed wants to merge 10 commits into
rasheed/dominance-anovafrom
rasheed/dependence-partition
Closed

gitRasheed wants to merge 10 commits into
rasheed/dominance-anovafrom
rasheed/dependence-partition

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

nns_copula(x, y) segfaulted on empty input and returned 1.0 for NaN. Its default targets used np.mean, which can differ from R's colMeans in the last ulp and flip the copula. The internal gravity capped its histogram at 1e6 bins, so partitions moved onto the outlier on data with one far value. nns_part with a noise reduction other than "off" was 8-20x slower than R, and nns_dep had no matrix form.

Stacked on #152; merge with a merge commit.

Changes

  • The bivariate nns_copula checks size and finiteness like R and validates the targets. Empty input crashed the interpreter, one NaN returned 1.0 and one Inf 0.728; they now raise R's errors.
  • nns_copula takes default targets with R's long-double colMeans. On a case with four observations on the mean, R gives 1 and Python gave 0.8819; it now gives 1. On R's dy.dx sin grid the copula goes from 1.0 to R's 0.5759122837209066.
  • The internal gravity counts only occupied bins, so it has no cap. On 1e4 normals with one value at -1e5 it returned -21558.3 against R's -2.510, and nns_dep Dependence goes from 0.5995 to R's 0.5733. The native kernels keep their 1e6 cap until Re-vendor NNS-core, bind its M.reg kernels and honour ncores #157 replaces them, so calls that could hit it fall back to the Python path.
  • nns_causation computes the asymmetric dependence with the native nns_dep. causal_matrix with 10 columns at n = 1000 goes from 1.109 s to 0.399 s (R 0.39 s). Values are identical.
  • nns_part with mean, median or mode noise reduction is vectorised and caps order at floor(log2(n)) before allocating. At n = 1e5, mean goes from 1.41 s to 0.038 s (R 0.057) and XONLY median from 30.5 s to 0.55 s. order=20000 at n = 1000 took 328 MB and now takes 0.1 MB. Results are identical on 26,624 calls.
  • nns_part accepts NumPy and integral float orders, and rejects order=0 and fractional obs_req as R does.
  • nns_dep(X) on a matrix or DataFrame returns R's Correlation and Dependence matrices, with asym=True oriented row to column as R returns it. causal_matrix keeps DataFrame labels.
  • conventions.md and api_reference.md describe the partition rules, the matrix copula and causal_matrix as the code has them.

Tests

New parity tests against R for the colMeans targets, matrix nns_dep (exact), and an outlier case for NNS.gravity, NNS.part (XY and XONLY) and NNS.dep. Invariants cover copula validation, the native causation path and the nns_part argument rules; the nns_part property test draws order from 1..5. The parity cache encodes unpatched R 13.3.

Update

This push adds a guard in _native_gravity_is_exact. For inputs like [5e-324, 1], range / 128 underflowed to 0 and the check raised ZeroDivisionError. R's NNS.gravity returns 0 on the same input. Hypothesis 6.170.14 found it through test_nns_m_reg_classification_bounds_hold. #157 deletes the function, so nothing above that slice changes.

Review

GPT-6.1 Sol reviewed this slice and raised two findings. We took one and rejected one.

  • Taken: the docs described the causal_matrix direction backwards. A positive [i, j] means column j causes column i. We fixed the text in all three places and added a test that checks the matrix against C(x--->y).
  • Rejected: the native-cap guard misses regression-point groups left behind by sibling splits. Re-vendor NNS-core, bind its M.reg kernels and honour ncores #157 removes the guard and uses NNS-core's gravity; the reviewer's case matches R (-15.624978228978252) from there up.

The fix commits have not been re-reviewed.

The bivariate nns_copula(x, y) path used co_moments._as_pair, which checks
neither size nor finiteness, and handed the data to the native copula_nd
kernel, which assumes n >= 1:

- nns_copula(np.array([]), np.array([])) crashed the interpreter (SIGSEGV)
- one NaN in x returned 1.0 (R: "You have some missing values")
- one Inf in x returned 0.728 (R: "`data` must be finite")
- NaN/Inf target_x/target_y were accepted

The matrix form already rejected these. The bivariate form now uses
nns._helpers._as_pair (non-empty, equal length, finite) and validates the
targets, and _copula refuses an empty matrix before calling the native
kernel. Results for valid input are unchanged.
NNS.copula takes target <- colMeans(X), which R computes as a sequential
long-double column sum divided by n. Python used np.mean (pairwise double
summation), which can differ in the last ulp. The copula then compares
sum(cupm + clpm) of the degree-0 PM.matrix to 1 exactly, and the ulp decides
whether observations sitting on the mean count as lower or upper.

Example (new parity test): x has np.mean 0.4999999999999999 and R colMeans
0.5, with four observations equal to 0.5. R NNS.copula(cbind(x, y)) = 1;
Python returned 0.8819; it now returns 1. On R's dy.dx sin grid used by the
regression default-order heuristic, nns_copula(m) goes from 1.0 to
0.5759122837209066, which is R's value to the last digit.

nns_copula (bivariate and matrix form) now uses _r_col_means for the default
targets. Explicit targets are unchanged; on continuous data the copula value
is unchanged. The internal callers _reg_engine._dependence and
regression._nns_copula_matrix still build their own np.mean targets; they
can switch to nns.copula._r_col_means to close the np.mean vs colMeans
gap fully.
Uni.caus needs NNS.dep(x, y, asym = TRUE) twice per direction. _asym_dep
rebuilt the X-only partition in pure Python (_xonly_partition appends to
lists and concatenates label strings per element) and then evaluated the
Python copula per cell; at n = 2e4 61% of nns_causation time was in
_xonly_partition. The native nns_dep_pair kernel computes the same value.

_asym_dep now calls nns_dep(x, y, asym=True) for finite input and keeps
the pure-Python version as _asym_dep_python for non-finite input.

A/B against the previous implementation on the 41 audit cases plus random,
small-n and integer data (tau 0/1/3, 135 calls, and causal_matrix with tau
0 and 2): every value identical. Timing (loaded 4-core box, one process):

  nns_causation n=1000          0.0176 s -> 0.0070 s
  causal_matrix 10 cols n=1000  1.109 s  -> 0.399 s   (R 0.39 s)
  causal_matrix 10 cols n=5000  3.578 s  -> 1.780 s   (R 1.59 s)

Results do not change.
nns_part rejected order=np.int64(3) and order=3.0 with "TypeError: order
must be an integer" (R accepts any integral numeric), silently treated
order=0 as 1 (R: "[order] must be NULL, 'max', or a positive integer."),
and accepted obs_req=2.5 (R: "[obs.req] must be a nonnegative integer.").

order and obs_req now accept numbers.Integral (Python and NumPy integers)
and integral, finite reals; order must be >= 1 and obs_req >= 0, otherwise
ValueError. Booleans and non-numeric values raise TypeError. R's NNS.reg
also errors for order = 0.

The hypothesis property test drew order from 0..5; it now draws 1..5.
Results for valid arguments are unchanged.
For noise_reduction other than "off" nns_part ran a Python loop that
called np.unique on growing string-label arrays at every depth, built a
boolean mask per group (O(n * groups)) to compute centres, assign digits
and aggregate regression points, and allocated the label arrays as
dtype "<U{order + 1}" although the depth never exceeds floor(log2(n)).

- Groups are dense integer ids whose numeric order is the lexicographic
  order of their labels (current labels are the leaves of the partition
  tree, so children sorted by (parent id, digit) keep label order); label
  strings are built once per group, not per point.
- Digits are assigned for all split points at once.
- Per-group reductions read a stable argsort-by-group slice, which holds
  exactly the values x[group == g] in the same order, so np.mean / the
  mode / gravity give bit-identical results; medians are computed for
  all groups from one lexsort (odd: middle value, even: mean of the two
  middle values, as np.median).
- order is capped at floor(log2(n)) before anything is allocated (and
  before it is passed to the native kernel, so a huge order cannot
  overflow its C int).

A/B against the previous implementation: 26,624 calls (41 audit cases plus
random n = 1..333, Inf/NaN, integer and outlier data x type None/XONLY x
order None/1/2/3/6/40 x obs_req 0/1/8/50 x min_obs_stop x all five noise
reductions, plus the pure-Python "off" path against the native kernel):
order, quadrant, prior.quadrant and regression points identical.

Timing, one core on a loaded box, before -> after (R 13.3 in parentheses):

  n = 1e4  mean 0.090 -> 0.005 s, median 0.172 -> 0.018 s,
           mode 0.090 -> 0.011 s, XONLY median 0.631 -> 0.033 s
  n = 1e5  mean 1.41 -> 0.038 s (R 0.057), median 3.21 -> 0.26 s (R 0.18),
           mode 1.78 -> 0.16 s (R 0.125), mode_class 3.21 -> 0.18 s,
           XONLY median 30.5 -> 0.55 s, XONLY mean 4.78 -> 0.074 s

Memory for n = 1000, noise_reduction="mean": order=20000 took
2.34 s and 328 MB peak, now 0.003 s and 0.1 MB (same result as order=9).
Label arrays are now as wide as the longest label rather than order + 1.
Results do not change.
R's NNS.dep(X) on a matrix or data frame returns Correlation and Dependence
matrices (NNS.dep.matrix); nns_dep(M) raised "missing 1 required
positional argument: 'y'" and nns_dep(M, None) "x and y must be 1D".
causal_matrix returned a bare ndarray even for a labelled DataFrame, while
R keeps the dimnames.

- nns_dep(x, y=None, asym=False): a 2-D array or DataFrame x with y=None
  returns {"Correlation": M, "Dependence": M} built pairwise like R:
  symmetric with a unit diagonal for asym=False; for asym=True entry
  [i, j] = nns_dep(X[:, i], X[:, j], asym=True) (row -> column, the
  orientation R actually returns; its man page says the reverse),
  with R's NaN -> 0 on the lower triangle. A DataFrame input returns
  DataFrames labelled with its columns. 1-D x without y, or a single
  column, raises ValueError.
- The nns_dep docstring now says what asym=True measures (y as a function
  of x) and that p.value is not implemented.
- causal_matrix returns a DataFrame labelled with the input columns for
  DataFrame input; ndarray input still returns an ndarray.

Verified against R NNS.dep(X) and NNS.dep(X, asym = TRUE) on a 4-column
matrix (new parity tests, exact). The bivariate API and all existing
results are unchanged.
The internal gravity (nns.dependence._gravity and the native gravity_exact
kernel behind nns_part "off", nns_dep and nns_causation) built a dense
histogram of width IQR / sqrt(n) and clipped it at 1e6 bins
(_MAX_DENSE_BINS / the C++ constant). R does not cap, so with an outlier
every bulk value beyond bin 1e6 was moved into the last bin and the mode
term landed on the outlier. Audit case (n = 1e4 normals, x[0] = -1e5):
_gravity(x) = -21558.3 vs R NNS.gravity -2.510; nns_part(x, y, order = 1)
regression point x = -21558 vs -2.51; nns_dep 0.3633 vs 0.3592.

- _gravity_python counts only the occupied bins (runs of floor((v - x1) /
  width) over the sorted values), so no cap and no allocation are needed.
  The bin-count rule, including R's int overflow to one bin (kept as in
  R), is unchanged. Wherever the old dense code fit, the arithmetic is
  the same (bit-identical to gravity_exact on 600 randomised arrays in the
  new test); beyond 1e6 bins with tied maxima it uses R's long-double sum.
- The native kernels cannot be rebuilt here, so they are guarded:
  _gravity uses gravity_exact only when the bin count of that call is below
  the cap (O(n) partition for the quartiles). For native partitions a cheap
  a-priori bound covers every subset of at least obs_req + 1 points
  (q3 - q1 >= min_j (s[j+k] - s[j]) over the full sample); if it does not
  prove safety, every gravity call of the returned partition is recomputed
  from the final labels and the prior.quadrant groups. If any would hit the
  cap, nns_part falls back to the exact Python partitioner and nns_dep to
  its Python path.
- docs/conventions.md described a 4 * len(input) cap that the code never
  had; it now describes this behaviour.

Checks: the guard never accepted a capped partition (3200 native vs Python
comparisons on outlier/Cauchy/rounded data, 0 violations; no plain-normal
case was flagged); the part A/B harness (26,624 calls) differs from the
previous code only on its outlier dataset with noise "off". New parity
tests match R on the outlier case for NNS.gravity, NNS.part (XY and XONLY)
and NNS.dep (sym and asym; n = 1e4 Dependence 0.5995 -> R's 0.5733).

Cost of the guards (one core): nns_dep unchanged; nns_part default
n = 1e3 1.1 -> 1.3 ms, 1e4 17 -> 24 ms, 1e5 0.33 -> 0.42 s; XONLY
obs_req = 0 (NNS.reg's call) 1.7 -> 2.3 ms at 1e3, 0.21 -> 0.25 s at 1e5;
nns_reg n = 1e3 5.4 -> 6.0 ms; _gravity of 10 values 0.9 -> ~11 us.
Removing the 1e6 clip in gravity_exact_impl (_nnscore_bindings.cpp) would
make the guards unnecessary.

Results change only where the cap was hit (they now match R).
- conventions.md "Partition" said R only distinguishes NULL from non-NULL
  type and that order="max" raises TypeError. Both R 13.3 and the code
  validate type (ValueError for anything but None/"XONLY") and support
  order="max"; it now also states the order/obs_req rules and that the
  returned order is the depth reached (<= floor(log2(n))).
- conventions.md "Copula" and api_reference.md described only a bivariate
  scalar form; the matrix form with any number of columns is documented,
  with the colMeans default targets and input validation.
- api_reference.md named the R API "Causal.matrix", which does not exist;
  it is NNS.caus on a matrix (internally NNS.caus.matrix). The
  orientation of causal_matrix and its DataFrame labels are documented.
- nns_dep: document the matrix form, what asym=True measures and the
  asymmetric matrix orientation (row -> column, as R returns it), and that
  p.value is not implemented. The nns_part docstring gains the argument
  rules.

(The stale "4 * len(input)" gravity cap text was replaced with the
gravity fix.) Documentation only.
For x = linspace(-2, 2, 100), y = x^2, C(x--->y) is +18.44 while
causal_matrix[0, 1] is -18.44, so the docs had the direction reversed.
They point at notes that don't live in this repo.
@gitRasheed
gitRasheed force-pushed the rasheed/dependence-partition branch from 7ac7c3b to dd188de Compare November 2, 2026 23:09
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head dd188de7e403a5647d46dda0497a6b6596f41d98 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/dependence-partition branch November 7, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant