Repository navigation
Copula, dependence and partitions: validation, R's arithmetic, matrix nns_dep - #153
Closed
gitRasheed wants to merge 10 commits into
Closed
gitRasheed wants to merge 10 commits into
gitRasheed wants to merge 10 commits into
Conversation
This was referenced Nov 2, 2026
The bivariate nns_copula(x, y) path used co_moments._as_pair, which checks neither size nor finiteness, and handed the data to the native copula_nd kernel, which assumes n >= 1: - nns_copula(np.array([]), np.array([])) crashed the interpreter (SIGSEGV) - one NaN in x returned 1.0 (R: "You have some missing values") - one Inf in x returned 0.728 (R: "`data` must be finite") - NaN/Inf target_x/target_y were accepted The matrix form already rejected these. The bivariate form now uses nns._helpers._as_pair (non-empty, equal length, finite) and validates the targets, and _copula refuses an empty matrix before calling the native kernel. Results for valid input are unchanged.
NNS.copula takes target <- colMeans(X), which R computes as a sequential long-double column sum divided by n. Python used np.mean (pairwise double summation), which can differ in the last ulp. The copula then compares sum(cupm + clpm) of the degree-0 PM.matrix to 1 exactly, and the ulp decides whether observations sitting on the mean count as lower or upper. Example (new parity test): x has np.mean 0.4999999999999999 and R colMeans 0.5, with four observations equal to 0.5. R NNS.copula(cbind(x, y)) = 1; Python returned 0.8819; it now returns 1. On R's dy.dx sin grid used by the regression default-order heuristic, nns_copula(m) goes from 1.0 to 0.5759122837209066, which is R's value to the last digit. nns_copula (bivariate and matrix form) now uses _r_col_means for the default targets. Explicit targets are unchanged; on continuous data the copula value is unchanged. The internal callers _reg_engine._dependence and regression._nns_copula_matrix still build their own np.mean targets; they can switch to nns.copula._r_col_means to close the np.mean vs colMeans gap fully.
Uni.caus needs NNS.dep(x, y, asym = TRUE) twice per direction. _asym_dep rebuilt the X-only partition in pure Python (_xonly_partition appends to lists and concatenates label strings per element) and then evaluated the Python copula per cell; at n = 2e4 61% of nns_causation time was in _xonly_partition. The native nns_dep_pair kernel computes the same value. _asym_dep now calls nns_dep(x, y, asym=True) for finite input and keeps the pure-Python version as _asym_dep_python for non-finite input. A/B against the previous implementation on the 41 audit cases plus random, small-n and integer data (tau 0/1/3, 135 calls, and causal_matrix with tau 0 and 2): every value identical. Timing (loaded 4-core box, one process): nns_causation n=1000 0.0176 s -> 0.0070 s causal_matrix 10 cols n=1000 1.109 s -> 0.399 s (R 0.39 s) causal_matrix 10 cols n=5000 3.578 s -> 1.780 s (R 1.59 s) Results do not change.
nns_part rejected order=np.int64(3) and order=3.0 with "TypeError: order must be an integer" (R accepts any integral numeric), silently treated order=0 as 1 (R: "[order] must be NULL, 'max', or a positive integer."), and accepted obs_req=2.5 (R: "[obs.req] must be a nonnegative integer."). order and obs_req now accept numbers.Integral (Python and NumPy integers) and integral, finite reals; order must be >= 1 and obs_req >= 0, otherwise ValueError. Booleans and non-numeric values raise TypeError. R's NNS.reg also errors for order = 0. The hypothesis property test drew order from 0..5; it now draws 1..5. Results for valid arguments are unchanged.
For noise_reduction other than "off" nns_part ran a Python loop that
called np.unique on growing string-label arrays at every depth, built a
boolean mask per group (O(n * groups)) to compute centres, assign digits
and aggregate regression points, and allocated the label arrays as
dtype "<U{order + 1}" although the depth never exceeds floor(log2(n)).
- Groups are dense integer ids whose numeric order is the lexicographic
order of their labels (current labels are the leaves of the partition
tree, so children sorted by (parent id, digit) keep label order); label
strings are built once per group, not per point.
- Digits are assigned for all split points at once.
- Per-group reductions read a stable argsort-by-group slice, which holds
exactly the values x[group == g] in the same order, so np.mean / the
mode / gravity give bit-identical results; medians are computed for
all groups from one lexsort (odd: middle value, even: mean of the two
middle values, as np.median).
- order is capped at floor(log2(n)) before anything is allocated (and
before it is passed to the native kernel, so a huge order cannot
overflow its C int).
A/B against the previous implementation: 26,624 calls (41 audit cases plus
random n = 1..333, Inf/NaN, integer and outlier data x type None/XONLY x
order None/1/2/3/6/40 x obs_req 0/1/8/50 x min_obs_stop x all five noise
reductions, plus the pure-Python "off" path against the native kernel):
order, quadrant, prior.quadrant and regression points identical.
Timing, one core on a loaded box, before -> after (R 13.3 in parentheses):
n = 1e4 mean 0.090 -> 0.005 s, median 0.172 -> 0.018 s,
mode 0.090 -> 0.011 s, XONLY median 0.631 -> 0.033 s
n = 1e5 mean 1.41 -> 0.038 s (R 0.057), median 3.21 -> 0.26 s (R 0.18),
mode 1.78 -> 0.16 s (R 0.125), mode_class 3.21 -> 0.18 s,
XONLY median 30.5 -> 0.55 s, XONLY mean 4.78 -> 0.074 s
Memory for n = 1000, noise_reduction="mean": order=20000 took
2.34 s and 328 MB peak, now 0.003 s and 0.1 MB (same result as order=9).
Label arrays are now as wide as the longest label rather than order + 1.
Results do not change.
R's NNS.dep(X) on a matrix or data frame returns Correlation and Dependence
matrices (NNS.dep.matrix); nns_dep(M) raised "missing 1 required
positional argument: 'y'" and nns_dep(M, None) "x and y must be 1D".
causal_matrix returned a bare ndarray even for a labelled DataFrame, while
R keeps the dimnames.
- nns_dep(x, y=None, asym=False): a 2-D array or DataFrame x with y=None
returns {"Correlation": M, "Dependence": M} built pairwise like R:
symmetric with a unit diagonal for asym=False; for asym=True entry
[i, j] = nns_dep(X[:, i], X[:, j], asym=True) (row -> column, the
orientation R actually returns; its man page says the reverse),
with R's NaN -> 0 on the lower triangle. A DataFrame input returns
DataFrames labelled with its columns. 1-D x without y, or a single
column, raises ValueError.
- The nns_dep docstring now says what asym=True measures (y as a function
of x) and that p.value is not implemented.
- causal_matrix returns a DataFrame labelled with the input columns for
DataFrame input; ndarray input still returns an ndarray.
Verified against R NNS.dep(X) and NNS.dep(X, asym = TRUE) on a 4-column
matrix (new parity tests, exact). The bivariate API and all existing
results are unchanged.
The internal gravity (nns.dependence._gravity and the native gravity_exact kernel behind nns_part "off", nns_dep and nns_causation) built a dense histogram of width IQR / sqrt(n) and clipped it at 1e6 bins (_MAX_DENSE_BINS / the C++ constant). R does not cap, so with an outlier every bulk value beyond bin 1e6 was moved into the last bin and the mode term landed on the outlier. Audit case (n = 1e4 normals, x[0] = -1e5): _gravity(x) = -21558.3 vs R NNS.gravity -2.510; nns_part(x, y, order = 1) regression point x = -21558 vs -2.51; nns_dep 0.3633 vs 0.3592. - _gravity_python counts only the occupied bins (runs of floor((v - x1) / width) over the sorted values), so no cap and no allocation are needed. The bin-count rule, including R's int overflow to one bin (kept as in R), is unchanged. Wherever the old dense code fit, the arithmetic is the same (bit-identical to gravity_exact on 600 randomised arrays in the new test); beyond 1e6 bins with tied maxima it uses R's long-double sum. - The native kernels cannot be rebuilt here, so they are guarded: _gravity uses gravity_exact only when the bin count of that call is below the cap (O(n) partition for the quartiles). For native partitions a cheap a-priori bound covers every subset of at least obs_req + 1 points (q3 - q1 >= min_j (s[j+k] - s[j]) over the full sample); if it does not prove safety, every gravity call of the returned partition is recomputed from the final labels and the prior.quadrant groups. If any would hit the cap, nns_part falls back to the exact Python partitioner and nns_dep to its Python path. - docs/conventions.md described a 4 * len(input) cap that the code never had; it now describes this behaviour. Checks: the guard never accepted a capped partition (3200 native vs Python comparisons on outlier/Cauchy/rounded data, 0 violations; no plain-normal case was flagged); the part A/B harness (26,624 calls) differs from the previous code only on its outlier dataset with noise "off". New parity tests match R on the outlier case for NNS.gravity, NNS.part (XY and XONLY) and NNS.dep (sym and asym; n = 1e4 Dependence 0.5995 -> R's 0.5733). Cost of the guards (one core): nns_dep unchanged; nns_part default n = 1e3 1.1 -> 1.3 ms, 1e4 17 -> 24 ms, 1e5 0.33 -> 0.42 s; XONLY obs_req = 0 (NNS.reg's call) 1.7 -> 2.3 ms at 1e3, 0.21 -> 0.25 s at 1e5; nns_reg n = 1e3 5.4 -> 6.0 ms; _gravity of 10 values 0.9 -> ~11 us. Removing the 1e6 clip in gravity_exact_impl (_nnscore_bindings.cpp) would make the guards unnecessary. Results change only where the cap was hit (they now match R).
- conventions.md "Partition" said R only distinguishes NULL from non-NULL type and that order="max" raises TypeError. Both R 13.3 and the code validate type (ValueError for anything but None/"XONLY") and support order="max"; it now also states the order/obs_req rules and that the returned order is the depth reached (<= floor(log2(n))). - conventions.md "Copula" and api_reference.md described only a bivariate scalar form; the matrix form with any number of columns is documented, with the colMeans default targets and input validation. - api_reference.md named the R API "Causal.matrix", which does not exist; it is NNS.caus on a matrix (internally NNS.caus.matrix). The orientation of causal_matrix and its DataFrame labels are documented. - nns_dep: document the matrix form, what asym=True measures and the asymmetric matrix orientation (row -> column, as R returns it), and that p.value is not implemented. The nns_part docstring gains the argument rules. (The stale "4 * len(input)" gravity cap text was replaced with the gravity fix.) Documentation only.
For x = linspace(-2, 2, 100), y = x^2, C(x--->y) is +18.44 while causal_matrix[0, 1] is -18.44, so the docs had the direction reversed.
They point at notes that don't live in this repo.
gitRasheed
force-pushed
the
rasheed/dependence-partition
branch
from
November 2, 2026 23:09
7ac7c3b to
dd188de
Compare
Contributor
|
Incorporated through NNS R 14.0 integration #192, merged as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
nns_copula(x, y)segfaulted on empty input and returned 1.0 for NaN. Its default targets usednp.mean, which can differ from R'scolMeansin the last ulp and flip the copula. The internal gravity capped its histogram at 1e6 bins, so partitions moved onto the outlier on data with one far value.nns_partwith a noise reduction other than "off" was 8-20x slower than R, andnns_dephad no matrix form.Stacked on #152; merge with a merge commit.
Changes
nns_copulachecks size and finiteness like R and validates the targets. Empty input crashed the interpreter, one NaN returned 1.0 and one Inf 0.728; they now raise R's errors.nns_copulatakes default targets with R's long-doublecolMeans. On a case with four observations on the mean, R gives 1 and Python gave 0.8819; it now gives 1. On R'sdy.dxsin grid the copula goes from 1.0 to R's 0.5759122837209066.nns_depDependence goes from 0.5995 to R's 0.5733. The native kernels keep their 1e6 cap until Re-vendor NNS-core, bind its M.reg kernels and honour ncores #157 replaces them, so calls that could hit it fall back to the Python path.nns_causationcomputes the asymmetric dependence with the nativenns_dep.causal_matrixwith 10 columns at n = 1000 goes from 1.109 s to 0.399 s (R 0.39 s). Values are identical.nns_partwith mean, median or mode noise reduction is vectorised and capsorderat floor(log2(n)) before allocating. At n = 1e5, mean goes from 1.41 s to 0.038 s (R 0.057) and XONLY median from 30.5 s to 0.55 s.order=20000at n = 1000 took 328 MB and now takes 0.1 MB. Results are identical on 26,624 calls.nns_partaccepts NumPy and integral float orders, and rejectsorder=0and fractionalobs_reqas R does.nns_dep(X)on a matrix or DataFrame returns R's Correlation and Dependence matrices, withasym=Trueoriented row to column as R returns it.causal_matrixkeeps DataFrame labels.causal_matrixas the code has them.Tests
New parity tests against R for the colMeans targets, matrix
nns_dep(exact), and an outlier case forNNS.gravity,NNS.part(XY and XONLY) andNNS.dep. Invariants cover copula validation, the native causation path and thenns_partargument rules; thenns_partproperty test drawsorderfrom 1..5. The parity cache encodes unpatched R 13.3.Update
This push adds a guard in
_native_gravity_is_exact. For inputs like[5e-324, 1],range / 128underflowed to 0 and the check raised ZeroDivisionError. R'sNNS.gravityreturns 0 on the same input. Hypothesis 6.170.14 found it throughtest_nns_m_reg_classification_bounds_hold. #157 deletes the function, so nothing above that slice changes.Review
GPT-6.1 Sol reviewed this slice and raised two findings. We took one and rejected one.
causal_matrixdirection backwards. A positive [i, j] means column j causes column i. We fixed the text in all three places and added a test that checks the matrix againstC(x--->y).The fix commits have not been re-reviewed.