Skip to content

Stochastic dominance and ANOVA: drop the dense matrices, follow R's sampling - #152

Closed
gitRasheed wants to merge 16 commits into
rasheed/regression-difffrom
rasheed/dominance-anova
Closed

gitRasheed wants to merge 16 commits into
rasheed/regression-difffrom
rasheed/dominance-anova

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

fsd_uni(type="continuous") built a dense (n_x + n_y) x n matrix and ran out of memory at n = 3e4. The efficient set built a k x k x n tensor and changed its answer when unrelated columns were added. nns_anova used its own optimiser for the VaR targets, NumPy's RNG and paired robust resampling, so seeded runs could not match R. We fix these and close the API gaps.

Stacked on #151; merge with a merge commit.

Changes

  • Continuous fsd_uni evaluates the curves from prefix sums, the same arithmetic as R's .uni walker. n = 1e4 goes from 10.6 s to 5.3 ms, and n = 3e4, which was OOM-killed, takes 16 ms. Verdicts are unchanged on 12,000 random pairs.
  • The efficient set and SD cluster find identical columns by hashing instead of a k x k x n tensor. 500 x 4232 at degree 1 goes from 7.6 s and 1124 MB to 2.8 s and 245 MB. Results do not change.
  • Below 75 columns, every pair is now tested on its own merged grid instead of one global grid, so unrelated columns no longer change the set. At degree 3 a pair that gave [0] alone gave [0, 1] with a third column; it now gives [0] in both, as R does. 11 assets at degree 1 goes from 23 ms to 0.9 ms.
  • sd_efficient_set truncates the degree like R's as.integer (1.5 runs FSD and returns all 11 assets, as R does) and raises ValueError outside 1-3. nns_sd_cluster(min_cluster=0) no longer appends an empty final cluster.
  • fsd takes type=, and a misspelt type warns and follows R's branch. The SD functions accept single-column 2-D input, and nns_sd_cluster labels DataFrame columns by name.
  • nns_anova takes the quantile targets from nns.var's exact lpm_var/upm_var. Certainty now agrees with R to 6.2e-15 on every iris and 11-asset case; the optimiser put it up to 2.7e-7 off.
  • nns_anova(random_seed=s) replays R's set.seed(s) Mersenne-Twister stream, so seeded effect-size bounds match R to rounding (atol 1e-12). Robust mode pairs the resamples as R does instead of reusing one index vector: SPY vs QQQ over 30 seeds went from 0.927 [0.890, 0.956] to R's 0.862 [0.674, 0.962]. The robust keys follow R ("Lower 95% CI" etc.), and confidence_interval=None returns only the estimate.
  • One-sided tails with a confidence interval raise ValueError instead of returning (-inf, inf). R crashes on the same input; Mirror the R fixes to robust ANOVA and stochastic dominance #160 replaces the error with R's fixed bounds. Infinite values raise instead of being dropped silently; NaN is still dropped.
  • nns_norm accepts a DataFrame and returns one with R's " Normalized" names, and nns_ss accepts NumPy booleans. Unused dominance helpers are removed.

Tests

New parity tests against R for continuous fsd_uni up to n = 30,000, efficient sets with unrelated columns, degree coercion, empty clusters, fsd(type=), the seeded and robust ANOVA streams and DataFrame nns_norm. The dominance prefix-pair invariants now check the production evaluators against a per-pair oracle built from the bivariate tests. The parity cache encodes unpatched R 13.3.

Review

GPT-6.1 Sol reviewed this slice and raised two findings. We took one and rejected one.

  • Taken: vignette 06 read the old robust interval keys and printed [nan, nan]. It now prints the robust estimate with its bounds.
  • Rejected: prefix-sum cancellation at tied minima turns a continuous FSD into non-dominance. R 13.3 has the same cancellation, and Stochastic dominance: midpoint check and compensated prefix sums #167 adds R's compensated sums; the reviewer's case returns 1 at the top of the stack.

The fix commits have not been re-reviewed.

fsd_uni(type="continuous") built the upper partial moment as
mean(max(0, values - grid[:, None])), a dense (n_x + n_y) x n matrix.
At n = 1e4 that took 2.7-10 s (R's NNS.FSD.uni: 1.5 ms), n = 3e4 was
OOM-killed and n = 1e5 asked for 149 GiB.

The LPM1/(LPM1 + UPM1) ratio is now evaluated with the existing
prefix-sum evaluator (_pair_curve_values_at_thresholds), which is the
same arithmetic as R's C++ .uni walker: sort once, searchsorted the
merged grid, O((n + grid) log n) time and O(n) memory.

Timing (x = y + 0.1, every gate passes, shared machine):
  n = 1e4: 10.6 s -> 5.3 ms
  n = 3e4: OOM    -> 16 ms
  n = 1e6: n/a    -> 444 ms

Verdicts are unchanged: identical to the old code on 12,000 directed
random pairs (ties, unequal lengths) and on the audit's 600-case
corpus, and equal to R's NNS.FSD.uni on 1,069 equal-length pairs
(excluding the R-only mutual-dominance case) and the 110 finance pairs.
Adds a parity test up to n = 30,000 and a dense-definition invariant.
The efficient-set and cluster precomputes built
np.all(sorted.T[:, None, :] == sorted.T[None, :, :], axis=2), a k x k x n
boolean tensor, to mark columns with identical sorted samples. At 500
columns x 4232 days that is 1.06 GB (peak RSS 1124-1157 MB); 2000 columns
would need about 17 GB.

Columns are now bucketed by the bytes of their sorted values (with -0.0
normalised to 0.0 so the keys agree with ==), and the k x k mask is built
from the bucket labels: O(k n) for the keys plus k^2 bytes.

500 x 4232 (audit fixture), before -> after:
  degree 1: 7.6 s, 1124 MB -> 2.8 s, 245 MB
  degree 2: 4.4 s, 1141 MB -> 0.8 s, 181 MB
  degree 3: 4.5 s, 1157 MB -> 0.9 s, 213 MB
  2000 x 4232, degree 2: previously ~17 GB -> 4.1 s, 524 MB
Efficient-set sizes are unchanged (451 / 27 / 12, as in R). Results do
not change; a test checks the mask against the dense comparison,
including permuted, duplicated and signed-zero columns.
Below 75 columns, sd_efficient_set and nns_sd_cluster evaluated every
column's curve on one global grid (np.unique of all columns) and only
switched to per-pair evaluation at 75 columns. For degree 3 (piecewise
quadratic LPM2) and continuous FSD (linear-fractional ratio), extra knots
from unrelated columns can expose a crossing between a pair's own knots,
so the set depended on which other columns were present:

  x = (-0.85, -1.52, 0.68), y = (-0.21, -0.33, -1.67), z = (-3, 0.16, 0.2)
  sd_efficient_set([x, y], 3)          -> [0]     (R: X_1)
  sd_efficient_set([x, y, z], 3)       -> [0, 1]  (R: X_1)
  [x, y, z] + 71 dominated junk cols   -> [0, 1]  (R: X_1)
  [x, y, z] + 72 dominated junk cols   -> [0]     (R: X_1)

docs/conventions.md already promised per-pair grids. All sizes now use
the paths previously reserved for >= 75 columns (order-statistic matrix
for degree-1 discrete, lazy kept-only prefix scan otherwise), and the
now-unused global-grid code and the column switch are removed. For
degree-1 discrete and degree 2 the two grids were already equivalent, so
those results do not change; degree 3 and continuous FSD now match R in
the cases above. (Checking TSD and continuous FSD only at the knots is
shared with R and is left as is.)

Matches R on the 11-asset finance sets and clusters (degrees 1-3,
continuous) and on 159 of the audit's 160 random matrices (the remaining
one is R's mutual-dominance bug). Faster at small sizes too:
11 assets x 4232 days, degree 1/2/3/continuous: 23/25/31/51 ms ->
0.9/4.9/4.9/11 ms; nns_sd_cluster degree 2: 30 -> 7.4 ms.

The tests that compared the removed global-grid path or toggled the
column switch now compare against the per-pair bivariate test, plus an
invariance test with unrelated columns and a parity test against R.
sd_efficient_set only checked 1 <= degree <= 3, so degree=1.5 fell
through _prefix/curve dispatch to the LPM2 (third-order) curve with the
degree > 1 mean gate on: on the audit's 11 assets it returned the 7-asset
set [5, 0, 4, 6, 7, 1, 8], where R (as.integer(1.5) = 1, FSD) returns all
11. degree="2" raised a TypeError from the comparison.

Both entry points (sd_efficient_set and nns_sd_cluster, which already
used int()) now share one helper that truncates toward zero like R's
as.integer and then requires 1, 2 or 3, raising ValueError otherwise
(0.5, 4, NaN, None, "abc"). Integer degrees behave as before. Parity
tests against R for 1.5, 2.9 and "2".
When the last peel empties the pool, nns_sd_cluster assigned
clusters["Cluster_<n>"] = [], creating an empty final cluster. R's
NNS.SD.cluster assigns colnames() of a 0-column matrix, which is NULL,
and assigning NULL to a list element adds nothing. With min_cluster >= 1
the empty cluster was later merged away, so only min_cluster=0 was
affected: five shifted copies gave {..., 'Cluster_5': ['X_5'],
'Cluster_6': []} where R stops at Cluster_5.

The final cluster is now only added when columns remain. Parity test
against R for min_cluster 0, 1 and 2.
API gaps against R found in the audit:

- fsd(x, y, type="continuous") raised a TypeError; R's NNS.FSD has
  type=. fsd now takes type and runs the same bivariate test on the
  degree-1 ratio LPM1/(LPM1 + UPM1). Where both partial moments vanish
  (a constant sample) the ratio is 0, R's C++ convention (R's
  R-level NNS.FSD returns NA there). On the audit's 798-case corpus it
  matches R's NNS.FSD(type="continuous") everywhere R is not NA, except
  one permuted-sample case where R's curve-identity check fails on
  summation order and R reports dominance between identical samples.
- A misspelt type (e.g. "contin") was silently treated as discrete. Like
  R it now warns "type needs to be either 'discrete' or 'continuous'"
  and follows R's branch: continuous for NNS.FSD, discrete for
  NNS.FSD.uni and the degree-1 efficient set / cluster.
- fsd/ssd/tsd and the *_uni wrappers rejected single-column 2-D inputs
  ("x must be 1D"); R accepts single-column matrices and data frames.
  They are squeezed now, and wider inputs get R's message.
- nns_sd_cluster(DataFrame) labelled columns X_1.. instead of the
  frame's column names (R uses colnames). An explicit names= still wins.

Default-argument results do not change. sd_efficient_set still returns
positional indices for DataFrames (R returns names) to keep its
documented list[int] return type.
_dominates_from_curves, _prefix_directional_candidates_to_target and
_sd_efficient_set_names had no references anywhere. The all-pairs
_dominance_matrix_from_prefix_pairs (with _prefix_pair_candidate_matrix,
_prefix_directional_candidate_matrix and its block-size constant) was
only called from tests, as a second vectorised evaluator to check the
lazy kept-only scan against.

The tests now check the production evaluators (_dominates_from_prefix_pair,
the order-statistic matrix and the kept-only scan) against a per-pair
oracle built from the bivariate fsd/ssd/tsd test, which is independent of
the prefix code. No production behaviour changes.
nns_anova had its own _lpm_var/_upm_var that located the 25% and 12.5%
partial-moment targets with scipy's bounded minimize_scalar, while R's
LPM.VaR/UPM.VaR (and nns.var.lpm_var/upm_var, already ported) invert the
integer-degree partial moments exactly. For SPY at 25% the optimiser gave
-3.556843063e-03 vs -3.556845568e-03 in R, which moved Certainty by up to
2.7e-7 (SPY vs TLT 1.3e-7, finance pairwise matrix 2.7e-7, iris multi-group
3e-11).

The targets now come from nns.var. The degree-0 bounds (effect-size and
robust-certainty quantiles) use the same functions behind a small wrapper
that keeps returning NaN when every certainty is NaN (degenerate groups).
Also drops _lower_area_share, which computed r / (r + (1 - r)) for
r = lpm_ratio(1, ...), i.e. lpm_ratio(1, ...) itself, as R does.

Differences against R after the change (audit script anova_py.py): every
binary, multi-group and pairwise case on iris and the 11-asset matrix is
within 6.2e-15 (was 2.7e-7). means.only results are unchanged.
nns_anova drew its effect-size bootstrap (and robust resamples) from
numpy's default_rng, so a seeded Python run could only agree with R in
distribution. R draws sample(x, n_boot * length(x), replace = TRUE) per
group from its Mersenne-Twister stream, which the package already
reproduces in nns._rrng for nns_stack/nns_boost.

- RRNG gains sample_int_replace(n, size): R's sample.int(n, size,
  replace = TRUE) with the R >= 3.6 rejection sampler, vectorised by
  loading R's MT state into numpy's MT19937 (the same generator) and
  cutting the stream into fixed-width attempts. It is draw-for-draw and
  state-for-state identical to the scalar sample_int (tested), and about
  40x faster (5M draws: ~0.12 s).
- nns_anova(random_seed=s) now replays set.seed(s): the 1000-replicate
  mean/median bootstrap is laid out as R's column-major matrix, and the
  effect-size bounds match R to rounding (atol 1e-12 in the new parity
  tests; e.g. seed 5: LB -0.3320447563778621 vs R -0.33204475637786213).
  Without a seed the stream starts from fresh entropy, as before.

Seeded effect-size bounds therefore change value (same distribution).
Certainty and the other keys do not change. The CI path costs a little
more than with numpy's choice (n = 5000 per group: ~95 -> ~290 ms; R 582
ms). The robust index draws move to the same stream; their pairing is
fixed separately. Test helper _call_r_anova_custom accepts optional
seed/ci/tails keys (old payloads unchanged).
With tails="left" or "right" and a confidence_interval, Python seeded
control_upper = inf / control_lower = -inf and returned
Effect_Size_LB = -inf and Effect_Size_UB = +inf for both tails, i.e. no
information (x ~ N(0,1), y ~ N(1,1), n = 50: left (-inf, inf), right
(-inf, inf), both (0.44, 1.41)). R's NNS.ANOVA.bin crashes on the same
input ("object 'control_upper' not found" / "'control_lower' not
found"), including robust = TRUE.

The correct one-sided bounds are coordinated with the R patch (R has
the same bug), so for now nns_anova raises a ValueError explaining the
limitation, before drawing anything, whenever tails is one-sided and a
confidence_interval is requested. tails="both" and confidence_interval=None
(any tails) are unchanged. Parity test pins the R failure alongside.
Robust nns_anova resampled control and treatment with the same index
vector, i.e. a paired bootstrap. R builds treatment_matrix as
cbind(resample 1, resamples 100..1), so control resample g is paired
with treatment resample 1 (g = 1) or 102 - g: 98 of the 100 replicates
resample the two groups independently, which is also what the book
describes. For correlated series the paired scheme kept the day-by-day
co-movement: SPY vs QQQ (first 300 days, mean of 30 seeds) gave
0.927 [0.890, 0.956] in Python vs 0.862 [0.674, 0.962] in R.

Now, on the R Mersenne-Twister stream from the previous commit:
- the 100 index vectors are drawn first and the effect-size bootstrap
  after, in R's order, and paired exactly as R does;
- the bounds are keyed like R, "Lower <level>% CI"/"Upper <level>% CI"
  (one-sided: "... bound", honouring tails), instead of
  "Lower/Upper Bound Robust Certainty";
- confidence_interval=None returns only "Robust Certainty Estimate",
  as R does, instead of 95% bounds.

This is exact stream parity, not just statistical: for SPY/QQQ, seeds
1..30, estimate and bounds agree with R to 2.2e-15 per seed (means
0.86152428 / 0.67369344 / 0.96238441 on both sides); the new parity
test covers means/medians, CI 0.95/0.9/None. R's robust truncation to
min(n1, n2) observations is shared with R and kept. The
effect-size keys stay "Effect_Size_LB"/"Effect_Size_UB" (R's
"Effect_Size_LB.2.5%" is an unlist() artefact). Public robust results
and key names change.
_as_group filtered values[np.isfinite(values)], so +-Inf was dropped as
silently as NaN: appending inf to a group left Certainty unchanged
(0.768). An infinite observation has no partial-moment meaning, and R's
binary NNS.ANOVA fails on it ("invalid 'xmax' value" inside LPM.VaR).

Infinite values now raise a ValueError naming the group; NaN is still
dropped (R's multi-group path does the same with na.omit; its binary
path propagates NaN, which is less useful). Both rules are now written
down in docs/conventions.md. Finite inputs are unaffected.
nns_norm treated any non-ndarray as a sequence of series, and iterating
a DataFrame yields its column labels, so nns_norm(prices_df) failed with
"could not convert string to float: 'SPY'". A data.frame is R's most
common NNS.norm input.

A DataFrame-like input (has columns and to_numpy) is now normalised
through the matrix path and returned as a DataFrame with the original
index and R's "<column> Normalized" column names. The numbers are
bitwise identical to passing the same values as a C-ordered ndarray
(and match R to 5e-15 on the audit's 11-asset prices). ndarray and list
inputs are unchanged.
nns_ss checked isinstance(confidence_interval, bool), so a flag computed
with numpy (np.True_ from a comparison or array element) was rejected
with "confidence_interval must be a single TRUE/FALSE value", although R
accepts any logical scalar. np.bool_ is now accepted; other types (e.g.
the integer 1) still raise. Results do not change.
The vignette still looked up "Lower/Upper Bound Robust Certainty" and printed
[nan, nan]; it now shows e.g. robust = 0.0113 [0.0000, 0.1288].
They point at notes that don't live in this repo.
@gitRasheed
gitRasheed force-pushed the rasheed/dominance-anova branch from aea6e7f to 31f812b Compare November 2, 2026 23:09
@gitRasheed
gitRasheed force-pushed the rasheed/regression-diff branch from 879ce04 to 256378c Compare November 2, 2026 23:09
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head 31f812b951ee3c8bfa5facc67e5e1272eacecb58 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/dominance-anova branch November 7, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant