Repository navigation
Stochastic dominance and ANOVA: drop the dense matrices, follow R's sampling - #152
Closed
gitRasheed wants to merge 16 commits into
Closed
gitRasheed wants to merge 16 commits into
gitRasheed wants to merge 16 commits into
Conversation
This was referenced Nov 2, 2026
fsd_uni(type="continuous") built the upper partial moment as mean(max(0, values - grid[:, None])), a dense (n_x + n_y) x n matrix. At n = 1e4 that took 2.7-10 s (R's NNS.FSD.uni: 1.5 ms), n = 3e4 was OOM-killed and n = 1e5 asked for 149 GiB. The LPM1/(LPM1 + UPM1) ratio is now evaluated with the existing prefix-sum evaluator (_pair_curve_values_at_thresholds), which is the same arithmetic as R's C++ .uni walker: sort once, searchsorted the merged grid, O((n + grid) log n) time and O(n) memory. Timing (x = y + 0.1, every gate passes, shared machine): n = 1e4: 10.6 s -> 5.3 ms n = 3e4: OOM -> 16 ms n = 1e6: n/a -> 444 ms Verdicts are unchanged: identical to the old code on 12,000 directed random pairs (ties, unequal lengths) and on the audit's 600-case corpus, and equal to R's NNS.FSD.uni on 1,069 equal-length pairs (excluding the R-only mutual-dominance case) and the 110 finance pairs. Adds a parity test up to n = 30,000 and a dense-definition invariant.
The efficient-set and cluster precomputes built np.all(sorted.T[:, None, :] == sorted.T[None, :, :], axis=2), a k x k x n boolean tensor, to mark columns with identical sorted samples. At 500 columns x 4232 days that is 1.06 GB (peak RSS 1124-1157 MB); 2000 columns would need about 17 GB. Columns are now bucketed by the bytes of their sorted values (with -0.0 normalised to 0.0 so the keys agree with ==), and the k x k mask is built from the bucket labels: O(k n) for the keys plus k^2 bytes. 500 x 4232 (audit fixture), before -> after: degree 1: 7.6 s, 1124 MB -> 2.8 s, 245 MB degree 2: 4.4 s, 1141 MB -> 0.8 s, 181 MB degree 3: 4.5 s, 1157 MB -> 0.9 s, 213 MB 2000 x 4232, degree 2: previously ~17 GB -> 4.1 s, 524 MB Efficient-set sizes are unchanged (451 / 27 / 12, as in R). Results do not change; a test checks the mask against the dense comparison, including permuted, duplicated and signed-zero columns.
Below 75 columns, sd_efficient_set and nns_sd_cluster evaluated every column's curve on one global grid (np.unique of all columns) and only switched to per-pair evaluation at 75 columns. For degree 3 (piecewise quadratic LPM2) and continuous FSD (linear-fractional ratio), extra knots from unrelated columns can expose a crossing between a pair's own knots, so the set depended on which other columns were present: x = (-0.85, -1.52, 0.68), y = (-0.21, -0.33, -1.67), z = (-3, 0.16, 0.2) sd_efficient_set([x, y], 3) -> [0] (R: X_1) sd_efficient_set([x, y, z], 3) -> [0, 1] (R: X_1) [x, y, z] + 71 dominated junk cols -> [0, 1] (R: X_1) [x, y, z] + 72 dominated junk cols -> [0] (R: X_1) docs/conventions.md already promised per-pair grids. All sizes now use the paths previously reserved for >= 75 columns (order-statistic matrix for degree-1 discrete, lazy kept-only prefix scan otherwise), and the now-unused global-grid code and the column switch are removed. For degree-1 discrete and degree 2 the two grids were already equivalent, so those results do not change; degree 3 and continuous FSD now match R in the cases above. (Checking TSD and continuous FSD only at the knots is shared with R and is left as is.) Matches R on the 11-asset finance sets and clusters (degrees 1-3, continuous) and on 159 of the audit's 160 random matrices (the remaining one is R's mutual-dominance bug). Faster at small sizes too: 11 assets x 4232 days, degree 1/2/3/continuous: 23/25/31/51 ms -> 0.9/4.9/4.9/11 ms; nns_sd_cluster degree 2: 30 -> 7.4 ms. The tests that compared the removed global-grid path or toggled the column switch now compare against the per-pair bivariate test, plus an invariance test with unrelated columns and a parity test against R.
sd_efficient_set only checked 1 <= degree <= 3, so degree=1.5 fell through _prefix/curve dispatch to the LPM2 (third-order) curve with the degree > 1 mean gate on: on the audit's 11 assets it returned the 7-asset set [5, 0, 4, 6, 7, 1, 8], where R (as.integer(1.5) = 1, FSD) returns all 11. degree="2" raised a TypeError from the comparison. Both entry points (sd_efficient_set and nns_sd_cluster, which already used int()) now share one helper that truncates toward zero like R's as.integer and then requires 1, 2 or 3, raising ValueError otherwise (0.5, 4, NaN, None, "abc"). Integer degrees behave as before. Parity tests against R for 1.5, 2.9 and "2".
When the last peel empties the pool, nns_sd_cluster assigned
clusters["Cluster_<n>"] = [], creating an empty final cluster. R's
NNS.SD.cluster assigns colnames() of a 0-column matrix, which is NULL,
and assigning NULL to a list element adds nothing. With min_cluster >= 1
the empty cluster was later merged away, so only min_cluster=0 was
affected: five shifted copies gave {..., 'Cluster_5': ['X_5'],
'Cluster_6': []} where R stops at Cluster_5.
The final cluster is now only added when columns remain. Parity test
against R for min_cluster 0, 1 and 2.
API gaps against R found in the audit:
- fsd(x, y, type="continuous") raised a TypeError; R's NNS.FSD has
type=. fsd now takes type and runs the same bivariate test on the
degree-1 ratio LPM1/(LPM1 + UPM1). Where both partial moments vanish
(a constant sample) the ratio is 0, R's C++ convention (R's
R-level NNS.FSD returns NA there). On the audit's 798-case corpus it
matches R's NNS.FSD(type="continuous") everywhere R is not NA, except
one permuted-sample case where R's curve-identity check fails on
summation order and R reports dominance between identical samples.
- A misspelt type (e.g. "contin") was silently treated as discrete. Like
R it now warns "type needs to be either 'discrete' or 'continuous'"
and follows R's branch: continuous for NNS.FSD, discrete for
NNS.FSD.uni and the degree-1 efficient set / cluster.
- fsd/ssd/tsd and the *_uni wrappers rejected single-column 2-D inputs
("x must be 1D"); R accepts single-column matrices and data frames.
They are squeezed now, and wider inputs get R's message.
- nns_sd_cluster(DataFrame) labelled columns X_1.. instead of the
frame's column names (R uses colnames). An explicit names= still wins.
Default-argument results do not change. sd_efficient_set still returns
positional indices for DataFrames (R returns names) to keep its
documented list[int] return type.
_dominates_from_curves, _prefix_directional_candidates_to_target and _sd_efficient_set_names had no references anywhere. The all-pairs _dominance_matrix_from_prefix_pairs (with _prefix_pair_candidate_matrix, _prefix_directional_candidate_matrix and its block-size constant) was only called from tests, as a second vectorised evaluator to check the lazy kept-only scan against. The tests now check the production evaluators (_dominates_from_prefix_pair, the order-statistic matrix and the kept-only scan) against a per-pair oracle built from the bivariate fsd/ssd/tsd test, which is independent of the prefix code. No production behaviour changes.
nns_anova had its own _lpm_var/_upm_var that located the 25% and 12.5% partial-moment targets with scipy's bounded minimize_scalar, while R's LPM.VaR/UPM.VaR (and nns.var.lpm_var/upm_var, already ported) invert the integer-degree partial moments exactly. For SPY at 25% the optimiser gave -3.556843063e-03 vs -3.556845568e-03 in R, which moved Certainty by up to 2.7e-7 (SPY vs TLT 1.3e-7, finance pairwise matrix 2.7e-7, iris multi-group 3e-11). The targets now come from nns.var. The degree-0 bounds (effect-size and robust-certainty quantiles) use the same functions behind a small wrapper that keeps returning NaN when every certainty is NaN (degenerate groups). Also drops _lower_area_share, which computed r / (r + (1 - r)) for r = lpm_ratio(1, ...), i.e. lpm_ratio(1, ...) itself, as R does. Differences against R after the change (audit script anova_py.py): every binary, multi-group and pairwise case on iris and the 11-asset matrix is within 6.2e-15 (was 2.7e-7). means.only results are unchanged.
nns_anova drew its effect-size bootstrap (and robust resamples) from numpy's default_rng, so a seeded Python run could only agree with R in distribution. R draws sample(x, n_boot * length(x), replace = TRUE) per group from its Mersenne-Twister stream, which the package already reproduces in nns._rrng for nns_stack/nns_boost. - RRNG gains sample_int_replace(n, size): R's sample.int(n, size, replace = TRUE) with the R >= 3.6 rejection sampler, vectorised by loading R's MT state into numpy's MT19937 (the same generator) and cutting the stream into fixed-width attempts. It is draw-for-draw and state-for-state identical to the scalar sample_int (tested), and about 40x faster (5M draws: ~0.12 s). - nns_anova(random_seed=s) now replays set.seed(s): the 1000-replicate mean/median bootstrap is laid out as R's column-major matrix, and the effect-size bounds match R to rounding (atol 1e-12 in the new parity tests; e.g. seed 5: LB -0.3320447563778621 vs R -0.33204475637786213). Without a seed the stream starts from fresh entropy, as before. Seeded effect-size bounds therefore change value (same distribution). Certainty and the other keys do not change. The CI path costs a little more than with numpy's choice (n = 5000 per group: ~95 -> ~290 ms; R 582 ms). The robust index draws move to the same stream; their pairing is fixed separately. Test helper _call_r_anova_custom accepts optional seed/ci/tails keys (old payloads unchanged).
With tails="left" or "right" and a confidence_interval, Python seeded
control_upper = inf / control_lower = -inf and returned
Effect_Size_LB = -inf and Effect_Size_UB = +inf for both tails, i.e. no
information (x ~ N(0,1), y ~ N(1,1), n = 50: left (-inf, inf), right
(-inf, inf), both (0.44, 1.41)). R's NNS.ANOVA.bin crashes on the same
input ("object 'control_upper' not found" / "'control_lower' not
found"), including robust = TRUE.
The correct one-sided bounds are coordinated with the R patch (R has
the same bug), so for now nns_anova raises a ValueError explaining the
limitation, before drawing anything, whenever tails is one-sided and a
confidence_interval is requested. tails="both" and confidence_interval=None
(any tails) are unchanged. Parity test pins the R failure alongside.
Robust nns_anova resampled control and treatment with the same index vector, i.e. a paired bootstrap. R builds treatment_matrix as cbind(resample 1, resamples 100..1), so control resample g is paired with treatment resample 1 (g = 1) or 102 - g: 98 of the 100 replicates resample the two groups independently, which is also what the book describes. For correlated series the paired scheme kept the day-by-day co-movement: SPY vs QQQ (first 300 days, mean of 30 seeds) gave 0.927 [0.890, 0.956] in Python vs 0.862 [0.674, 0.962] in R. Now, on the R Mersenne-Twister stream from the previous commit: - the 100 index vectors are drawn first and the effect-size bootstrap after, in R's order, and paired exactly as R does; - the bounds are keyed like R, "Lower <level>% CI"/"Upper <level>% CI" (one-sided: "... bound", honouring tails), instead of "Lower/Upper Bound Robust Certainty"; - confidence_interval=None returns only "Robust Certainty Estimate", as R does, instead of 95% bounds. This is exact stream parity, not just statistical: for SPY/QQQ, seeds 1..30, estimate and bounds agree with R to 2.2e-15 per seed (means 0.86152428 / 0.67369344 / 0.96238441 on both sides); the new parity test covers means/medians, CI 0.95/0.9/None. R's robust truncation to min(n1, n2) observations is shared with R and kept. The effect-size keys stay "Effect_Size_LB"/"Effect_Size_UB" (R's "Effect_Size_LB.2.5%" is an unlist() artefact). Public robust results and key names change.
_as_group filtered values[np.isfinite(values)], so +-Inf was dropped as
silently as NaN: appending inf to a group left Certainty unchanged
(0.768). An infinite observation has no partial-moment meaning, and R's
binary NNS.ANOVA fails on it ("invalid 'xmax' value" inside LPM.VaR).
Infinite values now raise a ValueError naming the group; NaN is still
dropped (R's multi-group path does the same with na.omit; its binary
path propagates NaN, which is less useful). Both rules are now written
down in docs/conventions.md. Finite inputs are unaffected.
nns_norm treated any non-ndarray as a sequence of series, and iterating a DataFrame yields its column labels, so nns_norm(prices_df) failed with "could not convert string to float: 'SPY'". A data.frame is R's most common NNS.norm input. A DataFrame-like input (has columns and to_numpy) is now normalised through the matrix path and returned as a DataFrame with the original index and R's "<column> Normalized" column names. The numbers are bitwise identical to passing the same values as a C-ordered ndarray (and match R to 5e-15 on the audit's 11-asset prices). ndarray and list inputs are unchanged.
nns_ss checked isinstance(confidence_interval, bool), so a flag computed with numpy (np.True_ from a comparison or array element) was rejected with "confidence_interval must be a single TRUE/FALSE value", although R accepts any logical scalar. np.bool_ is now accepted; other types (e.g. the integer 1) still raise. Results do not change.
The vignette still looked up "Lower/Upper Bound Robust Certainty" and printed [nan, nan]; it now shows e.g. robust = 0.0113 [0.0000, 0.1288].
They point at notes that don't live in this repo.
gitRasheed
force-pushed
the
rasheed/dominance-anova
branch
from
November 2, 2026 23:09
aea6e7f to
31f812b
Compare
gitRasheed
force-pushed
the
rasheed/regression-diff
branch
from
November 2, 2026 23:09
879ce04 to
256378c
Compare
Contributor
|
Incorporated through NNS R 14.0 integration #192, merged as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fsd_uni(type="continuous")built a dense (n_x + n_y) x n matrix and ran out of memory at n = 3e4. The efficient set built a k x k x n tensor and changed its answer when unrelated columns were added.nns_anovaused its own optimiser for the VaR targets, NumPy's RNG and paired robust resampling, so seeded runs could not match R. We fix these and close the API gaps.Stacked on #151; merge with a merge commit.
Changes
fsd_unievaluates the curves from prefix sums, the same arithmetic as R's.uniwalker. n = 1e4 goes from 10.6 s to 5.3 ms, and n = 3e4, which was OOM-killed, takes 16 ms. Verdicts are unchanged on 12,000 random pairs.sd_efficient_settruncates the degree like R'sas.integer(1.5 runs FSD and returns all 11 assets, as R does) and raisesValueErroroutside 1-3.nns_sd_cluster(min_cluster=0)no longer appends an empty final cluster.fsdtakestype=, and a misspelt type warns and follows R's branch. The SD functions accept single-column 2-D input, andnns_sd_clusterlabels DataFrame columns by name.nns_anovatakes the quantile targets fromnns.var's exactlpm_var/upm_var. Certainty now agrees with R to 6.2e-15 on every iris and 11-asset case; the optimiser put it up to 2.7e-7 off.nns_anova(random_seed=s)replays R'sset.seed(s)Mersenne-Twister stream, so seeded effect-size bounds match R to rounding (atol 1e-12). Robust mode pairs the resamples as R does instead of reusing one index vector: SPY vs QQQ over 30 seeds went from 0.927 [0.890, 0.956] to R's 0.862 [0.674, 0.962]. The robust keys follow R ("Lower 95% CI" etc.), andconfidence_interval=Nonereturns only the estimate.tailswith a confidence interval raiseValueErrorinstead of returning (-inf, inf). R crashes on the same input; Mirror the R fixes to robust ANOVA and stochastic dominance #160 replaces the error with R's fixed bounds. Infinite values raise instead of being dropped silently; NaN is still dropped.nns_normaccepts a DataFrame and returns one with R's " Normalized" names, andnns_ssaccepts NumPy booleans. Unused dominance helpers are removed.Tests
New parity tests against R for continuous
fsd_uniup to n = 30,000, efficient sets with unrelated columns, degree coercion, empty clusters,fsd(type=), the seeded and robust ANOVA streams and DataFramenns_norm. The dominance prefix-pair invariants now check the production evaluators against a per-pair oracle built from the bivariate tests. The parity cache encodes unpatched R 13.3.Review
GPT-6.1 Sol reviewed this slice and raised two findings. We took one and rejected one.
The fix commits have not been re-reviewed.