Repository navigation
Mirror the R fixes to robust ANOVA and stochastic dominance - #160
Closed
gitRasheed wants to merge 7 commits into
Closed
gitRasheed wants to merge 7 commits into
gitRasheed wants to merge 7 commits into
Conversation
This was referenced Nov 2, 2026
gitRasheed
force-pushed
the
rasheed/r-fixes-dominance-anova
branch
from
November 2, 2026 04:12
98b0faa to
22c4a06
Compare
gitRasheed
force-pushed
the
rasheed/r-fixes-timeseries
branch
from
November 2, 2026 04:12
c0118f0 to
72daf66
Compare
…of the R fix)
The robust path drew one index matrix with sample.int(min(n1, n2)) and
used it for both groups, so only the first min(n1, n2) observations of
the longer group were ever resampled and the result depended on row
order; Python reproduced R's draws and pairing exactly. R now
draws independent bootstrap indices for each group from its own full
sample and size (control first, then treatment); do the same, still on
the reproduced R Mersenne-Twister stream, so random_seed=s matches
set.seed(s) in the patched R.
Before -> after (Robust Certainty Estimate [95% CI], = patched R):
control = 50 x N(0,1) then 950 x N(5,1), treatment = 50 x N(0,1), seed 1:
0.677 [0.265, 0.903] -> 0 [0, 0]
control = 50 sin values then 150 shifted by 5, treatment 50 sin values:
0.785 [0.452, 0.926] -> 0 (base Certainty 0)
The base (non-robust) Certainty is unchanged. The NNS.ANOVA.custom shard
is regenerated from the patched R (the one-sided-tail entries keep their
13.3 values until the one-sided-tail R fix is mirrored next).
With confidence_interval and tails="left"/"right" R's NNS.ANOVA.bin needed a quantile it had not computed and stopped with "object 'control_upper' / 'control_lower' not found"; Python raised a ValueError pointing at that failure. R now computes both bootstrap quantiles of both groups for every tail choice; replace the ValueError with R's result: tails="left" (treatment_lower - control_upper, Inf) tails="right" (-Inf, treatment_upper - control_lower) also on the robust path (with its one-sided "Lower/Upper <level>% bound"). Before -> after (the 30-point parity groups, ci = 0.95, random_seed=1; = patched R with set.seed(1)): tails="left" ValueError -> Effect_Size_LB -0.443, Effect_Size_UB Inf tails="right" ValueError -> Effect_Size_LB -Inf, Effect_Size_UB 0.959 tails="both" unchanged (-0.545, 1.095) The NNS.ANOVA.custom shard is regenerated from the patched R.
… fixes)
The continuous FSD curve L1/(L1+U1) is 0/0 when every observation of a
sample equals the threshold (a constant sample, or a threshold at a
point mass). R's .uni code and Python mapped it to 0, which makes a
point mass look as if it had no mass at its own location and produced
spurious dominance. R now uses the degree-0 ratio there, as LPM.ratio
does; do the same in the prefix-sum curve that
fsd/fsd_uni(type="continuous") and the efficient set use.
Before -> after (= patched R):
a = [5, 5, 0, 4, 0, 2], b = [4] * 6: fsd_uni(b, a, "continuous") 1 -> 0
x = [0, 0], y = [0, 3]: fsd_uni(y, x, "continuous") 0 -> 1;
fsd(x, y, "continuous") 0 -> -1 ("Y FSD X")
R's other .uni fixes (walk each sample with its own length;
no dominance when the curves coincide) needed
no Python change: Python already agreed with R's bidirectional tests.
New parity cases pin all three against the patched R .uni routines.
The NNS.FSD, NNS.FSD.uni, NNS.SSD, NNS.TSD, NNS.SD.efficient.set,
NNS.SD.cluster and NNS.SD.cluster.dendrogram shards are regenerated from
the patched R (no existing entry changes); NNS.SSD.uni and NNS.TSD.uni
shards are added.
…ror of the R fix)
Dominance was checked only at the merged sample points. That is exact
for discrete FSD (step functions) and SSD (LPM1 is piecewise linear),
but LPM2 is piecewise quadratic and the continuous FSD ratio
L1/(L1+U1) is linear-fractional between points, so the difference of
the two curves can peak strictly between knots. tsd/tsd_uni, continuous
fsd/fsd_uni, the efficient set and SD clustering reported false
dominance, e.g. x = (-0.85, -1.52, 0.68), y = (-0.21, -0.33, -1.67):
tsd(x, y) = 1 although LPM2_x - LPM2_y = +0.0389 at t = 0.16.
All pairwise verdicts now come from one port of the patched R's native
pairwise test (sd_dom_pair): curves from prefix sums on the merged
unique thresholds, each sample with its own count, plus the closed-form
interior extremum on each interval between knots: the vertex b/a of
the quadratic when a < 0 (TSD) and the stationary point of the
difference of the two linear-fractional ratios (continuous FSD).
Candidates whose excess is at rounding level (64 eps relative to the
data scale) are ignored, as in R. The bidirectional tests take their
verdict from that test in both directions, as before.
Before -> after (A/B against the patched R, 2611 pairs: the audit's
counterexamples and SD cases plus 1800 small random rounded pairs):
8 TSD counterexamples: tsd and tsd_uni say X TSD Y in 8 -> 0
continuous FSD x = (0.15, 1.55, -0.86, -0.8, 0.25, 0.33),
y = (-0.15, 1.17, 0.26, -0.04, -2.44, 1.19): 1 -> 0
agreement with R: continuous FSD 2567 -> 2611 (all; the 44 changes
are curve crossings between knots, confirmed on an exact rational
grid, most on integer-valued data), discrete FSD 2611 -> 2611,
TSD 2590 -> 2599
SSD 2600 -> 2594: SSD/TSD curves are now computed from prefix sums
like R's native test (and Python's efficient set already did), which
moves rounding-level ties; the next commit adds R's tie tolerance,
after which every pair agrees.
The three NNS.SD.efficient.set cache entries of the TSD extra-knots
test change from "X_1" to ["X_1", "X_2"] after regeneration from the
patched R; every other SD entry the tests read is unchanged.
Efficient set 252 x 50: degree 2 2.1 -> 2.2 ms, degree 3 2.6 -> 3.6 ms,
continuous FSD 3.4 -> 5.7 ms; nns_sd_cluster degree 2 7.1 -> 11.7 ms.
…e R fix)
Dominance verdicts compared curve values exactly, so rounding noise of
order 1e-16 decided them. The textbook mean-preserving spread
x = qnorm(ppoints(100)), y = 2x and the same pair shifted by 0.37 flip
with the summation order, and x = (0.1, 0.7) vs y = (0, 0.8) gave no
SSD because 0.1 + 0.7 = 0.7999999999999999, while the same data times
10 gave X SSD Y.
As in the patched R's native pairwise test, differences up to
64 * eps * S^d are ties (S = largest |value| of the two samples; d = 1
for SSD curves and the mean gate, 2 for TSD, a ratio analogue
max(1, S / min(L1 + U1)) for continuous FSD); discrete FSD still
compares exact ECDF fractions. The bidirectional fsd/ssd/tsd already
take their verdict from the pairwise test in both directions, so they
and the *_uni functions keep agreeing.
Before -> after:
(0.1, 0.7) vs (0, 0.8): ssd/tsd 0 -> 1 (= the x10 case)
vs the patched R, 2611 pairs x 12 tests (fsd/ssd/tsd both types,
the .uni functions both ways): 58 disagreeing verdicts -> 0
(ssd 17, tsd 12, ssd_uni 14 + 3, tsd_uni 7 + 5)
efficient sets and nns_sd_cluster, degrees 1-3 and continuous FSD, on
the 11-asset daily returns, its last 252 rows, rounded / integer /
mean-preserving-spread matrices: 35 of 35 equal to R (32 before)
The SD cache entries the tests read were regenerated from the patched
R and are unchanged; entries for the new rounding-noise tests are added.
Every pairwise SD test built the merged threshold grid with np.union1d
and then counted each sample's observations <= every threshold with
np.searchsorted over the 2n thresholds; the two searches were about 40%
of a pairwise test at n = 1e4 (cProfile, tsd). One stable argsort of the
two concatenated sorted runs (a linear merge) plus a cumulative count now
gives the same thresholds and the same integer counts.
Bit-identical: thresholds and counts equal union1d/searchsorted (new
invariant test), and an A/B over 3000 random pairs (normal, integer ties,
rounded with signed zeros, near-duplicates; FSD discrete and continuous,
SSD, TSD, both directions and the .uni forms), the efficient set at
degrees 1-3 and nns_sd_cluster gave identical verdicts.
Timings (best of 7, shared 4-core box), before -> after:
ssd / ssd_uni n=1e4 1.83 / 1.67 ms -> 0.94 / 0.87 ms
tsd / tsd_uni n=1e4 2.55 / 2.34 ms -> 2.09 / 1.54 ms
fsd / ssd / tsd n=3e4 5.6 / 7.9 / 10.0 ms -> 3.5 / 4.4 / 6.1 ms
efficient set, 11 x 4232, degree 1/2/3: 1.48 / 9.12 / 8.44 ms
-> 1.44 / 3.72 / 4.70 ms
nns_sd_cluster degree 2: 8.0 ms -> 3.5 ms
At n = 1e3 the tests take 0.02-0.1 ms (no change), already at R's
native .uni cost; the fixed small-n overhead was removed earlier.
gitRasheed
force-pushed
the
rasheed/r-fixes-dominance-anova
branch
from
November 2, 2026 23:09
22c4a06 to
ed0c496
Compare
gitRasheed
force-pushed
the
rasheed/r-fixes-timeseries
branch
from
November 2, 2026 23:09
72daf66 to
ea29dbf
Compare
Contributor
|
Incorporated through NNS R 14.0 integration #192, merged as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Robust
nns_anovaresampled only the first min(n1, n2) observations of the longer group, so the result depended on row order, and one-sided tails with an interval had no answer. Dominance was checked only at the sample points and compared curves exactly, so TSD and continuous FSD reported false dominance and textbook SSD cases flipped on 1e-16 rounding. We mirror the patched R for each.Stacked on #159; merge with a merge commit.
Changes
random_seed=smatchesset.seed(s)in the patched R. For a control of 50 N(0,1) rows followed by 950 N(5,1) rows against 50 N(0,1) treatment rows, the estimate goes from 0.677 [0.265, 0.903] to 0 [0, 0].tails="left"/"right"with a confidence interval return R's one-sided bounds instead of theValueErrorfrom Stochastic dominance and ANOVA: drop the dense matrices, follow R's sampling #152. On the parity groups, left gives (-0.443, Inf) and right (-Inf, 0.959).LPM.ratiodoes, instead of 0.tsdandtsd_unireported dominance in 8 and now in 0.ssdat n = 1e4 goes from 1.83 ms to 0.94 ms.Tests
New parity cases against R 13.3 with the audit's R patches for the robust and one-sided ANOVA paths, the point-mass ratio, between-knot crossings, rounding-level ties and the patched
.uniroutines. The SD andNNS.ANOVA.customshards are regenerated from that build; the three TSD extra-knot efficient-set entries change from "X_1" to ["X_1", "X_2"].NNS.SSD.uniandNNS.TSD.unishards are new.test_public_nns_var_objective_max_matches_rstays xfailed until #161.Mirrors OVVO-Financial/NNS#81 and OVVO-Financial/NNS#82
Review
GPT-6.1 Sol reviewed this slice and raised two findings. We rejected both.