Skip to content

sd: efficient-set membership no longer depends on column order - #183

Closed
gitRasheed wants to merge 1 commit into
rasheed/py-arma-train-supportfrom
rasheed/py-sd-efficient-set-order
Closed

gitRasheed wants to merge 1 commit into
rasheed/py-arma-train-supportfrom
rasheed/py-sd-efficient-set-order

Conversation

@gitRasheed

Copy link
Copy Markdown
Collaborator

R's NNS.SD.efficient.set sorted columns by LPM(degree, tmax) and kept a column unless an earlier kept column dominated it. Continuous FSD does not follow that order: for x = (-1, -1, 2) and y = (-2, 1, 1), x dominates y but y sorts first, so the set was (y, x) or (x) depending on column order. Tolerance-level ties can also make dominance cyclic, and the scan kept whichever cycle members came first. OVVO-Financial/NNS#127 (merged) keeps every column of a dominance-graph strongly connected component that no column outside it dominates. This slice mirrors it.

Stacked on #182; merge with a merge commit.

Changes

  • The degree-1 discrete path already builds the full dominance matrix. _sd_efficient_active_indices_from_matrix now takes the undominated strongly connected components of its active submatrix (scipy.sparse.csgraph.connected_components). The output keeps the LPM-at-tmax order.
  • The prefix path (SSD, TSD, continuous FSD) used to test each column only against those already kept. Building the full matrix there made sd_efficient_set on 250 x 150 data 36x slower at degree 2 (0.02 s -> 0.73 s), so it now tests only the pairs it needs. A column that nothing dominates is efficient, one dominated by such a column is not, and only the leftover columns, which only dominated columns dominate, go through the component check. That costs about what the old scan did (0.03 s). nns_sd_cluster shares the pair verdicts across peels, since a pair's verdict does not depend on the other columns.
  • Before and after, with after equal to R main: sd_efficient_set([y, x], 1, "continuous") [0, 1] -> [1], nns_sd_cluster([y, x], 1, "continuous") {Cluster_1: [Y, X]} -> {Cluster_1: [X], Cluster_2: [Y]}, and R's tolerance cycle A > B > C > A now returns all three in R main's order for each of R's four column orders.

Tests

  • New invariants mirroring R's test-sd-dominance.R: the continuous-FSD order case, the tolerance cycle under four permutations (output order checked against R main), the cycle with a column below it (all three stay) and above it (only that column), and a permutation-invariance sweep over 100 random integer matrices at each degree and type. R's sweep uses set.seed(s) samples, so mine draws from NumPy.
  • The three tests that compared the kept-only prefix scan with the matrix path now compare the new prefix path with it. Off the suite, the two paths gave identical sets on 600 random small matrices (none of which happened to contain a cycle) and on 300 built around R's tolerance cycle with extra columns above, below and across it.
  • No cache entry moves.
  • Gates on the top of this stack (rasheed/ss-constant-group, which includes this slice), cache only: native 2935 passed, 4 skipped, 3 xfailed, 2 xpassed. With NNS_DISABLE_NATIVE=1: 2913 passed, 29 skipped, 2 xpassed. The 3 xfails are mode: break tied discrete modes by gravity, not their mean (mirrors R #111) #178's strict ones and the 2 xpasses the existing non-strict live-R VAR xfails. tests/parity/test_practical_examples.py against the usual R library: 9 passed, 2 xfailed, as on mode: break tied discrete modes by gravity, not their mean (mirrors R #111) #178. ruff check is clean on every changed file.

Review

GPT-6.1 Sol reviewed the #135, #127 and #126 mirrors together: SHIP. It matched R main on 1,050 efficient-set cases, 150 of them with cycles.

Mirrors OVVO-Financial/NNS#127

…efficient-set-order)

Mirrors R NNS.SD.efficient.set (OVVO-Financial/NNS sd-efficient-set-order).
Columns were sorted by LPM(degree, tmax) and a column was kept unless an
earlier kept column dominated it. Continuous FSD does not follow that order:
for x = [-1, -1, 2], y = [-2, 1, 1], x dominates y but y sorts first, so the
set was [y, x] or [x] depending on the column order. Tolerance-level ties
can also make dominance cyclic, and the scan kept whichever cycle members
came first.

The efficient set is now every column in a strongly connected component of
the dominance graph that no column outside the component dominates, which
is "not dominated by any column" when there are no cycles. Output keeps the
LPM-at-tmax order. The degree-1 discrete path has the full matrix already.
The prefix path (SSD, TSD, continuous FSD) tests only the pairs it needs:
a column nothing dominates is efficient, one dominated by such a column is
not, and only the columns left over get the component check, so it costs
about what the old scan did. nns_sd_cluster shares the pair verdicts across
its peels.

Before -> after (= R main):
  sd_efficient_set([y, x], 1, "continuous")   [0, 1] -> [1]
  nns_sd_cluster([y, x], 1, "continuous")     {Cluster_1: [Y, X]}
                                    -> {Cluster_1: [X], Cluster_2: [Y]}
  A > B > C > A at tolerance level: every column order now gives all three

No cache entry moves.
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head 91e14bd047adf03c222e7dfa2a0f4222a2706ebe is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/py-sd-efficient-set-order branch November 7, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant