Skip to content

sd: cluster keeps hclust's leaf order, and the variables when nothing is peeled - #184

Closed
gitRasheed wants to merge 1 commit into
rasheed/py-sd-efficient-set-orderfrom
rasheed/py-sd-cluster-fixes
Closed

gitRasheed wants to merge 1 commit into
rasheed/py-sd-efficient-set-orderfrom
rasheed/py-sd-cluster-fixes

Conversation

@gitRasheed

Copy link
Copy Markdown
Collaborator

Two fixes to R's NNS.SD.cluster from OVVO-Financial/NNS#126 (merged). The returned dendrogram's order was overwritten with each label's original column position, which is not a leaf order of the tree. And when min_cluster was at or above the number of columns nothing was peeled, and the final-cluster step also needed more than min_cluster columns, so the result had no clusters. This slice mirrors both.

Stacked on #183; merge with a merge commit.

Changes

  • "order" is hclust's own leaf order: the last merge expanded left before right, as R's hcass2 does, read from the R-convention merge matrix _r_hclust_merge already builds. On a 12-column test matrix this gives R main's order [1, 2, 3, 4, 9, 10, 11, 12, 5, 6, 7, 8].
  • Any variables left after peeling form the final cluster (if active instead of len(active) > min_cluster). nns_sd_cluster(4 columns, min_cluster=4) {} -> {Cluster_1: [X_1, ..., X_4]}, cbind(A = 1:5, B = 2:6) with min_cluster=2 {} -> {Cluster_1: [A, B]}, both as on R main. A single column still warns that there are too few variables for a dendrogram.
  • R also switched to plot.dendrogram for a two-leaf tree because plot.hclust rejects it. Python draws with scipy, which already handles two leaves, so there is nothing to mirror.

Tests

  • New invariants mirroring R's test-sd-cluster.R: the leftover-variables cases (two columns at min_cluster 2 and 3, one column with and without dendrogram) and R main's leaf order and labels on a deterministic 12-column matrix. R's first test builds its data with rnorm under set.seed(123), so I took R main's values on a deterministic matrix instead.
  • The invariant, property and parity tests that expected no clusters at min_cluster >= ncol now expect the single final cluster. The parity test of the too-few-variables warning used min_cluster=4 on four columns, which now clusters, so it uses one column.
  • 3 cache entries predate the R change: the two NNS.SD.cluster entries at min_cluster 4 and 5, and the NNS.SD.cluster.dendrogram entry whose order moved ([1, 2, 3, 4] now). I refilled them from an R build of the cache's base plus only Regenerate R parity cache for NNS 13.1 #126's R/SD_Cluster.R hunk, which also produced the new one-column entry. The min_cluster=4 dendrogram entry is no longer used and is gone; the other 8 NNS.SD.cluster entries came back identical. docs/parity.md records it.
  • Gates on the top of this stack (rasheed/ss-constant-group, which includes this slice), cache only: native 2935 passed, 4 skipped, 3 xfailed, 2 xpassed. With NNS_DISABLE_NATIVE=1: 2913 passed, 29 skipped, 2 xpassed. The 3 xfails are mode: break tied discrete modes by gravity, not their mean (mirrors R #111) #178's strict ones and the 2 xpasses the existing non-strict live-R VAR xfails. tests/parity/test_practical_examples.py against the usual R library: 9 passed, 2 xfailed, as on mode: break tied discrete modes by gravity, not their mean (mirrors R #111) #178. ruff check is clean on every changed file.

Review

GPT-6.1 Sol reviewed the #135, #127 and #126 mirrors together: SHIP.

Mirrors OVVO-Financial/NNS#126

… is peeled (R sd-cluster-fixes)

Mirrors R NNS.SD.cluster (OVVO-Financial/NNS sd-cluster-fixes).

The dendrogram's "order" was overwritten with each label's original column
position, which is not a leaf order of the tree, so a plot or cut drawn
from it misplaced the leaves. It is now hclust's own order: the last merge
expanded left before right, as R's hcass2 does, read from the R-convention
merge matrix the function already returns.

When min_cluster was at or above the number of columns nothing was peeled,
and the final-cluster step also required more than min_cluster columns, so
the result had no clusters. Any variables left now form the final cluster.

Before -> after (= R main):
  nns_sd_cluster(4 columns, min_cluster=4)   {} -> {Cluster_1: [X_1, ..., X_4]}
  nns_sd_cluster(cbind(A = 1:5, B = 2:6), min_cluster=2)
                                             {} -> {Cluster_1: [A, B]}
  dendrogram order of the 12-column test matrix (min_cluster=2)
                        original positions -> [1, 2, 3, 4, 9, 10, 11, 12, 5, 6, 7, 8]

3 cache entries predate the R change and were refilled from the cache's R
build plus that change alone: the two NNS.SD.cluster entries with
min_cluster 4 and 5 and the dendrogram entry whose order moved. The parity
test for the too-few-variables warning used min_cluster=4, which no longer
warns; it now uses one column, a new entry from the same build. The other
8 NNS.SD.cluster entries came back identical.
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head 6eacb45c21795f82ff391c30663d73763063ad44 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/py-sd-cluster-fixes branch November 7, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant