Skip to content

configs: ship PALETTE alone without the Voronoi split, the label remap or unblurred speckle - #150

Open
Hendrik-code wants to merge 1 commit into
mainfrom
hm/palette-clean-config
Open

Hendrik-code wants to merge 1 commit into
mainfrom
hm/palette-clean-config

Conversation

@Hendrik-code

Copy link
Copy Markdown
Collaborator

What

One config, transform_params_paper-palette-clean.json (hash ba46ce7b). It is the Palette alone arm plus three RandomPaletteGPU keys. Palette alone is transform_params_paper-synthseg.json with RandomSynthSegGPU replaced by RandomPaletteGPU at p: 1.0, hash 4b92045c, the same config #142 ships as transform_params_paper-palette.json.

key value removes
skip_sub_parc_prob 1.0 the Voronoi sub-parcellation, which cuts straight polygon edges through anatomy: edges no imaging modality produces
label_remap_prob 0.0 the per-label affine remap, which steps the intensity exactly on the ground-truth boundary
blur_sigmas [0.5, 0.8] unblurred speckle: the k-means bucketing is hard, so voxels next to a cluster threshold flip between two remaps

Every other key is Palette alone's, so the two differ only in these three. No code change: all three are existing RandomPaletteGPU parameters. Based on main rather than #142, so it does not wait on that PR; the .gitignore entry is its own.

Why

Seen in renders of the PALETTE arm on four SpineGAN datasets:

  • polygonal regions crossing vertebrae;
  • salt-and-pepper, worst on CT;
  • label-shaped steps.

In 9 of 40 measured feature-only samples, intensity no longer separated the labels at all. The config asks whether PALETTE's measured gain on Dataset804 survives without those artifacts.

Tests

unit_tests/test_palette_clean_config.py:

  • the config sets the three knobs and has no SynthSeg;
  • it builds and runs;
  • with Voronoi and label remap off (blur off for the check), voxels of equal input intensity map to equal outputs across both labels, over five seeds;
  • the defaults break that, so the check can fail.

Full suite: 477 passed, 13 skipped. Pre-commit passes.

🤖 Generated with Claude Code

…p or unblurred speckle

transform_params_paper-palette-clean.json is the `Palette alone` arm
(transform_params_paper-synthseg.json with RandomSynthSegGPU replaced by
RandomPaletteGPU at p=1, hash 4b92045c) plus three PALETTE keys:

* skip_sub_parc_prob 1.0: no Voronoi sub-parcellation, which cuts straight
  polygon edges through anatomy.
* label_remap_prob 0.0: no per-label affine remap, which steps the
  intensity exactly on the ground-truth boundary.
* blur_sigmas [0.5, 0.8]: a blur floor. The k-means bucketing is hard, so
  voxels next to a cluster threshold flip between two remaps and come out
  as salt-and-pepper.

With the first two off, the synthesis before the blur is a function of input
intensity alone; the test checks that, and that the defaults break it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Hendrik-code

Copy link
Copy Markdown
Collaborator Author

Results: Dataset803 and Dataset804, fold 0

Worse than PALETTE alone out-of-domain on both datasets. Removing the Voronoi split, the label remap and unblurred speckle costs −0.016 per scan out-of-domain on both 803 and 804. On Dataset804 the Dixon water cohort collapses, from 0.778 to 0.539. PALETTE's value for domain generalization sits in the parts this config removes.

Setup. nnUNetTrainerDAExtGPU, 3d_fullres, fold 0, 1000 epochs, one run per arm. Dataset803 is trained on SpineGAN Dixon in-phase MRI and Dataset804 on SpineGAN CT; out-of-domain is every other modality and sequence in the test split (803: 1077 scans out, 24 in; 804: 462 out, 639 in). Metric is global binary Dice, averaged over vertebra, IVD and spinal canal per scan. Verdicts are TOST equivalence at ±0.01 (α = 0.05) on paired per-scan deltas. For scale, three fold-0 runs of the paper config span 0.0043 (803) and 0.0079 (804) out-of-domain in cohort means.

arm config code GPU
palette-clean transform_params_paper-palette-clean.json (this PR) a89e42b (this branch) A40 (803), L40S (804)
palette-alone the SynthSeg config with PALETTE (p = 1) in SynthSeg's slot 7169464 A40
paper (replicate) transform_params_one-sequence-to-segment-them-all.json 7169464 A40
paper (published run) same b0110743 A40

palette-clean is palette-alone plus skip_sub_parc_prob: 1.0, label_remap_prob: 0.0 and blur_sigmas: [0.5, 0.8], and nothing else.

Palette-clean minus palette-alone, paired mean and 95% CI:

in-domain out-of-domain
Dataset803 −0.003 [−0.005, −0.002] −0.016 [−0.017, −0.015]
Dataset804 −0.000 [−0.001, −0.000] −0.016 [−0.021, −0.010]

In-domain is equivalent on both datasets. Out-of-domain is worse on both (TOST p = 1.000 and 0.967).

Against the paper config, paired mean:

803 in 803 out 804 in 804 out
palette-clean − paper (replicate) −0.011 −0.016 +0.004 −0.007
palette-clean − paper (published) −0.011 −0.015 +0.004 −0.003

Cohort means, unweighted across cohorts:

arm 803 in 803 out 804 in 804 out
paper (replicate) 0.901 0.844 0.903 0.781
palette-alone 0.894 0.844 0.904 0.791
palette-clean 0.890 0.835 0.904 0.736

Where it moves, per cohort against palette-alone:

  • Dataset803 (A40): myelom-ct −0.026 [−0.027, −0.024] and spinegan-ct −0.011. The Dixon cohorts are −0.003 to −0.008 and SPIDER −0.002 / −0.001.
  • Dataset804 (L40S): Dixon water −0.239 [−0.306, −0.172]. 22 of 24 scans drop by more than 0.05 and 14 by more than 0.2; spinal canal falls from 0.865 to 0.480. Dixon in-phase is −0.024 and fat −0.010. The CT and SPIDER cohorts are within ±0.003.

Caveat: GPU model. This arm's Dataset804 run trained on an L40S; every reference trained on an A40. On Dataset804 Dixon water, the three arms of this round that trained on the L40S (#147, #149, #150) scored 0.724, 0.698 and 0.539. That is below all 23 other full-capacity arms on 804 (0.756 to 0.778) and all four runs of the paper config (0.767 to 0.771). An earlier paper-family run on the same L40S scored 0.759, with no scan more than 0.05 below the paper config, so the card alone does not produce the drop. With one run per cell, an interaction between card and arm is not excluded; an A40 rerun on 804 would settle it.
For this PR the direction does not rest on the card: the Dataset803 run trained on an A40 and is worse as well.

In training. About 54 s/epoch on the A40 and 38 s/epoch on the L40S, no non-finite losses in either run. PALETTE fired in 20 of 20 pre-flight batches.

Bottom line: do not merge this config. Which of the three keys carries the loss is not identified. Any cleanup of PALETTE's look should be tested one key at a time.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant