Skip to content

feat: mutually exclusive paths through the GPU pipeline with pipeline.split - #145

Open
Hendrik-code wants to merge 1 commit into
mainfrom
hm/split-path
Open

Hendrik-code wants to merge 1 commit into
mainfrom
hm/split-path

Conversation

@Hendrik-code

@Hendrik-code Hendrik-code commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What

A new pipeline key, split, that makes groups of GPU transforms mutually exclusive:

"pipeline": {"split": {"paths": [["RandomSynthSegGPU"],
                                 ["RandomInverseGPU", "RandomHistogramEqualizationGPU",
                                  "RandomRedistributeSegGPU", "RandomScharrGPU",
                                  "RandomUnsharpMaskGPU", "RandomRandConvGPU"]],
                       "p": [0.5, 0.5]}}

Each batch takes exactly one path, path i with probability p[i]:

  • the transforms of the paths not taken are skipped for that batch (no parameter draw, image and mask untouched);
  • the transforms of the path taken run as usual, each still subject to its own p;
  • every transform the split does not name runs on every batch.

Pipeline order is unchanged: a split decides which transforms run, never when, so a path does not have to be contiguous. An empty path is allowed and means "none of the others".

Why

The question was "SynthSeg on half the batches, our transfer augmentations on the other half, the general enhancements on all of them". The config surface could not say "not both": adding RandomSynthSegGPU at p: 0.5 to the paper config stacks it, because SynthSeg sits ahead of every TA in PIPELINE_ORDER, so on the batches it fires the TAs then run on the synthetic image.

With the split above and SynthSeg at p: 1.0, each path reproduces one of the two shipped paper configs exactly: transform_params_one-sequence-to-segment-them-all.json and transform_params_paper-synthseg.json differ only in the SynthSeg block and in the p of those six TAs.

How

  • SmaugConfig.pipeline_split() returns a SplitSpec(paths, p). Validation (all problems reported together): at least two paths, one p per path, each in [0, 1], summing to 1; every name in the GPU section (with a did-you-mean) and in one path at most; sequential mode only, since the bucketing modes hide the members inside a RandomChooseXTransformsGPU; and not with same_on_batch: false, because the path is drawn once per batch.
  • AugmentationSequentialCustom(split=...) resolves each path to its child stages by class name (a config key is the class name) and refuses a name that is not exactly one stage. It overrides get_forward_sequence: on the parameter-drawing call it draws the path and filters out the other paths' stages. forward and the mask pass iterate the parameters that call produced, so a skipped stage is skipped for image and mask alike, and replays with explicit params are untouched. The same hook feat: shuffle a group's order in place with pipeline.shuffle #143 used.
  • The draw uses rng.shared_rand, so it follows torch.manual_seed and DDP ranks agree on the path.
  • The nnU-Net trainer and AugTransformsGPU both pass it through. The trainer prints the split under its config line.

Without split nothing changes: the override returns kornia's sequence as is.

Tests

unit_tests/test_pipeline_split.py (21 tests): parsing and every rejection; no split keeps every stage; over 4000 seeded draws the two paths never co-occur, the shared stages always run and the path frequency is within 5 sd of p; an empty path; the draw follows the torch seed; through a real forward with record_applications, exactly one path runs per batch and a skipped stage leaves the mask alone; a name that is not a stage raises; AugTransformsGPU and the trainer (constructed with nnU-Net's base __init__ patched out) carry the split. Removing the trainer's split= argument fails the trainer test.

Full CPU suite: 494 passed, 13 skipped. Ruff clean (local 0.17.1; CI pins 0.16.1). mypy 2.1.0 clean on smauglab/ (CI pins 2.4.0).

An nnU-Net training with the config above (nnUNetTrainerDAExtGPU, fold 0, 1000 epochs) started on Dataset803 on 2026-11-13, with Dataset804 queued behind it. The trainer logs the split at startup and runs at ~63 s/epoch on an A40, against ~64 s for the stacked SynthSeg p=0.5 arm. A pre-flight on 4000 parameter draws of that config took the paths 1973/2027 times, never both on one draw, with each TA at its paper p on its own path.

🤖 Generated with Claude Code

…e.split`

`"pipeline": {"split": {"paths": [[...], [...]], "p": [...]}}` draws one path
per batch. The transforms named in the paths not taken are left out of that
batch's forward sequence, so they neither draw parameters nor touch image or
mask; the path taken runs as usual, each transform still subject to its own
`p`; everything the split does not name runs on every batch. Pipeline order
is unchanged, so a path need not be contiguous.

The motivating case is SynthSeg on half the batches and the transfer
augmentations on the other half, the general enhancements on all of them.
Adding SynthSeg at p=0.5 to the paper config does not say that: it sits
ahead of every TA, so on the batches it fires the TAs then run on the
synthetic image. With paths [[SynthSeg], [the six TAs the SynthSeg paper
config zeroes]] and SynthSeg at p=1, each path reproduces one of the two
shipped paper configs exactly.

Validated at load time: at least two paths, one `p` each, summing to 1,
every name in the GPU section and in one path at most, sequential mode only
(the bucketing modes hide the members inside a RandomChooseX), and not with
`same_on_batch: false` (the draw is per batch). The container also refuses
a name that is not one of its stages. The draw uses `shared_rand`, so it
follows `torch.manual_seed` and DDP ranks agree on the path.

The trainer builds its own container rather than an `AugTransformsGPU`, so
it passes the split itself; a test constructs the trainer to pin that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Hendrik-code

Copy link
Copy Markdown
Collaborator Author

Results: Dataset803 and Dataset804, fold 0

The split arm is equivalent to the stacked SynthSeg arm within ±0.01 Dice on both datasets, in-domain and out-of-domain, and equivalent to the paper config as well. On these data, making SynthSeg and the transfer augmentations mutually exclusive changes nothing measurable.

Setup. nnUNetTrainerDAExtGPU, 3d_fullres, fold 0, 1000 epochs, A40, one run per arm. Dataset803 is trained on Dixon in-phase MRI and Dataset804 on CT; out-of-domain is every other modality and sequence in the test split (803: 1077 scans out, 24 in; 804: 462 out, 639 in). Metric is global binary Dice, averaged over vertebra, IVD and spinal canal per scan. Verdicts are TOST equivalence at ±0.01 (α = 0.05) on paired per-scan deltas.

arm config code
split this PR's example: paper config + RandomSynthSegGPU (p=1) in a 0.5 path 8be3e80 (this branch)
stacked paper config + the same SynthSeg block at p=0.5, no split 7169464
paper transform_params_one-sequence-to-segment-them-all.json 7169464
paper-synthseg transform_params_paper-synthseg.json 41ac414b (older)

Split and stacked differ in the split and in SynthSeg's p only. The code also differs by #141, which this trainer cannot reach: it maps nnU-Net's -1 label to 0 before any GPU transform.

Split minus stacked, paired mean and 95% CI:

in-domain out-of-domain
Dataset803 +0.001 [+0.000, +0.003] +0.004 [+0.004, +0.005]
Dataset804 +0.003 [+0.003, +0.003] −0.005 [−0.006, −0.004]

All four are equivalent (TOST p < 0.001).

Against the paper config, paired mean:

arm 803 in 803 out 804 in 804 out
stacked −0.005 −0.008 −0.004 +0.012
split −0.004 −0.004 −0.001 +0.007

Only stacked on 804 out-of-domain crosses the margin. Three fold-0 runs of the paper config span 0.0043 (803) and 0.0079 (804) out-of-domain in cohort means, so neither 804 gain clearly exceeds run-to-run spread. Where it comes from: the MRI test cohorts of 804, mostly the vertebra class, about +0.02 over the paper config in both SynthSeg arms.

Cohort means, unweighted across cohorts:

arm 803 in 803 out 804 in 804 out
paper 0.901 0.844 0.903 0.781
stacked 0.896 0.842 0.900 0.790
split 0.897 0.844 0.902 0.789
paper-synthseg 0.869 0.830 0.858 0.773

The split sits with the paper config, not with paper-synthseg; paired against paper-synthseg it is +0.011 (803) and +0.031 (804) out-of-domain. That reference is on an older checkout, so it carries a code-era term as well.

Caveat. In the split, the six path-B transforms fire on half the batches they fire on in the paper config. Equivalence to the paper config therefore means SynthSeg on half the batches makes up for halving those six, not that SynthSeg adds nothing.

The feature in training. Both runs logged the split at startup, ran at ~63.5 s/epoch with no non-finite losses, and a pre-flight on 4000 parameter draws of this config took the paths 1973/2027 times, never both on one draw.

Bottom line: a null at this margin. It does not argue for changing the default config, and the split is what made the comparison possible.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant