Repository navigation
feat: mutually exclusive paths through the GPU pipeline with pipeline.split - #145
Hendrik-code wants to merge 1 commit into
Conversation
…e.split`
`"pipeline": {"split": {"paths": [[...], [...]], "p": [...]}}` draws one path
per batch. The transforms named in the paths not taken are left out of that
batch's forward sequence, so they neither draw parameters nor touch image or
mask; the path taken runs as usual, each transform still subject to its own
`p`; everything the split does not name runs on every batch. Pipeline order
is unchanged, so a path need not be contiguous.
The motivating case is SynthSeg on half the batches and the transfer
augmentations on the other half, the general enhancements on all of them.
Adding SynthSeg at p=0.5 to the paper config does not say that: it sits
ahead of every TA, so on the batches it fires the TAs then run on the
synthetic image. With paths [[SynthSeg], [the six TAs the SynthSeg paper
config zeroes]] and SynthSeg at p=1, each path reproduces one of the two
shipped paper configs exactly.
Validated at load time: at least two paths, one `p` each, summing to 1,
every name in the GPU section and in one path at most, sequential mode only
(the bucketing modes hide the members inside a RandomChooseX), and not with
`same_on_batch: false` (the draw is per batch). The container also refuses
a name that is not one of its stages. The draw uses `shared_rand`, so it
follows `torch.manual_seed` and DDP ranks agree on the path.
The trainer builds its own container rather than an `AugTransformsGPU`, so
it passes the split itself; a test constructs the trainer to pin that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Results: Dataset803 and Dataset804, fold 0The split arm is equivalent to the stacked SynthSeg arm within ±0.01 Dice on both datasets, in-domain and out-of-domain, and equivalent to the paper config as well. On these data, making SynthSeg and the transfer augmentations mutually exclusive changes nothing measurable. Setup.
Split and stacked differ in the split and in SynthSeg's Split minus stacked, paired mean and 95% CI:
All four are equivalent (TOST p < 0.001). Against the paper config, paired mean:
Only stacked on 804 out-of-domain crosses the margin. Three fold-0 runs of the paper config span 0.0043 (803) and 0.0079 (804) out-of-domain in cohort means, so neither 804 gain clearly exceeds run-to-run spread. Where it comes from: the MRI test cohorts of 804, mostly the vertebra class, about +0.02 over the paper config in both SynthSeg arms. Cohort means, unweighted across cohorts:
The split sits with the paper config, not with paper-synthseg; paired against paper-synthseg it is +0.011 (803) and +0.031 (804) out-of-domain. That reference is on an older checkout, so it carries a code-era term as well. Caveat. In the split, the six path-B transforms fire on half the batches they fire on in the paper config. Equivalence to the paper config therefore means SynthSeg on half the batches makes up for halving those six, not that SynthSeg adds nothing. The feature in training. Both runs logged the split at startup, ran at ~63.5 s/epoch with no non-finite losses, and a pre-flight on 4000 parameter draws of this config took the paths 1973/2027 times, never both on one draw. Bottom line: a null at this margin. It does not argue for changing the default config, and the split is what made the comparison possible. |
What
A new
pipelinekey,split, that makes groups of GPU transforms mutually exclusive:Each batch takes exactly one path, path
iwith probabilityp[i]:p;Pipeline order is unchanged: a split decides which transforms run, never when, so a path does not have to be contiguous. An empty path is allowed and means "none of the others".
Why
The question was "SynthSeg on half the batches, our transfer augmentations on the other half, the general enhancements on all of them". The config surface could not say "not both": adding
RandomSynthSegGPUatp: 0.5to the paper config stacks it, because SynthSeg sits ahead of every TA inPIPELINE_ORDER, so on the batches it fires the TAs then run on the synthetic image.With the split above and SynthSeg at
p: 1.0, each path reproduces one of the two shipped paper configs exactly:transform_params_one-sequence-to-segment-them-all.jsonandtransform_params_paper-synthseg.jsondiffer only in the SynthSeg block and in thepof those six TAs.How
SmaugConfig.pipeline_split()returns aSplitSpec(paths, p). Validation (all problems reported together): at least two paths, onepper path, each in [0, 1], summing to 1; every name in the GPU section (with a did-you-mean) and in one path at most; sequential mode only, since the bucketing modes hide the members inside aRandomChooseXTransformsGPU; and not withsame_on_batch: false, because the path is drawn once per batch.AugmentationSequentialCustom(split=...)resolves each path to its child stages by class name (a config key is the class name) and refuses a name that is not exactly one stage. It overridesget_forward_sequence: on the parameter-drawing call it draws the path and filters out the other paths' stages.forwardand the mask pass iterate the parameters that call produced, so a skipped stage is skipped for image and mask alike, and replays with explicitparamsare untouched. The same hook feat: shuffle a group's order in place with pipeline.shuffle #143 used.rng.shared_rand, so it followstorch.manual_seedand DDP ranks agree on the path.AugTransformsGPUboth pass it through. The trainer prints the split under its config line.Without
splitnothing changes: the override returns kornia's sequence as is.Tests
unit_tests/test_pipeline_split.py(21 tests): parsing and every rejection; no split keeps every stage; over 4000 seeded draws the two paths never co-occur, the shared stages always run and the path frequency is within 5 sd ofp; an empty path; the draw follows the torch seed; through a real forward withrecord_applications, exactly one path runs per batch and a skipped stage leaves the mask alone; a name that is not a stage raises;AugTransformsGPUand the trainer (constructed with nnU-Net's base__init__patched out) carry the split. Removing the trainer'ssplit=argument fails the trainer test.Full CPU suite: 494 passed, 13 skipped. Ruff clean (local 0.17.1; CI pins 0.16.1). mypy 2.1.0 clean on
smauglab/(CI pins 2.4.0).An nnU-Net training with the config above (
nnUNetTrainerDAExtGPU, fold 0, 1000 epochs) started on Dataset803 on 2026-11-13, with Dataset804 queued behind it. The trainer logs the split at startup and runs at ~63 s/epoch on an A40, against ~64 s for the stacked SynthSeg p=0.5 arm. A pre-flight on 4000 parameter draws of that config took the paths 1973/2027 times, never both on one draw, with each TA at its paperpon its own path.🤖 Generated with Claude Code