Repository navigation
Expand benchmark regression suite with real HPC cases - #591
Merged
Merged
Conversation
Verbatim production configs (provenance in headers): 4 OPS flavors (pheno phase 3D, tracking 2D/3D at 5x, GFP fluorescence), kazansky birefringence, a549-hummingbird phase 2D/3D, cardiomyocytes-falcon high-NA phase 3D, zebrafish-daxi 760 nm phase 3D, EV71 infected neurons phase + FITC decon, mantis v1 neuromast phase. Deviation: OPS GFP config listifies input_channel_names (production bare string fails current schema validation).
Full-FOV fluorescence 3D on the 7x2048x2048 GFP input peaks at ~302 GB RSS (OOM-killed at 64G and 128G job limits). Crop keeps the case runnable; the memory footprint itself is a 4.0 refactor target.
Bounds are ~50% of the baseline midband power per case: loose enough for cross-node variation, tight enough to catch collapsed or empty reconstructions (e.g. the AMD torch.fft.ifftn normalization bug).
Three cases from the Sri-curated neuromast-mantis straight/oblique comparison: epi phase 3D, oblique light-sheet phase 3D, and the same oblique settings run through wo tile-stitch (the suite's first real-data tiled reconstruction; its stats match the untiled case). Harness: cases take cli: rec|tile-stitch, config introspection understands the nested recon block, and reading the output store handles both HCS plates (wo rec) and single positions (wo tile-stitch).
The EV71 FITC channel is near background (~70 counts of signal) in every position and timepoint, so it exercised the fluorescence path without testing quality. The a549-hummingbird FITC channel has real signal and a production config (kept verbatim, including its wavelength_emission 0.630 oddity, noted in the header). Neuromast cases move to position 0/1/0, whose rosette is centered rather than at the FOV edge. Bounds re-pinned in the next commit from a fresh baseline.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #591 +/- ##
=======================================
Coverage 56.75% 56.75%
=======================================
Files 77 77
Lines 13074 13074
=======================================
Hits 7420 7420
Misses 5654 5654 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
srivarra
reviewed
Oct 30, 2026
srivarra
left a comment
srivarra
left a comment
Contributor
There was a problem hiding this comment.
Just setup a couple of test runs, I will let you know when it's done and approve. Looks great otherwise.
Regenerates benchmarks/benchmark_datasets.png from any --save-all run; verified byte-identical to the committed figure.
srivarra
approved these changes
Oct 31, 2026
srivarra
left a comment
srivarra
left a comment
Contributor
There was a problem hiding this comment.
benchmark ran swimmingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expands the benchmark suite from 5 to 20 cases ahead of the 4.0 config refactor, so reconstructions that work today are gated against regressions. Each new case reconstructs existing HPC data with a verbatim copy of its production config (provenance noted in each config header).
New coverage: OPS production flavors (phenotyping phase 3D, tracking phase 2D/3D at 5x, GFP fluorescence deconvolution), birefringence (kazansky target, the suite's first polarization case), a549-hummingbird phase 2D/3D, cardiomyocytes high-NA phase 3D, zebrafish DaXi 760 nm phase 3D, EV71 infected-neuron phase 3D, a549 FITC fluorescence deconvolution, the neuromast straight/oblique comparison (epi and light-sheet phase 3D), and mantis v1 neuromast phase 3D.
Benchmark cases can now also run
wo tile-stitch(cli: tile-stitchon a case), and the neuromast oblique case has a tiled variant: the suite's first real-data tiled reconstruction, whose statistics match the untiled case (the two neuromast ls columns in the figure agreeing is the point: same data and settings through the tiled code path).All 20 cases pass on the 2026-10-29 baseline (full suite: ~11 min wall, ~33 GB peak RAM, ~60 MB stored per run). Reference bounds are pinned at roughly half the baseline midband power per case, loose enough for hardware variation but tight enough to catch collapsed reconstructions.
The figure above shows each case's raw input (top) and reconstruction (bottom), using the highest-contrast z-slice for 3D volumes.
Not covered yet: OPS 63x and mCherry (no dataset on hand), mantis v2 (deskewed intermediates were deleted; needs a cached crop), Sri's 2026 neuromast RL reconstructions (pending data permissions), and golden-reference volume comparison.