Skip to content

(re)Merge the 2.4 build branch changes into main - #519

Merged
sgosline merged 38 commits into
mainfrom
build-2.4-cn
Oct 27, 2026
Merged

sgosline merged 38 commits into
mainfrom
build-2.4-cn

Conversation

@jjacobson95

@jjacobson95 jjacobson95 commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Brings the 2.4 build code into main (again). Almost all of it was already reviewed and merged in #480–#493, but never reached main.

Why it isn't on main: the 2.4 changes went up as a stack of PRs, each targeting the branch of the PR below it. The bottom PR (#479) merged into main first. The rest (#480–#493) were then merged into the branch below them, not into main. GitHub lists them all as merged, but their code only exists on those branches and on build-2.4-cn. build-2.4-cn holds the whole stack, and main has been merged into it. The latest version of Figshare 2.4 was built from this branch plus the three fixes listed at the end.

Reviewed in the stack (merged, but not into main):

PR Change
#480 Retrying downloads, retry utilities, DepMap fetch
#481 Drug descriptors and curve fitting
#482 broad_sanger data sources and PSet handling
#483 Genes build that survives Ensembl outages
#484 Phosphosite reference builder
#485, #486 cNF organoid dataset (not in the default build)
#487, #488 MPNST treated samples, omics and drug combinations
#489 Docker image build fixes, resume fixes
#490 --continue resume, --depmap-ready check, default dataset list
#491 Schema for phosphoproteomics and new dataset outputs
#492 Faster schema check, ID mapping, IMPROVE export
#493 Package dataset metadata and loader

Not in a PR before (1 small commit made directly on build-2.4-cn):

Everything already merged into main (#499, #500, #505, #507, #508, #509, #511, #516, #517) is unchanged here. This merges without conflicts.

The validation suite, the NCI60/PRISM drug ID duplicate fix and the Novartis sample ID fix will follow as separate PRs to main.

jjacobson95 and others added 30 commits September 21, 2026 11:49
Also makes the broad_sanger downloads survive a flaky connection. These are
the largest transfers in the pipeline: Sanger ships rnaseq_all at ~897MB,
WES_pureCN_CNV_genes at ~935MB, mutations_all at ~107MB and the proteomics
archive at ~113MB.

Neither downloader could resume. robust_download() in the proteomics script
mounted a urllib3 Retry adapter, but Retry only covers connection setup and
HTTP status codes: once the body is streaming a truncated read raises out of
iter_content() and became a fatal RuntimeError. A build died exactly that way,
12MB into the 113MB file, three times, restarting from zero each time.
robust_download_httr2() in the omics script had the same shape -- httr2's
req_retry() restarts the request rather than resuming it -- so the 897MB and
935MB files would fail the same way, only slower.

Both now retry on the 1/3/10/15 minute schedule used elsewhere and resume with
a Range request, accumulating in "<dest>.part" and promoting the file only
once its size matches the server's Content-Length. All of these hosts send
accept-ranges: bytes. The omics script delegates to download_file_resumable()
from retry_utils.R rather than keeping a second implementation.

Dropped req_verbose(): full curl logging across ~2GB of transfers buried the
real errors in the build log.

The transcriptomics branch also now pairs rm() with gc(), as the copy_number
and mutations branches already did. Without it R still holds the previous
object when pivot_longer allocates the long frame, which is the peak of the
whole step: measured 1,351MB -> 1,219MB peak RSS at no run time cost.

The nci60 unzips check their exit status instead of failing later as a
confusing parse error on a stale DOSERESP.csv.

Verified against the live Sanger host: an interrupted transfer resumes with
206 Partial Content and completes byte-exact, and resuming from a 100MB
partial took 4.8s against 61.3s for the full download, confirming it fetches
only the remaining bytes.
Also fixes three orchestration bugs found in the v19 full build, and
redacts credentials from the build log.

Every process_* function awaited only the PREVIOUS future before submitting
the next, so the LAST dataset's exception was never retrieved -- and an
unretrieved exception in a ThreadPoolExecutor is discarded, not raised. In
--high_mem mode nothing was awaited at all. In v19 that let `hcmi omics`
fail all three attempts while the build logged "All omics files completed",
then published and validated a release with hcmi transcriptomics,
copy_number and mutations missing entirely. Every future is now collected
and retrieved through _await_all(), so no dataset failure can be dropped.

The resulting `validate failed` printed only "None": stderr is passed
through to sys.stderr, so subprocess never captures it and res.stderr is
always None. Report the exit code and point at the container output.

Finally, docker command lines were logged verbatim, and credentials are
passed as `-e NAME=value`, so build logs contained the live
SYNAPSE_AUTH_TOKEN in cleartext. Added _redact() and gitignored the build
logs and PID files.
…mics

Add MPNST treated-sample and treated-omics support
Harden build orchestration and set the default dataset list
…d-2.4-cn

# Conflicts:
#	coderbuild/build_all.py
…ld-2.4-cn

# Conflicts:
#	schema/coderdata.yaml
@jjacobson95 jjacobson95 added this to the 2.4 new build milestone Oct 27, 2026
@jjacobson95 jjacobson95 self-assigned this Oct 27, 2026
@jjacobson95 jjacobson95 changed the title Merge the 2.4 build branch into main (re)Merge the 2.4 build branch changes into main Oct 27, 2026
@jjacobson95
jjacobson95 marked this pull request as ready for review October 27, 2026 20:21
@jjacobson95
jjacobson95 requested a review from sgosline October 27, 2026 20:21
@sgosline
sgosline merged commit 136e6ef into main Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants