Repository navigation
(re)Merge the 2.4 build branch changes into main - #519
Merged
Merged
Conversation
Also makes the broad_sanger downloads survive a flaky connection. These are the largest transfers in the pipeline: Sanger ships rnaseq_all at ~897MB, WES_pureCN_CNV_genes at ~935MB, mutations_all at ~107MB and the proteomics archive at ~113MB. Neither downloader could resume. robust_download() in the proteomics script mounted a urllib3 Retry adapter, but Retry only covers connection setup and HTTP status codes: once the body is streaming a truncated read raises out of iter_content() and became a fatal RuntimeError. A build died exactly that way, 12MB into the 113MB file, three times, restarting from zero each time. robust_download_httr2() in the omics script had the same shape -- httr2's req_retry() restarts the request rather than resuming it -- so the 897MB and 935MB files would fail the same way, only slower. Both now retry on the 1/3/10/15 minute schedule used elsewhere and resume with a Range request, accumulating in "<dest>.part" and promoting the file only once its size matches the server's Content-Length. All of these hosts send accept-ranges: bytes. The omics script delegates to download_file_resumable() from retry_utils.R rather than keeping a second implementation. Dropped req_verbose(): full curl logging across ~2GB of transfers buried the real errors in the build log. The transcriptomics branch also now pairs rm() with gc(), as the copy_number and mutations branches already did. Without it R still holds the previous object when pivot_longer allocates the long frame, which is the peak of the whole step: measured 1,351MB -> 1,219MB peak RSS at no run time cost. The nci60 unzips check their exit status instead of failing later as a confusing parse error on a stale DOSERESP.csv. Verified against the live Sanger host: an interrupted transfer resumes with 206 Partial Content and completes byte-exact, and resuming from a 100MB partial took 4.8s against 61.3s for the full download, confirming it fetches only the remaining bytes.
Also fixes three orchestration bugs found in the v19 full build, and redacts credentials from the build log. Every process_* function awaited only the PREVIOUS future before submitting the next, so the LAST dataset's exception was never retrieved -- and an unretrieved exception in a ThreadPoolExecutor is discarded, not raised. In --high_mem mode nothing was awaited at all. In v19 that let `hcmi omics` fail all three attempts while the build logged "All omics files completed", then published and validated a release with hcmi transcriptomics, copy_number and mutations missing entirely. Every future is now collected and retrieved through _await_all(), so no dataset failure can be dropped. The resulting `validate failed` printed only "None": stderr is passed through to sys.stderr, so subprocess never captures it and res.stderr is always None. Report the exit code and point at the container output. Finally, docker command lines were logged verbatim, and credentials are passed as `-e NAME=value`, so build logs contained the live SYNAPSE_AUTH_TOKEN in cleartext. Added _redact() and gitignored the build logs and PID files.
…mics Add MPNST treated-sample and treated-omics support
Harden build orchestration and set the default dataset list
…s' into build-2.4-cn
…d-2.4-cn # Conflicts: # coderbuild/build_all.py
…2.4-cn # Conflicts: # coderdata/dataset.yml
…ld-2.4-cn # Conflicts: # schema/coderdata.yaml
…call bins in prepare_data_for_improve
jjacobson95
marked this pull request as ready for review
October 27, 2026 20:21
This was referenced Oct 27, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the 2.4 build code into
main(again). Almost all of it was already reviewed and merged in #480–#493, but never reachedmain.Why it isn't on
main: the 2.4 changes went up as a stack of PRs, each targeting the branch of the PR below it. The bottom PR (#479) merged intomainfirst. The rest (#480–#493) were then merged into the branch below them, not intomain. GitHub lists them all as merged, but their code only exists on those branches and onbuild-2.4-cn.build-2.4-cnholds the whole stack, andmainhas been merged into it. The latest version of Figshare 2.4 was built from this branch plus the three fixes listed at the end.Reviewed in the stack (merged, but not into
main):--continueresume,--depmap-readycheck, default dataset listNot in a PR before (1 small commit made directly on
build-2.4-cn):24ca8ac: stop converting DepMap copy number in02-broadSangerOmics.R, since DepMap files are already log2(relative CN + 1). This puts DepMap on the same scale Copy number Harmonization - all datasets. #507 uses for every other dataset. Copy number Harmonization - all datasets. #507 couldn't include it because this code (from Update broad_sanger data sources, PSet handling, and download resilience #482) wasn't onmain. Its bin change inprepare_data_for_improve.py, and the follow-up commit7d5db2b, are already onmainthrough Keep high copy-number values in amplification bin #508.Everything already merged into
main(#499, #500, #505, #507, #508, #509, #511, #516, #517) is unchanged here. This merges without conflicts.The validation suite, the NCI60/PRISM drug ID duplicate fix and the Novartis sample ID fix will follow as separate PRs to
main.