Repository navigation
Update broad_sanger data sources, PSet handling, and download resilience - #482
Merged
Merged
Conversation
Also makes the broad_sanger downloads survive a flaky connection. These are the largest transfers in the pipeline: Sanger ships rnaseq_all at ~897MB, WES_pureCN_CNV_genes at ~935MB, mutations_all at ~107MB and the proteomics archive at ~113MB. Neither downloader could resume. robust_download() in the proteomics script mounted a urllib3 Retry adapter, but Retry only covers connection setup and HTTP status codes: once the body is streaming a truncated read raises out of iter_content() and became a fatal RuntimeError. A build died exactly that way, 12MB into the 113MB file, three times, restarting from zero each time. robust_download_httr2() in the omics script had the same shape -- httr2's req_retry() restarts the request rather than resuming it -- so the 897MB and 935MB files would fail the same way, only slower. Both now retry on the 1/3/10/15 minute schedule used elsewhere and resume with a Range request, accumulating in "<dest>.part" and promoting the file only once its size matches the server's Content-Length. All of these hosts send accept-ranges: bytes. The omics script delegates to download_file_resumable() from retry_utils.R rather than keeping a second implementation. Dropped req_verbose(): full curl logging across ~2GB of transfers buried the real errors in the build log. The transcriptomics branch also now pairs rm() with gc(), as the copy_number and mutations branches already did. Without it R still holds the previous object when pivot_longer allocates the long frame, which is the peak of the whole step: measured 1,351MB -> 1,219MB peak RSS at no run time cost. The nci60 unzips check their exit status instead of failing later as a confusing parse error on a stale DOSERESP.csv. Verified against the live Sanger host: an interrupted transfer resumes with 206 Partial Content and completes byte-exact, and resuming from a 100MB partial took 4.8s against 61.3s for the full download, confirming it fetches only the remaining bytes.
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipeline Hardening/Debugging PR # 4
Update broad_sanger data sources, PSet handling, and download resilience
Reworks the broad_sanger (cell-line) pipeline so it keeps working after several upstream changes: DepMap moving behind a Cloudflare challenge, PharmacoGx publishing an incompatible dataset, DepMap's 26Q1 format changes, and large downloads that previously restarted from zero whenever they were interrupted. Builds on the shared download and drug-pipeline utilities from the previous two PRs.
DepMap now read from a Synapse copy
syn75028495), fetched byfetch_depmap.pyat the start of the samples and omics steps (build_samples.sh,build_omics.sh).fetch_depmap.pyneeds to run first, rather than failing further along. The Synapse copy is currently pinned to DepMap Public 26Q1, and the scripts document that it must be refreshed by hand for each new DepMap release.PharmacoGx PSets: pinned and validated
pinned_psets.R) instead of taken fromavailablePSets(), so a newly published upstream PSet can't silently change the build (03-createDrugFile.R,04a-drugResponseData.R).CTRPv2.1_2016change: it was verified to be the same 907 experiments as the existingCTRPv2_2015, so it is deliberately excluded.DepMap 26Q1 format changes handled (
02-broadSangerOmics.R)depmap_consume) and by usingdata.tablefor the large copy-number read and reshape.NCI-60 downloads fixed (
03a-nci60Drugs.py,04b-nci60-updated.py)DOSERESP.zip, ~346 MB) now downloads through the resumable helper with a browser User-Agent. Previously the default Python User-Agent was blocked (HTTP 403) and a single read truncated the large file. The download URL was updated to the current version 19 release.Dose-response fitting and resume (
04-drug_dosage_and_curves.py,04a-drugResponseData.R)*_doseResponse.tsvand fails loudly if none are found.ctrpv2DoseResponsevsCTRPv2doseResponse), so every attempt re-did all the work instead of resuming. Added a--workersoption and progress logging.Samples, proteomics, and dependencies
01-broadSangerSamples.R: keep theModelTypecolumn through the join so organoid models are classified correctly; downloadcellosaurus.xmlwith retries and fail if the file is truncated.02a-broad_sanger_proteomics.py: replace a fragile single-read download with a retrying, resumable transfer that keeps a.partfile between attempts, so the large proteomics file no longer fails on a truncated body.omics_requirements.r/exp_requirements.r: add a post-install check that fails the build if any required R package is missing; minor requirements additions.Scope: 15 files. Base: drug-pipeline-hardening.
Related issues
Closes #471