Skip to content

Update broad_sanger data sources, PSet handling, and download resilience - #482

Merged
sgosline merged 1 commit into
drug-pipeline-hardeningfrom
broad-sanger-data-sources
Sep 23, 2026
Merged

sgosline merged 1 commit into
drug-pipeline-hardeningfrom
broad-sanger-data-sources

Conversation

@jjacobson95

@jjacobson95 jjacobson95 commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Pipeline Hardening/Debugging PR # 4

Update broad_sanger data sources, PSet handling, and download resilience

Reworks the broad_sanger (cell-line) pipeline so it keeps working after several upstream changes: DepMap moving behind a Cloudflare challenge, PharmacoGx publishing an incompatible dataset, DepMap's 26Q1 format changes, and large downloads that previously restarted from zero whenever they were interrupted. Builds on the shared download and drug-pipeline utilities from the previous two PRs.

DepMap now read from a Synapse copy

  • The DepMap portal added a Cloudflare "verify you are a person" challenge, so it can no longer be downloaded by a script. The build now reads the required DepMap files from a static copy kept on Synapse (the "DepMap Raw" folder, syn75028495), fetched by fetch_depmap.py at the start of the samples and omics steps (build_samples.sh, build_omics.sh).
  • If those files are missing, the R scripts stop with a clear message explaining that fetch_depmap.py needs to run first, rather than failing further along. The Synapse copy is currently pinned to DepMap Public 26Q1, and the scripts document that it must be refreshed by hand for each new DepMap release.

PharmacoGx PSets: pinned and validated

  • The PSet list is now pinned (via pinned_psets.R) instead of taken from availablePSets(), so a newly published upstream PSet can't silently change the build (03-createDrugFile.R, 04a-drugResponseData.R).
  • A pinned PSet that comes back with an unexpected schema is now a hard failure with an explanatory error, rather than being silently skipped. This is what surfaced the August 2026 CTRPv2.1_2016 change: it was verified to be the same 907 experiments as the existing CTRPv2_2015, so it is deliberately excluded.

DepMap 26Q1 format changes handled (02-broadSangerOmics.R)

  • The "default model entry" flag changed from a true/false boolean to the strings "Yes"/"No"; the filter was updated to accept both, since the old check silently returned zero rows and produced empty mutation and transcriptomics files.
  • Variant annotations changed from MAF-style labels to VEP consequence terms; multi-term values are reduced to the single highest-impact consequence and the recognized-term list was extended.
  • Memory use during omics processing was reduced by reading and freeing each large DepMap input one at a time (depmap_consume) and by using data.table for the large copy-number read and reshape.

NCI-60 downloads fixed (03a-nci60Drugs.py, 04b-nci60-updated.py)

  • The NCI dose-response archive (DOSERESP.zip, ~346 MB) now downloads through the resumable helper with a browser User-Agent. Previously the default Python User-Agent was blocked (HTTP 403) and a single read truncated the large file. The download URL was updated to the current version 19 release.
  • The unzip step's exit status is now checked so a bad archive fails immediately instead of surfacing later as a confusing parse error on a stale CSV.
  • Fixed a re-run crash where the SMILES column was overwritten during de-duplication; separate variable names are now used so the column survives.

Dose-response fitting and resume (04-drug_dosage_and_curves.py, 04a-drugResponseData.R)

  • All dose-response studies are now fit, not just whichever one happened to be left in the working directory. The step scans the cache directory for every *_doseResponse.tsv and fails loudly if none are found.
  • Fixed a resume bug: the writer and reader used different letter-casing for the same filename (for example ctrpv2DoseResponse vs CTRPv2doseResponse), so every attempt re-did all the work instead of resuming. Added a --workers option and progress logging.

Samples, proteomics, and dependencies

  • 01-broadSangerSamples.R: keep the ModelType column through the join so organoid models are classified correctly; download cellosaurus.xml with retries and fail if the file is truncated.
  • 02a-broad_sanger_proteomics.py: replace a fragile single-read download with a retrying, resumable transfer that keeps a .part file between attempts, so the large proteomics file no longer fails on a truncated body.
  • omics_requirements.r / exp_requirements.r: add a post-install check that fails the build if any required R package is missing; minor requirements additions.

Scope: 15 files. Base: drug-pipeline-hardening.

Related issues

Closes #471

Also makes the broad_sanger downloads survive a flaky connection. These are
the largest transfers in the pipeline: Sanger ships rnaseq_all at ~897MB,
WES_pureCN_CNV_genes at ~935MB, mutations_all at ~107MB and the proteomics
archive at ~113MB.

Neither downloader could resume. robust_download() in the proteomics script
mounted a urllib3 Retry adapter, but Retry only covers connection setup and
HTTP status codes: once the body is streaming a truncated read raises out of
iter_content() and became a fatal RuntimeError. A build died exactly that way,
12MB into the 113MB file, three times, restarting from zero each time.
robust_download_httr2() in the omics script had the same shape -- httr2's
req_retry() restarts the request rather than resuming it -- so the 897MB and
935MB files would fail the same way, only slower.

Both now retry on the 1/3/10/15 minute schedule used elsewhere and resume with
a Range request, accumulating in "<dest>.part" and promoting the file only
once its size matches the server's Content-Length. All of these hosts send
accept-ranges: bytes. The omics script delegates to download_file_resumable()
from retry_utils.R rather than keeping a second implementation.

Dropped req_verbose(): full curl logging across ~2GB of transfers buried the
real errors in the build log.

The transcriptomics branch also now pairs rm() with gc(), as the copy_number
and mutations branches already did. Without it R still holds the previous
object when pivot_longer allocates the long frame, which is the peak of the
whole step: measured 1,351MB -> 1,219MB peak RSS at no run time cost.

The nci60 unzips check their exit status instead of failing later as a
confusing parse error on a stale DOSERESP.csv.

Verified against the live Sanger host: an interrupted transfer resumes with
206 Partial Content and completes byte-exact, and resuming from a 100MB
partial took 4.8s against 61.3s for the full download, confirming it fetches
only the remaining bytes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants