Skip to content

Add resilient download, retry, and DepMap-fetch utilities - #480

Merged
sgosline merged 3 commits into
dataset-build-fixesfrom
download-resilience-utils
Sep 29, 2026
Merged

sgosline merged 3 commits into
dataset-build-fixesfrom
download-resilience-utils

Conversation

@jjacobson95

@jjacobson95 jjacobson95 commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Pipeline Hardening/Debugging PR # 2

Add resilient download, retry, and DepMap-fetch utilities

Introduces the shared network-resilience layer used across dataset builders so long builds survive transient outages, truncated downloads, and providers that block scripted access. This is one half of the former shared-utilities work; the drug-descriptor pipeline is a separate PR.

retry_utils.R: retry and resumable PSet handling

  • with_retries(): runs a zero-arg expression with 1/3/10/15-minute backoff, logging each attempt and hard-failing when exhausted.
  • download_file_resumable(): if a download is interrupted, it continues from the bytes already saved on disk (using wget -c) instead of starting over. If the file still isn't complete, it raises an error so the next retry picks up where it left off.
  • PharmacoGx PSet helpers: a set of functions for downloading the large (~2 GB) PharmacoGx "PSet" data files reliably. They look up the download URL (pset_download_url()), read the expected file size and checksum from the server (pset_remote_meta()), download with resume support while making sure a half-finished file is never cached as if it were complete (fetch_pset_resumable()), and reuse an already-downloaded PSet when one is present (pset_load_cached()). The net effect: if the PSet step is interrupted, it resumes instead of starting the multi-gigabyte download over.

download_utils.py: resumable HTTP downloads (new)

  • retrieve_url() can resume an interrupted download: it asks the server for only the missing portion of the file (an HTTP "Range" request) and appends it to what was already downloaded, rather than downloading the whole file again. We confirmed the NCI server (wiki.nci.nih.gov) supports this. If a server doesn't support resuming, it safely downloads the file from scratch instead.
  • Uses the same 1/3/10/15-minute retry schedule as the R helper, so retry behavior is consistent across the build.

fetch_depmap.py: DepMap via Synapse (new)

  • Fetches the four DepMap inputs (model, mutations, transcriptomics, copy number) from the "DepMap Raw" Synapse folder syn75028495, since the DepMap portal is no longer scriptable.
  • The four Synapse file IDs have sensible defaults but can each be overridden with an environment variable, so a future DepMap release can be pointed at new files without editing code. If the required Synapse IDs or credentials aren't configured, it stops with a clear message instead of failing partway through, and default_dest() picks the standard output location when the caller doesn't specify one.

pinned_psets.R: explicit PSet selection (new)

  • psets_for_dataset() returns the fixed PSet list used for the release instead of trusting availablePSets().

Scope: 4 files, all new (no deletions). Base: dataset-build-fixes. Consumed by the broad_sanger and drug-pipeline PRs later in the stack.

@jjacobson95 jjacobson95 modified the milestones: 2.3 new build, 2.4 new build Sep 22, 2026
if not got or not os.path.exists(got):
sys.exit(f'Synapse returned no downloaded file for {fname} ({syn_id}). '
f'Check that the entity exists and that this account has '
f'download permission on {DEPMAP_SYNAPSE_FOLDER}.')

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

permissions arent always inherited and can be for the specific object, so this might not be true.

Harden the drug-descriptor and curve-fitting pipeline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants