Repository navigation
Add resilient download, retry, and DepMap-fetch utilities - #480
Merged
Merged
Conversation
sgosline
reviewed
Sep 23, 2026
| if not got or not os.path.exists(got): | ||
| sys.exit(f'Synapse returned no downloaded file for {fname} ({syn_id}). ' | ||
| f'Check that the entity exists and that this account has ' | ||
| f'download permission on {DEPMAP_SYNAPSE_FOLDER}.') |
Member
There was a problem hiding this comment.
permissions arent always inherited and can be for the specific object, so this might not be true.
Harden the drug-descriptor and curve-fitting pipeline
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipeline Hardening/Debugging PR # 2
Add resilient download, retry, and DepMap-fetch utilities
Introduces the shared network-resilience layer used across dataset builders so long builds survive transient outages, truncated downloads, and providers that block scripted access. This is one half of the former shared-utilities work; the drug-descriptor pipeline is a separate PR.
retry_utils.R: retry and resumable PSet handlingwith_retries(): runs a zero-arg expression with 1/3/10/15-minute backoff, logging each attempt and hard-failing when exhausted.download_file_resumable(): if a download is interrupted, it continues from the bytes already saved on disk (usingwget -c) instead of starting over. If the file still isn't complete, it raises an error so the next retry picks up where it left off.pset_download_url()), read the expected file size and checksum from the server (pset_remote_meta()), download with resume support while making sure a half-finished file is never cached as if it were complete (fetch_pset_resumable()), and reuse an already-downloaded PSet when one is present (pset_load_cached()). The net effect: if the PSet step is interrupted, it resumes instead of starting the multi-gigabyte download over.download_utils.py: resumable HTTP downloads (new)retrieve_url()can resume an interrupted download: it asks the server for only the missing portion of the file (an HTTP "Range" request) and appends it to what was already downloaded, rather than downloading the whole file again. We confirmed the NCI server (wiki.nci.nih.gov) supports this. If a server doesn't support resuming, it safely downloads the file from scratch instead.fetch_depmap.py: DepMap via Synapse (new)syn75028495, since the DepMap portal is no longer scriptable.default_dest()picks the standard output location when the caller doesn't specify one.pinned_psets.R: explicit PSet selection (new)psets_for_dataset()returns the fixed PSet list used for the release instead of trustingavailablePSets().Scope: 4 files, all new (no deletions). Base: dataset-build-fixes. Consumed by the broad_sanger and drug-pipeline PRs later in the stack.