Skip to content

Harden the drug-descriptor and curve-fitting pipeline - #481

Merged
sgosline merged 1 commit into
download-resilience-utilsfrom
drug-pipeline-hardening
Sep 23, 2026
Merged

sgosline merged 1 commit into
download-resilience-utilsfrom
drug-pipeline-hardening

Conversation

@jjacobson95

@jjacobson95 jjacobson95 commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Pipeline Hardening/Debugging PR # 3

Harden the drug-descriptor and curve-fitting pipeline

Fixes correctness, memory, and data-integrity problems in the shared drug pipeline: PubChem lookups, molecular-descriptor generation, dose-response curve fitting, and PDX response metrics. This is the second half of the former shared-utilities work.

pubchem_retrieval.py: reuse prior results and keep TSVs clean

  • Reuse previously fetched drug data. Adds an in-memory cache (_pubchem_data_cache) that can be pre-loaded from earlier drug files via load_pubchem_cache(). When a drug is already known, its properties and synonyms are returned from the cache instead of making a PubChem HTTP request, so re-runs are far faster and gentler on the API. The list of prior files is picked up automatically from /tmp/prev_drug_files.txt when present.
  • Prevent corrupted drug tables. A new _sanitise_field() strips stray tab, carriage-return, newline, and quote characters from every value before it is written, so a single malformed name can no longer shift columns or break the TSV on the next read.

build_drug_desc.py: fix the OOM crash and make output deterministic

  • Fix the out-of-memory crash in Docker. Points the multiprocessing temp directory at container-local storage (/opt/mp_tmp, overridable via CODERDATA_MP_TMPDIR). Previously the worker processes wrote scratch files to a bind-mounted /tmp, which triggered a fork/OSError crash on macOS during descriptor generation.
  • Process molecules in fixed-size chunks (_chunks) so memory stays bounded no matter how many drugs are processed, instead of loading everything at once.
  • Make descriptor output stable. Non-numeric Mordred descriptor values are now consistently written as "NaN" regardless of how molecules happen to fall into chunks, so the same input always produces byte-identical output (previously the formatting could vary run to run).

fit_curve.py: bounded memory and resilient parallel fitting

  • Stream drug groups in chunks rather than materializing the entire grouped dataset, so the parallel workers only copy one chunk of data at a time.
  • Default to using 3/4 of available CPU cores (instead of all-but-one), which avoids the memory spikes that were causing Docker to kill the process; added a --workers flag to override and a --chunk_size flag to tune batch size.
  • Recover from failed chunks: if a chunk fails, it retries up to 3 times, halving the worker count each time to reduce memory pressure; if it still fails, that chunk is skipped (and counted) rather than aborting the whole run. Added timestamped progress logging.

calc_pdx_metrics.py: match the schema's metric names

  • Renames the emitted metric labels to match the schema's controlled vocabulary: mRECIST becomes mrecist and TGI becomes tgi.

Scope: 4 files. Base: download-resilience-utils.

Related issues

Closes #468
Closes #473

@sgosline
sgosline merged commit 8943d24 into download-resilience-utils Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants