Repository navigation
Harden the drug-descriptor and curve-fitting pipeline - #481
Merged
Merged
Conversation
sgosline
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipeline Hardening/Debugging PR # 3
Harden the drug-descriptor and curve-fitting pipeline
Fixes correctness, memory, and data-integrity problems in the shared drug pipeline: PubChem lookups, molecular-descriptor generation, dose-response curve fitting, and PDX response metrics. This is the second half of the former shared-utilities work.
pubchem_retrieval.py: reuse prior results and keep TSVs clean_pubchem_data_cache) that can be pre-loaded from earlier drug files viaload_pubchem_cache(). When a drug is already known, its properties and synonyms are returned from the cache instead of making a PubChem HTTP request, so re-runs are far faster and gentler on the API. The list of prior files is picked up automatically from/tmp/prev_drug_files.txtwhen present._sanitise_field()strips stray tab, carriage-return, newline, and quote characters from every value before it is written, so a single malformed name can no longer shift columns or break the TSV on the next read.build_drug_desc.py: fix the OOM crash and make output deterministic/opt/mp_tmp, overridable viaCODERDATA_MP_TMPDIR). Previously the worker processes wrote scratch files to a bind-mounted/tmp, which triggered a fork/OSErrorcrash on macOS during descriptor generation._chunks) so memory stays bounded no matter how many drugs are processed, instead of loading everything at once."NaN"regardless of how molecules happen to fall into chunks, so the same input always produces byte-identical output (previously the formatting could vary run to run).fit_curve.py: bounded memory and resilient parallel fitting--workersflag to override and a--chunk_sizeflag to tune batch size.calc_pdx_metrics.py: match the schema's metric namesmRECISTbecomesmrecistandTGIbecomestgi.Scope: 4 files. Base: download-resilience-utils.
Related issues
Closes #468
Closes #473