Skip to content

Fix data-source and processing bugs across individual datasets - #479

Merged
sgosline merged 1 commit into
mainfrom
dataset-build-fixes
Sep 26, 2026
Merged

sgosline merged 1 commit into
mainfrom
dataset-build-fixes

Conversation

@jjacobson95

@jjacobson95 jjacobson95 commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Pipeline Hardening/Debugging PR # 1

Fix data-source and processing bugs across individual datasets

Corrects sample-ID handling, omics processing, and data-source fetching in the per-dataset builders (everything except broad_sanger and MPNST, which have their own PRs). Several of these were silent failures that produced empty or wrong output without erroring.

Sample-ID continuation (beatAML, bladder, colorectal, novartis, sarcoma)

  • Read the previous samples file and continue improve_sample_id from its max value instead of restarting, with a safe fallback to 0 when the prior file is empty/absent. Prevents ID collisions across incremental builds.

Fail-loud omics guards

  • colorectal (02-omics-colorectal.py): check the exit status of the TPM conversion (tpmFromCounts.py) and the CNV annotation step (CNV-segfile-annotation.R); raise if either exits non-zero or produces no output (the TPM step downloads the GRCh37 GTF, a common network failure point). Take the first Entrez match for ambiguous gene aliases.
  • novartis (02-omics-novartis.py): same TPM exit-status/empty-output guard.
  • beatAML (GetBeatAML.py): fix an indentation bug that had silently disabled all omics processing; add a TPM exit-status/empty-output guard.

Pancreatic

  • Samples (01-createPancreaticSamplesFile.py): stop dropping organoid IDs; fetch the sample mapping from Figshare (figstatic) with a browser User-Agent and 1/3/10/15-minute retry backoff; fail on an empty body.
  • Data (02-getPancreaticData.py): add drop_controlled_access() to filter out dbGaP controlled-access GDC files (they can never download without dbGaP auth and cannot be redistributed in an open dataset), plus partial-download guards.
  • Drugs (03-getPancreaticDrugs.py): fetch the supplementary drug table from Figshare with the same retry backoff; validate the expected drug count (26) and fail if it differs.
  • AUC helpers (05-addPrecalcAUC.py, 05-compare_with_scores.py): point at the Figshare table link and correct the merge/comparison.

CPTAC (getCptacData.py)

  • Stream omics per cohort, appending each cohort's rows to per-data-type gzip writers, so peak memory no longer scales with the full matrix; coerce entrez_id to int and drop null/zero rows.

Liver

  • Replace the live mygene.info API call (returning HTTP 500 mid-build) with a local genes.csv lookup and drop the mygene dependency (02-omics-liver.py, requirements.txt); sample-ID continuation (01-samples-liver.py).

HCMI

  • GDC partial-download guards and a cross-device (EXDEV) rename fix (02-getHCMIData.py); sample-file correction (01-createHCMISamplesFile.py).

Dependencies

  • Pin R requirements for bladder and colorectal and add a post-install check that fails the build if any required package (BiocManager, GEOquery, GenomicRanges, Homo.sapiens) is missing; drop mygene from liver.

Scope: 20 files. Independent of the rest of the upcoming PRs.

Also fixes an hcmi omics failure that only appears once TMPDIR is moved off
the /tmp bind mount (see the docker-infra branch):

  OSError: [Errno 18] Invalid cross-device link:
    '/opt/mp_tmp/tmpXXXX.csv.gz' -> '/tmp/hcmi_transcriptomics.csv.gz'

deduplicate_final_csv() created its temp file with tempfile.mkstemp() and no
dir=, so it landed in TMPDIR while the output lives on the bind mount, and
os.replace() cannot rename across filesystems. Creating the temp file beside
the destination keeps the replace atomic and avoids copying a multi-GB file.

Separately, stops several silent-failure paths that produced a SUCCESSFUL
build containing wrong or missing data -- the worst class of bug here,
because nothing downstream can tell the difference.

liver, novartis and colorectal convert counts to TPM by shelling out with
os.system(), which returns a status rather than raising, and the status was
never checked. All three then read the same /tmp/transcriptomics_tpm.tsv --
and /tmp is the bind mount shared by every container, so the filename is
global and nothing ever removed it between datasets. A failed conversion fell
through to the read and picked up whichever dataset wrote the file last, so
novartis could be built from liver's transcriptomics. tpmFromCounts.py
downloads the GRCh37 GTF from NCBI on every run, so one network blip is
enough. The stale output is now removed first, the status checked, and the
result confirmed non-empty. Same treatment for beatAML's TPM conversion and
colorectal's CNV annotation.

hcmi and pancreatic both ended their GDC download-and-verify loop with
"Proceeding with available files", so any number of files could go missing
and the build would carry on. Both now raise; ALLOW_INCOMPLETE_GDC_DOWNLOAD=1
overrides deliberately, mirroring ALLOW_GENE_FALLBACK in the genes step. The
MD5 retry loop above is unchanged -- this only changes what happens once it
is exhausted.
@jjacobson95 jjacobson95 added 'build' update bug Something isn't working labels Sep 22, 2026
@jjacobson95 jjacobson95 modified the milestones: 2.3 new build, 2.4 new build Sep 22, 2026

@sgosline sgosline left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixes are solid, just two questions remain, once addressed i will merge.

Comment thread coderbuild/beatAML/GetBeatAML.py Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working 'build' update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants