Repository navigation
Fix data-source and processing bugs across individual datasets - #479
Merged
Merged
Conversation
Also fixes an hcmi omics failure that only appears once TMPDIR is moved off
the /tmp bind mount (see the docker-infra branch):
OSError: [Errno 18] Invalid cross-device link:
'/opt/mp_tmp/tmpXXXX.csv.gz' -> '/tmp/hcmi_transcriptomics.csv.gz'
deduplicate_final_csv() created its temp file with tempfile.mkstemp() and no
dir=, so it landed in TMPDIR while the output lives on the bind mount, and
os.replace() cannot rename across filesystems. Creating the temp file beside
the destination keeps the replace atomic and avoids copying a multi-GB file.
Separately, stops several silent-failure paths that produced a SUCCESSFUL
build containing wrong or missing data -- the worst class of bug here,
because nothing downstream can tell the difference.
liver, novartis and colorectal convert counts to TPM by shelling out with
os.system(), which returns a status rather than raising, and the status was
never checked. All three then read the same /tmp/transcriptomics_tpm.tsv --
and /tmp is the bind mount shared by every container, so the filename is
global and nothing ever removed it between datasets. A failed conversion fell
through to the read and picked up whichever dataset wrote the file last, so
novartis could be built from liver's transcriptomics. tpmFromCounts.py
downloads the GRCh37 GTF from NCBI on every run, so one network blip is
enough. The stale output is now removed first, the status checked, and the
result confirmed non-empty. Same treatment for beatAML's TPM conversion and
colorectal's CNV annotation.
hcmi and pancreatic both ended their GDC download-and-verify loop with
"Proceeding with available files", so any number of files could go missing
and the build would carry on. Both now raise; ALLOW_INCOMPLETE_GDC_DOWNLOAD=1
overrides deliberately, mirroring ALLOW_GENE_FALLBACK in the genes step. The
MD5 retry loop above is unchanged -- this only changes what happens once it
is exhausted.
sgosline
reviewed
Sep 23, 2026
sgosline
left a comment
Member
There was a problem hiding this comment.
The fixes are solid, just two questions remain, once addressed i will merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipeline Hardening/Debugging PR # 1
Fix data-source and processing bugs across individual datasets
Corrects sample-ID handling, omics processing, and data-source fetching in the per-dataset builders (everything except broad_sanger and MPNST, which have their own PRs). Several of these were silent failures that produced empty or wrong output without erroring.
Sample-ID continuation (beatAML, bladder, colorectal, novartis, sarcoma)
improve_sample_idfrom its max value instead of restarting, with a safe fallback to 0 when the prior file is empty/absent. Prevents ID collisions across incremental builds.Fail-loud omics guards
02-omics-colorectal.py): check the exit status of the TPM conversion (tpmFromCounts.py) and the CNV annotation step (CNV-segfile-annotation.R); raise if either exits non-zero or produces no output (the TPM step downloads the GRCh37 GTF, a common network failure point). Take the first Entrez match for ambiguous gene aliases.02-omics-novartis.py): same TPM exit-status/empty-output guard.GetBeatAML.py): fix an indentation bug that had silently disabled all omics processing; add a TPM exit-status/empty-output guard.Pancreatic
01-createPancreaticSamplesFile.py): stop dropping organoid IDs; fetch the sample mapping from Figshare (figstatic) with a browser User-Agent and 1/3/10/15-minute retry backoff; fail on an empty body.02-getPancreaticData.py): adddrop_controlled_access()to filter out dbGaP controlled-access GDC files (they can never download without dbGaP auth and cannot be redistributed in an open dataset), plus partial-download guards.03-getPancreaticDrugs.py): fetch the supplementary drug table from Figshare with the same retry backoff; validate the expected drug count (26) and fail if it differs.05-addPrecalcAUC.py,05-compare_with_scores.py): point at the Figshare table link and correct the merge/comparison.CPTAC (
getCptacData.py)entrez_idto int and drop null/zero rows.Liver
mygene.infoAPI call (returning HTTP 500 mid-build) with a localgenes.csvlookup and drop themygenedependency (02-omics-liver.py,requirements.txt); sample-ID continuation (01-samples-liver.py).HCMI
EXDEV) rename fix (02-getHCMIData.py); sample-file correction (01-createHCMISamplesFile.py).Dependencies
mygenefrom liver.Scope: 20 files. Independent of the rest of the upcoming PRs.