Repository navigation
Make the gene-reference build resilient to Ensembl outages and API changes - #483
Merged
Merged
Conversation
sgosline
reviewed
Sep 23, 2026
| '<Dataset name="hsapiens_gene_ensembl" interface="default">', | ||
| '<Filter name="biotype" value="protein_coding"/>', | ||
| '<Attribute name="ensembl_gene_id"/>', | ||
| '</Dataset></Query>') |
Member
There was a problem hiding this comment.
how was this tested? how do you know it does the same thing?
Collaborator
Author
There was a problem hiding this comment.
Compared the new genes file to the previous version (from build 2.3) and there were 134,497 rows identical with just 5 new ones in this version (entrez 160365).
The query is effectively the same, just the file was totally rewritten because there were major site/API changes. And also added more retry logic.
Query:
useEnsembl(biomart="genes", dataset="hsapiens_gene_ensembl") + getBM(attributes='ensembl_gene_id', filters='biotype', values='protein_coding')
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipeline Hardening/Debugging PR # 5
Make the gene-reference build resilient to Ensembl outages and API changes
Reworks
genes/00-buildGeneFile.Rso the gene reference (genes.csv) can still be built after Ensembl's 2026 API changes and during mirror outages, and never silently ships a lower-quality gene list.Fix the Ensembl request that always failed (Accept header)
Acceptheader the client sends. The library the build uses (biomaRt, via httr) sendsapplication/jsonfirst by default, which Ensembl answers with a hard 404. That made every gene-build attempt fail with a misleading error, so no amount of retrying could ever succeed.Accept: text/xml, which makes Ensembl redirect to the BioMart endpoint biomaRt actually expects. Verified working on 2026-08-28.Retry across mirrors with backoff
www,useast,asia), waiting longer after each full cycle (exponential backoff) before trying again, so a temporary mirror outage no longer fails the build.Safe fallback that can't silently degrade quality
org.Hs.eg.dbinstead of failing outright.Build local ID mappings from org.Hs.eg.db
org.Hs.eg.db, with biomaRt used only to restrict the result to protein-coding genes.Dependencies
requirements.rinstallsAnnotationDbi,org.Hs.eg.db,biomaRt, andtidyr, and adds a post-install check that fails the Docker build if any required package is missing.Scope: 2 files. Base: broad-sanger-data-sources.
Addresses #472