Skip to content

Make the gene-reference build resilient to Ensembl outages and API changes - #483

Merged
sgosline merged 1 commit into
broad-sanger-data-sourcesfrom
genes-ensembl-resilience
Sep 29, 2026
Merged

sgosline merged 1 commit into
broad-sanger-data-sourcesfrom
genes-ensembl-resilience

Conversation

@jjacobson95

@jjacobson95 jjacobson95 commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Pipeline Hardening/Debugging PR # 5

Make the gene-reference build resilient to Ensembl outages and API changes

Reworks genes/00-buildGeneFile.R so the gene reference (genes.csv) can still be built after Ensembl's 2026 API changes and during mirror outages, and never silently ships a lower-quality gene list.

Fix the Ensembl request that always failed (Accept header)

  • Ensembl's BioMart endpoint decides what to return based on the Accept header the client sends. The library the build uses (biomaRt, via httr) sends application/json first by default, which Ensembl answers with a hard 404. That made every gene-build attempt fail with a misleading error, so no amount of retrying could ever succeed.
  • The fix forces Accept: text/xml, which makes Ensembl redirect to the BioMart endpoint biomaRt actually expects. Verified working on 2026-08-28.

Retry across mirrors with backoff

  • The Ensembl call is now wrapped in a retry loop that cycles through the three valid mirrors (www, useast, asia), waiting longer after each full cycle (exponential backoff) before trying again, so a temporary mirror outage no longer fails the build.

Safe fallback that can't silently degrade quality

  • If every mirror stays unreachable after all retry cycles, the script falls back to deriving the protein-coding gene list locally from org.Hs.eg.db instead of failing outright.
  • This fallback produces slightly broader coverage than the Ensembl filter, so it is clearly documented in the code and surfaced, ensuring a degraded gene list is never shipped unnoticed.

Build local ID mappings from org.Hs.eg.db

  • Entrez-to-symbol, Ensembl gene, and Ensembl transcript mappings are assembled locally from org.Hs.eg.db, with biomaRt used only to restrict the result to protein-coding genes.

Dependencies

  • requirements.r installs AnnotationDbi, org.Hs.eg.db, biomaRt, and tidyr, and adds a post-install check that fails the Docker build if any required package is missing.

Scope: 2 files. Base: broad-sanger-data-sources.

Addresses #472

@jjacobson95 jjacobson95 modified the milestones: 2.3 new build, 2.4 new build Sep 22, 2026
'<Dataset name="hsapiens_gene_ensembl" interface="default">',
'<Filter name="biotype" value="protein_coding"/>',
'<Attribute name="ensembl_gene_id"/>',
'</Dataset></Query>')

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how was this tested? how do you know it does the same thing?

@jjacobson95 jjacobson95 Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared the new genes file to the previous version (from build 2.3) and there were 134,497 rows identical with just 5 new ones in this version (entrez 160365).

The query is effectively the same, just the file was totally rewritten because there were major site/API changes. And also added more retry logic.

Query:
useEnsembl(biomart="genes", dataset="hsapiens_gene_ensembl") + getBM(attributes='ensembl_gene_id', filters='biotype', values='protein_coding')

@sgosline
sgosline merged commit 9806ce1 into broad-sanger-data-sources Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants