Skip to content

docs: add a draft page on federating workflows with Conquest - #121

Open
anuragxxd wants to merge 1 commit into
mainfrom
docs/conquest-federation
Open

anuragxxd wants to merge 1 commit into
mainfrom
docs/conquest-federation

Conversation

@anuragxxd

@anuragxxd anuragxxd commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Adds Federate Workflows with Conquest under Researchers, as Alex asked. It's a draft, marked as such on the page, to grow as the setup matures.

What's on the page

Same structure as the other Researchers pages: intro, a draft warning, then:

  • How it works: component table, plus the call chain. Client → Conquest WES → workflow engine → Conquest TES → routing → Kubernetes, Funnel or TESK, and the storage gateway that gives remote TES services presigned S3 access without credentials.
  • Route tasks: how the operator's rules and the run tags decide where each task runs.
  • What was tested:
    • workflow types and engines with versions: Nextflow 25.04.3 and 25.10.4 with nf-ga4gh; Snakemake 9.17.3; miniwdl 1.12.1;
    • workflows with revisions and data: nf-core/sarek 3.8.1, nf-core/methylseq 4.2.0, viral-pipelines assemble_refbased, rna-seq-star-deseq2;
    • TES services with implementations and versions: Conquest's Kubernetes backend, Funnel develop plus #1488, TESK with tesk-api#133 and tesk-core#134.
  • Results: each workflow ran federated and all-local, and the outputs were compared. sarek on GIAB HG002 chr20 is also scored against the GIAB truth set.
  • Known limitations and Where to go next.

The index table and the nav link to the page, and includes/references.md gains a [conquest] reference.

Checks

python scripts/check_names.py, mkdocs build --strict and pytest pass locally.

Summary by Sourcery

Document draft support for federating workflow tasks with Conquest across local and remote compute services.

New Features:

  • Add a draft Researchers guide explaining how Conquest federates Nextflow, Snakemake, and WDL workflow tasks across local Kubernetes and remote TES services.
  • Document tested workflow engines, workflows, TES implementations, result comparisons, limitations, and next steps for Conquest deployments.

Enhancements:

  • Link the new Conquest guide from the Researchers index and site navigation.

Documentation:

  • Add a Conquest reference link for the new documentation page.

Describes how Conquest (GA4GH WES and TES) runs Nextflow, Snakemake and
WDL workflows and sends each task to its own Kubernetes cluster or to a
remote TES service (Funnel, TESK), the call chain and the storage gateway,
and which workflow types, workflows, engines and TES implementations were
tested, with versions and results. Marked as a draft of test deployments.
@sourcery-ai

sourcery-ai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Documentation

Add the draft Conquest guide describing the federation flow, routing, tested workflows and services, results, and limitations.

  • docs/researchers/federated-workflows-conquest.md L1-165

List the new guide in the Researchers index and site navigation, and add its Conquest reference link.

  • docs/researchers/index.md L15
  • includes/references.md L1
  • mkdocs.yml L117

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 elixir-on-cloud-docs | 🛠️ Build #35789916 | 📁 Comparing e343e1b against latest (0ea20dd)

  🔍 Preview build  

2 files changed
+ researchers/federated-workflows-conquest/index.html
± researchers/index.html

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="docs/researchers/federated-workflows-conquest.md" line_range="127" />
<code_context>
+
+### Results
+
+Each workflow ran twice: with its tasks split across Kubernetes, Funnel and
+TESK, and with all tasks in Kubernetes. The results were compared.
+
</code_context>
<issue_to_address>
**Federated run counts are unexplained**

When the sarek `test` profile results are read alongside the stated run count, the results say each workflow ran once federated and once locally, but the sarek `test` row lists two separate federated task distributions without explaining them. Readers cannot tell how many runs were compared or reproduce that result.

Clarify what each sarek `test` task distribution represents and how many federated and local runs were compared.

Also at `docs/researchers/federated-workflows-conquest.md:133`.
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: docs/researchers/federated-workflows-conquest.md:127


Sourcery is free for open source - if you like our reviews please consider sharing them ✨


### Results

Each workflow ran twice: with its tasks split across Kubernetes, Funnel and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Federated run counts are unexplained

When the sarek test profile results are read alongside the stated run count, the results say each workflow ran once federated and once locally, but the sarek test row lists two separate federated task distributions without explaining them. Readers cannot tell how many runs were compared or reproduce that result.

Clarify what each sarek test task distribution represents and how many federated and local runs were compared.

Also at docs/researchers/federated-workflows-conquest.md:133.

Prompt for AI agents
In `docs/researchers/federated-workflows-conquest.md` at line 127:

**Federated run counts are unexplained**

When the sarek `test` profile results are read alongside the stated run count, the results say each workflow ran once federated and once locally, but the sarek `test` row lists two separate federated task distributions without explaining them. Readers cannot tell how many runs were compared or reproduce that result.

Clarify what each sarek `test` task distribution represents and how many federated and local runs were compared.

Also at `docs/researchers/federated-workflows-conquest.md:133`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant