Skip to content

docs: add a Node onboarding tutorial - #125

Merged
uniqueg merged 2 commits into
mainfrom
docs/113-node-onboarding-tutorial
Nov 10, 2026
Merged

uniqueg merged 2 commits into
mainfrom
docs/113-node-onboarding-tutorial

Conversation

@uniqueg

@uniqueg uniqueg commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds docs/operators/node-onboarding-tutorial.md, a step-by-step tutorial for Node operators that goes from a cluster to a TES service that is registered in the Cloud Registry, verified and behind the gateway:

  1. Prerequisites: cluster, namespace, storage, Ingress/HTTPS, LS Login account and client
  2. Deploy a TES service: TESK (chart 0.3.0, tesk-api 1.2.0), with Poiesis and Funnel as alternatives that link to their own docs, plus a table of what each one supports today
  3. LS Login and ECP_CLN access per product, with the gaps stated and linked (generic-issues#20, docs(contributors): update default branch info #21, docs: restructure docs and update branding #35, Update sandbox docu #46, docs: update Cloud Registry deployment notes #50)
  4. Check with a test task (TESK's hello.json)
  5. Register by import with sync on (links to the existing "Using the Cloud Registry" pages and doesn't repeat them)
  6. Get verified
  7. Get added behind proTES (configured or discovered; generic-issues#31)
  8. Optional: Pubgrade-Sidecar 0.2.0 (central Pubgrade not deployed: generic-issues#32)
  9. A short path for data centres: storage, and TRS-Filer 0.5.0. DRS-Filer has no release, so the page says it isn't covered

The page is in the nav ("Architects & Operators") and in the section index, and onboarding.md links to it in one sentence.

Refs #113

Acceptance not met by this PR: #113 needs the tutorial to be tested by an operator who didn't write it. I can't do that, so this PR uses "Refs" and the operator test stays open on the issue.

Light lane: docs only. I reviewed the diff myself against the product sources listed below.

Sources (released versions)

  • TESK: charts/tesk and docs/ on dev at 65db464 (chart version: 0.3.0, appVersion: 1.2.0, release notes docs/releases/0.3.0.md). The chart isn't released yet, so links point to tesk.readthedocs.io latest
  • tesk-api v1.2.0: docs/access-control.md, docs/configuration.md, src/main/resources/application.properties
  • Poiesis v0.2.0: docs/app/docs/deploy/*.md, deployment/helm/poiesis/values.yaml, poiesis/api/routes/service_info.py
  • Funnel v0.12.3: docs/security/oauth2.md, docs/compute/slurm.md, internal/server/taskservice/service.go, internal/server/auth/auth.go
  • Cloud Registry v1.0.0rc2: docs/api.md, docs/configuration.md
  • proTES v0.6.0: docs/configuration.md, docs/access-control.md
  • Pubgrade-Sidecar v0.2.0: docs/deployment.md, deployment/ (the chart is attached to the release)
  • TRS-Filer v0.5.0: docs/configuration.md#service-info
  • This repo: apis.md (token, LS Login issuer and discovery URL), ls-login.md (userinfo URL, SPREG), using-cloud-registry*.md (prod/test only, admin-only types)

Verification

  • python scripts/check_names.py: no findings
  • mkdocs build --strict: built without warnings, so internal links and anchors resolve
  • pytest: 25 passed, 100% coverage
  • External links: checked with curl. All return 200, and their anchors exist on the rendered pages (TESK latest, proTES v0.6.0, Cloud Registry v1.0.0rc2, TRS-Filer v0.5.0, Poiesis site, Pubgrade docs latest/sidecar)
  • CI: all checks pass (CI passed, Docs, Test, PR title, CodeQL, Read the Docs preview). The first run was cancelled by the concurrency group when the PR was edited right after it opened, and the next run passed

What's unverified

  • No command on the page was run against a cluster. They are copied from the product docs, and only names and namespaces are adapted (e.g., -n tesk for the sidecar)
  • TESK with LS Login: the chart has no value for tesk-api's user info endpoint (SECURITY_OAUTH2_RESOURCE_USER_INFO_URI, default ELIXIR AAI's login.elixir-czech.org). Nobody has checked whether LS Login tokens validate with the default. The page says so
  • Funnel's service info is public with auth on: I read this from the code of v0.12.3 (auth.go). Funnel's docs don't say it
  • The test-task curl lines combine TESK's docs/api.md example with the token pattern of this repo's apis.md
  • Pubgrade-Sidecar docs exist only as latest (there is no versioned build for 0.2.0)

Gaps found in the products (stated on the page; no issues opened)

  • TESK chart 0.3.0: no value for the user info endpoint, or for the group prefix and suffix of tesk-api. Their defaults are ELIXIR AAI's, so the page recommends claim mode. The page also notes that the chart's default service info has environment: dev, which this registry refuses
  • Funnel 0.12.3: the service info is hardcoded (id: org.ga4gh.funnel, environment: development). It can't be imported or synced here (invalid service info), so Funnel Nodes can't currently meet the verification policy
  • Poiesis 0.2.0: the service info is hardcoded (id: org.ga4gh.poiesis, no environment). Gateways that filter by environment skip it
  • TRS-Filer 0.5.0: the chart has no value for the service info, so an overlay is needed (the TRS-Filer docs already say this)
  • DRS-Filer: has no release

Review (Sourcery, round 1)

I fixed four of the seven findings in aeb47de:

  • auth is set before the first install, or the Ingress stays off until it is
  • the TESK checkout is pinned to 65db464
  • the values files go in the checkout
  • the page says plainly that Poiesis and Funnel can't enforce ECP_CLN yet, and to ask the admins first

The same commit also aligns services/tesk.md with the storage requirements of chart 0.3.0.

Declined review findings

  • "ReadWriteMany required": the TESK docs for chart 0.3.0 say ReadWriteOnce is enough for the per-task PVCs. The old note on this site's TESK page was the outlier, and that page is now updated
  • sleep 2 before curl: copied verbatim from the TESK deployment guide. A rerun is enough, and a sturdier check would belong in the TESK docs
  • Sidecar steps not retry-safe: these are the commands of the Pubgrade-Sidecar deployment guide for a one-time setup, and the errors show plainly what already exists

Open questions

None.

🤖 Generated with Claude Code

https://claude.ai/code/session_011FSoqMpmDCfbACLj3uy6xV

Take a Node from a cluster to a registered, verified TES service behind
the gateway: prerequisites, deploying TESK (Poiesis and Funnel as
alternatives), LS Login and ECP_CLN access, a test task, registration
and sync, verification, proTES, and Pubgrade-Sidecar, plus a short path
for data centres. Commands and values come from the docs of TESK chart
0.3.0 (tesk-api 1.2.0), Poiesis 0.2.0, Funnel 0.12.3, Cloud Registry
1.0.0rc2, proTES 0.6.0, Pubgrade-Sidecar 0.2.0 and TRS-Filer 0.5.0;
gaps of the products are stated with their tracking issues.

Refs #113

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FSoqMpmDCfbACLj3uy6xV
@uniqueg uniqueg self-assigned this Nov 10, 2026
@sourcery-ai

sourcery-ai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Configuration

Add the tutorial to the operator navigation so readers can find it.

  • mkdocs.yml L92

Documentation

Add the end-to-end Node onboarding tutorial, including product-specific deployment and authentication guidance, registry registration and verification, gateway integration, and the data-centre path.

  • docs/operators/node-onboarding-tutorial.md L1-481

Connect the tutorial to the existing onboarding and TESK guidance, and update TESK’s storage requirements for chart 0.3.0.

  • docs/operators/index.md L41
  • docs/operators/onboarding.md L6-7
  • docs/operators/services/tesk.md L20-28

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@uniqueg
uniqueg marked this pull request as ready for review November 10, 2026 17:16
@read-the-docs-community

read-the-docs-community Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 7 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="docs/operators/node-onboarding-tutorial.md" line_range="127" />
<code_context>
+```bash
+git clone https://github.com/elixir-cloud-aai/TESK.git
+cd TESK
+helm upgrade --install tesk charts/tesk \
+    --namespace tesk --create-namespace \
+    -f my-values.yaml -f secrets.yaml
</code_context>
<issue_to_address>
**TESK is exposed without authentication**

When an operator follows the documented install order with the public Ingress enabled, the first Helm install omits `auth.mode: auth` while enabling public Ingress, so TESK accepts unauthenticated task submissions until step 3's second install applies authentication. Anyone who reaches the endpoint during that interval can submit jobs to the cluster; stopping after the first install leaves it exposed.

Configure authentication before the first install, or keep Ingress disabled until authentication has been applied.

Also at `docs/operators/node-onboarding-tutorial.md:178`.
</issue_to_address>

### Comment 2
<location path="docs/operators/node-onboarding-tutorial.md" line_range="78" />
<code_context>
+|---|---|---|---|
+| Runs on | Kubernetes, OpenShift | Kubernetes 1.29+ | HPC, HTC (e.g., Slurm) |
+| Validates LS Login tokens | yes, with gaps (see [step 3](#3-enable-ls-login-and-ecp_cln-access)) | yes (OIDC issuer and audience) | yes (OIDC issuer and audience) |
+| Access by membership in `ECP_CLN` | by a claim of the user info | no, any valid token ([generic-issues#21][gi-21]) | no, any valid token ([generic-issues#20][gi-20]) |
+| Service info for the registry | set in a file of the chart | fixed by the code, without `environment` | fixed by the code, with an environment that this registry refuses |
+
</code_context>
<issue_to_address>
**Non-members can submit cluster tasks**

When a user outside `ECP_CLN` obtains a token with the configured issuer and audience for a Poiesis or Funnel service, poiesis and Funnel validate the token's issuer and audience but do not check `ECP_CLN` membership, so a caller outside the stated federation group can submit tasks and run workloads on the Node's compute.

Mark Poiesis and Funnel as unsuitable for this federation until they can enforce the `ECP_CLN` policy, or require an external access-control layer that enforces it.

Also at `docs/operators/node-onboarding-tutorial.md:222`, `docs/operators/node-onboarding-tutorial.md:236`.
</issue_to_address>

### Comment 3
<location path="docs/operators/node-onboarding-tutorial.md" line_range="125" />
<code_context>
+`-f secrets.yaml` if you need none.
+
+```bash
+git clone https://github.com/elixir-cloud-aai/TESK.git
+cd TESK
+helm upgrade --install tesk charts/tesk \
</code_context>
<issue_to_address>
**Unreviewed chart version gets deployed**

When the TESK default branch advances before an operator installs it, `git clone` checks out the current default branch and Helm deploys its chart, so the installed version can differ from the one described and leave the service misconfigured or unable to start.

Pin the checkout or chart installation to the reviewed release or revision described by the tutorial.
</issue_to_address>

### Comment 4
<location path="docs/operators/node-onboarding-tutorial.md" line_range="51" />
<code_context>
+  [Funnel](services/funnel.md)).
+- **A namespace** of its own for the TES service, e.g., `tesk`.
+- **Storage.** A default storage class (TESK creates a temporary volume
+  claim for each task with inputs or outputs; `ReadWriteOnce` is enough),
+  and a storage backend for the inputs and outputs of tasks, e.g., S3 or FTP
+  (see [Deploy storage](onboarding.md#2-deploy-storage)).
</code_context>
<issue_to_address>
**Tasks use unsupported storage**

When an operator provisions a cluster with only a `ReadWriteOnce` storage class based on the tutorial, the tutorial says `ReadWriteOnce` is enough, while the existing TESK guidance requires `ReadWriteMany`, so TESK task volume mounts fail instead of tasks completing.

Change the tutorial prerequisite to require `ReadWriteMany`, consistent with the existing TESK guidance.
</issue_to_address>

### Comment 5
<location path="docs/operators/node-onboarding-tutorial.md" line_range="129" />
<code_context>
+cd TESK
+helm upgrade --install tesk charts/tesk \
+    --namespace tesk --create-namespace \
+    -f my-values.yaml -f secrets.yaml
+```
+
</code_context>
<issue_to_address>
**Helm cannot find the values files**

When an operator creates the values files in their starting directory before following the clone-and-install commands, `cd TESK` changes the working directory before Helm resolves the relative file paths, so Helm cannot open the values files and the deployment fails.

Tell operators to move the values files into `TESK` or pass paths that resolve from that directory.
</issue_to_address>

### Comment 6
<location path="docs/operators/node-onboarding-tutorial.md" line_range="142" />
<code_context>
+helm list -n tesk
+kubectl get all -n tesk
+kubectl port-forward -n tesk service/tesk-api 8080:8080 &
+sleep 2
+curl http://localhost:8080/ga4gh/tes/v1/service-info
+kill %1  # stop the port forward
</code_context>
<issue_to_address>
**Health check fails before startup**

When the API or port-forward takes longer than two seconds to become ready, `sleep 2` does not ensure the port-forward is listening, so `curl` fails with a connection error even when TESK is healthy.

Replace the fixed sleep with a readiness check that retries the endpoint until it responds or a timeout is reached.
</issue_to_address>

### Comment 7
<location path="docs/operators/node-onboarding-tutorial.md" line_range="374" />
<code_context>
+of its own. `secrets.yaml` holds credentials, e.g., of FTP: leave out
+`-f secrets.yaml` if you need none.
+
+```bash
+git clone https://github.com/elixir-cloud-aai/TESK.git
+cd TESK
</code_context>
<issue_to_address>
**Sidecar setup retries stop**

When two operators run these steps concurrently, or an operator retries after a partial or ambiguous result, `kubectl create secret` returns `AlreadyExists` on a retry or when another operator creates the shared Secret first; if the Helm install already succeeded but its result was lost, `helm install` also rejects the existing release. The documented sequence therefore stops instead of converging without manual recovery.

Make the Secret and release steps retry-safe, or document how to detect and reuse resources created by an earlier attempt.

Also at `docs/operators/node-onboarding-tutorial.md:401`.
</issue_to_address>

Sourcery assessment

Approval pending. 7 findings to address first.

Blocking findings: docs/operators/node-onboarding-tutorial.md:127, docs/operators/node-onboarding-tutorial.md:78, docs/operators/node-onboarding-tutorial.md:125, docs/operators/node-onboarding-tutorial.md:51, docs/operators/node-onboarding-tutorial.md:129, and 2 more


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Set auth before the first install (or keep the Ingress off), pin the
TESK checkout to the commit the tutorial was checked against, work in
the checkout so Helm finds the values files, say plainly that Poiesis
and Funnel can't enforce the ECP_CLN policy yet, and align the TESK
service page with the storage requirements of chart 0.3.0.

Refs #113

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FSoqMpmDCfbACLj3uy6xV
@uniqueg
uniqueg merged commit e3c9e73 into main Nov 10, 2026
15 of 17 checks passed
@uniqueg
uniqueg deleted the docs/113-node-onboarding-tutorial branch November 10, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant