From f908ab51a1cb8b34ee61b18ee72fe466b5c5ee0a Mon Sep 17 00:00:00 2001 From: Alex Kanitz Date: Tue, 17 Nov 2026 10:19:14 +0100 Subject: [PATCH 1/4] docs: add verified dates and registry steps Pages with operational facts carry `last_verified` in their front matter, a date or `null` until verified, and show it at their end (override of Material's `content` partial, with a build test). The README tells contributors how to set it. About, TESK and Funnel were checked against their sources today; the TESK page now links the chart on `dev` and the S3 section of the TESK docs. Every deployment and upgrade procedure (onboarding, both tutorials, the service pages, the registry's own upgrades) now ends with updating the service's Cloud Registry entry or checking that sync did, described once in "After a deployment or an upgrade". Closes #161 Closes #162 Co-Authored-By: Claude Opus 5.5 --- README.md | 24 ++++++ docs/about.md | 1 + docs/operators/cloud-registry.md | 12 +++ .../data-centre-onboarding-tutorial.md | 13 +++ docs/operators/index.md | 4 + docs/operators/ls-login.md | 4 + docs/operators/node-onboarding-tutorial.md | 26 +++++- docs/operators/onboarding.md | 17 +++- docs/operators/service-info.md | 4 + docs/operators/services/drsfiler.md | 4 + docs/operators/services/funnel.md | 8 +- docs/operators/services/index.md | 4 + docs/operators/services/poiesis.md | 9 +- docs/operators/services/protes.md | 9 +- docs/operators/services/prowes.md | 4 + docs/operators/services/tesk.md | 15 +++- docs/operators/services/trsfiler.md | 10 ++- docs/operators/using-cloud-registry-ui.md | 4 + docs/operators/using-cloud-registry.md | 25 ++++++ docs/researchers/index.md | 4 + overrides/partials/content.html | 30 +++++++ tests/integration/test_last_verified.py | 84 +++++++++++++++++++ 22 files changed, 305 insertions(+), 10 deletions(-) create mode 100644 overrides/partials/content.html create mode 100644 tests/integration/test_last_verified.py diff --git a/README.md b/README.md index f3799da..85365f6 100644 --- a/README.md +++ b/README.md @@ -93,6 +93,30 @@ The hook refuses `DOCS_REGISTRY_OFFLINE=1` in CI and on Read the Docs (where `CI` or `READTHEDOCS` is set), so published builds always show the services. +### Last verified dates + +Pages with operational facts that can't be generated, such as onboarding +steps, access procedures, addresses, versions and contacts, say when +someone last checked those facts against their sources. Their front matter +holds the date, which the page shows at its end (see +[`overrides/partials/content.html`](overrides/partials/content.html)): + +```markdown +--- +last_verified: 2026-11-17 +--- +``` + +- Set the date (`YYYY-MM-DD`) only after you have checked every operational + fact of the page against its source: the Cloud Registry, the repository + and docs of the product, or the live service. Fix what is wrong in the + same pull request. +- `last_verified: null` marks a page with operational facts that nobody has + verified yet; the page says "Facts not verified yet". +- Give new pages with operational facts the key too, with `null` until you + have verified them. +- Editing a page doesn't change its date: only verifying it does. + ## Contributing Contributions are welcome! See the organisation's [contributing diff --git a/docs/about.md b/docs/about.md index 4602863..f4b716b 100644 --- a/docs/about.md +++ b/docs/about.md @@ -1,4 +1,5 @@ --- +last_verified: 2026-11-17 hide: - feedback --- diff --git a/docs/operators/cloud-registry.md b/docs/operators/cloud-registry.md index f364cfd..2d4cda9 100644 --- a/docs/operators/cloud-registry.md +++ b/docs/operators/cloud-registry.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Cloud Registry ## Synopsis @@ -1755,6 +1759,14 @@ deployment][cr-ui-updating]). Pre-releases can need the front end to be upgraded with the registry even within a minor version, as 1.0.0-rc2 needed 1.0.0rc2: follow the release notes of both. +Finally, check the registry's own entry among its services, as for any +other service (see [After a deployment or an +upgrade](using-cloud-registry.md#after-a-deployment-or-an-upgrade)): while +the registry is listed (see [Listing the registry +itself](#listing-the-registry-itself)), the app brings the entry up to date +when it starts, so it must show the new version. An entry of the registry +that was registered like other services instead has to be edited by hand. + Rolling back from 1.0.0rc2 to an earlier version needs the database of that version, as it was (see [Upgrading from 1.0.0rc1](#upgrading-from-100rc1)). Don't roll back to an image older than 0.4.0 (e.g., that of the previous diff --git a/docs/operators/data-centre-onboarding-tutorial.md b/docs/operators/data-centre-onboarding-tutorial.md index 080e866..46e737f 100644 --- a/docs/operators/data-centre-onboarding-tutorial.md +++ b/docs/operators/data-centre-onboarding-tutorial.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Tutorial: Onboard a Data Centre ## Synopsis @@ -309,6 +313,15 @@ Register your tools and workflows from a trusted client, as described in [Registering tools][trsfiler-registering-tools] in the TRS-Filer docs. Researchers then find them through the TRS API of your service. +## 7. Keep the registry entry up to date + +End every upgrade, move or change of TRS-Filer, e.g., with `helm upgrade`, +with this step: **update the service's Cloud Registry entry, or check that +sync did.** With sync on, check that the entry shows the new version, +labelled `synced`; after a new URL or API type, ask the +[admins][elixir-on-cloud-email] to verify it again. See [After a deployment +or an upgrade](using-cloud-registry.md#after-a-deployment-or-an-upgrade). + ## Planned - **Writes with tokens** in TRS-Filer ([trs-filer#7][trsfiler-7]), which diff --git a/docs/operators/index.md b/docs/operators/index.md index 39652f2..b7df9a5 100644 --- a/docs/operators/index.md +++ b/docs/operators/index.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Architects & Operators: Start Here This section is primarily for architects, who decide whether to adopt GA4GH diff --git a/docs/operators/ls-login.md b/docs/operators/ls-login.md index 75869ab..34165c8 100644 --- a/docs/operators/ls-login.md +++ b/docs/operators/ls-login.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # LS Login [LS Login](https://lifescience-ri.eu/ls-login/) (Life Science Login) is the diff --git a/docs/operators/node-onboarding-tutorial.md b/docs/operators/node-onboarding-tutorial.md index 974f413..9dc54d5 100644 --- a/docs/operators/node-onboarding-tutorial.md +++ b/docs/operators/node-onboarding-tutorial.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Tutorial: Onboard a Node ## Synopsis @@ -453,7 +457,27 @@ TESK, as described in [Deployment][sidecar-deployment]: The central Pubgrade service of the ELIXIR On-Cloud isn't deployed yet ([generic-issues#32][gi-32]), so nothing notifies the sidecar so far. Until then, upgrade TESK by hand, following the [upgrade - notes][tesk-upgrading]. + notes][tesk-upgrading], and end with [step + 9](#9-keep-the-registry-entry-up-to-date). + +## 9. Keep the registry entry up to date + +End every upgrade, move or change of your TES service, by hand or by +Pubgrade-Sidecar, with this step: **update the service's Cloud Registry +entry, or check that sync did.** Others find your service through its +entry, so it must follow your service: + +- With sync on (TESK): check that the entry shows the new version, labelled + `synced`. To sync at once, rather than waiting for the next sync, use + `PUT /services//sync` of the registry. +- Without sync (Poiesis and Funnel, registered by hand): edit the entry + so that it matches the service again. +- After a new URL or API type: ask the + [admins][elixir-on-cloud-email] to verify the entry again. + +See [After a deployment or an +upgrade](using-cloud-registry.md#after-a-deployment-or-an-upgrade) for +the details. ## Data centres diff --git a/docs/operators/onboarding.md b/docs/operators/onboarding.md index b851f92..03baade 100644 --- a/docs/operators/onboarding.md +++ b/docs/operators/onboarding.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Onboarding Follow these steps to connect your Node to the ELIXIR On-Cloud federation. @@ -104,7 +108,16 @@ For what the admins check before they verify an entry, and who uses the verified entries, see [What the admins check](using-cloud-registry.md#what-the-admins-check). -## 4. Next steps +## 4. Keep the registry entry up to date + +Each time you upgrade, move or change your TES service, end with this step: +**update the service's Cloud Registry entry, or check that sync did** (see +[After a deployment or an +upgrade](using-cloud-registry.md#after-a-deployment-or-an-upgrade)). Others +find your service through its entry, so it must follow your service. + +## 5. Next steps -- Deploy more [services](services/index.md). +- Deploy more [services](services/index.md), and end each deployment with + the same step. - Connect your own services to [LS Login](ls-login.md). diff --git a/docs/operators/service-info.md b/docs/operators/service-info.md index ffe92d5..b22cac2 100644 --- a/docs/operators/service-info.md +++ b/docs/operators/service-info.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Prepare Your Service Info ## Synopsis diff --git a/docs/operators/services/drsfiler.md b/docs/operators/services/drsfiler.md index 4990a24..1af332a 100644 --- a/docs/operators/services/drsfiler.md +++ b/docs/operators/services/drsfiler.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # DRS-Filer !!! warning "Being revived" diff --git a/docs/operators/services/funnel.md b/docs/operators/services/funnel.md index fecc517..2e204ff 100644 --- a/docs/operators/services/funnel.md +++ b/docs/operators/services/funnel.md @@ -1,3 +1,7 @@ +--- +last_verified: 2026-11-17 +--- + # Funnel [Funnel](https://github.com/calypr/funnel) is a [GA4GH TES][ga4gh-tes] @@ -8,7 +12,9 @@ PBS. Funnel is the TES service for Nodes with an HPC or HTC cluster. After you deploy it, register it in the [Cloud Registry][elixir-cloud-registry] (see -[Onboarding](../onboarding.md)). +[Onboarding](../onboarding.md)). End every upgrade or change of it by +updating its entry, which has no sync (see [After a deployment or an +upgrade](../using-cloud-registry.md#after-a-deployment-or-an-upgrade)). ## Docs diff --git a/docs/operators/services/index.md b/docs/operators/services/index.md index fd8498b..ec62e93 100644 --- a/docs/operators/services/index.md +++ b/docs/operators/services/index.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Services These are the services that a Node can run in the ELIXIR On-Cloud. Each diff --git a/docs/operators/services/poiesis.md b/docs/operators/services/poiesis.md index 2f83b9b..45fac55 100644 --- a/docs/operators/services/poiesis.md +++ b/docs/operators/services/poiesis.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Poiesis [Poiesis](https://github.com/JaeAeich/poiesis) is a [GA4GH TES][ga4gh-tes] @@ -7,7 +11,10 @@ service for Kubernetes. It runs each task in its own pod. Poiesis is a TES service for Nodes with a Kubernetes cluster, next to [TESK](tesk.md). After you deploy it, register it in the [Cloud -Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)). +Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)). End +every upgrade or change of it by updating its entry, which has no sync (see +[After a deployment or an +upgrade](../using-cloud-registry.md#after-a-deployment-or-an-upgrade)). ## Docs diff --git a/docs/operators/services/protes.md b/docs/operators/services/protes.md index 1437780..7c192df 100644 --- a/docs/operators/services/protes.md +++ b/docs/operators/services/protes.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # proTES [proTES](https://github.com/elixir-cloud-aai/proTES) is a gateway for @@ -12,7 +16,10 @@ to proTES, and proTES distributes them to the Nodes. It can read the verified TES services of a [Cloud Registry][elixir-cloud-registry], so that a new Node is used once it is registered and verified; the ELIXIR On-Cloud's proTES lists its TES services in its configuration instead -(see below). +(see below). Register a proTES that you deploy in the Cloud Registry like +any other TES service, and end every upgrade or change of it by updating +its entry, or checking that sync did (see [After a deployment or an +upgrade](../using-cloud-registry.md#after-a-deployment-or-an-upgrade)). ## Docs diff --git a/docs/operators/services/prowes.md b/docs/operators/services/prowes.md index 1acf136..dedb584 100644 --- a/docs/operators/services/prowes.md +++ b/docs/operators/services/prowes.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # proWES !!! warning "Being revived" diff --git a/docs/operators/services/tesk.md b/docs/operators/services/tesk.md index 73daeef..f1a6ce7 100644 --- a/docs/operators/services/tesk.md +++ b/docs/operators/services/tesk.md @@ -1,3 +1,7 @@ +--- +last_verified: 2026-11-17 +--- + # TESK [TESK](https://github.com/elixir-cloud-aai/TESK) is a [GA4GH TES][ga4gh-tes] @@ -7,12 +11,15 @@ service for Kubernetes. It runs each task as a Kubernetes Job. TESK is the TES service for Nodes with a Kubernetes or OpenShift cluster. After you deploy it, register it in the [Cloud -Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)). +Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)). End +every upgrade or change of it by checking that sync updated its entry (see +[After a deployment or an +upgrade](../using-cloud-registry.md#after-a-deployment-or-an-upgrade)). ## Docs - [TESK docs](https://tesk.readthedocs.io/) -- [Helm chart and its values](https://github.com/elixir-cloud-aai/TESK/tree/master/charts/tesk) +- [Helm chart and its values](https://github.com/elixir-cloud-aai/TESK/tree/dev/charts/tesk) ## Notes for the deployment @@ -30,4 +37,6 @@ Registry][elixir-cloud-registry] (see [Onboarding](../onboarding.md)). - For a step-by-step setup, see the [Node onboarding tutorial](../node-onboarding-tutorial.md). - To use the Node's S3 storage (see [Onboarding](../onboarding.md)), set the - chart's `storage` value to `s3` and follow the chart's README. + chart's `storage` value to `s3` and follow + [S3](https://tesk.readthedocs.io/en/latest/deployment/#s3) in the TESK + docs. diff --git a/docs/operators/services/trsfiler.md b/docs/operators/services/trsfiler.md index 3a5bdce..928881b 100644 --- a/docs/operators/services/trsfiler.md +++ b/docs/operators/services/trsfiler.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # TRS-Filer [TRS-Filer](https://github.com/elixir-cloud-aai/trs-filer) is a lightweight @@ -7,7 +11,11 @@ workflows. ## Role in the deployment TRS-Filer is the TRS implementation of the ELIXIR On-Cloud project. Use it -to offer tools and workflows through the TRS API. +to offer tools and workflows through the TRS API. After you deploy it, +register it in the [Cloud Registry][elixir-cloud-registry], and end every +upgrade or change of it by checking that sync updated its entry (see [After +a deployment or an +upgrade](../using-cloud-registry.md#after-a-deployment-or-an-upgrade)). ## Docs diff --git a/docs/operators/using-cloud-registry-ui.md b/docs/operators/using-cloud-registry-ui.md index cbfd738..0a0143e 100644 --- a/docs/operators/using-cloud-registry-ui.md +++ b/docs/operators/using-cloud-registry-ui.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Using the Cloud Registry UI ## Synopsis diff --git a/docs/operators/using-cloud-registry.md b/docs/operators/using-cloud-registry.md index 2d0f456..3577b77 100644 --- a/docs/operators/using-cloud-registry.md +++ b/docs/operators/using-cloud-registry.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Using the Cloud Registry ## Synopsis @@ -184,6 +188,27 @@ The [user guide of the UI][cr-ui-user-guide] covers registering, editing and sharing services step by step; [Using the Cloud Registry UI](using-cloud-registry-ui.md) adds what applies to this deployment. +### After a deployment or an upgrade + +The registry is the source of truth for the services of the deployment, and +these docs list services from it. It stays right only if every change of a +service reaches its entry. **End every deployment, upgrade or move of a +service with this step: update its entry, or check that sync did.** + +- **With sync on:** once the new version answers, sync the entry at once + (`PUT /services//sync`) or wait for the next sync. Then check that + the entry shows the new values, e.g., the version, with the label + `synced`. If it shows `out-of-sync`, see [Out-of-sync + reasons](#out-of-sync-reasons). +- **Without sync** (a manual entry, e.g., of Funnel or Poiesis): edit the + entry by hand, so that it matches the service again. +- **A new URL or API type:** with sync on, the URL can't change (see + [Labels](#labels)), and a new API type needs a new entry. Either change + removes the verification: ask the + [admins][elixir-on-cloud-email] to verify the entry again. +- **A service that you take down:** delete its entry, or tell the + [admins][elixir-on-cloud-email]. + [cr-access-control]: [cr-admin-mode]: [cr-api]: diff --git a/docs/researchers/index.md b/docs/researchers/index.md index 9b3e307..002347a 100644 --- a/docs/researchers/index.md +++ b/docs/researchers/index.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Researchers: Start Here This section is for researchers who want to try the federated analytics diff --git a/overrides/partials/content.html b/overrides/partials/content.html new file mode 100644 index 0000000..a1bf2f1 --- /dev/null +++ b/overrides/partials/content.html @@ -0,0 +1,30 @@ +{#- + Copy of the partial of Material for MkDocs, extended: pages with + `last_verified` in their front matter show when their operational facts + were last verified against their sources (a date), or that they haven't + been yet (`null`); see "Last verified dates" in `README.md` +-#} +{% include "partials/tags.html" %} +{% include "partials/actions.html" %} +{% if "{{ page.title | d(config.site_name, true)}} +{% endif %} +{{ page.content }} +{% if page.meta and "last_verified" in page.meta %} + +{% endif %} +{% include "partials/source-file.html" %} +{% include "partials/feedback.html" %} +{% include "partials/comments.html" %} diff --git a/tests/integration/test_last_verified.py b/tests/integration/test_last_verified.py new file mode 100644 index 0000000..8b23845 --- /dev/null +++ b/tests/integration/test_last_verified.py @@ -0,0 +1,84 @@ +"""Builds a small site with the theme overrides and checks the "last verified" note. + +Pages with operational facts carry `last_verified` in their front matter: a +date, or `null` until someone has verified them. The override of the +`content` partial in `overrides/partials/` shows it at the end of the page. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from mkdocs.commands.build import build +from mkdocs.config import load_config + +ROOT = Path(__file__).parent.parent.parent +OVERRIDES = ROOT / "overrides" + +PAGES = { + "verified.md": "---\nlast_verified: 2026-11-17\n---\n\n# Verified\n", + "unverified.md": "---\nlast_verified: null\n---\n\n# Unverified\n", + "other.md": "# Other\n", +} + + +@pytest.fixture(scope="module") +def site(tmp_path_factory: pytest.TempPathFactory) -> Path: + """Builds a site with one page of each kind. + + Args: + tmp_path_factory: Factory of temporary directories. + + Returns: + The directory of the built site. + """ + tmp_path = tmp_path_factory.mktemp("site") + docs = tmp_path / "docs" + docs.mkdir() + for name, text in PAGES.items(): + (docs / name).write_text(text, encoding="utf-8") + config = tmp_path / "mkdocs.yml" + config.write_text( + "site_name: Test\n" + f"theme: {{name: material, custom_dir: '{OVERRIDES}'}}\n", + encoding="utf-8", + ) + cfg = load_config(str(config), strict=True) + build(cfg) + return Path(cfg.site_dir) + + +def read_page(site: Path, name: str) -> str: + """Reads a built page. + + Args: + site: The directory of the built site. + name: The name of the page, without `.md`. + + Returns: + The HTML of the page. + """ + return (site / name / "index.html").read_text(encoding="utf-8") + + +def test_shows_date(site: Path) -> None: + """Shows the date of a verified page, as a machine-readable `time`.""" + html = read_page(site, "verified") + assert 'class="md-source-file last-verified"' in html + assert '' in html + assert "not verified yet" not in html + + +def test_shows_not_verified(site: Path) -> None: + """Says so on a page whose facts haven't been verified yet.""" + html = read_page(site, "unverified") + assert 'class="md-source-file last-verified"' in html + assert "Facts not verified yet" in html + assert " None: + """Adds nothing to pages without `last_verified`.""" + html = read_page(site, "other") + assert "last-verified" not in html From 617a3c1fef9539348d53f6afd4d5f456c003c8ee Mon Sep 17 00:00:00 2001 From: Alex Kanitz Date: Tue, 17 Nov 2026 10:35:39 +0100 Subject: [PATCH 2/4] docs: check verified dates and cover more pages Address the Sourcery review: a hook fails the build on malformed `last_verified` values (anything but an unquoted date or null), with unit and build tests. The researchers' guides, the API guide and the demos index get `last_verified: null`. The README says that a pull request setting a date verifies the page as on `main`, and the registry section says to delete an entry replaced after an API type change. Co-Authored-By: Claude Opus 5.5 --- README.md | 7 ++- docs/demos/index.md | 4 ++ docs/operators/using-cloud-registry.md | 4 +- docs/partners-and-developers/apis.md | 4 ++ docs/researchers/cli.md | 4 ++ docs/researchers/crypt4gh-with-protes.md | 4 ++ docs/researchers/federated-workflows.md | 4 ++ docs/researchers/krini.md | 4 ++ docs/researchers/tutorial-cli.md | 4 ++ docs/researchers/tutorial-krini.md | 4 ++ .../tes-services/interactive_services_tes.md | 4 ++ hooks/last_verified.py | 44 ++++++++++++++ mkdocs.yml | 6 +- pyproject.toml | 6 +- ...erified.py => test_last_verified_build.py} | 39 ++++++++++--- tests/unit/test_last_verified.py | 58 +++++++++++++++++++ 16 files changed, 185 insertions(+), 15 deletions(-) create mode 100644 hooks/last_verified.py rename tests/integration/{test_last_verified.py => test_last_verified_build.py} (72%) create mode 100644 tests/unit/test_last_verified.py diff --git a/README.md b/README.md index 85365f6..ca27a6d 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,12 @@ last_verified: 2026-11-17 verified yet; the page says "Facts not verified yet". - Give new pages with operational facts the key too, with `null` until you have verified them. -- Editing a page doesn't change its date: only verifying it does. +- Editing a page doesn't change its date, but check every operational fact + that you add or change against its source, as for any change of the + docs. A pull request that sets a date verifies the page as it is on + `main` when it merges: update its branch first, and check what changed. +- Anything else than an unquoted date or `null` fails the build (hook + [`hooks/last_verified.py`](hooks/last_verified.py)). ## Contributing diff --git a/docs/demos/index.md b/docs/demos/index.md index 1857006..da6c50c 100644 --- a/docs/demos/index.md +++ b/docs/demos/index.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Demos: Start Here This section lists the demos and test beds of ELIXIR On-Cloud: what each one diff --git a/docs/operators/using-cloud-registry.md b/docs/operators/using-cloud-registry.md index 3577b77..91816b4 100644 --- a/docs/operators/using-cloud-registry.md +++ b/docs/operators/using-cloud-registry.md @@ -205,7 +205,9 @@ service with this step: update its entry, or check that sync did.** - **A new URL or API type:** with sync on, the URL can't change (see [Labels](#labels)), and a new API type needs a new entry. Either change removes the verification: ask the - [admins][elixir-on-cloud-email] to verify the entry again. + [admins][elixir-on-cloud-email] to verify the entry again. If you + registered a new entry, delete the old one, so that it isn't listed next + to its replacement. - **A service that you take down:** delete its entry, or tell the [admins][elixir-on-cloud-email]. diff --git a/docs/partners-and-developers/apis.md b/docs/partners-and-developers/apis.md index 495b9fb..d570556 100644 --- a/docs/partners-and-developers/apis.md +++ b/docs/partners-and-developers/apis.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Using the APIs This page shows how to send a request to a service of the ELIXIR On-Cloud diff --git a/docs/researchers/cli.md b/docs/researchers/cli.md index b20c244..9531a42 100644 --- a/docs/researchers/cli.md +++ b/docs/researchers/cli.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Use the Command Line This page shows how to use the services of ELIXIR On-Cloud from your terminal diff --git a/docs/researchers/crypt4gh-with-protes.md b/docs/researchers/crypt4gh-with-protes.md index e9d6dde..53d0ad3 100644 --- a/docs/researchers/crypt4gh-with-protes.md +++ b/docs/researchers/crypt4gh-with-protes.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Crypt4GH with proTES This page shows how researchers can analyse Crypt4GH-encrypted sensitive diff --git a/docs/researchers/federated-workflows.md b/docs/researchers/federated-workflows.md index c32d62c..c6cffd4 100644 --- a/docs/researchers/federated-workflows.md +++ b/docs/researchers/federated-workflows.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Run Workflows Across Nodes This page shows how to run a Snakemake workflow on the TES services of diff --git a/docs/researchers/krini.md b/docs/researchers/krini.md index 82fe9ab..7d85488 100644 --- a/docs/researchers/krini.md +++ b/docs/researchers/krini.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Use Krini [Krini][krini-repo] is the web portal of ELIXIR On-Cloud. It replaces the TES diff --git a/docs/researchers/tutorial-cli.md b/docs/researchers/tutorial-cli.md index b909297..31221ee 100644 --- a/docs/researchers/tutorial-cli.md +++ b/docs/researchers/tutorial-cli.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Tutorial: Your First Run from the Command Line ## Synopsis diff --git a/docs/researchers/tutorial-krini.md b/docs/researchers/tutorial-krini.md index 06ea978..23222fc 100644 --- a/docs/researchers/tutorial-krini.md +++ b/docs/researchers/tutorial-krini.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Tutorial: Your First Run in Krini ## Synopsis diff --git a/docs/tutorials/tes-services/interactive_services_tes.md b/docs/tutorials/tes-services/interactive_services_tes.md index 2000f45..a63a1bc 100644 --- a/docs/tutorials/tes-services/interactive_services_tes.md +++ b/docs/tutorials/tes-services/interactive_services_tes.md @@ -1,3 +1,7 @@ +--- +last_verified: null +--- + # Run Interactive Services on TES This page shows how to run interactive web services, such as RStudio Server, diff --git a/hooks/last_verified.py b/hooks/last_verified.py new file mode 100644 index 0000000..e090556 --- /dev/null +++ b/hooks/last_verified.py @@ -0,0 +1,44 @@ +"""MkDocs hook that checks the `last_verified` dates of the fact pages. + +Pages with operational facts carry `last_verified` in their front matter: the +date when someone last verified those facts against their sources +(`YYYY-MM-DD`, which YAML reads as a date), or `null` until someone has. The +page shows it at its end (`overrides/partials/content.html`). Any other value +fails the build, so that no page shows a malformed date as verified. +""" + +from __future__ import annotations + +import datetime +from typing import Any + +from mkdocs.exceptions import PluginError +from mkdocs.structure.pages import Page + + +def on_page_markdown(markdown: str, *, page: Page, **kwargs: Any) -> str: + """Fails the build if `last_verified` is neither a date nor `null`. + + Args: + markdown: The Markdown of the page, returned unchanged. + page: The page, with its front matter in `meta`. + **kwargs: The other arguments of the event, unused. + + Returns: + The Markdown of the page. + + Raises: + PluginError: If `last_verified` is set to anything but a date + (without time) or `null`. + """ + if "last_verified" not in page.meta: + return markdown + value = page.meta["last_verified"] + if value is None or ( + isinstance(value, datetime.date) and not isinstance(value, datetime.datetime) + ): + return markdown + raise PluginError( + f"last_verified hook: {page.file.src_uri}: `last_verified` must be a " + f"date (YYYY-MM-DD, unquoted) or null, not {value!r}" + ) diff --git a/mkdocs.yml b/mkdocs.yml index 9317b09..7baaec5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -133,10 +133,12 @@ nav: - About: "about.md" extra_css: - stylesheets/extra.css -# Replaces `` markers with tables of the -# services in the Cloud Registry (see `README.md`) +# `registry.py` replaces `` markers with tables +# of the services in the Cloud Registry; `last_verified.py` fails the build +# on malformed `last_verified` dates of fact pages (see `README.md`) hooks: - hooks/registry.py + - hooks/last_verified.py plugins: - search repo_name: elixir-cloud-aai/docs diff --git a/pyproject.toml b/pyproject.toml index 856307d..d558758 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,8 +25,8 @@ docs = [ "pymdown-extensions>=12.3,<14", ] -# Tests of `scripts/check_names.py` and `hooks/registry.py` (the hook's tests -# need the `docs` dependencies too) +# Tests of `scripts/check_names.py` and the hooks in `hooks/` (the hooks' +# tests need the `docs` dependencies too) tests = [ "pytest>=8,<10", "pytest-cov>=6,<8", @@ -39,4 +39,4 @@ packages = [] [tool.pytest.ini_options] testpaths = ["tests"] pythonpath = ["scripts", "hooks"] -addopts = "--cov=check_names --cov=registry --cov-branch --cov-report=term-missing --cov-fail-under=100" +addopts = "--cov=check_names --cov=registry --cov=last_verified --cov-branch --cov-report=term-missing --cov-fail-under=100" diff --git a/tests/integration/test_last_verified.py b/tests/integration/test_last_verified_build.py similarity index 72% rename from tests/integration/test_last_verified.py rename to tests/integration/test_last_verified_build.py index 8b23845..12b7789 100644 --- a/tests/integration/test_last_verified.py +++ b/tests/integration/test_last_verified_build.py @@ -2,7 +2,8 @@ Pages with operational facts carry `last_verified` in their front matter: a date, or `null` until someone has verified them. The override of the -`content` partial in `overrides/partials/` shows it at the end of the page. +`content` partial in `overrides/partials/` shows it at the end of the page; +the hook `hooks/last_verified.py` fails the build on any other value. """ from __future__ import annotations @@ -12,9 +13,11 @@ import pytest from mkdocs.commands.build import build from mkdocs.config import load_config +from mkdocs.exceptions import Abort ROOT = Path(__file__).parent.parent.parent OVERRIDES = ROOT / "overrides" +HOOK = ROOT / "hooks" / "last_verified.py" PAGES = { "verified.md": "---\nlast_verified: 2026-11-17\n---\n\n# Verified\n", @@ -23,25 +26,25 @@ } -@pytest.fixture(scope="module") -def site(tmp_path_factory: pytest.TempPathFactory) -> Path: - """Builds a site with one page of each kind. +def build_site(tmp_path: Path, pages: dict[str, str]) -> Path: + """Builds a site with the overrides and the hook, strictly. Args: - tmp_path_factory: Factory of temporary directories. + tmp_path: Directory to write to. + pages: The Markdown of each page, by file name. Returns: The directory of the built site. """ - tmp_path = tmp_path_factory.mktemp("site") docs = tmp_path / "docs" docs.mkdir() - for name, text in PAGES.items(): + for name, text in pages.items(): (docs / name).write_text(text, encoding="utf-8") config = tmp_path / "mkdocs.yml" config.write_text( "site_name: Test\n" - f"theme: {{name: material, custom_dir: '{OVERRIDES}'}}\n", + f"theme: {{name: material, custom_dir: '{OVERRIDES}'}}\n" + f"hooks: ['{HOOK}']\n", encoding="utf-8", ) cfg = load_config(str(config), strict=True) @@ -49,6 +52,19 @@ def site(tmp_path_factory: pytest.TempPathFactory) -> Path: return Path(cfg.site_dir) +@pytest.fixture(scope="module") +def site(tmp_path_factory: pytest.TempPathFactory) -> Path: + """Builds a site with one page of each kind. + + Args: + tmp_path_factory: Factory of temporary directories. + + Returns: + The directory of the built site. + """ + return build_site(tmp_path_factory.mktemp("site"), PAGES) + + def read_page(site: Path, name: str) -> str: """Reads a built page. @@ -82,3 +98,10 @@ def test_leaves_other_pages_alone(site: Path) -> None: """Adds nothing to pages without `last_verified`.""" html = read_page(site, "other") assert "last-verified" not in html + + +@pytest.mark.parametrize("value", ["'2026-11-17'", "17.11.2026", "never"]) +def test_fails_on_malformed_date(tmp_path: Path, value: str) -> None: + """Fails the build for a value that isn't a date or `null`.""" + with pytest.raises(Abort): + build_site(tmp_path, {"bad.md": f"---\nlast_verified: {value}\n---\n\n# Bad\n"}) diff --git a/tests/unit/test_last_verified.py b/tests/unit/test_last_verified.py new file mode 100644 index 0000000..480d93b --- /dev/null +++ b/tests/unit/test_last_verified.py @@ -0,0 +1,58 @@ +"""Tests of the hook `hooks/last_verified.py`.""" + +from __future__ import annotations + +import datetime +from typing import Any +from unittest.mock import MagicMock + +import pytest +from last_verified import on_page_markdown +from mkdocs.exceptions import PluginError + + +def make_page(meta: dict[str, Any]) -> MagicMock: + """Builds a page with the given front matter. + + Args: + meta: The front matter of the page. + + Returns: + A stand-in for an MkDocs page. + """ + page = MagicMock() + page.meta = meta + page.file.src_uri = "operators/page.md" + return page + + +@pytest.mark.parametrize( + "meta", + [ + {}, + {"title": "Other"}, + {"last_verified": None}, + {"last_verified": datetime.date(2026, 11, 17)}, + ], +) +def test_accepts_valid_values(meta: dict[str, Any]) -> None: + """Leaves the Markdown of pages without the key or with valid values.""" + assert on_page_markdown("# Page\n", page=make_page(meta)) == "# Page\n" + + +@pytest.mark.parametrize( + "value", + [ + "2026-11-17", + "17.11.2026", + "never", + "", + False, + 20261117, + datetime.datetime(2026, 11, 17, 12, 0), + ], +) +def test_refuses_malformed_values(value: object) -> None: + """Fails the build for anything but a date or `null`, naming the page.""" + with pytest.raises(PluginError, match="operators/page.md"): + on_page_markdown("# Page\n", page=make_page({"last_verified": value})) From 167cb995fb061c096e7cd9ad49c3d3152c9a5a80 Mon Sep 17 00:00:00 2001 From: Alex Kanitz Date: Tue, 17 Nov 2026 10:42:23 +0100 Subject: [PATCH 3/4] fix: catch unreadable verified dates An impossible date such as 2026-13-01 makes YAML fail, and MkDocs then drops the whole front matter silently; the hook now fails the build when unread front matter sets `last_verified`. Build tests cover the values YAML reads differently (datetimes, unpadded dates) and check the hook's message. Also from the review: the partial names the Material version it was copied from, the README lists the pages that carry the key and says it is a convention, and the registry section tells a new API version (synced into the same entry) from another API (a new entry). Co-Authored-By: Claude Opus 5.5 --- README.md | 9 +++++-- docs/operators/using-cloud-registry.md | 10 +++++--- hooks/last_verified.py | 25 ++++++++++++++++--- overrides/partials/content.html | 6 +++-- tests/integration/test_last_verified_build.py | 20 ++++++++++++--- tests/unit/test_last_verified.py | 19 ++++++++++++++ 6 files changed, 75 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index ca27a6d..694656c 100644 --- a/README.md +++ b/README.md @@ -113,14 +113,19 @@ last_verified: 2026-11-17 same pull request. - `last_verified: null` marks a page with operational facts that nobody has verified yet; the page says "Facts not verified yet". -- Give new pages with operational facts the key too, with `null` until you - have verified them. +- Which pages carry the key: the operator pages (`docs/operators/`), the + researchers' start page and guides, the API guide + (`partners-and-developers/apis.md`), the demos index and About. Give new + pages with operational facts the key too, with `null` until you have + verified them. This is a convention: the build doesn't require the key. - Editing a page doesn't change its date, but check every operational fact that you add or change against its source, as for any change of the docs. A pull request that sets a date verifies the page as it is on `main` when it merges: update its branch first, and check what changed. - Anything else than an unquoted date or `null` fails the build (hook [`hooks/last_verified.py`](hooks/last_verified.py)). +- The partial is a copy of Material's: when you upgrade Material, compare + it with the new release's `partials/content.html`. ## Contributing diff --git a/docs/operators/using-cloud-registry.md b/docs/operators/using-cloud-registry.md index 91816b4..175535f 100644 --- a/docs/operators/using-cloud-registry.md +++ b/docs/operators/using-cloud-registry.md @@ -202,9 +202,13 @@ service with this step: update its entry, or check that sync did.** reasons](#out-of-sync-reasons). - **Without sync** (a manual entry, e.g., of Funnel or Poiesis): edit the entry by hand, so that it matches the service again. -- **A new URL or API type:** with sync on, the URL can't change (see - [Labels](#labels)), and a new API type needs a new entry. Either change - removes the verification: ask the +- **A new version of the API** (`type.version`): sync applies it to the + same entry, after an admin has approved it if it is new (see [Out-of-sync + reasons](#out-of-sync-reasons)). Ask for a new version in time. +- **A new URL or another API** (`type.group` or `type.artifact`): with sync + on, the URL can't change: turn sync off first, or import the service + anew (see [Labels](#labels)). Another API needs a new entry. Either + change removes the verification: ask the [admins][elixir-on-cloud-email] to verify the entry again. If you registered a new entry, delete the old one, so that it isn't listed next to its replacement. diff --git a/hooks/last_verified.py b/hooks/last_verified.py index e090556..0b9195a 100644 --- a/hooks/last_verified.py +++ b/hooks/last_verified.py @@ -5,22 +5,31 @@ (`YYYY-MM-DD`, which YAML reads as a date), or `null` until someone has. The page shows it at its end (`overrides/partials/content.html`). Any other value fails the build, so that no page shows a malformed date as verified. + +An impossible date, such as `2026-13-01`, makes YAML fail, and MkDocs then +drops the whole front matter of the page without a warning and leaves it in +the Markdown. The hook catches that case too. """ from __future__ import annotations import datetime +import re from typing import Any from mkdocs.exceptions import PluginError from mkdocs.structure.pages import Page +from mkdocs.utils.meta import YAML_RE + +KEY_RE = re.compile(r"^last_verified\s*:", re.MULTILINE) def on_page_markdown(markdown: str, *, page: Page, **kwargs: Any) -> str: """Fails the build if `last_verified` is neither a date nor `null`. Args: - markdown: The Markdown of the page, returned unchanged. + markdown: The Markdown of the page, returned unchanged. It still + starts with the front matter if MkDocs couldn't read it. page: The page, with its front matter in `meta`. **kwargs: The other arguments of the event, unused. @@ -29,9 +38,17 @@ def on_page_markdown(markdown: str, *, page: Page, **kwargs: Any) -> str: Raises: PluginError: If `last_verified` is set to anything but a date - (without time) or `null`. + (without time) or `null`, or the front matter that sets it can't + be read. """ + where = f"last_verified hook: {page.file.src_uri}" if "last_verified" not in page.meta: + front_matter = YAML_RE.match(markdown) + if front_matter and KEY_RE.search(front_matter.group(1)): + raise PluginError( + f"{where}: the front matter can't be read, e.g., because of an " + "impossible date in `last_verified`" + ) return markdown value = page.meta["last_verified"] if value is None or ( @@ -39,6 +56,6 @@ def on_page_markdown(markdown: str, *, page: Page, **kwargs: Any) -> str: ): return markdown raise PluginError( - f"last_verified hook: {page.file.src_uri}: `last_verified` must be a " - f"date (YYYY-MM-DD, unquoted) or null, not {value!r}" + f"{where}: `last_verified` must be a date (YYYY-MM-DD, unquoted) or " + f"null, not {value!r}" ) diff --git a/overrides/partials/content.html b/overrides/partials/content.html index a1bf2f1..a1d209c 100644 --- a/overrides/partials/content.html +++ b/overrides/partials/content.html @@ -1,8 +1,10 @@ {#- - Copy of the partial of Material for MkDocs, extended: pages with + Copy of the partial of Material for MkDocs 9.8.1, extended: pages with `last_verified` in their front matter show when their operational facts were last verified against their sources (a date), or that they haven't - been yet (`null`); see "Last verified dates" in `README.md` + been yet (`null`); see "Last verified dates" in `README.md`. Compare it + with the partial of each new Material release. The class `last-verified` + is a hook for styles and tests -#} {% include "partials/tags.html" %} {% include "partials/actions.html" %} diff --git a/tests/integration/test_last_verified_build.py b/tests/integration/test_last_verified_build.py index 12b7789..7cab87c 100644 --- a/tests/integration/test_last_verified_build.py +++ b/tests/integration/test_last_verified_build.py @@ -100,8 +100,22 @@ def test_leaves_other_pages_alone(site: Path) -> None: assert "last-verified" not in html -@pytest.mark.parametrize("value", ["'2026-11-17'", "17.11.2026", "never"]) -def test_fails_on_malformed_date(tmp_path: Path, value: str) -> None: - """Fails the build for a value that isn't a date or `null`.""" +@pytest.mark.parametrize( + "value", + [ + "'2026-11-17'", + "17.11.2026", + "never", + "2026-13-01", + "2026-11-17T10:00:00Z", + "2026-11-17 10:00", + "2026-1-5", + ], +) +def test_fails_on_malformed_date( + tmp_path: Path, value: str, caplog: pytest.LogCaptureFixture +) -> None: + """Fails the build for a value that isn't a date or `null`, as YAML reads it.""" with pytest.raises(Abort): build_site(tmp_path, {"bad.md": f"---\nlast_verified: {value}\n---\n\n# Bad\n"}) + assert "last_verified hook: bad.md" in caplog.text diff --git a/tests/unit/test_last_verified.py b/tests/unit/test_last_verified.py index 480d93b..80b62ad 100644 --- a/tests/unit/test_last_verified.py +++ b/tests/unit/test_last_verified.py @@ -56,3 +56,22 @@ def test_refuses_malformed_values(value: object) -> None: """Fails the build for anything but a date or `null`, naming the page.""" with pytest.raises(PluginError, match="operators/page.md"): on_page_markdown("# Page\n", page=make_page({"last_verified": value})) + + +def test_refuses_unreadable_front_matter() -> None: + """Fails the build if MkDocs dropped front matter that sets the key.""" + markdown = "---\nlast_verified: 2026-13-01\n---\n\n# Page\n" + with pytest.raises(PluginError, match="can't be read"): + on_page_markdown(markdown, page=make_page({})) + + +@pytest.mark.parametrize( + "markdown", + [ + "---\ntitle: x: y\n---\n\n# Page\n", + "# Page\n\nlast_verified: 2026-13-01\n", + ], +) +def test_ignores_other_unread_text(markdown: str) -> None: + """Leaves pages alone whose unread front matter or text has no key.""" + assert on_page_markdown(markdown, page=make_page({})) == markdown From df728a479736a79a09d7691f47ac9fc9745f47c2 Mon Sep 17 00:00:00 2001 From: Alex Kanitz Date: Tue, 17 Nov 2026 10:45:50 +0100 Subject: [PATCH 4/4] build: require the Material of the partial copy The copy of Material's `content` partial comes from 9.8.1, so require that version at least. Clarify that the `last-verified` class has no styles, and list the interactive services tutorial among the pages with the key. Co-Authored-By: Claude Opus 5.5 --- README.md | 3 ++- overrides/partials/content.html | 5 +++-- pyproject.toml | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 694656c..6008793 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,8 @@ last_verified: 2026-11-17 - `last_verified: null` marks a page with operational facts that nobody has verified yet; the page says "Facts not verified yet". - Which pages carry the key: the operator pages (`docs/operators/`), the - researchers' start page and guides, the API guide + researchers' start page and guides, the tutorial on interactive services + (`tutorials/tes-services/`), the API guide (`partners-and-developers/apis.md`), the demos index and About. Give new pages with operational facts the key too, with `null` until you have verified them. This is a convention: the build doesn't require the key. diff --git a/overrides/partials/content.html b/overrides/partials/content.html index a1d209c..a1643bf 100644 --- a/overrides/partials/content.html +++ b/overrides/partials/content.html @@ -3,8 +3,9 @@ `last_verified` in their front matter show when their operational facts were last verified against their sources (a date), or that they haven't been yet (`null`); see "Last verified dates" in `README.md`. Compare it - with the partial of each new Material release. The class `last-verified` - is a hook for styles and tests + with the partial of each new Material release (`pyproject.toml` requires + 9.8.1 or later). The class `last-verified` has no styles of its own; the + tests find the note by it -#} {% include "partials/tags.html" %} {% include "partials/actions.html" %} diff --git a/pyproject.toml b/pyproject.toml index d558758..c40119f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,7 +21,7 @@ requires-python = ">=3.11" # releases still fail it) docs = [ "mkdocs>=1.6.1,<2", - "mkdocs-material>=9.7.7,<10", + "mkdocs-material>=9.8.1,<10", "pymdown-extensions>=12.3,<14", ]