From de339f27769a47b30f70c25d06ee9f8bc93bea68 Mon Sep 17 00:00:00 2001 From: Simon Scatton Date: Thu, 15 Oct 2026 19:00:37 +0200 Subject: [PATCH 1/4] ci(release): promote qualified prereleases to stable Signed-off-by: Simon Scatton --- .agents/skills/release-openshell/SKILL.md | 41 ++++- .agents/skills/watch-github-actions/SKILL.md | 16 +- .github/workflows/branch-docs.yml | 1 + .github/workflows/release-build.yml | 10 ++ .github/workflows/release-publish.yml | 20 ++- .github/workflows/release-tag.yml | 72 --------- .github/workflows/release.yml | 128 ++++++++++++++++ CI.md | 39 ++++- scripts/baseline_workflow_metrics.py | 3 +- tasks/scripts/select-release-candidate.py | 152 +++++++++++++++++++ 10 files changed, 386 insertions(+), 96 deletions(-) delete mode 100644 .github/workflows/release-tag.yml create mode 100644 .github/workflows/release.yml create mode 100644 tasks/scripts/select-release-candidate.py diff --git a/.agents/skills/release-openshell/SKILL.md b/.agents/skills/release-openshell/SKILL.md index 2f641aea3c..2416aa6066 100644 --- a/.agents/skills/release-openshell/SKILL.md +++ b/.agents/skills/release-openshell/SKILL.md @@ -7,15 +7,40 @@ metadata: # Release OpenShell -For the intended release version, pick the latest prerelease whose Release -Qualification job passed. Confirm that its qualification summary matches the -tag and commit. Present the candidate and changes since the previous stable -release to the maintainer; once approved, tag that exact commit as `vX.Y.Z`. +For the intended release version and branch, inspect the latest passing +prerelease and changes since the previous stable release. Present the candidate +to the maintainer. Once approved, dispatch Release from that branch: + +```shell +gh workflow run release.yml --ref main -f tag=vX.Y.Z +``` + +Use the support branch instead of `main` for a previous minor version. Release +selects the highest-numbered `vX.Y.Z-pre.N` reachable from that branch with a +successful Pre-release run triggered by that tag or dispatched on the same branch, +and matching passing `release-tag-v1` qualification evidence. Missing or expired evidence blocks +promotion. Inspect the selected candidate in the promotion job summary. + +Release creates `vX.Y.Z` at that exact commit, then rebuilds and publishes all +stable artifacts without rerunning qualification. Embedded versions change, so +stable artifacts are not byte-identical to the qualified prerelease. An existing +stable tag may be retried only if it already points to the selected commit. If no prerelease exists, agree on the version and source commit, then create and -push `vX.Y.Z-pre.1`. Wait for qualification to pass before promoting it. +push `vX.Y.Z-pre.1`. Prerelease tag pushes start Pre-release. If the tag was +created using `GITHUB_TOKEN`, dispatch Pre-release explicitly on the intended branch: + +```shell +gh workflow run release-prerelease.yml --ref main -f tag=vX.Y.Z-pre.1 +``` + +Wait for qualification to pass before promoting it. Stable tag pushes do not +start Release. Release Auto-Tag creates and dispatches prereleases on `main`. + +Stable releases serialize across branches. A release older than the highest +stable Git tag publishes versioned artifacts without moving GHCR `latest`, the +GitHub latest release, npm `latest`, or snap `latest/stable`. Its npm package +uses `vX.Y`; snap files remain available in its GitHub Release. Docs retain an +immutable version and advance latest only for a newer version. -Prerelease tag pushes start Pre-release; stable tag pushes start Release. -Release Auto-Tag uses an explicit dispatch because its `GITHUB_TOKEN` pushes do -not trigger workflows. Watch the stable release run and report its result. diff --git a/.agents/skills/watch-github-actions/SKILL.md b/.agents/skills/watch-github-actions/SKILL.md index 6a5893a234..8901ac96d8 100644 --- a/.agents/skills/watch-github-actions/SKILL.md +++ b/.agents/skills/watch-github-actions/SKILL.md @@ -125,8 +125,9 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId ## View Job Logs -`Release Tag` temporarily omits Codex Security from its `Security Scan` job. -Other scanners and qualification gates still run. Use the standalone +`Pre-release` temporarily omits Codex Security from its `Security Scan` job. +`Release` uses the passing prerelease qualification result. Other scanners and +qualification gates still run. Use the standalone `Codex Security Release Qualification` workflow for manual investigation. For `Codex Security`, an exit code of 2 with partial coverage is an incomplete @@ -167,10 +168,13 @@ During `0.x`, a minor train permits compatibility findings as warnings; a patch train or no active train rejects them. Compare the current train's version with the latest stable release; commit messages are irrelevant. Compilation, baseline, and tool errors remain fatal. The `protobuf_compatibility` suite participates in -the `release-tag-v1` qualification profile. Release and Pre-release call the reusable -`release-qualification.yml` workflow to run the checks and collect their results. -Release builds, qualifies, then publishes. Pre-release builds, then publishes -and qualifies in parallel in the same run. Failed, cancelled, or skipped required suites fail +the `release-tag-v1` qualification profile. Pre-release calls the reusable +`release-qualification.yml` workflow after building, in parallel with publication. Release +selects the latest passing prerelease for the requested version and branch, +verifies its qualification evidence, creates the stable tag at its exact commit, +then rebuilds and publishes without rerunning qualification. The promotion job +summary and `stable-promotion-*` artifact record the candidate and evidence. +Failed, cancelled, or skipped required suites fail qualification; stable publication is blocked, while published pre-release artifacts remain available. The Pre-release failure notification includes the run link. Both callers share `release-build.yml` and `release-publish.yml`. diff --git a/.github/workflows/branch-docs.yml b/.github/workflows/branch-docs.yml index f42402743b..b6ad5efeea 100644 --- a/.github/workflows/branch-docs.yml +++ b/.github/workflows/branch-docs.yml @@ -12,6 +12,7 @@ on: - "tasks/scripts/check_docs_nav_test.py" - ".github/workflows/branch-docs.yml" - ".github/workflows/release-tag.yml" + - ".github/workflows/release.yml" - ".github/workflows/release-prerelease.yml" - ".github/workflows/release-publish.yml" diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml index 957d9130ec..2915770f4e 100644 --- a/.github/workflows/release-build.yml +++ b/.github/workflows/release-build.yml @@ -10,6 +10,11 @@ on: description: Git tag to build and publish required: true type: string + expected-source-sha: + description: Qualified source commit that the release tag must resolve to + required: false + default: "" + type: string is-prerelease: description: Whether the caller is publishing a pre-release required: true @@ -81,8 +86,13 @@ jobs: id: v env: EXPECTED_PRERELEASE: ${{ inputs.is-prerelease }} + EXPECTED_SOURCE_SHA: ${{ inputs.expected-source-sha }} run: | set -euo pipefail + if [[ -n "${EXPECTED_SOURCE_SHA}" && "$(git rev-parse HEAD)" != "${EXPECTED_SOURCE_SHA}" ]]; then + echo "Release tag ${RELEASE_TAG} does not match the qualified commit ${EXPECTED_SOURCE_SHA}." >&2 + exit 1 + fi if [[ "${RELEASE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[1-9][0-9]*$ ]]; then is_prerelease=true elif [[ "${RELEASE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml index 83302a8873..9c123e2534 100644 --- a/.github/workflows/release-publish.yml +++ b/.github/workflows/release-publish.yml @@ -14,6 +14,11 @@ on: description: Whether the caller is publishing a pre-release required: true type: boolean + make-latest: + description: Whether this stable release should update shared latest channels + required: false + default: false + type: boolean source-sha: description: Source commit of the built artifacts in the calling run required: true @@ -226,7 +231,7 @@ jobs: - name: Tag images with release version env: - IS_PRERELEASE: ${{ inputs.is-prerelease }} + MAKE_LATEST: ${{ !inputs.is-prerelease && inputs.make-latest }} SOURCE_TAG: ${{ inputs.source-sha }} VERSION: ${{ inputs.semver }} run: | @@ -238,7 +243,7 @@ jobs: --prefer-index=false \ -t "${REGISTRY}/${component}:${VERSION}" \ "${REGISTRY}/${component}:${SOURCE_TAG}" - if [[ "${IS_PRERELEASE}" != "true" ]]; then + if [[ "${MAKE_LATEST}" == "true" ]]; then echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as latest..." docker buildx imagetools create \ --prefer-index=false \ @@ -312,6 +317,7 @@ jobs: with: name: OpenShell ${{ env.RELEASE_TAG }} prerelease: false + make_latest: ${{ inputs.make-latest && 'true' || 'false' }} tag_name: ${{ env.RELEASE_TAG }} generate_release_notes: true body: | @@ -415,7 +421,7 @@ jobs: publish-snap: name: Publish Snap needs: release - if: ${{ !inputs.is-prerelease }} + if: ${{ !inputs.is-prerelease && inputs.make-latest }} uses: ./.github/workflows/snap-publish.yml with: upload-channel: latest/stable @@ -462,8 +468,14 @@ jobs: - name: Publish env: OPENSHELL_NPM_VERSION: ${{ inputs.npm-version }} + MAKE_LATEST: ${{ inputs.make-latest }} NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: mise run sdk:ts:publish + run: | + set -euo pipefail + if [[ "${MAKE_LATEST}" != "true" ]]; then + export OPENSHELL_NPM_PUBLISH_ARGS="--tag v${OPENSHELL_NPM_VERSION%.*}" + fi + mise run sdk:ts:publish trigger-wheel-publish: name: Trigger Wheel Publish diff --git a/.github/workflows/release-tag.yml b/.github/workflows/release-tag.yml deleted file mode 100644 index e3ac6f5947..0000000000 --- a/.github/workflows/release-tag.yml +++ /dev/null @@ -1,72 +0,0 @@ -name: Release - -on: - push: - tags: - - "v*.*.*" - - "!v*.*.*-pre.*" - workflow_dispatch: - inputs: - tag: - description: "Stable tag to release (e.g. v0.0.4)" - required: true - type: string - -permissions: - contents: read - -jobs: - build: - name: Build - permissions: - actions: read - contents: read - packages: write - uses: ./.github/workflows/release-build.yml - with: - release-tag: ${{ inputs.tag || github.ref_name }} - is-prerelease: false - secrets: - CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} - - qualify: - name: Qualify - needs: build - permissions: - actions: read - contents: read - packages: write - security-events: write - pull-requests: read - uses: ./.github/workflows/release-qualification.yml - with: - release-tag: ${{ inputs.tag || github.ref_name }} - source-sha: ${{ needs.build.outputs.source_sha }} - is-prerelease: false - secrets: - CODEX_SECURITY_API_KEY: ${{ secrets.CODEX_SECURITY_API_KEY }} - CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} - - publish: - name: Publish - needs: [build, qualify] - permissions: - actions: read - contents: write - packages: write - id-token: write - attestations: write - artifact-metadata: write - uses: ./.github/workflows/release-publish.yml - with: - release-tag: ${{ inputs.tag || github.ref_name }} - source-sha: ${{ needs.build.outputs.source_sha }} - semver: ${{ needs.build.outputs.semver }} - python-version: ${{ needs.build.outputs.python_version }} - npm-version: ${{ needs.build.outputs.npm_version }} - is-prerelease: false - secrets: - SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }} - FERN_TOKEN: ${{ secrets.FERN_TOKEN }} - GITLAB_CI_TRIGGER_TOKEN: ${{ secrets.GITLAB_CI_TRIGGER_TOKEN }} - GITLAB_CI_TRIGGER_URL: ${{ secrets.GITLAB_CI_TRIGGER_URL }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000000..7f4088f0ef --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Release + +on: + workflow_dispatch: + inputs: + tag: + description: "Stable version to promote from its latest passing prerelease (e.g. v0.0.4)" + required: true + type: string + +permissions: + contents: read + +concurrency: + group: release-stable + cancel-in-progress: false + queue: max + +jobs: + promote: + name: Select Qualified Prerelease + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + actions: read + contents: write + outputs: + tag: ${{ steps.candidate.outputs.tag }} + source-sha: ${{ steps.candidate.outputs.source_sha }} + make-latest: ${{ steps.candidate.outputs.make_latest }} + steps: + - name: Require a release branch + env: + REF_TYPE: ${{ github.ref_type }} + run: test "${REF_TYPE}" = branch + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + with: + version: "0.10.12" + python-version: "3.13" + + - name: Select candidate and verify qualification evidence + id: candidate + env: + GH_TOKEN: ${{ github.token }} + RELEASE_TAG: ${{ inputs.tag }} + RELEASE_BRANCH: ${{ github.ref_name }} + run: | + set -euo pipefail + uv run --no-project python tasks/scripts/select-release-candidate.py \ + "${RELEASE_TAG}" --branch "${RELEASE_BRANCH}" > promotion.json + jq -r 'to_entries[] | "\(.key)=\(.value)"' promotion.json >> "${GITHUB_OUTPUT}" + { + echo "## Stable release promotion" + jq -r '"- Prerelease: `\(.candidate_tag)`\n- Stable tag: `\(.tag)`\n- Source commit: `\(.source_sha)`\n- Qualification: \(.qualification_run_url)\n- Update shared latest channels: \(.make_latest)"' promotion.json + echo + echo "Stable artifacts are rebuilt from this qualified commit with the stable version." + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Retain promotion record + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: stable-promotion-${{ inputs.tag }}-${{ github.run_id }}-${{ github.run_attempt }} + path: promotion.json + retention-days: 90 + if-no-files-found: error + + - name: Create stable tag at the qualified commit + env: + GH_TOKEN: ${{ github.token }} + RELEASE_TAG: ${{ steps.candidate.outputs.tag }} + SOURCE_SHA: ${{ steps.candidate.outputs.source_sha }} + run: | + set -euo pipefail + if git show-ref --verify --quiet "refs/tags/${RELEASE_TAG}"; then + test "$(git rev-parse "refs/tags/${RELEASE_TAG}^{commit}")" = "${SOURCE_SHA}" + else + gh api --method POST "repos/${GITHUB_REPOSITORY}/git/refs" \ + -f "ref=refs/tags/${RELEASE_TAG}" -f "sha=${SOURCE_SHA}" + fi + + build: + name: Build + needs: promote + permissions: + actions: read + contents: read + packages: write + uses: ./.github/workflows/release-build.yml + with: + release-tag: ${{ needs.promote.outputs.tag }} + expected-source-sha: ${{ needs.promote.outputs.source-sha }} + is-prerelease: false + secrets: + CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} + + publish: + name: Publish + needs: [promote, build] + permissions: + actions: read + contents: write + packages: write + id-token: write + attestations: write + artifact-metadata: write + uses: ./.github/workflows/release-publish.yml + with: + release-tag: ${{ needs.promote.outputs.tag }} + source-sha: ${{ needs.build.outputs.source_sha }} + semver: ${{ needs.build.outputs.semver }} + python-version: ${{ needs.build.outputs.python_version }} + npm-version: ${{ needs.build.outputs.npm_version }} + is-prerelease: false + make-latest: ${{ needs.promote.outputs.make-latest == 'true' }} + secrets: + SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }} + FERN_TOKEN: ${{ secrets.FERN_TOKEN }} + GITLAB_CI_TRIGGER_TOKEN: ${{ secrets.GITLAB_CI_TRIGGER_TOKEN }} + GITLAB_CI_TRIGGER_URL: ${{ secrets.GITLAB_CI_TRIGGER_URL }} diff --git a/CI.md b/CI.md index b886bd70e3..2b04ed4c3d 100644 --- a/CI.md +++ b/CI.md @@ -19,7 +19,7 @@ Merge queue validation is a second integration gate for `main`. After a PR has p ### Protobuf API compatibility -`Protobuf Compatibility` runs in Branch Checks, Release, and Pre-release through the shared +`Protobuf Compatibility` runs in Branch Checks and Pre-release through the shared `check-protobuf-compatibility` action and Nix app. The app supplies Python, Buf, and Git from the flake lockfile. It uses Buf's `FILE` policy for the `proto/` @@ -322,9 +322,9 @@ jobs: ``` Set `needs: security` on a downstream promotion job to require successful scans. -`Release` and `Pre-release` share `release-build.yml`, `release-publish.yml`, and -`release-qualification.yml`. Release builds, qualifies, then publishes. Pre-release -builds, then publishes and qualifies in parallel within the same run, using native job dependencies. +`Release` and `Pre-release` share `release-build.yml` and `release-publish.yml`. +Pre-release builds, then publishes and calls `release-qualification.yml` in +parallel within the same run, using native job dependencies. Its platform package artifacts are uploaded after versioned images and the chart, so the installer can use them while qualification runs or after it fails. The implemented `release-tag-v1` profile is an incremental subset of RFC 0014 @@ -338,6 +338,35 @@ The summary contains qualification results and policy coverage only; artifact identity belongs in the release manifest. The run ID and attempt distinguish reruns without overwriting earlier evidence. +Dispatch Release from the intended release branch with a stable version: + +```shell +gh workflow run release.yml --ref main -f tag=v0.4.3 +``` + +For a support release, replace `main` with its support branch. Release chooses +the highest-numbered `v0.4.3-pre.N` reachable from the selected branch with a +successful Pre-release run triggered by the tag or dispatched on that branch. +It verifies the qualification summary's +tag, commit, profile, run, attempt, and passing result before creating `v0.4.3` +at the same commit. Missing or expired evidence blocks promotion. The promotion +summary and `stable-promotion-*` artifact retain the selected candidate and evidence link. +Successful reruns can retain passing evidence from an earlier attempt of the +same run; the promotion record links to the attempt that produced that evidence. +Release rebuilds all stable artifacts with the stable version and publishes +without rerunning qualification. These artifacts have different embedded versions +from the qualified prerelease. Existing stable tags are never moved; a retry must +resolve to the same qualified candidate. + +Release is manually dispatched; pushing a stable tag does not trigger publication. +Stable runs serialize across branches. A version older than the highest stable +Git tag publishes versioned artifacts while preserving shared latest channels: +GHCR `latest`, GitHub's latest release, npm `latest`, and snap `latest/stable`. +Its npm package uses a `vX.Y` dist-tag, and its snap files remain in the GitHub +Release. Docs publish an immutable version and already prevent latest from moving +backward. Release Auto-Tag continues on `main`; support branch prereleases can +use tag pushes or manual Pre-release dispatch from that branch. + `CACHIX_AUTH_TOKEN` is optional. The parent publishes SARIF from CodeQL, Trivy, and workflow reports against the candidate tag and commit. Existing standalone triggers stay active. @@ -580,7 +609,7 @@ These workflows run after merge to publish dev/tagged artifacts and verify them. | File | Role | |---|---| | `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | -| `.github/workflows/release-tag.yml` | Release orchestrator for stable tags: build, qualify, publish. Automatic tag triggers exclude `-pre.*`. | +| `.github/workflows/release.yml` | Manually promote the latest qualified prerelease for a version and branch, create its stable tag, then rebuild and publish. Support releases preserve shared latest channels. | | `.github/workflows/release-prerelease.yml` | Pre-release orchestrator, triggered by prerelease tag pushes or dispatched by Release Auto-Tag or manually: build, then publish and qualify in parallel. Qualification failure fails the run and sends the failure notification; published artifacts remain available. | | `.github/workflows/release-build.yml` | Shared version computation, binary and image builds, and package builds. Validates that the tag matches the caller's release kind. | | `.github/workflows/release-publish.yml` | Shared artifact assembly, OCI image tags and Helm chart publication, and package publication. Stable releases also publish to GitHub Releases, snap, npm, PyPI, and Fern. | diff --git a/scripts/baseline_workflow_metrics.py b/scripts/baseline_workflow_metrics.py index 02d1448f8d..8548900ee8 100755 --- a/scripts/baseline_workflow_metrics.py +++ b/scripts/baseline_workflow_metrics.py @@ -52,8 +52,9 @@ "package-release-binaries.yml", "release-canary.yml", "release-dev.yml", - "release-tag.yml", + "release-prerelease.yml", "release-vm-kernel.yml", + "release.yml", ] # Reusable workflows (workflow_call targets). The Actions API returns no runs diff --git a/tasks/scripts/select-release-candidate.py b/tasks/scripts/select-release-candidate.py new file mode 100644 index 0000000000..41daf90bb7 --- /dev/null +++ b/tasks/scripts/select-release-candidate.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 + +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Select a qualified prerelease reachable from the checked-out release branch.""" + +from __future__ import annotations + +import argparse +import io +import json +import os +import re +import subprocess +import zipfile +from urllib.parse import urlencode + + +def git(*args: str) -> str: + return subprocess.check_output(["git", *args], text=True).strip() + + +def api_pages(path: str, key: str) -> list[dict]: + pages = json.loads( + subprocess.check_output(["gh", "api", "--paginate", "--slurp", path]) + ) + return [item for page in pages for item in page[key]] + + +def verify_summary(summary: dict, tag: str, sha: str, run: dict, attempt: int) -> None: + if not ( + summary.get("schema_version") == 1 + and summary.get("tag") == tag + and summary.get("source_sha") == sha + and summary.get("is_prerelease") is True + and summary.get("current_profile_passed") is True + and summary.get("profile", {}).get("name") == "release-tag-v1" + and summary.get("run", {}).get("id") == str(run["id"]) + and summary.get("run", {}).get("attempt") == str(attempt) + ): + raise ValueError(f"Qualification evidence does not match {tag} ({sha})") + + +def is_latest(tag: str) -> bool: + version = tuple(map(int, tag.removeprefix("v").split("."))) + return all( + tuple(map(int, stable.removeprefix("v").split("."))) <= version + for stable in git("tag", "--list", "v*").splitlines() + if re.fullmatch(r"v[0-9]+\.[0-9]+\.[0-9]+", stable) + ) + + +def select_candidate(tag: str, branch: str) -> dict: + if not re.fullmatch(r"v[0-9]+\.[0-9]+\.[0-9]+", tag): + raise ValueError("Stable tag must match vMAJOR.MINOR.PATCH") + + candidates = { + candidate: git("rev-parse", f"refs/tags/{candidate}^{{commit}}") + for candidate in git( + "tag", "--merged", "HEAD", "--list", f"{tag}-pre.*" + ).splitlines() + if re.fullmatch(re.escape(tag) + r"-pre\.[1-9][0-9]*", candidate) + } + if not candidates: + raise ValueError(f"No {tag} prereleases are reachable from {branch}") + + repository = os.environ["GITHUB_REPOSITORY"] + query = urlencode({"status": "success", "per_page": 100}) + runs = api_pages( + f"repos/{repository}/actions/workflows/release-prerelease.yml/runs?{query}", + "workflow_runs", + ) + evidence = {} + for run in sorted(runs, key=lambda item: item["id"], reverse=True): + if run["event"] not in {"push", "workflow_dispatch"}: + continue + if run["event"] == "workflow_dispatch" and run["head_branch"] != branch: + continue + if run["event"] == "push" and run["head_branch"] not in candidates: + continue + artifacts = api_pages( + f"repos/{repository}/actions/runs/{run['id']}/artifacts?per_page=100", + "artifacts", + ) + for artifact in artifacts: + match = re.fullmatch( + rf"qualification-({re.escape(tag)}-pre\.[1-9][0-9]*)" + rf"-run-{run['id']}-attempt-([1-9][0-9]*)", + artifact["name"], + ) + if not match: + continue + candidate, attempt_text = match.groups() + attempt = int(attempt_text) + if attempt > run["run_attempt"]: + continue + if run["event"] == "push" and ( + run["head_branch"] != candidate + or run["head_sha"] != candidates.get(candidate) + ): + continue + if candidate in candidates and not artifact["expired"]: + previous = evidence.get(candidate) + if previous is None or (run["id"], attempt) > ( + previous[0]["id"], + previous[2], + ): + evidence[candidate] = (run, artifact, attempt) + + for candidate in sorted( + candidates, key=lambda item: int(item.rsplit(".", 1)[1]), reverse=True + ): + if candidate not in evidence: + continue + run, artifact, attempt = evidence[candidate] + archive = subprocess.check_output( + ["gh", "api", f"repos/{repository}/actions/artifacts/{artifact['id']}/zip"] + ) + with zipfile.ZipFile(io.BytesIO(archive)) as bundle: + summary = json.loads(bundle.read("qualification-summary.json")) + sha = candidates[candidate] + verify_summary(summary, candidate, sha, run, attempt) + + # An existing stable tag may be retried, but must never be moved. + existing = git("tag", "--list", tag) + if existing and git("rev-parse", f"refs/tags/{tag}^{{commit}}") != sha: + raise ValueError(f"Stable tag {tag} already exists at a different commit") + return { + "tag": tag, + "candidate_tag": candidate, + "source_sha": sha, + "qualification_run_url": f"{run['html_url']}/attempts/{attempt}", + "qualification_artifact_id": artifact["id"], + "make_latest": is_latest(tag), + } + + raise ValueError( + f"No passing {tag} prerelease with available qualification evidence on {branch}" + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("tag") + parser.add_argument("--branch", required=True) + args = parser.parse_args() + print(json.dumps(select_candidate(args.tag, args.branch))) + + +if __name__ == "__main__": + main() From c55431a8fce6744069ba7bb9a8165b97be3fc14f Mon Sep 17 00:00:00 2001 From: Simon Scatton Date: Mon, 19 Oct 2026 11:38:58 +0200 Subject: [PATCH 2/4] fix(release): preserve qualified images during stable publication Signed-off-by: Simon Scatton --- .agents/skills/release-openshell/SKILL.md | 10 +- .agents/skills/sbom/SKILL.md | 4 + .github/actions/build-docker-image/action.yml | 16 +++- .github/workflows/build-images.yml | 6 ++ .github/workflows/release-build.yml | 3 +- .github/workflows/release-publish.yml | 54 +++++++++-- CI.md | 12 ++- docs/security/verify-image-contents.mdx | 4 + skills/debug-openshell-cluster/SKILL.md | 4 + tasks/scripts/select-release-candidate.py | 96 ++++++++++--------- 10 files changed, 147 insertions(+), 62 deletions(-) diff --git a/.agents/skills/release-openshell/SKILL.md b/.agents/skills/release-openshell/SKILL.md index 2416aa6066..bdd81e725a 100644 --- a/.agents/skills/release-openshell/SKILL.md +++ b/.agents/skills/release-openshell/SKILL.md @@ -20,11 +20,16 @@ selects the highest-numbered `vX.Y.Z-pre.N` reachable from that branch with a successful Pre-release run triggered by that tag or dispatched on the same branch, and matching passing `release-tag-v1` qualification evidence. Missing or expired evidence blocks promotion. Inspect the selected candidate in the promotion job summary. +The entire Pre-release run must succeed, including publication. If qualification +passed but publication failed, rerun the failed jobs before promoting the candidate. Release creates `vX.Y.Z` at that exact commit, then rebuilds and publishes all stable artifacts without rerunning qualification. Embedded versions change, so stable artifacts are not byte-identical to the qualified prerelease. An existing stable tag may be retried only if it already points to the selected commit. +Stable images are saved as OCI archives during the build and pushed under the +stable version tags during publication. Their SBOMs are verified before updating +`latest`; the qualified prerelease's commit-SHA image tags remain unchanged. If no prerelease exists, agree on the version and source commit, then create and push `vX.Y.Z-pre.1`. Prerelease tag pushes start Pre-release. If the tag was @@ -34,13 +39,14 @@ created using `GITHUB_TOKEN`, dispatch Pre-release explicitly on the intended br gh workflow run release-prerelease.yml --ref main -f tag=vX.Y.Z-pre.1 ``` -Wait for qualification to pass before promoting it. Stable tag pushes do not +Wait for the entire Pre-release run to pass before promoting it. Stable tag pushes do not start Release. Release Auto-Tag creates and dispatches prereleases on `main`. Stable releases serialize across branches. A release older than the highest stable Git tag publishes versioned artifacts without moving GHCR `latest`, the GitHub latest release, npm `latest`, or snap `latest/stable`. Its npm package -uses `vX.Y`; snap files remain available in its GitHub Release. Docs retain an +uses `release-X.Y`; snap files remain available in its GitHub Release rather than +updating a Snap Store support track. Docs retain an immutable version and advance latest only for a newer version. Watch the stable release run and report its result. diff --git a/.agents/skills/sbom/SKILL.md b/.agents/skills/sbom/SKILL.md index 9e80b0838e..85bf4a7b31 100644 --- a/.agents/skills/sbom/SKILL.md +++ b/.agents/skills/sbom/SKILL.md @@ -17,6 +17,10 @@ SBOMs are **release artifacts only** -- they are generated on demand and not com Pushed gateway, sandbox, and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, Release, and Pre-release image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates. +Stable Release builds retain images and attestations in OCI archive artifacts. +Publication copies every platform and attestation to GHCR with preserved digests, +then runs the SBOM verification before updating `latest`. + ## Prerequisites - `mise install` has been run (installs Syft and other tools) diff --git a/.github/actions/build-docker-image/action.yml b/.github/actions/build-docker-image/action.yml index b3b19f2447..2f4bc41a4e 100644 --- a/.github/actions/build-docker-image/action.yml +++ b/.github/actions/build-docker-image/action.yml @@ -2,7 +2,7 @@ # SPDX-License-Identifier: Apache-2.0 name: Build Docker Image -description: Build, push, and export a multi-platform OpenShell image +description: Build and export a multi-platform OpenShell image, optionally pushing it inputs: component: @@ -17,6 +17,9 @@ inputs: image-tag: description: Docker image tag required: true + push: + description: Push the image to GHCR during the build + default: "true" github-token: description: Token used to push the image required: true @@ -29,6 +32,7 @@ runs: buildkitd-config: /etc/buildkit/buildkitd.toml - name: Log in to GHCR + if: inputs.push == 'true' uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 with: registry: ghcr.io @@ -60,8 +64,13 @@ runs: env: COMPONENT: ${{ inputs.component }} IMAGE_TAG: ${{ inputs.image-tag }} + PUSH_IMAGE: ${{ inputs.push }} run: | install -d artifacts/images + push_args=() + if [[ "${PUSH_IMAGE}" == "true" ]]; then + push_args+=(--output "type=image,push=true,oci-mediatypes=true,oci-artifact=true") + fi docker buildx build \ --builder openshell \ --platform linux/amd64,linux/arm64 \ @@ -72,11 +81,12 @@ runs: --cache-to type=gha,mode=max,scope=${COMPONENT} \ --provenance=mode=min \ --attest type=sbom \ - --output type=image,push=true,oci-mediatypes=true,oci-artifact=true \ - --output type=oci,dest=artifacts/images/${COMPONENT}.tar \ + "${push_args[@]}" \ + --output "type=oci,dest=artifacts/images/${COMPONENT}.tar,oci-mediatypes=true,oci-artifact=true" \ . - name: Verify SBOM attestation + if: inputs.push == 'true' shell: bash env: IMAGE_REF: ghcr.io/nvidia/openshell/${{ inputs.component }}:${{ inputs.image-tag }} diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml index 1aa513d578..e43b5bf62e 100644 --- a/.github/workflows/build-images.yml +++ b/.github/workflows/build-images.yml @@ -14,6 +14,11 @@ on: required: false type: string default: "" + push-images: + description: Push images to GHCR during the build + required: false + type: boolean + default: true permissions: contents: read @@ -39,6 +44,7 @@ jobs: binary: ${{ env.BINARY }} target-suffix: ${{ env.TARGET_SUFFIX }} image-tag: ${{ inputs.image-tag || github.sha }} + push: ${{ inputs.push-images }} github-token: ${{ secrets.GITHUB_TOKEN }} sandbox: diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml index 2915770f4e..f5d41249b8 100644 --- a/.github/workflows/release-build.yml +++ b/.github/workflows/release-build.yml @@ -157,7 +157,8 @@ jobs: packages: write uses: ./.github/workflows/build-images.yml with: - image-tag: ${{ needs.compute-versions.outputs.source_sha }} + image-tag: ${{ inputs.is-prerelease && needs.compute-versions.outputs.source_sha || needs.compute-versions.outputs.semver }} + push-images: ${{ inputs.is-prerelease }} checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} build-python-wheel: diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml index 9c123e2534..954cabbb2e 100644 --- a/.github/workflows/release-publish.yml +++ b/.github/workflows/release-publish.yml @@ -219,30 +219,68 @@ jobs: if-no-files-found: error tag-ghcr-release: - name: Tag GHCR Images + name: Publish GHCR Images needs: assemble runs-on: linux-amd64-cpu8 timeout-minutes: 10 permissions: + actions: read + contents: read packages: write steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + if: ${{ !inputs.is-prerelease }} + with: + persist-credentials: false + ref: ${{ inputs.source-sha }} + + - uses: ./.github/actions/setup-nix + if: ${{ !inputs.is-prerelease }} + + - name: Download stable image archives + if: ${{ !inputs.is-prerelease }} + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + pattern: image-* + path: artifacts/ + merge-multiple: true + - name: Log in to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${GITHUB_ACTOR}" --password-stdin + - name: Publish stable image archives + if: ${{ !inputs.is-prerelease }} + env: + VERSION: ${{ inputs.semver }} + run: | + set -euo pipefail + for component in gateway sandbox supervisor; do + image="ghcr.io/nvidia/openshell/${component}:${VERSION}" + # Archives are build artifacts from this run, not externally signed images. + nix shell --inputs-from . nixpkgs#skopeo -c \ + skopeo --insecure-policy copy --all --preserve-digests \ + --authfile "${HOME}/.docker/config.json" \ + "oci-archive:artifacts/images/${component}.tar" "docker://${image}" + tasks/scripts/verify-image-sbom.sh "${image}" --require-cargo + done + - name: Tag images with release version env: MAKE_LATEST: ${{ !inputs.is-prerelease && inputs.make-latest }} - SOURCE_TAG: ${{ inputs.source-sha }} + IS_PRERELEASE: ${{ inputs.is-prerelease }} + SOURCE_TAG: ${{ inputs.is-prerelease && inputs.source-sha || inputs.semver }} VERSION: ${{ inputs.semver }} run: | set -euo pipefail REGISTRY="ghcr.io/nvidia/openshell" for component in gateway sandbox supervisor; do - echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as ${VERSION}..." - docker buildx imagetools create \ - --prefer-index=false \ - -t "${REGISTRY}/${component}:${VERSION}" \ - "${REGISTRY}/${component}:${SOURCE_TAG}" + if [[ "${IS_PRERELEASE}" == "true" ]]; then + echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as ${VERSION}..." + docker buildx imagetools create \ + --prefer-index=false \ + -t "${REGISTRY}/${component}:${VERSION}" \ + "${REGISTRY}/${component}:${SOURCE_TAG}" + fi if [[ "${MAKE_LATEST}" == "true" ]]; then echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as latest..." docker buildx imagetools create \ @@ -473,7 +511,7 @@ jobs: run: | set -euo pipefail if [[ "${MAKE_LATEST}" != "true" ]]; then - export OPENSHELL_NPM_PUBLISH_ARGS="--tag v${OPENSHELL_NPM_VERSION%.*}" + export OPENSHELL_NPM_PUBLISH_ARGS="--tag release-${OPENSHELL_NPM_VERSION%.*}" fi mise run sdk:ts:publish diff --git a/CI.md b/CI.md index 2b04ed4c3d..7e622ac5ba 100644 --- a/CI.md +++ b/CI.md @@ -347,6 +347,8 @@ gh workflow run release.yml --ref main -f tag=v0.4.3 For a support release, replace `main` with its support branch. Release chooses the highest-numbered `v0.4.3-pre.N` reachable from the selected branch with a successful Pre-release run triggered by the tag or dispatched on that branch. +The entire Pre-release run must succeed, including publication. If qualification +passes but publication fails, rerun the failed jobs before promoting the candidate. It verifies the qualification summary's tag, commit, profile, run, attempt, and passing result before creating `v0.4.3` at the same commit. Missing or expired evidence blocks promotion. The promotion @@ -358,12 +360,18 @@ without rerunning qualification. These artifacts have different embedded version from the qualified prerelease. Existing stable tags are never moved; a retry must resolve to the same qualified candidate. +Stable image builds save multi-platform OCI archives as workflow artifacts without +pushing to GHCR. Publication copies those archives, including their attestations, +to the stable version tags and verifies their SBOMs before updating `latest`. +The qualified prerelease's commit-SHA image tags remain unchanged. + Release is manually dispatched; pushing a stable tag does not trigger publication. Stable runs serialize across branches. A version older than the highest stable Git tag publishes versioned artifacts while preserving shared latest channels: GHCR `latest`, GitHub's latest release, npm `latest`, and snap `latest/stable`. -Its npm package uses a `vX.Y` dist-tag, and its snap files remain in the GitHub -Release. Docs publish an immutable version and already prevent latest from moving +Its npm package uses a `release-X.Y` dist-tag, and its snap files remain in the +GitHub Release rather than updating a Snap Store support track. +Docs publish an immutable version and already prevent latest from moving backward. Release Auto-Tag continues on `main`; support branch prereleases can use tag pushes or manual Pre-release dispatch from that branch. diff --git a/docs/security/verify-image-contents.mdx b/docs/security/verify-image-contents.mdx index 948caec03f..86273a4762 100644 --- a/docs/security/verify-image-contents.mdx +++ b/docs/security/verify-image-contents.mdx @@ -11,6 +11,10 @@ position: 2 Published gateway and supervisor images carry one SPDX SBOM per platform as OCI attestations. +Stable releases publish rebuilt images under version tags such as `0.4.3`. +Promoting a prerelease preserves its commit-SHA image tags, which continue to +identify the images used for qualification. + ## Inspect an Image Read a platform's document without pulling the image: diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 87ab4d9cb6..1b9054ec27 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -99,6 +99,10 @@ vulnerability findings, check the deployed image digest and package version; deploy a rebuilt gateway image with the patched base. Updating the gateway binary alone does not update the libraries supplied by its container image. +For stable releases, inspect the version tag or deployed digest. Promotion +preserves the qualified prerelease's commit-SHA image tags; those tags do not +identify the stable rebuild. See [image contents verification](https://docs.nvidia.com/openshell/latest/security/verify-image-contents). + For resource-admission failures, distinguish disabled caller driver config from missing resource approval. Helm defaults `server.drivers.kubernetes.allowDriverConfig` to false and `resourceAdmission.enabled` to true. Existing PVCs, RuntimeClasses, diff --git a/tasks/scripts/select-release-candidate.py b/tasks/scripts/select-release-candidate.py index 41daf90bb7..2030bdbdb5 100644 --- a/tasks/scripts/select-release-candidate.py +++ b/tasks/scripts/select-release-candidate.py @@ -14,18 +14,27 @@ import re import subprocess import zipfile +from typing import TYPE_CHECKING from urllib.parse import urlencode +if TYPE_CHECKING: + from collections.abc import Iterator + def git(*args: str) -> str: return subprocess.check_output(["git", *args], text=True).strip() -def api_pages(path: str, key: str) -> list[dict]: - pages = json.loads( - subprocess.check_output(["gh", "api", "--paginate", "--slurp", path]) - ) - return [item for page in pages for item in page[key]] +def api_pages(path: str, key: str) -> Iterator[dict]: + page = 1 + while True: + items = json.loads( + subprocess.check_output(["gh", "api", f"{path}&page={page}"]) + )[key] + yield from items + if len(items) < 100: + return + page += 1 def verify_summary(summary: dict, tag: str, sha: str, run: dict, attempt: int) -> None: @@ -67,59 +76,54 @@ def select_candidate(tag: str, branch: str) -> dict: repository = os.environ["GITHUB_REPOSITORY"] query = urlencode({"status": "success", "per_page": 100}) - runs = api_pages( - f"repos/{repository}/actions/workflows/release-prerelease.yml/runs?{query}", - "workflow_runs", - ) - evidence = {} - for run in sorted(runs, key=lambda item: item["id"], reverse=True): - if run["event"] not in {"push", "workflow_dispatch"}: - continue - if run["event"] == "workflow_dispatch" and run["head_branch"] != branch: - continue - if run["event"] == "push" and run["head_branch"] not in candidates: - continue - artifacts = api_pages( - f"repos/{repository}/actions/runs/{run['id']}/artifacts?per_page=100", - "artifacts", + artifacts_by_run = {} + for candidate in sorted( + candidates, key=lambda item: int(item.rsplit(".", 1)[1]), reverse=True + ): + sha = candidates[candidate] + runs = api_pages( + f"repos/{repository}/actions/workflows/release-prerelease.yml/runs?{query}", + "workflow_runs", ) - for artifact in artifacts: - match = re.fullmatch( - rf"qualification-({re.escape(tag)}-pre\.[1-9][0-9]*)" - rf"-run-{run['id']}-attempt-([1-9][0-9]*)", - artifact["name"], - ) - if not match: + for run in runs: + if run["event"] not in {"push", "workflow_dispatch"}: continue - candidate, attempt_text = match.groups() - attempt = int(attempt_text) - if attempt > run["run_attempt"]: + if run["event"] == "workflow_dispatch" and run["head_branch"] != branch: continue if run["event"] == "push" and ( - run["head_branch"] != candidate - or run["head_sha"] != candidates.get(candidate) + run["head_branch"] != candidate or run["head_sha"] != sha ): continue - if candidate in candidates and not artifact["expired"]: - previous = evidence.get(candidate) - if previous is None or (run["id"], attempt) > ( - previous[0]["id"], - previous[2], - ): - evidence[candidate] = (run, artifact, attempt) - - for candidate in sorted( - candidates, key=lambda item: int(item.rsplit(".", 1)[1]), reverse=True - ): - if candidate not in evidence: + if run["id"] not in artifacts_by_run: + artifacts_by_run[run["id"]] = list( + api_pages( + f"repos/{repository}/actions/runs/{run['id']}/artifacts?per_page=100", + "artifacts", + ) + ) + evidence = {} + for artifact in artifacts_by_run[run["id"]]: + match = re.fullmatch( + rf"qualification-{re.escape(candidate)}" + rf"-run-{run['id']}-attempt-([1-9][0-9]*)", + artifact["name"], + ) + if not match or artifact["expired"]: + continue + attempt = int(match[1]) + if attempt <= run["run_attempt"]: + evidence[attempt] = artifact + if evidence: + attempt = max(evidence) + artifact = evidence[attempt] + break + else: continue - run, artifact, attempt = evidence[candidate] archive = subprocess.check_output( ["gh", "api", f"repos/{repository}/actions/artifacts/{artifact['id']}/zip"] ) with zipfile.ZipFile(io.BytesIO(archive)) as bundle: summary = json.loads(bundle.read("qualification-summary.json")) - sha = candidates[candidate] verify_summary(summary, candidate, sha, run, attempt) # An existing stable tag may be retried, but must never be moved. From 7eda44bf3ec3cf9d301bc8a8bd4cbf4af9125f45 Mon Sep 17 00:00:00 2001 From: Simon Scatton Date: Mon, 19 Oct 2026 11:56:19 +0200 Subject: [PATCH 3/4] ci(release): run shared release jobs with Nix Signed-off-by: Simon Scatton --- .github/workflows/release-build.yml | 46 +++++++++++++-------------- .github/workflows/release-publish.yml | 23 +++++++------- 2 files changed, 35 insertions(+), 34 deletions(-) diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml index f5d41249b8..e92a05df87 100644 --- a/.github/workflows/release-build.yml +++ b/.github/workflows/release-build.yml @@ -52,11 +52,9 @@ jobs: name: Compute Versions runs-on: linux-amd64-cpu8 timeout-minutes: 5 - container: - image: ghcr.io/nvidia/openshell/ci:9cb72baa2e61a1b5f12407e6e82da7fdba0aa722@sha256:67a9a0c32cb99825e6d3e9d9eec45d67149ea1ff7c11a1b1e1b3480d2d3df684 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} + defaults: + run: + shell: nix develop -c bash -euo pipefail {0} outputs: python_version: ${{ steps.v.outputs.python }} cargo_version: ${{ steps.v.outputs.cargo }} @@ -76,8 +74,9 @@ jobs: persist-credentials: false ref: ${{ inputs.release-tag }} - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + - uses: ./.github/actions/setup-nix + with: + prepare-shell: "true" - name: Fetch tags run: git fetch --tags --force @@ -107,12 +106,12 @@ jobs: fi { echo "is_prerelease=${is_prerelease}" - echo "python=$(uv run python tasks/scripts/release.py get-version --python)" - echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)" - echo "npm=$(uv run python tasks/scripts/release.py get-version --npm)" - echo "deb=$(uv run python tasks/scripts/release.py get-version --deb)" - echo "rpm_version=$(uv run python tasks/scripts/release.py get-version --rpm-version)" - echo "rpm_release=$(uv run python tasks/scripts/release.py get-version --rpm-release)" + echo "python=$(uv run --no-project python tasks/scripts/release.py get-version --python)" + echo "cargo=$(uv run --no-project python tasks/scripts/release.py get-version --cargo)" + echo "npm=$(uv run --no-project python tasks/scripts/release.py get-version --npm)" + echo "deb=$(uv run --no-project python tasks/scripts/release.py get-version --deb)" + echo "rpm_version=$(uv run --no-project python tasks/scripts/release.py get-version --rpm-version)" + echo "rpm_release=$(uv run --no-project python tasks/scripts/release.py get-version --rpm-release)" echo "semver=${RELEASE_TAG#v}" echo "source_sha=$(git rev-parse HEAD)" } >> "$GITHUB_OUTPUT" @@ -166,13 +165,10 @@ jobs: needs: [compute-versions] runs-on: linux-amd64-cpu8 timeout-minutes: 20 - container: - image: ghcr.io/nvidia/openshell/ci:9cb72baa2e61a1b5f12407e6e82da7fdba0aa722@sha256:67a9a0c32cb99825e6d3e9d9eec45d67149ea1ff7c11a1b1e1b3480d2d3df684 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} + defaults: + run: + shell: nix develop -c bash -euo pipefail {0} env: - MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL: ${{ needs.compute-versions.outputs.python_version }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -181,16 +177,20 @@ jobs: persist-credentials: false ref: ${{ needs.compute-versions.outputs.source_sha }} - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + - uses: ./.github/actions/setup-nix + with: + prepare-shell: "true" - name: Sync Python dependencies - run: uv sync + run: uv sync --frozen - name: Build Python wheel run: | set -euo pipefail - mise run python:build + uv run --frozen python tasks/scripts/generate_python_proto.py + uv build --wheel --out-dir target/wheels + uv run --no-project python tasks/scripts/verify-python-wheel.py target/wheels \ + --expected-version "${SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL}" ls -la target/wheels/*.whl - name: Upload wheel artifact diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml index 954cabbb2e..ddd8b930ae 100644 --- a/.github/workflows/release-publish.yml +++ b/.github/workflows/release-publish.yml @@ -476,22 +476,20 @@ jobs: permissions: contents: read packages: write - container: - image: ghcr.io/nvidia/openshell/ci:9cb72baa2e61a1b5f12407e6e82da7fdba0aa722@sha256:67a9a0c32cb99825e6d3e9d9eec45d67149ea1ff7c11a1b1e1b3480d2d3df684 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} + defaults: + run: + shell: nix shell --inputs-from . nixpkgs#nodejs_24 -c bash -euo pipefail {0} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false ref: ${{ inputs.source-sha }} - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + - uses: ./.github/actions/setup-nix - - name: Install tools - run: mise install --locked + - name: Install TypeScript dependencies + working-directory: ./sdk/typescript + run: npm ci - name: Configure npm auth for GitHub Packages working-directory: ./sdk/typescript @@ -504,16 +502,19 @@ jobs: } > .npmrc - name: Publish + working-directory: ./sdk/typescript env: OPENSHELL_NPM_VERSION: ${{ inputs.npm-version }} MAKE_LATEST: ${{ inputs.make-latest }} NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set -euo pipefail + npm pkg set version="${OPENSHELL_NPM_VERSION}" + dist_tag=latest if [[ "${MAKE_LATEST}" != "true" ]]; then - export OPENSHELL_NPM_PUBLISH_ARGS="--tag release-${OPENSHELL_NPM_VERSION%.*}" + dist_tag="release-${OPENSHELL_NPM_VERSION%.*}" fi - mise run sdk:ts:publish + npm publish --tag "${dist_tag}" trigger-wheel-publish: name: Trigger Wheel Publish From 0024def8cd15535cba9f881401d3eef8da9edb8c Mon Sep 17 00:00:00 2001 From: Simon Scatton Date: Mon, 19 Oct 2026 12:21:29 +0200 Subject: [PATCH 4/4] fix(release): recheck latest before publishing shared channels Signed-off-by: Simon Scatton --- .../actions/check-latest-release/action.yml | 46 ++++++++++++++++ .github/workflows/release-publish.yml | 52 +++++++++++++++++-- .github/workflows/snap-publish.yml | 21 ++++++++ CI.md | 3 ++ 4 files changed, 119 insertions(+), 3 deletions(-) create mode 100644 .github/actions/check-latest-release/action.yml diff --git a/.github/actions/check-latest-release/action.yml b/.github/actions/check-latest-release/action.yml new file mode 100644 index 0000000000..dc8006f93e --- /dev/null +++ b/.github/actions/check-latest-release/action.yml @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Check Latest Release +description: Check current remote stable tags before updating a shared latest channel + +inputs: + release-tag: + description: Stable version being published + required: true + +outputs: + make-latest: + description: Whether this is still the highest stable version in the repository + value: ${{ steps.latest.outputs.make-latest }} + +runs: + using: composite + steps: + - uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + id: latest + env: + RELEASE_TAG: ${{ inputs.release-tag }} + with: + script: | + const tag = process.env.RELEASE_TAG; + const parse = value => /^v([0-9]+)\.([0-9]+)\.([0-9]+)$/.exec(value)?.slice(1).map(BigInt); + const version = parse(tag); + if (!version) { + throw new Error(`Invalid stable release tag: ${tag}`); + } + const tags = await github.paginate(github.rest.repos.listTags, { + ...context.repo, + per_page: 100, + }); + if (!tags.some(candidate => candidate.name === tag)) { + throw new Error(`Stable release tag ${tag} is missing from the repository`); + } + const newer = tags.find(candidate => { + const other = parse(candidate.name); + if (!other) return false; + const index = other.findIndex((part, i) => part !== version[i]); + return index !== -1 && other[index] > version[index]; + }); + core.info(newer ? `${tag} is superseded by ${newer.name}; preserving latest` : `${tag} is the latest stable version`); + core.setOutput('make-latest', !newer); diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml index ddd8b930ae..83bfd95978 100644 --- a/.github/workflows/release-publish.yml +++ b/.github/workflows/release-publish.yml @@ -264,9 +264,24 @@ jobs: tasks/scripts/verify-image-sbom.sh "${image}" --require-cargo done + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + if: ${{ !inputs.is-prerelease && inputs.make-latest }} + with: + persist-credentials: false + ref: ${{ github.workflow_sha }} + path: release-workflow + sparse-checkout: .github/actions/check-latest-release + + - name: Check current latest release + id: latest + if: ${{ !inputs.is-prerelease && inputs.make-latest }} + uses: ./release-workflow/.github/actions/check-latest-release + with: + release-tag: ${{ inputs.release-tag }} + - name: Tag images with release version env: - MAKE_LATEST: ${{ !inputs.is-prerelease && inputs.make-latest }} + MAKE_LATEST: ${{ steps.latest.outputs.make-latest }} IS_PRERELEASE: ${{ inputs.is-prerelease }} SOURCE_TAG: ${{ inputs.is-prerelease && inputs.source-sha || inputs.semver }} VERSION: ${{ inputs.semver }} @@ -319,6 +334,14 @@ jobs: outputs: wheel_filenames: ${{ needs.assemble.outputs.wheel_filenames }} steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + if: ${{ !inputs.is-prerelease && inputs.make-latest }} + with: + persist-credentials: false + ref: ${{ github.workflow_sha }} + path: release-workflow + sparse-checkout: .github/actions/check-latest-release + - name: Download assembled release assets uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -349,13 +372,20 @@ jobs: } } + - name: Check current latest release + id: latest + if: ${{ !inputs.is-prerelease && inputs.make-latest }} + uses: ./release-workflow/.github/actions/check-latest-release + with: + release-tag: ${{ inputs.release-tag }} + - name: Create GitHub Release if: ${{ !inputs.is-prerelease }} uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3.0.2 with: name: OpenShell ${{ env.RELEASE_TAG }} prerelease: false - make_latest: ${{ inputs.make-latest && 'true' || 'false' }} + make_latest: ${{ steps.latest.outputs.make-latest == 'true' && 'true' || 'false' }} tag_name: ${{ env.RELEASE_TAG }} generate_release_notes: true body: | @@ -464,6 +494,7 @@ jobs: with: upload-channel: latest/stable github-environment: latest/stable + release-tag: ${{ inputs.release-tag }} secrets: publish-credentials: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }} @@ -501,11 +532,26 @@ jobs: echo '//npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}' } > .npmrc + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + if: inputs.make-latest + with: + persist-credentials: false + ref: ${{ github.workflow_sha }} + path: release-workflow + sparse-checkout: .github/actions/check-latest-release + + - name: Check current latest release + id: latest + if: inputs.make-latest + uses: ./release-workflow/.github/actions/check-latest-release + with: + release-tag: ${{ inputs.release-tag }} + - name: Publish working-directory: ./sdk/typescript env: OPENSHELL_NPM_VERSION: ${{ inputs.npm-version }} - MAKE_LATEST: ${{ inputs.make-latest }} + MAKE_LATEST: ${{ steps.latest.outputs.make-latest }} NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set -euo pipefail diff --git a/.github/workflows/snap-publish.yml b/.github/workflows/snap-publish.yml index 27892bbc14..4b571985c6 100644 --- a/.github/workflows/snap-publish.yml +++ b/.github/workflows/snap-publish.yml @@ -12,6 +12,11 @@ on: github-environment: required: true type: string + release-tag: + description: Stable tag to recheck before updating the stable channel + required: false + type: string + default: "" secrets: publish-credentials: required: true @@ -37,6 +42,14 @@ jobs: timeout-minutes: 10 environment: ${{ inputs.github-environment }} steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + if: inputs.release-tag != '' + with: + persist-credentials: false + ref: ${{ github.workflow_sha }} + path: release-workflow + sparse-checkout: .github/actions/check-latest-release + - name: Download built snap and components uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -52,7 +65,15 @@ jobs: sudo snap wait system seed.loaded sudo snap install snapcraft --classic + - name: Check current latest release + id: latest + if: inputs.release-tag != '' + uses: ./release-workflow/.github/actions/check-latest-release + with: + release-tag: ${{ inputs.release-tag }} + - name: Upload snap to Snap Store + if: ${{ inputs.release-tag == '' || steps.latest.outputs.make-latest == 'true' }} env: SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.publish-credentials }} INPUTS_UPLOAD_CHANNEL: ${{ inputs.upload-channel }} diff --git a/CI.md b/CI.md index 7e622ac5ba..f742fcb3f8 100644 --- a/CI.md +++ b/CI.md @@ -369,6 +369,9 @@ Release is manually dispatched; pushing a stable tag does not trigger publicatio Stable runs serialize across branches. A version older than the highest stable Git tag publishes versioned artifacts while preserving shared latest channels: GHCR `latest`, GitHub's latest release, npm `latest`, and snap `latest/stable`. +Each publisher rechecks the current remote stable tags immediately before +updating its shared channel, including when a past publishing job is rerun. +This protection applies to runs created with these workflow definitions. Its npm package uses a `release-X.Y` dist-tag, and its snap files remain in the GitHub Release rather than updating a Snap Store support track. Docs publish an immutable version and already prevent latest from moving