From 0af0b681c99b16be7284cad45d22b9ed05959743 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 12 Oct 2026 10:18:02 -0700 Subject: [PATCH 01/20] feat(skills): add release-captain skill for the weekly release cadence Add an internal skill with Monday pre-flight, Tuesday stable cut, and recovery steps for the RFC 0014 release train. All tag, workflow, merge, and Slack actions require explicit maintainer approval. Closes #4433 Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 251 +++++++++++++++++++++++ .agents/skills/sync-agent-infra/SKILL.md | 1 + CONTRIBUTING.md | 1 + 3 files changed, 253 insertions(+) create mode 100644 .agents/skills/release-captain/SKILL.md diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md new file mode 100644 index 0000000000..f6d1010e9b --- /dev/null +++ b/.agents/skills/release-captain/SKILL.md @@ -0,0 +1,251 @@ +--- +name: release-captain +description: Run the weekly OpenShell release captain checklist. Monday checks that the current pre-release train is qualified, triages qualification, compatibility, and security reports, and prepares the release coordination post. Tuesday selects the final qualified pre-release, tags its commit as stable after maintainer approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release. +metadata: + internal: true +--- + +# Release Captain + +OpenShell publishes a stable release every Tuesday from a qualified +pre-release ([RFC 0014](../../../rfc/0014-release-stability/README.md)). The +release captain keeps the week's pre-release train green, coordinates the last +merges, and cuts the release. `CI.md` describes the release workflows, +qualification profile, and scanner policies. This skill covers what the captain +does with them. + +## Guardrails + +Without explicit approval from the release captain or a maintainer for that +specific action, never: + +- push, move, or delete a tag; +- dispatch, cancel, or rerun any workflow; +- merge, approve, or enqueue a PR; +- post in Slack or comment on GitHub; +- install OpenShell or change the local machine. + +Ask for approval with the exact command and target commit SHA. Every other +step in this skill is read-only. + +Never repoint a stable tag at a different commit, and never re-dispatch a +release while an earlier run for the same tag might still start. `Release Tag` +has no concurrency guard, so two runs race to publish. Don't delete `-pre.N` +tags; the RFC's cleanup step is not implemented and tag deletion is restricted. + +## How the train works + +- A train has one base version, such as `v0.1.4`, fixed by its `-pre.1` tag. A + patch train can't take breaking changes to Stable interfaces. Those wait for a + minor train. +- `Release Auto-Tag` runs Monday through Friday at 14:00 Europe/Zurich. If + `main` has commits since the latest `-pre.N`, it tags the next one and + dispatches `Release Tag` (about 45 minutes). It skips after the base version + gets a stable tag, and fails if the latest pre-release isn't an ancestor of + `main`. +- Pushing a `-pre.*` tag does not start `Release Tag`. A pre-release tagged by + hand must be dispatched. +- Pushing a stable `vX.Y.Z` tag starts `Release Tag`, which rebuilds and + requalifies the commit. Codex Security runs only for pre-releases, so the + stable release relies on the selected pre-release's security result. Always + tag the commit of a pre-release that passed. + +## Setup + +Run from an OpenShell checkout with an authenticated `gh` CLI. Fetch tags with +`--force` so cached tags match the remote: + +```shell +export GH_REPO=NVIDIA/OpenShell +git fetch origin main --tags --force +PREV=$(git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1) +echo "Latest stable: $PREV" +git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[0-9]+$' | head -5 +``` + +Find the `Release Tag` run for a pre-release. Dispatched runs use `main`, so +their head commit is the tagged commit: + +```shell +TAG=vX.Y.Z-pre.N +gh run list --workflow release-tag.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")" +``` + +If that commit has several runs, the one whose Codex Security job name contains +`($TAG)` is the match. That job appears only after the images build. + +## Monday: pre-flight + +Finish these on Monday so Tuesday is only the cut. + +1. **Check the train.** Confirm that the latest `-pre.N` targets the version you + plan to ship. Check that the most recent `Release Auto-Tag` run succeeded + (today's runs at 14:00 Europe/Zurich): + + ```shell + gh run list --workflow release-auto-tag.yml --limit 3 + ``` + + If the next version has no `-pre.1`, start the train (Tuesday step 6). + +2. **Read the qualification summary for the latest attempt.** The summary's + `tag` and `source_sha` must match the candidate, and `current_profile_passed` + must be `true`: + + ```shell + RUN= + ATTEMPT=$(gh run view "$RUN" --json attempt --jq .attempt) + dir=$(mktemp -d) + gh run download "$RUN" -n "qualification-$TAG-run-$RUN-attempt-$ATTEMPT" -D "$dir/q" + jq '{tag, source_sha, current_profile_passed, suites}' "$dir/q/qualification-summary.json" + git rev-parse "$TAG^{commit}" + ``` + + For any failed suite, read the log with `gh run view "$RUN" --log-failed`. + `CI.md` lists which scanners block. Use the `watch-github-actions` skill for + CI diagnosis. + +3. **Triage the advisory compatibility review.** It never blocks publication, + and a green job doesn't mean it found no breaks: + + ```shell + gh run download "$RUN" -n "compatibility-review-run-$RUN-attempt-$ATTEMPT" -D "$dir/c" + jq '{status, verdict: .review.verdict}' "$dir/c/report.json" + cat "$dir/c/report.md" + ``` + + Only `complete` with `no_unaddressed_breaks` is a clean review. A missing + report or any other result means there's no assessment. For each finding, + decide whether it's a real break, a false positive, or unclear: + + - **Real break:** needs a fix or revert before Tuesday, or must wait for a + minor train. + - **Unclear:** ask an owner of that code. + - **False positive:** send it to the owner of + `.github/prompts/compatibility-review.md` so the prompt can be tuned. + +4. **Check the dev canary.** `Release Canary` smoke-tests `main` dev artifacts. + It doesn't gate the tag, but treat a red canary as a release risk. Use the + `test-release-canary` skill to diagnose it: + + ```shell + gh run list --workflow release-canary.yml --limit 3 + ``` + +5. **Plan fixes.** A transient infrastructure failure can be rerun after + approval. A code defect needs a merged fix and a new pre-release. + +6. **Draft the coordination post.** Release coordination happens in a thread + in the OpenShell CNCF Slack channel, started on Monday. Draft it for the + captain to post: + + - the target version and current candidate (tag, commit, run link); + - qualification and compatibility status, with an owner for each blocker; + - ready but unmerged PRs that should make the release, in merge order; + - the merge cutoff in UTC and the captain's time zone. Set it before + Tuesday's 14:00 Europe/Zurich Auto-Tag run so the last merges make that + pre-release. + + Candidate PRs: + + ```shell + gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"' + ``` + +## Tuesday: cut the release + +1. **Select the final candidate.** After the cutoff and Tuesday's Auto-Tag run, + pick the newest `-pre.N` whose qualification passed, and repeat Monday steps + 2 and 3 for it. If the newest pre-release failed, either merge a fix and + qualify another pre-release, or pick an earlier one that passed and confirm + that leaving out the later commits is acceptable. + +2. **Request approval.** Present the candidate tag, commit SHA, qualification + run, compatibility assessment, and the commits since `$PREV`: + + ```shell + git log --oneline --no-merges "$PREV..$TAG" + ``` + +3. **Tag the stable release.** After approval, check the candidate, then tag its + exact commit and push only that tag: + + ```shell + VERSION=vX.Y.Z + SHA=$(git rev-parse "$TAG^{commit}") + [[ "$VERSION" == "${TAG%-pre.*}" ]] || echo "version does not match the train" + [[ "$SHA" == "$(jq -r .source_sha "$dir/q/qualification-summary.json")" ]] || echo "commit does not match qualification" + git tag --points-at "$SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag" + git tag "$VERSION" "$SHA" + git push origin "refs/tags/$VERSION" + ``` + + Stop if any check prints a message. + +4. **Watch the release.** The tag push starts `Release Tag`. It runs for about + 45 minutes, so poll the status, or run the watch in the background: + + ```shell + gh run list --workflow release-tag.yml --branch "$VERSION" --limit 1 + gh run watch --exit-status + ``` + + If no run appears within a few minutes, or the run stays queued, see + Recovery. + +5. **Verify publication.** Confirm that the GitHub Release exists as a + non-pre-release with assets and generated notes, and that no job failed: + + ```shell + gh release view "$VERSION" --json name,isPrerelease,publishedAt,assets --jq '{name, isPrerelease, publishedAt, assets: (.assets | length)}' + gh run view --json jobs --jq '.jobs[] | select(.conclusion != "success" and .conclusion != "skipped") | "\(.conclusion)\t\(.name)"' + ``` + + `Trigger Wheel Publish` only starts the GitLab pipeline, so confirm that the + Python wheel reached PyPI separately. + +6. **Start the next train.** Auto-Tag waits until the next `-pre.1` exists. + Agree with the maintainers whether the next train is patch (the default) or + minor (notable features or planned breaking changes; see the + [version selection supplement](../../../rfc/0014-release-stability/release-version-selection.md)). + + Seed the train on a `main` commit that: + + - comes after `$VERSION`; + - has green branch checks; + - has no stable tag of its own. + + Release versioning prefers a stable tag over a pre-release tag on the same + commit, which would mislabel the packages. If nothing has merged since the + release, wait. After approval: + + ```shell + NEXT=vX.Y.Z-pre.1 + NEXT_SHA=
+ git merge-base --is-ancestor "$VERSION" "$NEXT_SHA" || echo "commit does not contain the release" + git tag --points-at "$NEXT_SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag" + git tag "$NEXT" "$NEXT_SHA" + git push origin "refs/tags/$NEXT" + gh workflow run release-tag.yml --ref main -f tag="$NEXT" + ``` + +7. **Close the loop.** Draft a reply to the coordination thread with the release + link and any PRs deferred to the next train. + +## Recovery + +| Symptom | Action | +|---|---| +| Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the new attempt's summary. | +| Code defect in a pre-release | Merge the fix. Let Auto-Tag create the next `-pre.N`, or, after approval, tag one by hand and dispatch it. | +| Stable run stuck queued | Don't re-dispatch or delete the tag. Check for a GitHub Actions incident and ask a maintainer to cancel the run (`gh run cancel `, or force-cancel through the API). Only after it shows `cancelled` or `failed`, rerun it after approval. | +| Stable run fails before the `Release` job, transient | Nothing was published. After approval, run `gh run rerun --failed`. | +| Stable run fails before the `Release` job, code defect | Nothing was published, and Auto-Tag won't create pre-releases for a base version that has a stable tag. Ask the maintainers to choose: remove the unpublished tag (needs a ruleset bypass) and recut from a qualified fix, or skip to the next patch version. | +| Stable run fails after the `Release` job | The GitHub Release exists. Check which publish jobs succeeded. A rerun of an npm or Snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. | + +## Related + +- `watch-github-actions`: general `gh run` monitoring and log retrieval. +- `test-release-canary`: diagnose dev artifact smoke-test failures. +- `CI.md`: release workflows, qualification profile, and scanner policies. +- `rfc/0014-release-stability/`: cadence, qualification, and version selection. diff --git a/.agents/skills/sync-agent-infra/SKILL.md b/.agents/skills/sync-agent-infra/SKILL.md index a34c973fa3..5f2a843197 100644 --- a/.agents/skills/sync-agent-infra/SKILL.md +++ b/.agents/skills/sync-agent-infra/SKILL.md @@ -27,6 +27,7 @@ Run after adding, removing, or renaming a skill or crate; changing issue or PR c | Inference providers or native model endpoints | `debug-inference`, `openshell-cli` | | TUI behavior | `tui-development` | | Release artifacts or smoke coverage | `test-release-canary` | +| Release cadence, tagging, or qualification workflows | `release-captain` | | CI diagnostics | `watch-github-actions` | | Gator supervision | `launch-openshell-gator` | | SBOM workflow | `sbom` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9f05b5fd50..73d0600729 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -117,6 +117,7 @@ Contributor and maintainer skills live in `.agents/skills/`. They are marked int | Platform | `build-openshell-mxc-windows` | Maintain and validate the build-only x64 and ARM64 Windows MSVC lane | | Documentation | `update-docs-from-commits` | Scan recent commits and draft doc updates for user-facing changes | | Maintenance | `sync-agent-infra` | Detect and fix drift across agent-first infrastructure files | +| Maintenance | `release-captain` | Run the weekly Monday pre-flight and Tuesday stable release checklist | | Reference | `sbom` | Generate SBOMs and resolve dependency licenses | ### Issue Workflow From 46e8ab0f1343855ad2a8cbcb0411b204385fd187 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Wed, 14 Oct 2026 13:02:19 -0700 Subject: [PATCH 02/20] docs(skills): document starting a release train after tag-push triggers Pre-release tag pushes now start Release Tag, so the skill no longer dispatches after pushing. Add seed-commit selection from Release Dev results, a commit-type summary for the patch/minor decision, and a recovery row for a tag push that starts no run. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 62 +++++++++++++++++++------ 1 file changed, 48 insertions(+), 14 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index f6d1010e9b..814b6c448f 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -43,9 +43,10 @@ tags; the RFC's cleanup step is not implemented and tag deletion is restricted. dispatches `Release Tag` (about 45 minutes). It skips after the base version gets a stable tag, and fails if the latest pre-release isn't an ancestor of `main`. -- Pushing a `-pre.*` tag does not start `Release Tag`. A pre-release tagged by - hand must be dispatched. -- Pushing a stable `vX.Y.Z` tag starts `Release Tag`, which rebuilds and +- Pushing any release tag by hand, pre-release or stable, starts `Release Tag`. + Don't also dispatch it, or two runs race to publish. Auto-Tag dispatches + explicitly only because its `GITHUB_TOKEN` push can't trigger workflows. +- A stable `vX.Y.Z` tag push rebuilds and requalifies the commit. Codex Security runs only for pre-releases, so the stable release relies on the selected pre-release's security result. Always tag the commit of a pre-release that passed. @@ -63,16 +64,18 @@ echo "Latest stable: $PREV" git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[0-9]+$' | head -5 ``` -Find the `Release Tag` run for a pre-release. Dispatched runs use `main`, so -their head commit is the tagged commit: +Find the `Release Tag` run for a tag. A tag pushed by hand runs on the tag +itself. An Auto-Tag run is dispatched on `main`, so its head commit is the +tagged commit: ```shell TAG=vX.Y.Z-pre.N +gh run list --workflow release-tag.yml --branch "$TAG" gh run list --workflow release-tag.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")" ``` -If that commit has several runs, the one whose Codex Security job name contains -`($TAG)` is the match. That job appears only after the images build. +If a commit has several dispatched runs, the one whose Codex Security job name +contains `($TAG)` is the match. That job appears only after the images build. ## Monday: pre-flight @@ -204,20 +207,44 @@ Finish these on Monday so Tuesday is only the cut. `Trigger Wheel Publish` only starts the GitLab pipeline, so confirm that the Python wheel reached PyPI separately. -6. **Start the next train.** Auto-Tag waits until the next `-pre.1` exists. +6. **Start the next train.** Until someone pushes the next `-pre.1`, Auto-Tag + logs `vX.Y.Z is already stable — skipping tag creation` and does nothing. + Start the train soon after the stable release; it can be any weekday. + Agree with the maintainers whether the next train is patch (the default) or minor (notable features or planned breaking changes; see the [version selection supplement](../../../rfc/0014-release-stability/release-version-selection.md)). + List the commit types since the release to inform the decision: + + ```shell + git log --format=%s --no-merges "$VERSION..origin/main" | sed -E 's/^([a-z]+)(\(.*\))?(!?):.*/\1\3/' | sort | uniq -c + ``` Seed the train on a `main` commit that: - comes after `$VERSION`; - - has green branch checks; - - has no stable tag of its own. + - has no stable tag of its own; + - has a successful `Release Dev` run. Release versioning prefers a stable tag over a pre-release tag on the same commit, which would mislabel the packages. If nothing has merged since the - release, wait. After approval: + release, wait. + + Branch Checks already passed in the merge queue before each merge. + `Release Dev` runs the post-merge E2E and dev publication on every commit. A + newer push cancels the older run, so `cancelled` means superseded, not + failed. Pick the newest commit whose run succeeded, or wait for the tip's + run to finish: + + ```shell + for sha in $(git rev-list --max-count=10 "$VERSION..origin/main"); do + state=$(gh run list --workflow release-dev.yml --commit "$sha" --json status,conclusion --jq '.[0] | if .status == "completed" then .conclusion else .status end') + echo "${sha:0:9} ${state:-none} $(git log -1 --format=%s "$sha")" + done + ``` + + After approval, push the tag. The push starts `Release Tag`; don't dispatch + it as well: ```shell NEXT=vX.Y.Z-pre.1 @@ -226,9 +253,13 @@ Finish these on Monday so Tuesday is only the cut. git tag --points-at "$NEXT_SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag" git tag "$NEXT" "$NEXT_SHA" git push origin "refs/tags/$NEXT" - gh workflow run release-tag.yml --ref main -f tag="$NEXT" + gh run list --workflow release-tag.yml --branch "$NEXT" ``` + Stop if either check prints a message. When the run finishes, read its + qualification summary (Monday step 2). From the next weekday, Auto-Tag + creates `-pre.2` onward when `main` has new commits. + 7. **Close the loop.** Draft a reply to the coordination thread with the release link and any PRs deferred to the next train. @@ -237,14 +268,17 @@ Finish these on Monday so Tuesday is only the cut. | Symptom | Action | |---|---| | Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the new attempt's summary. | -| Code defect in a pre-release | Merge the fix. Let Auto-Tag create the next `-pre.N`, or, after approval, tag one by hand and dispatch it. | +| Code defect in a pre-release | Merge the fix. Let Auto-Tag create the next `-pre.N`, or, after approval, tag and push one by hand. | +| A pushed tag started no `Release Tag` run | Wait a few minutes and check `gh run list --workflow release-tag.yml --branch "$TAG"` again. Only after confirming that no run exists, and with approval, dispatch it with `gh workflow run release-tag.yml --ref main -f tag="$TAG"`. | | Stable run stuck queued | Don't re-dispatch or delete the tag. Check for a GitHub Actions incident and ask a maintainer to cancel the run (`gh run cancel `, or force-cancel through the API). Only after it shows `cancelled` or `failed`, rerun it after approval. | | Stable run fails before the `Release` job, transient | Nothing was published. After approval, run `gh run rerun --failed`. | | Stable run fails before the `Release` job, code defect | Nothing was published, and Auto-Tag won't create pre-releases for a base version that has a stable tag. Ask the maintainers to choose: remove the unpublished tag (needs a ruleset bypass) and recut from a qualified fix, or skip to the next patch version. | -| Stable run fails after the `Release` job | The GitHub Release exists. Check which publish jobs succeeded. A rerun of an npm or Snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. | +| Stable run fails after the `Release` job | The GitHub Release exists. Check which publish jobs succeeded. A rerun of an npm or snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. | ## Related +- `release-openshell`: the short procedure for selecting a qualified + pre-release and tagging it stable. - `watch-github-actions`: general `gh run` monitoring and log retrieval. - `test-release-canary`: diagnose dev artifact smoke-test failures. - `CI.md`: release workflows, qualification profile, and scanner policies. From b50bb2b64be76b7a84181f14f796b6f3f3261f96 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Wed, 14 Oct 2026 13:09:30 -0700 Subject: [PATCH 03/20] docs(skills): create lightweight release tags regardless of signing config With tag.gpgSign enabled, git tag creates an annotated tag and blocks in an editor. Disable signing for the release tag commands so they create the same lightweight tags as Auto-Tag. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 814b6c448f..f127c42447 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -64,6 +64,11 @@ echo "Latest stable: $PREV" git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[0-9]+$' | head -5 ``` +Release tags are lightweight, matching the tags Auto-Tag creates. The tag +commands below pass `-c tag.gpgSign=false` and read from `/dev/null`. With +`tag.gpgSign` set, `git tag` would otherwise create a signed annotated tag and +wait in an editor for a message. + Find the `Release Tag` run for a tag. A tag pushed by hand runs on the tag itself. An Auto-Tag run is dispatched on `main`, so its head commit is the tagged commit: @@ -179,7 +184,7 @@ Finish these on Monday so Tuesday is only the cut. [[ "$VERSION" == "${TAG%-pre.*}" ]] || echo "version does not match the train" [[ "$SHA" == "$(jq -r .source_sha "$dir/q/qualification-summary.json")" ]] || echo "commit does not match qualification" git tag --points-at "$SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag" - git tag "$VERSION" "$SHA" + git -c tag.gpgSign=false tag "$VERSION" "$SHA" git merge-base --is-ancestor "$VERSION" "$NEXT_SHA" || echo "commit does not contain the release" git tag --points-at "$NEXT_SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag" - git tag "$NEXT" "$NEXT_SHA" + git -c tag.gpgSign=false tag "$NEXT" "$NEXT_SHA" Date: Wed, 14 Oct 2026 13:15:16 -0700 Subject: [PATCH 04/20] docs(skills): capture release captain lessons from starting 0.1.5 Note that Auto-Tag skips report success and show how to tell from the log, the UTC time of the schedule across DST, where pre-releases appear instead of the Releases page, why tags need a forced fetch, reading workflow triggers from origin/main, the rolling dev tag, and the common compatibility-review false positive for unpublished internal crates. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 44 ++++++++++++++++++++----- 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index f127c42447..643da77161 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -38,11 +38,12 @@ tags; the RFC's cleanup step is not implemented and tag deletion is restricted. - A train has one base version, such as `v0.1.4`, fixed by its `-pre.1` tag. A patch train can't take breaking changes to Stable interfaces. Those wait for a minor train. -- `Release Auto-Tag` runs Monday through Friday at 14:00 Europe/Zurich. If +- `Release Auto-Tag` runs Monday through Friday at 14:00 Europe/Zurich. That's + 12:00 UTC during Central European Summer Time and 13:00 UTC in winter. If `main` has commits since the latest `-pre.N`, it tags the next one and dispatches `Release Tag` (about 45 minutes). It skips after the base version gets a stable tag, and fails if the latest pre-release isn't an ancestor of - `main`. + `main`. A skip still counts as a successful run. - Pushing any release tag by hand, pre-release or stable, starts `Release Tag`. Don't also dispatch it, or two runs race to publish. Auto-Tag dispatches explicitly only because its `GITHUB_TOKEN` push can't trigger workflows. @@ -50,11 +51,18 @@ tags; the RFC's cleanup step is not implemented and tag deletion is restricted. requalifies the commit. Codex Security runs only for pre-releases, so the stable release relies on the selected pre-release's security result. Always tag the commit of a pre-release that passed. +- Only stable tags create a GitHub Release, so pre-releases never appear on the + Releases page. A passing pre-release publishes its images, Helm chart, and + qualification summary to GHCR. Its Debian, RPM, and macOS packages stay as + artifacts on the `Release Tag` run. To check a pre-release, look at the tag + and that run. A failed pre-release notifies the OpenShell triage Slack + channel. ## Setup Run from an OpenShell checkout with an authenticated `gh` CLI. Fetch tags with -`--force` so cached tags match the remote: +`--force`, because release tags have been removed and recreated, and a plain +fetch keeps stale local copies: ```shell export GH_REPO=NVIDIA/OpenShell @@ -64,6 +72,14 @@ echo "Latest stable: $PREV" git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[0-9]+$' | head -5 ``` +Release workflows change often, and a local branch may be days behind. Before +relying on trigger or job behavior, including anything in this skill, read the +current workflow from `origin/main`: + +```shell +git show origin/main:.github/workflows/release-tag.yml | sed -n '/^on:/,/^permissions:/p' +``` + Release tags are lightweight, matching the tags Auto-Tag creates. The tag commands below pass `-c tag.gpgSign=false` and read from `/dev/null`. With `tag.gpgSign` set, `git tag` would otherwise create a signed annotated tag and @@ -87,11 +103,13 @@ contains `($TAG)` is the match. That job appears only after the images build. Finish these on Monday so Tuesday is only the cut. 1. **Check the train.** Confirm that the latest `-pre.N` targets the version you - plan to ship. Check that the most recent `Release Auto-Tag` run succeeded - (today's runs at 14:00 Europe/Zurich): + plan to ship. Check that today's `Release Auto-Tag` run (14:00 + Europe/Zurich) succeeded, and check from its log whether it tagged or + skipped: ```shell gh run list --workflow release-auto-tag.yml --limit 3 + gh run view "$(gh run list --workflow release-auto-tag.yml --limit 1 --json databaseId --jq '.[0].databaseId')" --log | grep -E 'Next tag:|— skipping' | grep -v echo ``` If the next version has no `-pre.1`, start the train (Tuesday step 6). @@ -132,6 +150,12 @@ Finish these on Monday so Tuesday is only the cut. - **False positive:** send it to the owner of `.github/prompts/compatibility-review.md` so the prompt can be tuned. + The reviewer treats every `pub` Rust item as a Stable interface. No + workspace crate is published to crates.io, so a finding in an internal + crate such as `openshell-core` or `openshell-isolation-interface` is often a + false positive. Check whether a supported SDK surface, such as + `openshell-sdk`, exposes the item before deciding. + 4. **Check the dev canary.** `Release Canary` smoke-tests `main` dev artifacts. It doesn't gate the tag, but treat a red canary as a release risk. Use the `test-release-canary` skill to diagnose it: @@ -261,9 +285,13 @@ Finish these on Monday so Tuesday is only the cut. gh run list --workflow release-tag.yml --branch "$NEXT" ``` - Stop if either check prints a message. When the run finishes, read its - qualification summary (Monday step 2). From the next weekday, Auto-Tag - creates `-pre.2` onward when `main` has new commits. + Stop if either check prints a message. The rolling `dev` tag may point at + the same commit; it isn't a release tag and doesn't affect versioning. + + The run should appear as `queued` within a minute. Watch it as in Tuesday + step 4, then read its qualification summary (Monday step 2). It won't appear + on the Releases page. From the next weekday, Auto-Tag creates `-pre.2` onward + when `main` has new commits. 7. **Close the loop.** Draft a reply to the coordination thread with the release link and any PRs deferred to the next train. From e3d0caad8cc57a78602f375850607efa38462fab Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Wed, 14 Oct 2026 13:47:54 -0700 Subject: [PATCH 05/20] docs(skills): add recovery for pre-releases failing on new RustSec advisories Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 643da77161..344092be13 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -302,6 +302,7 @@ Finish these on Monday so Tuesday is only the cut. |---|---| | Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the new attempt's summary. | | Code defect in a pre-release | Merge the fix. Let Auto-Tag create the next `-pre.N`, or, after approval, tag and push one by hand. | +| `Cargo Deny` fails a pre-release with `error[vulnerability]` | A new RustSec advisory was published against a locked dependency. Code that passed earlier can start failing this way. Rerunning won't help. Find the advisory IDs with `gh run view --log-failed \| grep -E 'ID: RUSTSEC'`, look for an open dependency bump PR, and treat the merged bump as a code-defect fix. | | A pushed tag started no `Release Tag` run | Wait a few minutes and check `gh run list --workflow release-tag.yml --branch "$TAG"` again. Only after confirming that no run exists, and with approval, dispatch it with `gh workflow run release-tag.yml --ref main -f tag="$TAG"`. | | Stable run stuck queued | Don't re-dispatch or delete the tag. Check for a GitHub Actions incident and ask a maintainer to cancel the run (`gh run cancel `, or force-cancel through the API). Only after it shows `cancelled` or `failed`, rerun it after approval. | | Stable run fails before the `Release` job, transient | Nothing was published. After approval, run `gh run rerun --failed`. | From 7ec8363cb6a2c6912d0704f99ac9019464169d18 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Wed, 14 Oct 2026 14:27:21 -0700 Subject: [PATCH 06/20] docs(skills): recommend fixing forward to a green pre-release A failed pre-release doesn't end the train. Document merging the fix and tagging the next pre-release by hand once its Release Dev run passes, letting the failed run finish for its remaining signal. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 344092be13..d47f9757b4 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -164,8 +164,18 @@ Finish these on Monday so Tuesday is only the cut. gh run list --workflow release-canary.yml --limit 3 ``` -5. **Plan fixes.** A transient infrastructure failure can be rerun after - approval. A code defect needs a merged fix and a new pre-release. +5. **Plan fixes.** A failed pre-release doesn't end the train. The goal is a + green pre-release before Tuesday, so fix forward: + + - A transient infrastructure failure can be rerun after approval. + - A code defect, or a new advisory against a locked dependency, needs a + merged fix and a new pre-release. Don't rerun or promote the failed one. + - Let the failed run finish rather than cancelling it. Its other suites + still show whether the rest of the commit is healthy. + - Once the fix merges and its `Release Dev` run passes, tag the next + `-pre.N` on that commit by hand after approval (as in Tuesday step 6, with + the next number), instead of waiting for the next Auto-Tag run. That + gives a qualifiable candidate a day sooner. 6. **Draft the coordination post.** Release coordination happens in a thread in the OpenShell CNCF Slack channel, started on Monday. Draft it for the @@ -301,7 +311,7 @@ Finish these on Monday so Tuesday is only the cut. | Symptom | Action | |---|---| | Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the new attempt's summary. | -| Code defect in a pre-release | Merge the fix. Let Auto-Tag create the next `-pre.N`, or, after approval, tag and push one by hand. | +| Code defect in a pre-release | Fix forward (Monday step 5): merge the fix, then, after approval, tag and push the next `-pre.N` on the fix commit once its `Release Dev` run passes. Waiting for Auto-Tag also works but costs up to a day. | | `Cargo Deny` fails a pre-release with `error[vulnerability]` | A new RustSec advisory was published against a locked dependency. Code that passed earlier can start failing this way. Rerunning won't help. Find the advisory IDs with `gh run view --log-failed \| grep -E 'ID: RUSTSEC'`, look for an open dependency bump PR, and treat the merged bump as a code-defect fix. | | A pushed tag started no `Release Tag` run | Wait a few minutes and check `gh run list --workflow release-tag.yml --branch "$TAG"` again. Only after confirming that no run exists, and with approval, dispatch it with `gh workflow run release-tag.yml --ref main -f tag="$TAG"`. | | Stable run stuck queued | Don't re-dispatch or delete the tag. Check for a GitHub Actions incident and ask a maintainer to cancel the run (`gh run cancel `, or force-cancel through the API). Only after it shows `cancelled` or `failed`, rerun it after approval. | From f8fbb864dbaa65c919d48ababa6e228b0c8be200 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 15 Oct 2026 10:19:00 -0700 Subject: [PATCH 07/20] docs(skills): describe the triage channel and thread replies for pre-release failures Name #openshell-triage, where TriageBot posts each failed Release Tag attempt, and ask the captain to reply in the failure's thread so others know a fix is underway. List common transient failure types and rerun once when a failure looks unrelated to the commit. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index d47f9757b4..2cb58c7d72 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -55,8 +55,10 @@ tags; the RFC's cleanup step is not implemented and tag deletion is restricted. Releases page. A passing pre-release publishes its images, Helm chart, and qualification summary to GHCR. Its Debian, RPM, and macOS packages stay as artifacts on the `Release Tag` run. To check a pre-release, look at the tag - and that run. A failed pre-release notifies the OpenShell triage Slack - channel. + and that run. +- OpenShell TriageBot posts a top-level message in `#openshell-triage` + (`C0C20SDLDNW`) for every failed `Release Tag` attempt, with a link to the + run. Each rerun that fails gets its own message. ## Setup @@ -167,7 +169,14 @@ Finish these on Monday so Tuesday is only the cut. 5. **Plan fixes.** A failed pre-release doesn't end the train. The goal is a green pre-release before Tuesday, so fix forward: - - A transient infrastructure failure can be rerun after approval. + - Reply in the thread of the failure's `#openshell-triage` message, so + others know someone is working on a fix. Post the cause and the fix or + rerun, then post again when a pre-release passes. Draft these replies for + the captain to post. + - A transient infrastructure failure can be rerun after approval. Runner + setup errors, registry push errors, and integration tests that fail only + some of the time are common. If a failure looks unrelated to the commit's + changes, rerun once before treating it as a code defect. - A code defect, or a new advisory against a locked dependency, needs a merged fix and a new pre-release. Don't rerun or promote the failed one. - Let the failed run finish rather than cancelling it. Its other suites @@ -310,7 +319,7 @@ Finish these on Monday so Tuesday is only the cut. | Symptom | Action | |---|---| -| Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the new attempt's summary. | +| Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the new attempt's summary. Reply in the failure's `#openshell-triage` thread with the result. | | Code defect in a pre-release | Fix forward (Monday step 5): merge the fix, then, after approval, tag and push the next `-pre.N` on the fix commit once its `Release Dev` run passes. Waiting for Auto-Tag also works but costs up to a day. | | `Cargo Deny` fails a pre-release with `error[vulnerability]` | A new RustSec advisory was published against a locked dependency. Code that passed earlier can start failing this way. Rerunning won't help. Find the advisory IDs with `gh run view --log-failed \| grep -E 'ID: RUSTSEC'`, look for an open dependency bump PR, and treat the merged bump as a code-defect fix. | | A pushed tag started no `Release Tag` run | Wait a few minutes and check `gh run list --workflow release-tag.yml --branch "$TAG"` again. Only after confirming that no run exists, and with approval, dispatch it with `gh workflow run release-tag.yml --ref main -f tag="$TAG"`. | From 4e99714fa11bf34d74a49aa099bbb81f078b1987 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 15 Oct 2026 10:27:22 -0700 Subject: [PATCH 08/20] docs(skills): drop Codex Security from release captain run matching Codex Security no longer runs in release qualification (#4469), so the stable release can't rely on a pre-release's Codex result and runs can't be matched by its job name. Match dispatched runs by the qualification artifact name instead. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 2cb58c7d72..9a6dd7ef20 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -47,10 +47,10 @@ tags; the RFC's cleanup step is not implemented and tag deletion is restricted. - Pushing any release tag by hand, pre-release or stable, starts `Release Tag`. Don't also dispatch it, or two runs race to publish. Auto-Tag dispatches explicitly only because its `GITHUB_TOKEN` push can't trigger workflows. -- A stable `vX.Y.Z` tag push rebuilds and - requalifies the commit. Codex Security runs only for pre-releases, so the - stable release relies on the selected pre-release's security result. Always - tag the commit of a pre-release that passed. +- A stable `vX.Y.Z` tag push rebuilds and requalifies the commit. Always tag + the commit of a pre-release that passed. +- Codex Security is temporarily excluded from release qualification (#4469). + `CI.md` explains how to run it by hand against a `-pre.N` candidate. - Only stable tags create a GitHub Release, so pre-releases never appear on the Releases page. A passing pre-release publishes its images, Helm chart, and qualification summary to GHCR. Its Debian, RPM, and macOS packages stay as @@ -97,8 +97,13 @@ gh run list --workflow release-tag.yml --branch "$TAG" gh run list --workflow release-tag.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")" ``` -If a commit has several dispatched runs, the one whose Codex Security job name -contains `($TAG)` is the match. That job appears only after the images build. +If a commit has several dispatched runs, the one with a +`qualification-$TAG-run-*` artifact is the match. That artifact appears only +after qualification finishes: + +```shell +gh api "repos/NVIDIA/OpenShell/actions/runs//artifacts?per_page=100" --jq '.artifacts[].name' | grep "qualification-$TAG-" +``` ## Monday: pre-flight From a5834d2689d89a595b1a35fe4a01178fdf7dfc70 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 19 Oct 2026 10:18:18 -0700 Subject: [PATCH 09/20] docs(skills): describe where and how to post the release coordination thread Name the CNCF Slack channel, note that the release ships from main with no release branch or cherry-picks, and add an example post. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 9a6dd7ef20..eade3e2ff7 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -192,8 +192,13 @@ Finish these on Monday so Tuesday is only the cut. gives a qualifiable candidate a day sooner. 6. **Draft the coordination post.** Release coordination happens in a thread - in the OpenShell CNCF Slack channel, started on Monday. Draft it for the - captain to post: + the captain starts on Monday in the OpenShell channel of the CNCF Slack + (`cloud-native.slack.com`, channel `C0B6Y8WQHU1`). Also link the thread in + the internal team channel. The thread asks contributors to link ready PRs + they want in the release. The release ships from `main`; there's no release + branch and no cherry-picking. The list exists so the captain can get those + PRs reviewed and merged in order before the cutoff. Tag reviewers on the + linked PRs. Draft it for the captain to post: - the target version and current candidate (tag, commit, run link); - qualification and compatibility status, with an owner for each blocker; @@ -208,6 +213,16 @@ Finish these on Monday so Tuesday is only the cut. gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"' ``` + Example post: + + ```text + :thread: OpenShell vX.Y.Z release, Tuesday . I'm release captain this week. + + Status: vX.Y.Z-pre.N () passed qualification. + + Reply here with any ready PRs you'd like in vX.Y.Z. Merge cutoff is () Tuesday, before the next pre-release is cut. Anything merged on main by then ships; anything later goes in the next release. + ``` + ## Tuesday: cut the release 1. **Select the final candidate.** After the cutoff and Tuesday's Auto-Tag run, From 58a369a696892301e3c76254fe40e5154d567565 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 19 Oct 2026 10:23:51 -0700 Subject: [PATCH 10/20] docs(skills): make seeding the next train the outgoing captain's handoff Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index eade3e2ff7..437089d7ac 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -277,7 +277,10 @@ Finish these on Monday so Tuesday is only the cut. 6. **Start the next train.** Until someone pushes the next `-pre.1`, Auto-Tag logs `vX.Y.Z is already stable — skipping tag creation` and does nothing. - Start the train soon after the stable release; it can be any weekday. + Starting it is this week's captain's job, not next week's: once `-pre.1` + exists, Auto-Tag cuts `-pre.2` onward every weekday, so next week's captain + starts Monday with pre-releases already qualifying. Start the train soon + after the stable release; it can be any weekday. Agree with the maintainers whether the next train is patch (the default) or minor (notable features or planned breaking changes; see the From 9a3f3ae0c2a997b393b8b1d041abddbba2f944ea Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 19 Oct 2026 10:29:24 -0700 Subject: [PATCH 11/20] docs(skills): update release captain for the Release workflow and add usage guide Stable releases are now dispatched with release.yml, which selects the qualified pre-release and creates the tag, and pre-releases run in release-prerelease.yml. Replace the stable tag push steps and stale recovery rows, add a preview of the selected candidate, add a section on when to use each part of the skill, and trim redundant explanations. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 312 ++++++++++-------------- 1 file changed, 132 insertions(+), 180 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 437089d7ac..3ad67e20ae 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -1,6 +1,6 @@ --- name: release-captain -description: Run the weekly OpenShell release captain checklist. Monday checks that the current pre-release train is qualified, triages qualification, compatibility, and security reports, and prepares the release coordination post. Tuesday selects the final qualified pre-release, tags its commit as stable after maintainer approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release. +description: Run the weekly OpenShell release captain checklist. Monday checks that the current pre-release train is qualified, triages qualification, compatibility, and security reports, and prepares the release coordination post. Tuesday previews the qualified pre-release that Release will promote, dispatches Release after maintainer approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release. metadata: internal: true --- @@ -14,6 +14,19 @@ merges, and cuts the release. `CI.md` describes the release workflows, qualification profile, and scanner policies. This skill covers what the captain does with them. +## How to use this skill + +Ask for the part of the week you're in: + +| When | Ask | What happens | +|---|---|---| +| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists and Auto-Tag is cutting pre-releases. If not, ask this week's captain to start the train (Tuesday step 6). Nothing else is needed until Monday. | +| Monday | "Run the release captain Monday checklist." | Monday steps 1–6: candidate status, failure triage, compatibility review, canary, and a draft coordination post. | +| A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 5 and Recovery: classify the failure, propose a rerun or fix, and draft the `#openshell-triage` reply. | +| Tuesday | "Cut the release." | Tuesday steps 1–7: preview the candidate, dispatch `Release` after approval, verify, and start the next train. | + +Every command that changes something is shown for approval first. + ## Guardrails Without explicit approval from the release captain or a maintainer for that @@ -28,43 +41,37 @@ specific action, never: Ask for approval with the exact command and target commit SHA. Every other step in this skill is read-only. -Never repoint a stable tag at a different commit, and never re-dispatch a -release while an earlier run for the same tag might still start. `Release Tag` -has no concurrency guard, so two runs race to publish. Don't delete `-pre.N` -tags; the RFC's cleanup step is not implemented and tag deletion is restricted. +Never repoint a stable tag, and don't delete `-pre.N` tags; tag deletion is +restricted. `Pre-release` has no concurrency guard, so never start a second run +for a tag while an earlier one might still start. ## How the train works - A train has one base version, such as `v0.1.4`, fixed by its `-pre.1` tag. A patch train can't take breaking changes to Stable interfaces. Those wait for a minor train. -- `Release Auto-Tag` runs Monday through Friday at 14:00 Europe/Zurich. That's - 12:00 UTC during Central European Summer Time and 13:00 UTC in winter. If - `main` has commits since the latest `-pre.N`, it tags the next one and - dispatches `Release Tag` (about 45 minutes). It skips after the base version - gets a stable tag, and fails if the latest pre-release isn't an ancestor of - `main`. A skip still counts as a successful run. -- Pushing any release tag by hand, pre-release or stable, starts `Release Tag`. - Don't also dispatch it, or two runs race to publish. Auto-Tag dispatches - explicitly only because its `GITHUB_TOKEN` push can't trigger workflows. -- A stable `vX.Y.Z` tag push rebuilds and requalifies the commit. Always tag - the commit of a pre-release that passed. -- Codex Security is temporarily excluded from release qualification (#4469). - `CI.md` explains how to run it by hand against a `-pre.N` candidate. -- Only stable tags create a GitHub Release, so pre-releases never appear on the - Releases page. A passing pre-release publishes its images, Helm chart, and - qualification summary to GHCR. Its Debian, RPM, and macOS packages stay as - artifacts on the `Release Tag` run. To check a pre-release, look at the tag - and that run. -- OpenShell TriageBot posts a top-level message in `#openshell-triage` - (`C0C20SDLDNW`) for every failed `Release Tag` attempt, with a link to the - run. Each rerun that fails gets its own message. +- `Release Auto-Tag` runs Monday through Friday at 14:00 Europe/Zurich (12:00 + UTC in summer, 13:00 UTC in winter). If `main` has commits since the latest + `-pre.N`, it tags the next one and dispatches `Pre-release` + (`release-prerelease.yml`, about 45 minutes). It skips when the train has no + `-pre.1` or the base version is already stable. A skip still counts as a + successful run. +- `Pre-release` builds, then publishes and qualifies in parallel. Its artifacts + are published even when qualification fails. Pushing a `-pre.N` tag by hand + starts it; don't also dispatch it. +- `Release` (`release.yml`) is dispatched by hand with the stable version. It + picks the highest `-pre.N` on `main` whose whole `Pre-release` run passed, + creates the stable tag on that commit, then rebuilds and publishes without + requalifying. Pushing a stable tag does nothing. +- Pre-releases never appear on the Releases page. Their images, Helm chart, and + qualification summary go to GHCR; packages stay as run artifacts. +- OpenShell TriageBot posts in `#openshell-triage` (`C0C20SDLDNW`) for every + failed `Pre-release` attempt, with links to the failed jobs. ## Setup Run from an OpenShell checkout with an authenticated `gh` CLI. Fetch tags with -`--force`, because release tags have been removed and recreated, and a plain -fetch keeps stale local copies: +`--force`, because release tags have been removed and recreated: ```shell export GH_REPO=NVIDIA/OpenShell @@ -74,35 +81,32 @@ echo "Latest stable: $PREV" git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[0-9]+$' | head -5 ``` -Release workflows change often, and a local branch may be days behind. Before -relying on trigger or job behavior, including anything in this skill, read the -current workflow from `origin/main`: +Release workflows change often. Before relying on anything in this skill, check +the release workflows on `origin/main` and their recent history: ```shell -git show origin/main:.github/workflows/release-tag.yml | sed -n '/^on:/,/^permissions:/p' +git ls-tree --name-only origin/main .github/workflows/ | grep release +git log --oneline -5 origin/main -- '.github/workflows/release*.yml' ``` -Release tags are lightweight, matching the tags Auto-Tag creates. The tag -commands below pass `-c tag.gpgSign=false` and read from `/dev/null`. With -`tag.gpgSign` set, `git tag` would otherwise create a signed annotated tag and -wait in an editor for a message. - -Find the `Release Tag` run for a tag. A tag pushed by hand runs on the tag -itself. An Auto-Tag run is dispatched on `main`, so its head commit is the -tagged commit: +Find the `Pre-release` run for a tag. A tag pushed by hand runs on the tag; an +Auto-Tag run is dispatched on `main` at the tagged commit: ```shell TAG=vX.Y.Z-pre.N -gh run list --workflow release-tag.yml --branch "$TAG" -gh run list --workflow release-tag.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")" +gh run list --workflow release-prerelease.yml --branch "$TAG" +gh run list --workflow release-prerelease.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")" ``` -If a commit has several dispatched runs, the one with a -`qualification-$TAG-run-*` artifact is the match. That artifact appears only -after qualification finishes: +Preview the candidate `Release` would promote. This runs the same selection +script, read-only, from a temporary `origin/main` checkout: ```shell -gh api "repos/NVIDIA/OpenShell/actions/runs//artifacts?per_page=100" --jq '.artifacts[].name' | grep "qualification-$TAG-" +VERSION=vX.Y.Z +wt=$(mktemp -d) +git worktree add -q --detach "$wt" origin/main +(cd "$wt" && GITHUB_REPOSITORY=NVIDIA/OpenShell uv run -q --no-project python tasks/scripts/select-release-candidate.py "$VERSION" --branch main | jq .) +git worktree remove --force "$wt" ``` ## Monday: pre-flight @@ -110,46 +114,40 @@ gh api "repos/NVIDIA/OpenShell/actions/runs//artifacts?per_page=100" --j Finish these on Monday so Tuesday is only the cut. 1. **Check the train.** Confirm that the latest `-pre.N` targets the version you - plan to ship. Check that today's `Release Auto-Tag` run (14:00 - Europe/Zurich) succeeded, and check from its log whether it tagged or - skipped: + plan to ship, and whether today's Auto-Tag run tagged or skipped: ```shell - gh run list --workflow release-auto-tag.yml --limit 3 gh run view "$(gh run list --workflow release-auto-tag.yml --limit 1 --json databaseId --jq '.[0].databaseId')" --log | grep -E 'Next tag:|— skipping' | grep -v echo ``` If the next version has no `-pre.1`, start the train (Tuesday step 6). -2. **Read the qualification summary for the latest attempt.** The summary's - `tag` and `source_sha` must match the candidate, and `current_profile_passed` - must be `true`: +2. **Check the candidate.** Run the preview from Setup. It shows the newest + pre-release that would ship. If the latest `-pre.N` isn't the candidate, + its run failed; read why: ```shell RUN= - ATTEMPT=$(gh run view "$RUN" --json attempt --jq .attempt) - dir=$(mktemp -d) - gh run download "$RUN" -n "qualification-$TAG-run-$RUN-attempt-$ATTEMPT" -D "$dir/q" - jq '{tag, source_sha, current_profile_passed, suites}' "$dir/q/qualification-summary.json" - git rev-parse "$TAG^{commit}" + gh run view "$RUN" --json jobs --jq '.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out") | "\(.name)\t\(.url)"' + gh run view "$RUN" --log-failed ``` - For any failed suite, read the log with `gh run view "$RUN" --log-failed`. `CI.md` lists which scanners block. Use the `watch-github-actions` skill for CI diagnosis. -3. **Triage the advisory compatibility review.** It never blocks publication, - and a green job doesn't mean it found no breaks: +3. **Triage the advisory compatibility review** for the candidate's run. It + never blocks publication, and a green job doesn't mean it found no breaks: ```shell - gh run download "$RUN" -n "compatibility-review-run-$RUN-attempt-$ATTEMPT" -D "$dir/c" - jq '{status, verdict: .review.verdict}' "$dir/c/report.json" - cat "$dir/c/report.md" + ATTEMPT=$(gh run view "$RUN" --json attempt --jq .attempt) + dir=$(mktemp -d) + gh run download "$RUN" -n "compatibility-review-run-$RUN-attempt-$ATTEMPT" -D "$dir" + jq '{status, verdict: .review.verdict}' "$dir/report.json" + cat "$dir/report.md" ``` - Only `complete` with `no_unaddressed_breaks` is a clean review. A missing - report or any other result means there's no assessment. For each finding, - decide whether it's a real break, a false positive, or unclear: + Only `complete` with `no_unaddressed_breaks` is a clean review. Anything + else means there's no assessment. For each finding: - **Real break:** needs a fix or revert before Tuesday, or must wait for a minor train. @@ -159,55 +157,43 @@ Finish these on Monday so Tuesday is only the cut. The reviewer treats every `pub` Rust item as a Stable interface. No workspace crate is published to crates.io, so a finding in an internal - crate such as `openshell-core` or `openshell-isolation-interface` is often a - false positive. Check whether a supported SDK surface, such as - `openshell-sdk`, exposes the item before deciding. + crate such as `openshell-core` is often a false positive. Check whether a + supported SDK surface, such as `openshell-sdk`, exposes the item. 4. **Check the dev canary.** `Release Canary` smoke-tests `main` dev artifacts. - It doesn't gate the tag, but treat a red canary as a release risk. Use the + It doesn't gate the release, but treat a red canary as a risk. Use the `test-release-canary` skill to diagnose it: ```shell gh run list --workflow release-canary.yml --limit 3 ``` -5. **Plan fixes.** A failed pre-release doesn't end the train. The goal is a - green pre-release before Tuesday, so fix forward: - - - Reply in the thread of the failure's `#openshell-triage` message, so - others know someone is working on a fix. Post the cause and the fix or - rerun, then post again when a pre-release passes. Draft these replies for - the captain to post. - - A transient infrastructure failure can be rerun after approval. Runner - setup errors, registry push errors, and integration tests that fail only - some of the time are common. If a failure looks unrelated to the commit's - changes, rerun once before treating it as a code defect. +5. **Plan fixes.** A failed pre-release doesn't end the train. Fix forward to a + green pre-release before Tuesday: + + - Reply in the thread of the failure's `#openshell-triage` message with the + cause and the fix or rerun, then again when a pre-release passes. Draft + these replies for the captain to post. + - Rerun a transient infrastructure failure once, after approval: runner + setup errors, registry push errors, and intermittent integration tests. - A code defect, or a new advisory against a locked dependency, needs a - merged fix and a new pre-release. Don't rerun or promote the failed one. - - Let the failed run finish rather than cancelling it. Its other suites - still show whether the rest of the commit is healthy. + merged fix and a new pre-release. Don't rerun the failed one. + - Let a failed run finish rather than cancelling it. Its other suites still + show whether the rest of the commit is healthy. - Once the fix merges and its `Release Dev` run passes, tag the next - `-pre.N` on that commit by hand after approval (as in Tuesday step 6, with - the next number), instead of waiting for the next Auto-Tag run. That - gives a qualifiable candidate a day sooner. - -6. **Draft the coordination post.** Release coordination happens in a thread - the captain starts on Monday in the OpenShell channel of the CNCF Slack - (`cloud-native.slack.com`, channel `C0B6Y8WQHU1`). Also link the thread in - the internal team channel. The thread asks contributors to link ready PRs - they want in the release. The release ships from `main`; there's no release - branch and no cherry-picking. The list exists so the captain can get those - PRs reviewed and merged in order before the cutoff. Tag reviewers on the - linked PRs. Draft it for the captain to post: - - - the target version and current candidate (tag, commit, run link); - - qualification and compatibility status, with an owner for each blocker; - - ready but unmerged PRs that should make the release, in merge order; - - the merge cutoff in UTC and the captain's time zone. Set it before - Tuesday's 14:00 Europe/Zurich Auto-Tag run so the last merges make that - pre-release. - - Candidate PRs: + `-pre.N` on that commit by hand after approval (as in Tuesday step 6), + instead of waiting a day for Auto-Tag. + +6. **Draft the coordination post.** The captain starts a thread on Monday in the + OpenShell channel of the CNCF Slack (`cloud-native.slack.com`, channel + `C0B6Y8WQHU1`) and links it in the internal team channel. Contributors reply + with ready PRs they want in the release. The release ships from `main`, with + no release branch or cherry-picks, so the list is for getting those PRs + reviewed and merged before the cutoff. Tag reviewers on the linked PRs. + + Set the cutoff before the last Auto-Tag run you'll take on Tuesday (14:00 + Europe/Zurich), or before a pre-release you'll tag by hand. Approved open + PRs: ```shell gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"' @@ -225,87 +211,62 @@ Finish these on Monday so Tuesday is only the cut. ## Tuesday: cut the release -1. **Select the final candidate.** After the cutoff and Tuesday's Auto-Tag run, - pick the newest `-pre.N` whose qualification passed, and repeat Monday steps - 2 and 3 for it. If the newest pre-release failed, either merge a fix and - qualify another pre-release, or pick an earlier one that passed and confirm - that leaving out the later commits is acceptable. +1. **Select the final candidate.** After the cutoff and the last pre-release + has finished, run the preview from Setup and repeat Monday step 3 for its + run. If the newest pre-release failed, either fix it (Monday step 5) or + confirm that shipping the earlier candidate without the later commits is + acceptable. If qualification passed but publication failed, rerun the + failed jobs first; `Release` needs the whole run to pass. -2. **Request approval.** Present the candidate tag, commit SHA, qualification - run, compatibility assessment, and the commits since `$PREV`: +2. **Request approval.** Present the candidate tag, commit, qualification run, + compatibility assessment, and the commits since `$PREV`: ```shell - git log --oneline --no-merges "$PREV..$TAG" + git log --oneline --no-merges "$PREV.." ``` -3. **Tag the stable release.** After approval, check the candidate, then tag its - exact commit and push only that tag: +3. **Dispatch the release.** After approval: ```shell - VERSION=vX.Y.Z - SHA=$(git rev-parse "$TAG^{commit}") - [[ "$VERSION" == "${TAG%-pre.*}" ]] || echo "version does not match the train" - [[ "$SHA" == "$(jq -r .source_sha "$dir/q/qualification-summary.json")" ]] || echo "commit does not match qualification" - git tag --points-at "$SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag" - git -c tag.gpgSign=false tag "$VERSION" "$SHA" --exit-status ``` - If no run appears within a few minutes, or the run stays queued, see - Recovery. - -5. **Verify publication.** Confirm that the GitHub Release exists as a - non-pre-release with assets and generated notes, and that no job failed: +5. **Verify publication.** Confirm the GitHub Release has assets and that no + job failed: ```shell - gh release view "$VERSION" --json name,isPrerelease,publishedAt,assets --jq '{name, isPrerelease, publishedAt, assets: (.assets | length)}' + gh release view "$VERSION" --json isPrerelease,publishedAt,assets --jq '{isPrerelease, publishedAt, assets: (.assets | length)}' gh run view --json jobs --jq '.jobs[] | select(.conclusion != "success" and .conclusion != "skipped") | "\(.conclusion)\t\(.name)"' ``` `Trigger Wheel Publish` only starts the GitLab pipeline, so confirm that the - Python wheel reached PyPI separately. + wheel reached PyPI separately. -6. **Start the next train.** Until someone pushes the next `-pre.1`, Auto-Tag - logs `vX.Y.Z is already stable — skipping tag creation` and does nothing. - Starting it is this week's captain's job, not next week's: once `-pre.1` - exists, Auto-Tag cuts `-pre.2` onward every weekday, so next week's captain - starts Monday with pre-releases already qualifying. Start the train soon - after the stable release; it can be any weekday. +6. **Start the next train.** This is this week's captain's job, not next + week's. Until the next `-pre.1` exists, Auto-Tag skips. Once it does, + Auto-Tag cuts `-pre.2` onward every weekday, so next week's captain starts + Monday with qualified pre-releases. Do it soon after the release. Agree with the maintainers whether the next train is patch (the default) or minor (notable features or planned breaking changes; see the - [version selection supplement](../../../rfc/0014-release-stability/release-version-selection.md)). - List the commit types since the release to inform the decision: + [version selection supplement](../../../rfc/0014-release-stability/release-version-selection.md)): ```shell git log --format=%s --no-merges "$VERSION..origin/main" | sed -E 's/^([a-z]+)(\(.*\))?(!?):.*/\1\3/' | sort | uniq -c ``` - Seed the train on a `main` commit that: - - - comes after `$VERSION`; - - has no stable tag of its own; - - has a successful `Release Dev` run. - - Release versioning prefers a stable tag over a pre-release tag on the same - commit, which would mislabel the packages. If nothing has merged since the - release, wait. - - Branch Checks already passed in the merge queue before each merge. - `Release Dev` runs the post-merge E2E and dev publication on every commit. A - newer push cancels the older run, so `cancelled` means superseded, not - failed. Pick the newest commit whose run succeeded, or wait for the tip's - run to finish: + Seed it on a `main` commit after `$VERSION`, without a stable tag of its own + (versioning prefers the stable tag), and with a successful `Release Dev` run. + `cancelled` means a newer push superseded the run. If nothing has merged + since the release, wait. ```shell for sha in $(git rev-list --max-count=10 "$VERSION..origin/main"); do @@ -314,27 +275,18 @@ Finish these on Monday so Tuesday is only the cut. done ``` - After approval, push the tag. The push starts `Release Tag`; don't dispatch - it as well: + After approval, push the tag, which starts `Pre-release`. The `-c` and + `/dev/null` keep `git` from signing the tag and opening an editor: ```shell NEXT=vX.Y.Z-pre.1 NEXT_SHA=
git merge-base --is-ancestor "$VERSION" "$NEXT_SHA" || echo "commit does not contain the release" - git tag --points-at "$NEXT_SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag" git -c tag.gpgSign=false tag "$NEXT" "$NEXT_SHA" --failed`, then recheck the new attempt's summary. Reply in the failure's `#openshell-triage` thread with the result. | -| Code defect in a pre-release | Fix forward (Monday step 5): merge the fix, then, after approval, tag and push the next `-pre.N` on the fix commit once its `Release Dev` run passes. Waiting for Auto-Tag also works but costs up to a day. | -| `Cargo Deny` fails a pre-release with `error[vulnerability]` | A new RustSec advisory was published against a locked dependency. Code that passed earlier can start failing this way. Rerunning won't help. Find the advisory IDs with `gh run view --log-failed \| grep -E 'ID: RUSTSEC'`, look for an open dependency bump PR, and treat the merged bump as a code-defect fix. | -| A pushed tag started no `Release Tag` run | Wait a few minutes and check `gh run list --workflow release-tag.yml --branch "$TAG"` again. Only after confirming that no run exists, and with approval, dispatch it with `gh workflow run release-tag.yml --ref main -f tag="$TAG"`. | -| Stable run stuck queued | Don't re-dispatch or delete the tag. Check for a GitHub Actions incident and ask a maintainer to cancel the run (`gh run cancel `, or force-cancel through the API). Only after it shows `cancelled` or `failed`, rerun it after approval. | -| Stable run fails before the `Release` job, transient | Nothing was published. After approval, run `gh run rerun --failed`. | -| Stable run fails before the `Release` job, code defect | Nothing was published, and Auto-Tag won't create pre-releases for a base version that has a stable tag. Ask the maintainers to choose: remove the unpublished tag (needs a ruleset bypass) and recut from a qualified fix, or skip to the next patch version. | -| Stable run fails after the `Release` job | The GitHub Release exists. Check which publish jobs succeeded. A rerun of an npm or snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. | +| Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the candidate. Reply in the failure's `#openshell-triage` thread with the result. | +| Pre-release qualification passed but publication failed | Usually a registry error, such as GHCR `failed to perform "Tag"`. Rerun failed jobs after approval; `Release` won't select the run until it fully passes. | +| Code defect in a pre-release | Fix forward (Monday step 5): merge the fix, then tag the next `-pre.N` on it after its `Release Dev` run passes. | +| `Cargo Deny` fails with `error[vulnerability]` | A new RustSec advisory hit a locked dependency; rerunning won't help. Find the IDs with `gh run view --log-failed \| grep -E 'ID: RUSTSEC'`, find or open the bump PR, and treat it as a code defect. | +| A pushed `-pre.N` tag started no run | Wait a few minutes and check again. Only after confirming no run exists, and with approval, run `gh workflow run release-prerelease.yml --ref main -f tag="$TAG"`. | +| `Release` fails in `Select Qualified Prerelease` | No candidate qualified, or its evidence expired. Nothing was tagged or published. Fix the train, then dispatch again. | +| `Release` fails after selection, transient | After approval, run `gh run rerun --failed`. A stable tag that already points at the selected commit can be retried. | +| `Release` fails after the `Release` job | The GitHub Release exists. A rerun of an npm or snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. | +| `Release` stuck queued | Releases queue behind each other. Check for another running `Release` or a GitHub Actions incident before asking a maintainer to cancel. Don't dispatch again. | ## Related -- `release-openshell`: the short procedure for selecting a qualified - pre-release and tagging it stable. +- `release-openshell`: the short procedure behind Tuesday steps 1–5. - `watch-github-actions`: general `gh run` monitoring and log retrieval. - `test-release-canary`: diagnose dev artifact smoke-test failures. - `CI.md`: release workflows, qualification profile, and scanner policies. From 20001958112253e4c92a90843be45ea120eff677 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 19 Oct 2026 10:45:09 -0700 Subject: [PATCH 12/20] docs(skills): explain what the release cutoff guarantees The release ships the final pre-release's commit only if it passes, so the coordination post must not promise that every PR merged by the cutoff ships. Describe the Auto-Tag and hand-tagged cutoffs, and tag the final pre-release by hand on Tuesday for a later cutoff. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 32 +++++++++++++++++++------ 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 3ad67e20ae..c0ccd595e9 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -191,9 +191,21 @@ Finish these on Monday so Tuesday is only the cut. no release branch or cherry-picks, so the list is for getting those PRs reviewed and merged before the cutoff. Tag reviewers on the linked PRs. - Set the cutoff before the last Auto-Tag run you'll take on Tuesday (14:00 - Europe/Zurich), or before a pre-release you'll tag by hand. Approved open - PRs: + The release ships one commit: the final pre-release's, if it passes. A PR + ships only if that pre-release includes it. If the final pre-release fails + and can't be fixed in time, `Release` falls back to an earlier passing one, + and later merges miss the release. Pick one of two cutoffs: + + - **Before Tuesday's Auto-Tag run** (14:00 Europe/Zurich): Auto-Tag tags the + tip of `main`, so everything merged by the cutoff is in it, along with + anything merged before the run starts. Suits a captain in Europe. + - **Later on Tuesday:** after the cutoff, the captain tags the final + pre-release by hand on a commit they choose, then waits about 45 minutes + for it to qualify. Suits a captain in the Americas, where the Auto-Tag run + is before the workday. + + Don't promise that everything merged by the cutoff ships, or name the next + version; it's a patch by default but may be a minor. Approved open PRs: ```shell gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"' @@ -206,14 +218,20 @@ Finish these on Monday so Tuesday is only the cut. Status: vX.Y.Z-pre.N () passed qualification. - Reply here with any ready PRs you'd like in vX.Y.Z. Merge cutoff is () Tuesday, before the next pre-release is cut. Anything merged on main by then ships; anything later goes in the next release. + Reply here with any ready PRs you'd like in vX.Y.Z. Merge cutoff is () Tuesday. I'll cut the final pre-release after the cutoff, and vX.Y.Z ships from it if it passes qualification. Anything that misses it goes in the next release. ``` + For an Auto-Tag cutoff, replace the second sentence with "Tuesday's + automatic pre-release picks up everything merged by then, and vX.Y.Z ships + from it if it passes qualification." + ## Tuesday: cut the release -1. **Select the final candidate.** After the cutoff and the last pre-release - has finished, run the preview from Setup and repeat Monday step 3 for its - run. If the newest pre-release failed, either fix it (Monday step 5) or +1. **Select the final candidate.** With a cutoff after Auto-Tag, first tag the + final `-pre.N` by hand after approval, on the newest commit with a + successful `Release Dev` run (commands as in Tuesday step 6). Once the final + pre-release has finished, run the preview from Setup and repeat Monday + step 3 for its run. If the newest pre-release failed, either fix it (Monday step 5) or confirm that shipping the earlier candidate without the later commits is acceptable. If qualification passed but publication failed, rerun the failed jobs first; `Release` needs the whole run to pass. From c17528e6d478635628b27dc899cf3d0537907a54 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 08:49:33 -0700 Subject: [PATCH 13/20] docs(skills): add release notes review and v0.1.5 release day lessons Review the release notes preview from the Pre-release run before approval, and describe how Release regenerates them. Correct the thread when the shipped pre-release was cut before the posted cutoff, require maintainer agreement for accepted-risk breaks in a patch release, and add recovery rows for startup failures and release notes failures. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 36 +++++++++++++++++++++---- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index c0ccd595e9..d6d4c884e5 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -150,7 +150,10 @@ Finish these on Monday so Tuesday is only the cut. else means there's no assessment. For each finding: - **Real break:** needs a fix or revert before Tuesday, or must wait for a - minor train. + minor train. A break whose author accepted the risk on the PR can ship + in a patch only if the maintainers agree to the exception (as for the + Go SDK change in v0.1.5). Check that the release notes preview (Tuesday + step 2) gives upgrade steps for it. - **Unclear:** ask an owner of that code. - **False positive:** send it to the owner of `.github/prompts/compatibility-review.md` so the prompt can be tuned. @@ -225,22 +228,43 @@ Finish these on Monday so Tuesday is only the cut. automatic pre-release picks up everything merged by then, and vX.Y.Z ships from it if it passes qualification." + The posted cutoff must match the pre-release that ships. If you end up + shipping one cut before the posted cutoff, such as Auto-Tag's or a fix-up + tagged early, reply in the thread with its commit and time, and say that + later merges go in the next release. For v0.1.5, the post said 16:00 UTC, + but `pre.7` was cut at 12:38 UTC. + ## Tuesday: cut the release 1. **Select the final candidate.** With a cutoff after Auto-Tag, first tag the final `-pre.N` by hand after approval, on the newest commit with a successful `Release Dev` run (commands as in Tuesday step 6). Once the final pre-release has finished, run the preview from Setup and repeat Monday - step 3 for its run. If the newest pre-release failed, either fix it (Monday step 5) or - confirm that shipping the earlier candidate without the later commits is - acceptable. If qualification passed but publication failed, rerun the + step 3 for its run. If the newest pre-release failed, either fix it (Monday + step 5) or confirm that shipping the earlier candidate without the later + commits is acceptable. If qualification passed but publication failed, rerun the failed jobs first; `Release` needs the whole run to pass. 2. **Request approval.** Present the candidate tag, commit, qualification run, - compatibility assessment, and the commits since `$PREV`: + compatibility assessment, release notes preview, and the commits since + `$PREV`: ```shell git log --oneline --no-merges "$PREV.." + gh run download "$RUN" -n release-notes -D "$dir/notes" && cat "$dir/notes/release-notes.md" + ``` + + Release notes are generated from the merged PRs, including each PR's + "Release Notes" section, by `release-notes.yml`. Every `Pre-release` run + writes an advisory preview as its `release-notes` artifact. `Release` + writes them again in `Publish / Write Release Notes` and uses them as the + GitHub Release body, so the final text can differ slightly from the + preview. Check that upgrade notes cover every accepted break. To fix the + notes, edit the PR's Release Notes section and preview again without + publishing: + + ```shell + gh workflow run release.yml --ref main -f tag="$VERSION" -f notes-only=true ``` 3. **Dispatch the release.** After approval: @@ -314,10 +338,12 @@ Finish these on Monday so Tuesday is only the cut. |---|---| | Transient infrastructure failure in a pre-release | After approval, run `gh run rerun --failed`, then recheck the candidate. Reply in the failure's `#openshell-triage` thread with the result. | | Pre-release qualification passed but publication failed | Usually a registry error, such as GHCR `failed to perform "Tag"`. Rerun failed jobs after approval; `Release` won't select the run until it fully passes. | +| Pre-release run ends in `startup_failure` with no jobs | The workflow file is invalid at that commit, for example a missing permission (`v0.1.5-pre.6`). Rerunning won't help. Merge the workflow fix, then tag the next `-pre.N` on it. | | Code defect in a pre-release | Fix forward (Monday step 5): merge the fix, then tag the next `-pre.N` on it after its `Release Dev` run passes. | | `Cargo Deny` fails with `error[vulnerability]` | A new RustSec advisory hit a locked dependency; rerunning won't help. Find the IDs with `gh run view --log-failed \| grep -E 'ID: RUSTSEC'`, find or open the bump PR, and treat it as a code defect. | | A pushed `-pre.N` tag started no run | Wait a few minutes and check again. Only after confirming no run exists, and with approval, run `gh workflow run release-prerelease.yml --ref main -f tag="$TAG"`. | | `Release` fails in `Select Qualified Prerelease` | No candidate qualified, or its evidence expired. Nothing was tagged or published. Fix the train, then dispatch again. | +| `Release` fails in `Write Release Notes` | The GitHub Release isn't created until the notes succeed. After approval, rerun failed jobs. If it keeps failing, check the inference API key and variables named in the job log. | | `Release` fails after selection, transient | After approval, run `gh run rerun --failed`. A stable tag that already points at the selected commit can be retried. | | `Release` fails after the `Release` job | The GitHub Release exists. A rerun of an npm or snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. | | `Release` stuck queued | Releases queue behind each other. Check for another running `Release` or a GitHub Actions incident before asking a maintainer to cancel. Don't dispatch again. | From 2a5e5364685f0bd047e4c9d8b4f14c9a49ecd2fb Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 10:30:34 -0700 Subject: [PATCH 14/20] docs(skills): make starting the next train the captain's final duty Move starting the next train after closing the loop, so it is the last Tuesday step. The captain's week ends when the next -pre.1 Pre-release run passes or its failure is handed off. Also move open milestone items when closing the loop. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 28 +++++++++++++++---------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index d6d4c884e5..7d80103c41 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -20,10 +20,10 @@ Ask for the part of the week you're in: | When | Ask | What happens | |---|---|---| -| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists and Auto-Tag is cutting pre-releases. If not, ask this week's captain to start the train (Tuesday step 6). Nothing else is needed until Monday. | +| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists and Auto-Tag is cutting pre-releases. If not, ask this week's captain to start the train (Tuesday step 7). Nothing else is needed until Monday. | | Monday | "Run the release captain Monday checklist." | Monday steps 1–6: candidate status, failure triage, compatibility review, canary, and a draft coordination post. | | A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 5 and Recovery: classify the failure, propose a rerun or fix, and draft the `#openshell-triage` reply. | -| Tuesday | "Cut the release." | Tuesday steps 1–7: preview the candidate, dispatch `Release` after approval, verify, and start the next train. | +| Tuesday | "Cut the release." | Tuesday steps 1–7: preview the candidate, dispatch `Release` after approval, verify, close the loop, and finish by starting the next train. | Every command that changes something is shown for approval first. @@ -120,7 +120,7 @@ Finish these on Monday so Tuesday is only the cut. gh run view "$(gh run list --workflow release-auto-tag.yml --limit 1 --json databaseId --jq '.[0].databaseId')" --log | grep -E 'Next tag:|— skipping' | grep -v echo ``` - If the next version has no `-pre.1`, start the train (Tuesday step 6). + If the next version has no `-pre.1`, start the train (Tuesday step 7). 2. **Check the candidate.** Run the preview from Setup. It shows the newest pre-release that would ship. If the latest `-pre.N` isn't the candidate, @@ -184,7 +184,7 @@ Finish these on Monday so Tuesday is only the cut. - Let a failed run finish rather than cancelling it. Its other suites still show whether the rest of the commit is healthy. - Once the fix merges and its `Release Dev` run passes, tag the next - `-pre.N` on that commit by hand after approval (as in Tuesday step 6), + `-pre.N` on that commit by hand after approval (as in Tuesday step 7), instead of waiting a day for Auto-Tag. 6. **Draft the coordination post.** The captain starts a thread on Monday in the @@ -238,7 +238,7 @@ Finish these on Monday so Tuesday is only the cut. 1. **Select the final candidate.** With a cutoff after Auto-Tag, first tag the final `-pre.N` by hand after approval, on the newest commit with a - successful `Release Dev` run (commands as in Tuesday step 6). Once the final + successful `Release Dev` run (commands as in Tuesday step 7). Once the final pre-release has finished, run the preview from Setup and repeat Monday step 3 for its run. If the newest pre-release failed, either fix it (Monday step 5) or confirm that shipping the earlier candidate without the later @@ -292,10 +292,16 @@ Finish these on Monday so Tuesday is only the cut. `Trigger Wheel Publish` only starts the GitLab pipeline, so confirm that the wheel reached PyPI separately. -6. **Start the next train.** This is this week's captain's job, not next - week's. Until the next `-pre.1` exists, Auto-Tag skips. Once it does, - Auto-Tag cuts `-pre.2` onward every weekday, so next week's captain starts - Monday with qualified pre-releases. Do it soon after the release. +6. **Close the loop.** Draft a reply to the coordination thread with the release + link and any PRs deferred to the next train. After approval, move the + release milestone's open items to the next milestone. + +7. **Start the next train.** This is the last duty of this week's captain, not + the first of next week's. Until the next `-pre.1` exists, Auto-Tag skips. + Once it does, Auto-Tag cuts `-pre.2` onward every weekday, so next week's + captain starts Monday with qualified pre-releases. Do it the same day as + the release. Your week isn't done until the `-pre.1` `Pre-release` run + passes, or its failure is triaged and handed to next week's captain. Agree with the maintainers whether the next train is patch (the default) or minor (notable features or planned breaking changes; see the @@ -329,8 +335,8 @@ Finish these on Monday so Tuesday is only the cut. gh run list --workflow release-prerelease.yml --branch "$NEXT" ``` -7. **Close the loop.** Draft a reply to the coordination thread with the release - link and any PRs deferred to the next train. + Watch the run to the end, then tell next week's captain the train is + running, and which `-pre.1` run to check on Monday. ## Recovery From 9137d48cf4db2a3bb069039162df78062d09a12d Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 10:40:55 -0700 Subject: [PATCH 15/20] docs(skills): trim the release captain skill to its durable steps Remove repeated guidance, week-specific examples, fragile commands, and debugging detail an agent can derive from the workflows. Keep the approval guardrails, the weekly steps, and a short recovery table. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 304 ++++++------------------ 1 file changed, 75 insertions(+), 229 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 7d80103c41..48354de1d2 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -1,6 +1,6 @@ --- name: release-captain -description: Run the weekly OpenShell release captain checklist. Monday checks that the current pre-release train is qualified, triages qualification, compatibility, and security reports, and prepares the release coordination post. Tuesday previews the qualified pre-release that Release will promote, dispatches Release after maintainer approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release. +description: Run the weekly OpenShell release captain checklist. Monday checks the pre-release train, triages failures and the compatibility review, and drafts the coordination post. Tuesday promotes the qualified pre-release after approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release. metadata: internal: true --- @@ -10,87 +10,51 @@ metadata: OpenShell publishes a stable release every Tuesday from a qualified pre-release ([RFC 0014](../../../rfc/0014-release-stability/README.md)). The release captain keeps the week's pre-release train green, coordinates the last -merges, and cuts the release. `CI.md` describes the release workflows, -qualification profile, and scanner policies. This skill covers what the captain -does with them. +merges, cuts the release, and starts the next train. ## How to use this skill -Ask for the part of the week you're in: - | When | Ask | What happens | |---|---|---| -| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists and Auto-Tag is cutting pre-releases. If not, ask this week's captain to start the train (Tuesday step 7). Nothing else is needed until Monday. | -| Monday | "Run the release captain Monday checklist." | Monday steps 1–6: candidate status, failure triage, compatibility review, canary, and a draft coordination post. | -| A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 5 and Recovery: classify the failure, propose a rerun or fix, and draft the `#openshell-triage` reply. | -| Tuesday | "Cut the release." | Tuesday steps 1–7: preview the candidate, dispatch `Release` after approval, verify, close the loop, and finish by starting the next train. | - -Every command that changes something is shown for approval first. +| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists. If not, ask this week's captain to start the train (Tuesday step 6). | +| Monday | "Run the release captain Monday checklist." | Monday steps 1–4. | +| A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 3 and Recovery. | +| Tuesday | "Cut the release." | Tuesday steps 1–6, ending with starting the next train. | ## Guardrails -Without explicit approval from the release captain or a maintainer for that -specific action, never: - -- push, move, or delete a tag; -- dispatch, cancel, or rerun any workflow; -- merge, approve, or enqueue a PR; -- post in Slack or comment on GitHub; -- install OpenShell or change the local machine. - -Ask for approval with the exact command and target commit SHA. Every other -step in this skill is read-only. - -Never repoint a stable tag, and don't delete `-pre.N` tags; tag deletion is -restricted. `Pre-release` has no concurrency guard, so never start a second run -for a tag while an earlier one might still start. +Get explicit approval, with the exact command and target SHA, before pushing a +tag, dispatching, rerunning, or cancelling a workflow, merging or approving a +PR, or posting in Slack or GitHub. Never repoint a stable tag, delete a `-pre.N` +tag, or start a second `Pre-release` run for a tag; it has no concurrency guard. ## How the train works - A train has one base version, such as `v0.1.4`, fixed by its `-pre.1` tag. A - patch train can't take breaking changes to Stable interfaces. Those wait for a - minor train. -- `Release Auto-Tag` runs Monday through Friday at 14:00 Europe/Zurich (12:00 - UTC in summer, 13:00 UTC in winter). If `main` has commits since the latest - `-pre.N`, it tags the next one and dispatches `Pre-release` - (`release-prerelease.yml`, about 45 minutes). It skips when the train has no - `-pre.1` or the base version is already stable. A skip still counts as a - successful run. -- `Pre-release` builds, then publishes and qualifies in parallel. Its artifacts - are published even when qualification fails. Pushing a `-pre.N` tag by hand - starts it; don't also dispatch it. + patch train can't take breaking changes to Stable interfaces. +- `Release Auto-Tag` runs on weekdays at 14:00 Europe/Zurich. If `main` has + commits since the latest `-pre.N`, it tags the next one and dispatches + `Pre-release` (`release-prerelease.yml`, about 45 minutes). It skips when the + train has no `-pre.1` or the base version is already stable. +- `Pre-release` builds, then publishes and qualifies in parallel, and writes a + `release-notes` preview artifact. Pushing a `-pre.N` tag by hand starts it. - `Release` (`release.yml`) is dispatched by hand with the stable version. It picks the highest `-pre.N` on `main` whose whole `Pre-release` run passed, - creates the stable tag on that commit, then rebuilds and publishes without - requalifying. Pushing a stable tag does nothing. -- Pre-releases never appear on the Releases page. Their images, Helm chart, and - qualification summary go to GHCR; packages stay as run artifacts. -- OpenShell TriageBot posts in `#openshell-triage` (`C0C20SDLDNW`) for every - failed `Pre-release` attempt, with links to the failed jobs. + creates the stable tag on that commit, and publishes without requalifying. +- OpenShell TriageBot posts every failed `Pre-release` attempt in + `#openshell-triage` (`C0C20SDLDNW`), with links to the failed jobs. ## Setup -Run from an OpenShell checkout with an authenticated `gh` CLI. Fetch tags with -`--force`, because release tags have been removed and recreated: - ```shell export GH_REPO=NVIDIA/OpenShell git fetch origin main --tags --force PREV=$(git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1) -echo "Latest stable: $PREV" -git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[0-9]+$' | head -5 -``` - -Release workflows change often. Before relying on anything in this skill, check -the release workflows on `origin/main` and their recent history: - -```shell -git ls-tree --name-only origin/main .github/workflows/ | grep release -git log --oneline -5 origin/main -- '.github/workflows/release*.yml' +git tag -l 'v*' --sort=-v:refname | grep -E -- '-pre\.[0-9]+$' | head -5 ``` Find the `Pre-release` run for a tag. A tag pushed by hand runs on the tag; an -Auto-Tag run is dispatched on `main` at the tagged commit: +Auto-Tag run is dispatched on `main`: ```shell TAG=vX.Y.Z-pre.N @@ -98,8 +62,7 @@ gh run list --workflow release-prerelease.yml --branch "$TAG" gh run list --workflow release-prerelease.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")" ``` -Preview the candidate `Release` would promote. This runs the same selection -script, read-only, from a temporary `origin/main` checkout: +Preview the candidate `Release` would promote, read-only: ```shell VERSION=vX.Y.Z @@ -111,20 +74,8 @@ git worktree remove --force "$wt" ## Monday: pre-flight -Finish these on Monday so Tuesday is only the cut. - -1. **Check the train.** Confirm that the latest `-pre.N` targets the version you - plan to ship, and whether today's Auto-Tag run tagged or skipped: - - ```shell - gh run view "$(gh run list --workflow release-auto-tag.yml --limit 1 --json databaseId --jq '.[0].databaseId')" --log | grep -E 'Next tag:|— skipping' | grep -v echo - ``` - - If the next version has no `-pre.1`, start the train (Tuesday step 7). - -2. **Check the candidate.** Run the preview from Setup. It shows the newest - pre-release that would ship. If the latest `-pre.N` isn't the candidate, - its run failed; read why: +1. **Check the candidate.** Run the preview. If the latest `-pre.N` isn't the + candidate, its run failed; read why: ```shell RUN= @@ -132,11 +83,10 @@ Finish these on Monday so Tuesday is only the cut. gh run view "$RUN" --log-failed ``` - `CI.md` lists which scanners block. Use the `watch-github-actions` skill for - CI diagnosis. + If the next version has no `-pre.1`, start the train (Tuesday step 6). -3. **Triage the advisory compatibility review** for the candidate's run. It - never blocks publication, and a green job doesn't mean it found no breaks: +2. **Triage the compatibility review** for the candidate's run. It's advisory, + and a green job doesn't mean it found no breaks: ```shell ATTEMPT=$(gh run view "$RUN" --json attempt --jq .attempt) @@ -146,126 +96,58 @@ Finish these on Monday so Tuesday is only the cut. cat "$dir/report.md" ``` - Only `complete` with `no_unaddressed_breaks` is a clean review. Anything - else means there's no assessment. For each finding: + Only `complete` with `no_unaddressed_breaks` is clean. A real break needs a + fix, a revert, a minor train, or a maintainer-approved exception with upgrade + steps in the release notes. - - **Real break:** needs a fix or revert before Tuesday, or must wait for a - minor train. A break whose author accepted the risk on the PR can ship - in a patch only if the maintainers agree to the exception (as for the - Go SDK change in v0.1.5). Check that the release notes preview (Tuesday - step 2) gives upgrade steps for it. - - **Unclear:** ask an owner of that code. - - **False positive:** send it to the owner of - `.github/prompts/compatibility-review.md` so the prompt can be tuned. +3. **Fix forward.** A failed pre-release doesn't end the train. Draft a reply in + the failure's `#openshell-triage` thread with the cause and the fix. Let the + failed run finish rather than cancelling it. Once the fix merges and its + `Release Dev` run passes, tag the next `-pre.N` on it by hand (as in Tuesday + step 6). See Recovery for reruns. - The reviewer treats every `pub` Rust item as a Stable interface. No - workspace crate is published to crates.io, so a finding in an internal - crate such as `openshell-core` is often a false positive. Check whether a - supported SDK surface, such as `openshell-sdk`, exposes the item. - -4. **Check the dev canary.** `Release Canary` smoke-tests `main` dev artifacts. - It doesn't gate the release, but treat a red canary as a risk. Use the - `test-release-canary` skill to diagnose it: - - ```shell - gh run list --workflow release-canary.yml --limit 3 - ``` - -5. **Plan fixes.** A failed pre-release doesn't end the train. Fix forward to a - green pre-release before Tuesday: - - - Reply in the thread of the failure's `#openshell-triage` message with the - cause and the fix or rerun, then again when a pre-release passes. Draft - these replies for the captain to post. - - Rerun a transient infrastructure failure once, after approval: runner - setup errors, registry push errors, and intermittent integration tests. - - A code defect, or a new advisory against a locked dependency, needs a - merged fix and a new pre-release. Don't rerun the failed one. - - Let a failed run finish rather than cancelling it. Its other suites still - show whether the rest of the commit is healthy. - - Once the fix merges and its `Release Dev` run passes, tag the next - `-pre.N` on that commit by hand after approval (as in Tuesday step 7), - instead of waiting a day for Auto-Tag. - -6. **Draft the coordination post.** The captain starts a thread on Monday in the - OpenShell channel of the CNCF Slack (`cloud-native.slack.com`, channel - `C0B6Y8WQHU1`) and links it in the internal team channel. Contributors reply - with ready PRs they want in the release. The release ships from `main`, with - no release branch or cherry-picks, so the list is for getting those PRs - reviewed and merged before the cutoff. Tag reviewers on the linked PRs. - - The release ships one commit: the final pre-release's, if it passes. A PR - ships only if that pre-release includes it. If the final pre-release fails - and can't be fixed in time, `Release` falls back to an earlier passing one, - and later merges miss the release. Pick one of two cutoffs: - - - **Before Tuesday's Auto-Tag run** (14:00 Europe/Zurich): Auto-Tag tags the - tip of `main`, so everything merged by the cutoff is in it, along with - anything merged before the run starts. Suits a captain in Europe. - - **Later on Tuesday:** after the cutoff, the captain tags the final - pre-release by hand on a commit they choose, then waits about 45 minutes - for it to qualify. Suits a captain in the Americas, where the Auto-Tag run - is before the workday. - - Don't promise that everything merged by the cutoff ships, or name the next - version; it's a patch by default but may be a minor. Approved open PRs: +4. **Draft the coordination post.** Start a thread in the OpenShell channel of + the CNCF Slack (`C0B6Y8WQHU1`) and link it in the internal team channel. + The release ships from `main` with no release branch, so the thread is for + getting ready PRs reviewed and merged before the cutoff. List approved PRs: ```shell gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"' ``` - Example post: + The release ships the final pre-release's commit, if it passes. Pick a + cutoff before Tuesday's Auto-Tag run, or a later one after which you tag the + final pre-release by hand. Don't promise that everything merged by the + cutoff ships, or name the next version. ```text :thread: OpenShell vX.Y.Z release, Tuesday . I'm release captain this week. Status: vX.Y.Z-pre.N () passed qualification. - Reply here with any ready PRs you'd like in vX.Y.Z. Merge cutoff is () Tuesday. I'll cut the final pre-release after the cutoff, and vX.Y.Z ships from it if it passes qualification. Anything that misses it goes in the next release. + Reply here with any ready PRs you'd like in vX.Y.Z. Merge cutoff is () Tuesday. <"Tuesday's automatic pre-release picks up everything merged by then" | "I'll cut the final pre-release after the cutoff">, and vX.Y.Z ships from it if it passes qualification. Anything that misses it goes in the next release. ``` - For an Auto-Tag cutoff, replace the second sentence with "Tuesday's - automatic pre-release picks up everything merged by then, and vX.Y.Z ships - from it if it passes qualification." - - The posted cutoff must match the pre-release that ships. If you end up - shipping one cut before the posted cutoff, such as Auto-Tag's or a fix-up - tagged early, reply in the thread with its commit and time, and say that - later merges go in the next release. For v0.1.5, the post said 16:00 UTC, - but `pre.7` was cut at 12:38 UTC. + If the pre-release that ships was cut before the posted cutoff, say so in + the thread, with its commit and time. ## Tuesday: cut the release -1. **Select the final candidate.** With a cutoff after Auto-Tag, first tag the - final `-pre.N` by hand after approval, on the newest commit with a - successful `Release Dev` run (commands as in Tuesday step 7). Once the final - pre-release has finished, run the preview from Setup and repeat Monday - step 3 for its run. If the newest pre-release failed, either fix it (Monday - step 5) or confirm that shipping the earlier candidate without the later - commits is acceptable. If qualification passed but publication failed, rerun the - failed jobs first; `Release` needs the whole run to pass. +1. **Select the final candidate.** For a late cutoff, tag the final `-pre.N` by + hand on the newest commit with a passing `Release Dev` run. When it + finishes, run the preview and repeat Monday step 2 for its run. 2. **Request approval.** Present the candidate tag, commit, qualification run, - compatibility assessment, release notes preview, and the commits since - `$PREV`: + compatibility assessment, commits since `$PREV`, and release notes preview: ```shell git log --oneline --no-merges "$PREV.." gh run download "$RUN" -n release-notes -D "$dir/notes" && cat "$dir/notes/release-notes.md" ``` - Release notes are generated from the merged PRs, including each PR's - "Release Notes" section, by `release-notes.yml`. Every `Pre-release` run - writes an advisory preview as its `release-notes` artifact. `Release` - writes them again in `Publish / Write Release Notes` and uses them as the - GitHub Release body, so the final text can differ slightly from the - preview. Check that upgrade notes cover every accepted break. To fix the - notes, edit the PR's Release Notes section and preview again without - publishing: - - ```shell - gh workflow run release.yml --ref main -f tag="$VERSION" -f notes-only=true - ``` + The notes come from each PR's "Release Notes" section. To fix them, edit the + PR, then preview with + `gh workflow run release.yml --ref main -f tag="$VERSION" -f notes-only=true`. 3. **Dispatch the release.** After approval: @@ -273,48 +155,24 @@ Finish these on Monday so Tuesday is only the cut. gh workflow run release.yml --ref main -f tag="$VERSION" ``` -4. **Watch the release.** It runs for about 45 minutes. Check that the - `Select Qualified Prerelease` job summary names the approved candidate: +4. **Watch and verify.** Check that `Select Qualified Prerelease` names the + approved candidate, the GitHub Release has assets, no job failed, and the + wheel reached PyPI: ```shell - gh run list --workflow release.yml --limit 1 gh run watch --exit-status - ``` - -5. **Verify publication.** Confirm the GitHub Release has assets and that no - job failed: - - ```shell gh release view "$VERSION" --json isPrerelease,publishedAt,assets --jq '{isPrerelease, publishedAt, assets: (.assets | length)}' - gh run view --json jobs --jq '.jobs[] | select(.conclusion != "success" and .conclusion != "skipped") | "\(.conclusion)\t\(.name)"' ``` - `Trigger Wheel Publish` only starts the GitLab pipeline, so confirm that the - wheel reached PyPI separately. - -6. **Close the loop.** Draft a reply to the coordination thread with the release - link and any PRs deferred to the next train. After approval, move the - release milestone's open items to the next milestone. - -7. **Start the next train.** This is the last duty of this week's captain, not - the first of next week's. Until the next `-pre.1` exists, Auto-Tag skips. - Once it does, Auto-Tag cuts `-pre.2` onward every weekday, so next week's - captain starts Monday with qualified pre-releases. Do it the same day as - the release. Your week isn't done until the `-pre.1` `Pre-release` run - passes, or its failure is triaged and handed to next week's captain. - - Agree with the maintainers whether the next train is patch (the default) or - minor (notable features or planned breaking changes; see the - [version selection supplement](../../../rfc/0014-release-stability/release-version-selection.md)): - - ```shell - git log --format=%s --no-merges "$VERSION..origin/main" | sed -E 's/^([a-z]+)(\(.*\))?(!?):.*/\1\3/' | sort | uniq -c - ``` +5. **Close the loop.** Draft a reply to the coordination thread with the release + link and the PRs deferred to the next train. After approval, move the + milestone's open items to the next milestone. - Seed it on a `main` commit after `$VERSION`, without a stable tag of its own - (versioning prefers the stable tag), and with a successful `Release Dev` run. - `cancelled` means a newer push superseded the run. If nothing has merged - since the release, wait. +6. **Start the next train.** This is this week's captain's last duty. Until the + next `-pre.1` exists, Auto-Tag skips. Agree with the maintainers on patch + (the default) or minor + ([version selection](../../../rfc/0014-release-stability/release-version-selection.md)). + Seed it on a commit after `$VERSION` with a passing `Release Dev` run: ```shell for sha in $(git rev-list --max-count=10 "$VERSION..origin/main"); do @@ -323,41 +181,29 @@ Finish these on Monday so Tuesday is only the cut. done ``` - After approval, push the tag, which starts `Pre-release`. The `-c` and - `/dev/null` keep `git` from signing the tag and opening an editor: + After approval, push the tag. The `-c` and `/dev/null` keep `git` from + signing the tag and opening an editor: ```shell - NEXT=vX.Y.Z-pre.1 - NEXT_SHA=
- git merge-base --is-ancestor "$VERSION" "$NEXT_SHA" || echo "commit does not contain the release" + NEXT=vX.Y.Z-pre.1 NEXT_SHA= git -c tag.gpgSign=false tag "$NEXT" "$NEXT_SHA" --failed`, then recheck the candidate. Reply in the failure's `#openshell-triage` thread with the result. | -| Pre-release qualification passed but publication failed | Usually a registry error, such as GHCR `failed to perform "Tag"`. Rerun failed jobs after approval; `Release` won't select the run until it fully passes. | -| Pre-release run ends in `startup_failure` with no jobs | The workflow file is invalid at that commit, for example a missing permission (`v0.1.5-pre.6`). Rerunning won't help. Merge the workflow fix, then tag the next `-pre.N` on it. | -| Code defect in a pre-release | Fix forward (Monday step 5): merge the fix, then tag the next `-pre.N` on it after its `Release Dev` run passes. | -| `Cargo Deny` fails with `error[vulnerability]` | A new RustSec advisory hit a locked dependency; rerunning won't help. Find the IDs with `gh run view --log-failed \| grep -E 'ID: RUSTSEC'`, find or open the bump PR, and treat it as a code defect. | -| A pushed `-pre.N` tag started no run | Wait a few minutes and check again. Only after confirming no run exists, and with approval, run `gh workflow run release-prerelease.yml --ref main -f tag="$TAG"`. | -| `Release` fails in `Select Qualified Prerelease` | No candidate qualified, or its evidence expired. Nothing was tagged or published. Fix the train, then dispatch again. | -| `Release` fails in `Write Release Notes` | The GitHub Release isn't created until the notes succeed. After approval, rerun failed jobs. If it keeps failing, check the inference API key and variables named in the job log. | -| `Release` fails after selection, transient | After approval, run `gh run rerun --failed`. A stable tag that already points at the selected commit can be retried. | -| `Release` fails after the `Release` job | The GitHub Release exists. A rerun of an npm or snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. | -| `Release` stuck queued | Releases queue behind each other. Check for another running `Release` or a GitHub Actions incident before asking a maintainer to cancel. Don't dispatch again. | +| Transient infrastructure failure in a pre-release, including a registry push | After approval, `gh run rerun --failed`. `Release` won't select the run until all of it passes. | +| Pre-release run ends in `startup_failure` | The workflow file is invalid at that commit. Merge the fix, then tag the next `-pre.N`. | +| Code defect or `Cargo Deny` advisory | Rerunning won't help. Merge a fix, then tag the next `-pre.N` (Monday step 3). | +| `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. | ## Related -- `release-openshell`: the short procedure behind Tuesday steps 1–5. +- `release-openshell`: the short procedure behind Tuesday steps 1–4. - `watch-github-actions`: general `gh run` monitoring and log retrieval. -- `test-release-canary`: diagnose dev artifact smoke-test failures. - `CI.md`: release workflows, qualification profile, and scanner policies. -- `rfc/0014-release-stability/`: cadence, qualification, and version selection. From 218c4136b9d88840ecadd9e65d5ad535139f9cec Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 10:44:20 -0700 Subject: [PATCH 16/20] docs(ci): document Release Auto-Tag and pre-release notes previews Add Release Auto-Tag to the workflow table, including that each train's -pre.1 is pushed by hand, and note that every Pre-release run writes a release notes preview. Signed-off-by: Matthew Grossman --- CI.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CI.md b/CI.md index 78e98fc016..5154fdf82b 100644 --- a/CI.md +++ b/CI.md @@ -634,10 +634,11 @@ These workflows run after merge to publish dev/tagged artifacts and verify them. |---|---| | `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | | `.github/workflows/release.yml` | Manually promote the latest qualified prerelease for a version and branch, create its stable tag, then rebuild and publish. Support releases preserve shared latest channels. The notes-only mode generates a release notes preview artifact. | +| `.github/workflows/release-auto-tag.yml` | Weekdays at 14:00 Europe/Zurich, tags the next `-pre.N` on `main` when commits landed since the latest one, then dispatches Pre-release. It skips when the train has no `-pre.1` or its version is already stable, so push each train's `-pre.1` tag by hand. | | `.github/workflows/release-prerelease.yml` | Pre-release orchestrator, triggered by prerelease tag pushes or dispatched by Release Auto-Tag or manually: build, then publish and qualify in parallel. Qualification failure fails the run and sends the failure notification; published artifacts remain available. | | `.github/workflows/release-build.yml` | Shared version computation, binary and image builds, and package builds. Validates that the tag matches the caller's release kind. | | `.github/workflows/release-publish.yml` | Shared artifact assembly, OCI image tags and Helm chart publication, and package publication. Stable releases also publish to GitHub Releases, snap, npm, PyPI, and Fern. | -| `.github/workflows/release-notes.yml` | Shared prompt-driven release notes generation through Inference Hub, used by stable publication and notes-only previews. | +| `.github/workflows/release-notes.yml` | Shared prompt-driven release notes generation through Inference Hub, used by stable publication, notes-only previews, and the advisory preview in each Pre-release run. | | `.github/workflows/release-qualification.yml` | Reusable qualification suites and result aggregation. Runs security scans, conformance and feature integration, Docker and VM E2E, protobuf compatibility, and advisory compatibility review. | | `.github/workflows/snap-package.yml` | Builds snap and component artifacts for Release Dev, Release, and Pre-release without Store credentials or publication. | | `.github/workflows/snap-publish.yml` | Uploads existing snap and component artifacts to the Store without rebuilding. Release Dev calls it directly for `latest/edge` after snap builds and requires it to succeed before creating the dev release. Release calls it through `release-publish.yml` for `latest/stable` after qualification and release publication succeed. | From 4c5a04b7b00e09871448a4f89532156cfbe10eae Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 11:02:26 -0700 Subject: [PATCH 17/20] docs(skills): add a template for the release announcement reply Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 48354de1d2..e76905d5e9 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -164,9 +164,20 @@ git worktree remove --force "$wt" gh release view "$VERSION" --json isPrerelease,publishedAt,assets --jq '{isPrerelease, publishedAt, assets: (.assets | length)}' ``` -5. **Close the loop.** Draft a reply to the coordination thread with the release - link and the PRs deferred to the next train. After approval, move the - milestone's open items to the next milestone. +5. **Close the loop.** Draft a short reply to the coordination thread. Point + to the release notes rather than repeating them: + + ```text + :white_check_mark: :ship: OpenShell vX.Y.Z is out: https://github.com/NVIDIA/OpenShell/releases/tag/vX.Y.Z + + It shipped from vX.Y.Z-pre.N (). I'll be starting the -pre.* jobs shortly for next week. + + Please check the release notes for upgrade notes that operators might care about. I'm also moving everything still open in the X.Y.Z milestone to . + + Thanks everyone! + ``` + + After approval, move the milestone's open items to the next milestone. 6. **Start the next train.** This is this week's captain's last duty. Until the next `-pre.1` exists, Auto-Tag skips. Agree with the maintainers on patch From bace38b8878b665534632238e07c6701b30638b0 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 12:07:36 -0700 Subject: [PATCH 18/20] docs(skills): watch pre-releases in the week before your turn Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index e76905d5e9..bd7e746d4a 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -16,7 +16,7 @@ merges, cuts the release, and starts the next train. | When | Ask | What happens | |---|---|---| -| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists. If not, ask this week's captain to start the train (Tuesday step 6). | +| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists. If not, ask this week's captain to start the train (Tuesday step 6). Then watch `#openshell-triage` each weekday after Auto-Tag, and fix failures as they happen (Monday step 3) instead of on Monday. | | Monday | "Run the release captain Monday checklist." | Monday steps 1–4. | | A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 3 and Recovery. | | Tuesday | "Cut the release." | Tuesday steps 1–6, ending with starting the next train. | From 9d46fc53d38c2ee7be24ef1542a691c8a2ee70ce Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 12:09:47 -0700 Subject: [PATCH 19/20] docs(skills): remove internal channel references from release captain Check pre-release runs directly instead of relying on an internal notification channel, and drop internal Slack channel IDs. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index bd7e746d4a..b9adca0210 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -16,7 +16,7 @@ merges, cuts the release, and starts the next train. | When | Ask | What happens | |---|---|---| -| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists. If not, ask this week's captain to start the train (Tuesday step 6). Then watch `#openshell-triage` each weekday after Auto-Tag, and fix failures as they happen (Monday step 3) instead of on Monday. | +| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists. If not, ask this week's captain to start the train (Tuesday step 6). Then check each weekday that Auto-Tag's pre-release ran and passed (`gh run list --workflow release-prerelease.yml --limit 3`), and fix failures as they happen (Monday step 3) instead of on Monday. | | Monday | "Run the release captain Monday checklist." | Monday steps 1–4. | | A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 3 and Recovery. | | Tuesday | "Cut the release." | Tuesday steps 1–6, ending with starting the next train. | @@ -41,8 +41,6 @@ tag, or start a second `Pre-release` run for a tag; it has no concurrency guard. - `Release` (`release.yml`) is dispatched by hand with the stable version. It picks the highest `-pre.N` on `main` whose whole `Pre-release` run passed, creates the stable tag on that commit, and publishes without requalifying. -- OpenShell TriageBot posts every failed `Pre-release` attempt in - `#openshell-triage` (`C0C20SDLDNW`), with links to the failed jobs. ## Setup @@ -100,16 +98,15 @@ git worktree remove --force "$wt" fix, a revert, a minor train, or a maintainer-approved exception with upgrade steps in the release notes. -3. **Fix forward.** A failed pre-release doesn't end the train. Draft a reply in - the failure's `#openshell-triage` thread with the cause and the fix. Let the - failed run finish rather than cancelling it. Once the fix merges and its - `Release Dev` run passes, tag the next `-pre.N` on it by hand (as in Tuesday - step 6). See Recovery for reruns. +3. **Fix forward.** A failed pre-release doesn't end the train. Find the cause, + and let the failed run finish rather than cancelling it. Once the fix merges + and its `Release Dev` run passes, tag the next `-pre.N` on it by hand (as in + Tuesday step 6). See Recovery for reruns. 4. **Draft the coordination post.** Start a thread in the OpenShell channel of - the CNCF Slack (`C0B6Y8WQHU1`) and link it in the internal team channel. - The release ships from `main` with no release branch, so the thread is for - getting ready PRs reviewed and merged before the cutoff. List approved PRs: + the CNCF Slack. The release ships from `main` with no release branch, so the + thread is for getting ready PRs reviewed and merged before the cutoff. List + approved PRs: ```shell gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"' From a3fd474391634fd861a3bed33ddeec3548157917 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Tue, 20 Oct 2026 12:22:05 -0700 Subject: [PATCH 20/20] docs(skills): fix pre-release failures in a new number via Auto-Tag A rerun repeats the same commit, so a failure that needs a fix ships in the next -pre.N. Cut it by dispatching Release Auto-Tag from the tip of main, and tag a specific commit by hand only when needed. Signed-off-by: Matthew Grossman --- .agents/skills/release-captain/SKILL.md | 27 +++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index b9adca0210..6ad534404f 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -99,9 +99,20 @@ git worktree remove --force "$wt" steps in the release notes. 3. **Fix forward.** A failed pre-release doesn't end the train. Find the cause, - and let the failed run finish rather than cancelling it. Once the fix merges - and its `Release Dev` run passes, tag the next `-pre.N` on it by hand (as in - Tuesday step 6). See Recovery for reruns. + and let the failed run finish rather than cancelling it. + + A rerun repeats the same commit and workflow, so it only helps with a + transient failure (see Recovery). Anything that needs a code or workflow + fix ships in a new number: that `-pre.N` stays failed, and `Release` skips + it. Merge the fix, then cut the next `-pre.N` from the tip of `main` after + approval: + + ```shell + gh workflow run release-auto-tag.yml --ref main + ``` + + Tag a specific commit by hand (as in Tuesday step 6) only when the tip of + `main` isn't the commit you want. 4. **Draft the coordination post.** Start a thread in the OpenShell channel of the CNCF Slack. The release ships from `main` with no release branch, so the @@ -130,9 +141,9 @@ git worktree remove --force "$wt" ## Tuesday: cut the release -1. **Select the final candidate.** For a late cutoff, tag the final `-pre.N` by - hand on the newest commit with a passing `Release Dev` run. When it - finishes, run the preview and repeat Monday step 2 for its run. +1. **Select the final candidate.** For a late cutoff, cut the final `-pre.N` + after the cutoff as in Monday step 3. When it finishes, run the preview and + repeat Monday step 2 for its run. 2. **Request approval.** Present the candidate tag, commit, qualification run, compatibility assessment, commits since `$PREV`, and release notes preview: @@ -206,8 +217,8 @@ git worktree remove --force "$wt" | Symptom | Action | |---|---| | Transient infrastructure failure in a pre-release, including a registry push | After approval, `gh run rerun --failed`. `Release` won't select the run until all of it passes. | -| Pre-release run ends in `startup_failure` | The workflow file is invalid at that commit. Merge the fix, then tag the next `-pre.N`. | -| Code defect or `Cargo Deny` advisory | Rerunning won't help. Merge a fix, then tag the next `-pre.N` (Monday step 3). | +| Pre-release run ends in `startup_failure` | The workflow file is invalid at that commit. Fix it in a new `-pre.N` (Monday step 3). | +| Code defect or `Cargo Deny` advisory | Rerunning won't help. Fix it in a new `-pre.N` (Monday step 3). | | `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. | ## Related