From dbb941f68b5c16777142349c9c207e0a6dc750fd Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 08:23:34 -0500 Subject: [PATCH 01/12] Recover blocked release PR checks with a targeted Fix run (#2016) --- docs/guide.md | 26 ++- scripts/release.sh | 169 +++++++++++--- site/architecture/index.html | 1 + site/docs/index.html | 24 +- site/how-it-works/index.html | 1 + site/index.html | 1 + tests/release.rs | 17 +- tests/release/checks.rs | 424 +++++++++++++++++++++++++++++++++++ tests/release/experiments.rs | 15 +- 9 files changed, 630 insertions(+), 48 deletions(-) create mode 100644 tests/release/checks.rs diff --git a/docs/guide.md b/docs/guide.md index 399380d8a..4114144fc 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -4227,8 +4227,9 @@ qualify. The script uses a qualifying commit immediately even if the tip's CI is pending or failed. If none qualifies, it waits for a run to succeed, still considering older pending commits when `main` moves. The Experiment's Candidate is that release base. The tag names exactly that content -plus the version bump, so merges to `main` during a release do not change what -is measured or shipped. +plus the version bump. A repair or external head update can change that content; +the script validates it and refreshes the requested Experiment and release notes +before publishing. From a clone of this repo, signed in to `gh` and the User config's default Harness, run the release script with the new version: @@ -4236,7 +4237,26 @@ From a clone of this repo, signed in to `gh` and the User config's default Harne scripts/release.sh 0.4.0 ``` -It works from `origin/main` in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from `release-` that bumps the version in `Cargo.toml` and `Cargo.lock` and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in [`scripts/release-summary.md`](../scripts/release-summary.md). The script uses `cargo run` in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With `--review` (`scripts/release.sh --review 0.4.0`), it then prints the summary and asks `[y]es / [e]dit / [n]o`: `y` carries on, `e` opens the summary in `$EDITOR` and carries on with what you save, and `n` stops with no branch, pull request or tag pushed. Without `--review`, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of `release-` as `v` and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into `main` with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it names the open PR for the Day shift to merge and still exits `0`. CI's run on `main` for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as `HH:MM:SS` as thirdshift's own progress lines are, so a long wait can be told from a stalled one. If the PR checks fail, it stops before tagging and leaves the pull request open and untagged. If the release workflow fails, it exits `1`, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Neither wait has an overall timeout. +It works from `origin/main` in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from `release-` that bumps the version in `Cargo.toml` and `Cargo.lock` and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in [`scripts/release-summary.md`](../scripts/release-summary.md). The script uses `cargo run` in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With `--review` (`scripts/release.sh --review 0.4.0`), it then prints the summary and asks `[y]es / [e]dit / [n]o`: `y` carries on, `e` opens the summary in `$EDITOR` and carries on with what you save, and `n` stops with no branch, pull request or tag pushed. Without `--review`, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of `release-` as `v` and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into `main` with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it names the open PR for the Day shift to merge and still exits `0`. CI's run on `main` for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as `HH:MM:SS` as thirdshift's own progress lines are, so a long wait can be told from a stalled one. A Merge conflict or completed failing PR checks can start the bounded Fix run described below. If the release workflow fails, it exits `1`, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Running CI and publication have no overall timeout. + +While waiting for PR CI, the script observes that PR's current head and mergeability. +A Merge conflict, including one with no workflows, or completed failing CI starts +at most one targeted `thirdshift fix --no-merge --no-base-fix` per release +invocation, using this checkout's runner and User config. Those flags override +Self-merge and Base fix defaults: the release script still tags, waits for +publication, then merges. Healthy, queued and running CI start no Fix run. +Missing workflows without a conflict have a five-minute startup grace period +(`RELEASE_WORKFLOW_STARTUP_SECONDS=300`, adjustable for testing), then stop with +the PR, head and recovery command. This is not a deadline on CI runtime. + +After a Fix run or external push, the script fetches and validates the current +PR identity, open state, readiness, origin, branches, head and exact version-bump +tree against main. Extra source changes or a wrong version require human review +before tagging. A changed content base reruns the Experiment when requested and +regenerates the summary and version diff, honoring `--review`. More than five +head updates, a failed or interrupted Fix, or another conflict or CI failure +after the automatic Fix stops with recovery instructions and no new tag pushed. +Ctrl-C and rerunning the same version use the usual resume behavior. Before it pushes anything, it refuses a release that can't be cut and says why: a version that isn't plain `X.Y.Z` (no leading `v`), a version that isn't higher than the one in `Cargo.toml` on `origin/main`, or a `v` tag that already exists locally or on origin. A resumed PR must belong to origin and match its branch tip while open. Its tree (or a pushed branch without a PR) must be exactly the version bump in `Cargo.toml` and `Cargo.lock` from its merge base; extra content is refused even when already tagged. diff --git a/scripts/release.sh b/scripts/release.sh index 1e7782cbc..341907e38 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -68,6 +68,13 @@ # version is no longer higher than main. An invalid pin is selected again. # Main can gain content after selection: the tagged tree is still exactly the # Candidate plus the version bump, and merging later preserves it in history. +# While waiting for PR CI, a Merge conflict or completed failing CI starts at +# most one targeted Fix run, with --no-merge and --no-base-fix. After a Fix or +# external push, the current PR/head must still be an exact version bump. A +# changed content base refreshes the requested Experiment, summary and diff. +# More than five head changes, an unsuccessful Fix, or another blockage stops +# before tagging with recovery instructions. Workflows have 300 seconds to +# appear (RELEASE_WORKFLOW_STARTUP_SECONDS); queued/running CI has no deadline. # # Prints a progress line on stderr for each step, stamped with the local time # as HH:MM:SS, with the release workflow run's URL once it waits for that run, @@ -75,12 +82,12 @@ # Exits 0 once the release workflow run for the tag has succeeded: the GitHub # Release is published with its final body and the crate is on crates.io. It # exits 1 with nothing more pushed if it refuses or the review says no, exits -# 1 leaving the PR open if the PR's checks fail, and exits 1 leaving the tag in +# 1 leaving the PR open if automatic CI recovery fails, and exits 1 leaving the tag in # place and the PR open if the release workflow run fails, naming its failed # jobs and run URL. Recovery is only attempted on a subsequent invocation. # A failed merge after publication leaves the release standing, names # the PR for the Day shift and still exits 0. -# It waits with no overall timeout, so Ctrl-C and running it again +# Running CI and publication wait with no overall timeout, so Ctrl-C and running it again # with the same version carries on waiting. set -euo pipefail @@ -90,10 +97,14 @@ summary_manifest=$(dirname "$summary_prompt")/../Cargo.toml # Seconds between reads of main's CI, the PR's runs and the release workflow. poll_seconds=${RELEASE_POLL_SECONDS:-15} +# Only workflow startup is timed out, never queued/running CI. +workflow_startup_seconds=${RELEASE_WORKFLOW_STARTUP_SECONDS:-300} main() { review= with_experiment= + fix_attempted= + head_updates=0 while :; do case ${1-} in --review) review=1 ;; @@ -117,6 +128,7 @@ main() { # first, including macOS's older Bash, which does not have this option. shopt -s inherit_errexit + release_origin=$(git remote get-url origin) fetch_main find_earlier_run case $pr_state in @@ -136,7 +148,7 @@ main() { validate_bump_pr recut_untagged_bump if [ -z "$recut" ]; then - if [ -z "$tagged" ]; then wait_for_experiment "$(git rev-parse "$head^1")"; fi + if [ -z "$tagged" ]; then wait_for_experiment "$(release_content_base)"; fi write_summary "no PR was opened" fi fi @@ -145,11 +157,11 @@ main() { OPEN) progress "resuming: $url is open" recut_untagged_bump - if [ -z "$tagged" ] && [ -z "$recut" ]; then wait_for_experiment "$(git rev-parse "$head^1")"; fi + if [ -z "$tagged" ] && [ -z "$recut" ]; then wait_for_experiment "$(release_content_base)"; fi ;; MERGED) progress "resuming: $url was merged as ${merge:0:7}" - if [ -z "$tagged" ]; then wait_for_experiment "$(git rev-parse "$head^1")"; fi + if [ -z "$tagged" ]; then wait_for_experiment "$(release_content_base)"; fi ;; *) refuse "$url was closed without merging, so there is nothing to resume; reopen it or delete $branch to start over" @@ -178,7 +190,7 @@ recut_untagged_bump() { recut= if [ -n "$tagged" ]; then return; fi previous=$(last_release_tag) - if [ -z "$previous" ] || git merge-base --is-ancestor "$previous" "$head^1"; then return; fi + if [ -z "$previous" ] || git merge-base --is-ancestor "$previous" "$(release_content_base)"; then return; fi base=$(check_can_release "$version" "$tag") progress "re-cutting untagged $tag from main at ${base:0:7}" commit_bump "$base" @@ -202,13 +214,13 @@ recover_failed_release() { run=$(release_run "$tagged") read -r id run_url status conclusion attempt <<<"$run" if [ "$status" != completed ] || [ "$conclusion" = success ]; then return; fi - base=$(newest_green_main "$(git rev-parse "$head^1")") + base=$(newest_green_main "$(release_content_base)") if [ -n "$base" ] || [ "$head" != "$tagged" ]; then if [ -n "$base" ]; then progress "re-cutting unpublished $tag from main at ${base:0:7}" commit_bump "$base" else - wait_for_experiment "$(git rev-parse "$head^1")" + wait_for_experiment "$(release_content_base)" fi write_summary "the release branch and tag were not moved" refuse_if_published @@ -304,21 +316,25 @@ find_earlier_run() { # bump_version result from its merge base. A merged PR uses main just before # its merge, since its head is already an ancestor of main now. validate_bump_pr() { - local base target expected candidate=${url:-$branch on origin} + local base expected candidate=${url:-$branch on origin} if ! git fetch --quiet origin "$head"; then refuse "$candidate head could not be fetched from origin" fi - target=origin/main - if [ "$pr_state" = MERGED ]; then - target=$merge^1 - fi - base=$(git merge-base "$head" "$target") + base=$(release_content_base) expected=$(bump_tree "$base") if [ "$expected" != "$(git rev-parse "$head^{tree}")" ]; then - refuse "$candidate changes more than the version bump to $version in Cargo.toml and Cargo.lock" + refuse "$candidate changes more than the version bump to $version in Cargo.toml and Cargo.lock; human review is required before tagging" fi } +# A repaired bump can be a merge commit whose first parent is the old bump. +# Its content is the merge base with main, not that first parent. +release_content_base() { + local target=origin/main + if [ "$pr_state" = MERGED ]; then target=$merge^1; fi + git merge-base "$head" "$target" +} + # bump_tree # The tree commit_bump would make, without committing or touching the caller's # checkout. The subshell owns the temporary worktree and its cleanup trap. @@ -511,18 +527,104 @@ edit_pr() ( ) # wait_for_pr_checks -# Leaves the PR open and untagged and exits 1 if its checks failed. +# Observe this PR throughout CI. One targeted Fix run shares Delivery's Repair +# budget; a second blockage or continued head churn returns to the operator. wait_for_pr_checks() { - local failed tag_state="no tag was pushed" + local runs failed content_base next_base startup=$SECONDS + [[ "$workflow_startup_seconds" =~ ^[0-9]+$ ]] || refuse "RELEASE_WORKFLOW_STARTUP_SECONDS must be a nonnegative integer" + content_base=$(git merge-base "$head" origin/main) progress "waiting for CI on $url" - failed=$(wait_for_workflow_runs "$head") - if [ -n "$failed" ]; then - if [ -n "$tagged" ]; then tag_state="the existing tag is unchanged"; fi - progress "CI failed on $url ($failed), so the PR is left open and unmerged; $tag_state" - exit 1 + while :; do + observe_release_pr + if [ "$observed_head" != "$head" ]; then + head_updates=$((head_updates + 1)) + [ "$head_updates" -le 5 ] || release_blocked "the PR head changed more than five times" + head=$observed_head + fetch_main + validate_bump_pr + next_base=$(git merge-base "$head" origin/main) + if [ "$next_base" != "$content_base" ]; then + wait_for_experiment "$next_base" + write_summary "no new tag was pushed" + # A Harness or Experiment can take time. Check identity again + # before editing, and never publish stale notes on a newer head. + observe_release_pr + if [ "$observed_head" != "$head" ]; then continue; fi + edit_pr + fi + content_base=$next_base + startup=$SECONDS + progress "adopted $url at ${head:0:7}" + fi + if [ "$observed_mergeable" = CONFLICTING ]; then + fix_release_pr "Merge conflict" + continue + fi + runs=$(workflow_runs "$head") + # A workflow read can race an external push or Fix completion. + observe_release_pr + if [ "$observed_head" != "$head" ] || [ "$observed_mergeable" = CONFLICTING ]; then continue; fi + if [ -n "$runs" ] && ! grep -qv '^completed ' <<<"$runs"; then + failed=$(failed_workflows <<<"$runs") + if [ -n "$failed" ]; then + fix_release_pr "CI failed ($failed)" + continue + fi + if [ "$observed_mergeable" = MERGEABLE ]; then + fetch_main + validate_bump_pr + # A newly merged release can invalidate ancestry during CI. + local previous + previous=$(last_release_tag) + if [ -n "$previous" ] && ! git merge-base --is-ancestor "$previous" "$content_base"; then + release_blocked "the content base predates the latest release $previous" + fi + observe_release_pr + if [ "$observed_head" = "$head" ] && [ "$observed_mergeable" = MERGEABLE ]; then return; fi + continue + fi + elif [ -z "$runs" ] && [ "$((SECONDS - startup))" -ge "$workflow_startup_seconds" ]; then + release_blocked "no workflows appeared within ${workflow_startup_seconds}s; inspect GitHub Actions" + fi + sleep "$poll_seconds" + done +} + +# Re-read the captured PR, never search for a replacement. Refuse identity, +# state, origin or branch changes before adopting a head or publishing a tag. +observe_release_pr() { + local observation origin_head + [ "$(git remote get-url origin)" = "$release_origin" ] || release_blocked "origin changed" + if ! observation=$(gh pr view "$url" --json url,state,headRefName,baseRefName,headRefOid,isCrossRepository,isDraft,mergeable | + python3 -c ' +import json, sys +pr = json.load(sys.stdin) +url, branch = sys.argv[1:] +if (pr["url"] != url or pr["state"] != "OPEN" or pr["headRefName"] != branch + or pr["baseRefName"] != "main" or pr["isCrossRepository"] or pr["isDraft"]): + sys.exit("release PR identity/state/branch changed or PR is draft") +print(pr["headRefOid"], pr["mergeable"]) +' "$url" "$branch"); then + release_blocked "could not validate the release PR identity/state/branch" + fi + read -r observed_head observed_mergeable <<<"$observation" + origin_head=$(git ls-remote origin "refs/heads/$branch" | cut -f 1) + [ -n "$origin_head" ] && [ "$observed_head" = "$origin_head" ] || release_blocked "PR head does not match refs/heads/$branch on origin" +} + +fix_release_pr() { + [ -z "$fix_attempted" ] || release_blocked "$1 after the automatic Fix run" + fix_attempted=1 + progress "$1 on $url; starting one targeted Fix run" + if ! cargo run --quiet --manifest-path "$summary_manifest" -- fix "$url" --no-merge --no-base-fix; then + release_blocked "$1; the targeted Fix run failed or was interrupted" fi } +release_blocked() { + refuse "$url at $head: $1; the PR is left open for human review and no new tag was pushed. Recover with thirdshift fix $url --no-merge --no-base-fix, then scripts/release.sh${with_experiment:+ --with-experiment} $version" +} + # merge_bump_pr # The tag is already public. A failed merge leaves the release standing and # names the PR for the Day shift; publication has already succeeded. @@ -821,24 +923,19 @@ summary_input() { done } -# wait_for_workflow_runs -# Waits until has workflow runs and all of them have completed, then -# prints the names of those that did not succeed, comma-separated, or nothing. +# workflow_runs +# One observation; the PR wait owns startup grace, head refresh and recovery. # A workflow run completes only once all its jobs have, including jobs that # start after others finish, which a commit's check runs would not show yet. -wait_for_workflow_runs() { - local runs - while :; do - runs=$(gh api "repos/{owner}/{repo}/actions/runs?head_sha=$1&per_page=100" \ - --jq '.workflow_runs[] | "\(.status) \(.conclusion) \(.name)"') - if [ -n "$runs" ] && ! grep -qv '^completed ' <<<"$runs"; then - break - fi - sleep "$poll_seconds" - done +workflow_runs() { + gh api "repos/{owner}/{repo}/actions/runs?head_sha=$1&per_page=100" \ + --jq '.workflow_runs[] | "\(.status) \(.conclusion) \(.name)"' +} + +failed_workflows() { awk '$2 != "success" && $2 != "skipped" && $2 != "neutral" { sub(/^[^ ]+ [^ ]+ /, ""); printf "%s%s", sep, $0; sep = ", " - }' <<<"$runs" + }' } # Only when run, so the tests can source the script and call its functions. diff --git a/site/architecture/index.html b/site/architecture/index.html index bb5164f47..a24ae1290 100644 --- a/site/architecture/index.html +++ b/site/architecture/index.html @@ -281,6 +281,7 @@

Architecture

The Experiment owns recheck history beside its immutable Grader Scores. thirdshift experiments <id> --json exposes pending borderline and decisive Scores, naming each decisive flip and its resulting provisional Verdict. thirdshift recheck <id> <trial> pass|fail --reason '…' takes the Experiment's exclusive lock and atomically saves the history, recalculated Verdict and pending list. thirdshift recheck <id> --pending lists the pending Scores and their fingerprint; thirdshift recheck <id> --batch <fingerprint> pass|fail --reason '…' checks that list under the same lock and records every listed Score in one atomic save. A stale fingerprint changes nothing. Rechecks stay in the CLI. They also update the latest invocation's Command outcome and Inbox group, preserving its completion time. The Verdict waits until the list is empty; flipping either pass/fail or a wrong-finding flag can make a Score decisive. Waiting rechecks place that invocation in Needs you; a confirmed pass moves it to Done.

scripts/release.sh runs on Linux. It refuses other hosts before Git or GitHub work. On macOS, use an Ubuntu 24.04 Linux VM such as Lima; on Windows, use WSL2.

thirdshift experiment --release <commit> compares the content commit with the last release tag on touched suites. The release script skips Experiments by default. Add --with-experiment to scripts/release.sh on each invocation to run one: it waits for rechecks and asks permission for a fail or inconclusive Verdict before pushing. Reruns with the flag reuse the Experiment, including before opening a pushed branch's PR. It selects the newest main commit descending from the last release tag whose latest push CI passed, tags that content plus the version bump, waits for publication, then merges the bump PR. A saved Candidate is selected again if a new release tag on main invalidates its ancestry; the requested version must still be higher than main. An untagged branch or open bump PR predating that tag is re-cut from a green main descendant, with a new summary and version diff, honoring --review before force-pushing the branch and updating the open PR body. Tagged or merged releases keep their resume behavior. After a failed unpublished release, rerunning re-cuts from newer green main content or automatically reruns failed jobs; published releases are never moved. If the publication check cannot be completed, the script refuses and moves nothing. Recovery retains the prior release boundary for the summary and Experiment. Main can move during the release; a failed merge leaves the release standing and the PR for the Day shift. No touched suite or a last release without the Trial flag carries on with a note. Releasing.

+

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

diff --git a/site/docs/index.html b/site/docs/index.html index f5b2fd017..d179b9d1e 100644 --- a/site/docs/index.html +++ b/site/docs/index.html @@ -3917,12 +3917,30 @@ if the tip's CI is pending or failed. If none qualifies, it waits for a run to succeed, still considering older pending commits when main moves. The Experiment's Candidate is that release base. The tag names exactly that content -plus the version bump, so merges to main during a release do not change what -is measured or shipped.

+plus the version bump. A repair or external head update can change that content; +the script validates it and refreshes the requested Experiment and release notes +before publishing.

From a clone of this repo, signed in to gh and the User config's default Harness, run the release script with the new version:

scripts/release.sh 0.4.0
-

It works from origin/main in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from release-<version> that bumps the version in Cargo.toml and Cargo.lock and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in scripts/release-summary.md. The script uses cargo run in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With --review (scripts/release.sh --review 0.4.0), it then prints the summary and asks [y]es / [e]dit / [n]o: y carries on, e opens the summary in $EDITOR and carries on with what you save, and n stops with no branch, pull request or tag pushed. Without --review, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of release-<version> as v<version> and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into main with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it names the open PR for the Day shift to merge and still exits 0. CI's run on main for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as HH:MM:SS as thirdshift's own progress lines are, so a long wait can be told from a stalled one. If the PR checks fail, it stops before tagging and leaves the pull request open and untagged. If the release workflow fails, it exits 1, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Neither wait has an overall timeout.

+

It works from origin/main in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from release-<version> that bumps the version in Cargo.toml and Cargo.lock and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in scripts/release-summary.md. The script uses cargo run in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With --review (scripts/release.sh --review 0.4.0), it then prints the summary and asks [y]es / [e]dit / [n]o: y carries on, e opens the summary in $EDITOR and carries on with what you save, and n stops with no branch, pull request or tag pushed. Without --review, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of release-<version> as v<version> and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into main with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it names the open PR for the Day shift to merge and still exits 0. CI's run on main for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as HH:MM:SS as thirdshift's own progress lines are, so a long wait can be told from a stalled one. A Merge conflict or completed failing PR checks can start the bounded Fix run described below. If the release workflow fails, it exits 1, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Running CI and publication have no overall timeout.

+

While waiting for PR CI, the script observes that PR's current head and mergeability. +A Merge conflict, including one with no workflows, or completed failing CI starts +at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix per release +invocation, using this checkout's runner and User config. Those flags override +Self-merge and Base fix defaults: the release script still tags, waits for +publication, then merges. Healthy, queued and running CI start no Fix run. +Missing workflows without a conflict have a five-minute startup grace period +(RELEASE_WORKFLOW_STARTUP_SECONDS=300, adjustable for testing), then stop with +the PR, head and recovery command. This is not a deadline on CI runtime.

+

After a Fix run or external push, the script fetches and validates the current +PR identity, open state, readiness, origin, branches, head and exact version-bump +tree against main. Extra source changes or a wrong version require human review +before tagging. A changed content base reruns the Experiment when requested and +regenerates the summary and version diff, honoring --review. More than five +head updates, a failed or interrupted Fix, or another conflict or CI failure +after the automatic Fix stops with recovery instructions and no new tag pushed. +Ctrl-C and rerunning the same version use the usual resume behavior.

Before it pushes anything, it refuses a release that can't be cut and says why: a version that isn't plain X.Y.Z (no leading v), a version that isn't higher than the one in Cargo.toml on origin/main, or a v<version> tag that already exists locally or on origin. A resumed PR must belong to origin and match its branch tip while open. Its tree (or a pushed branch without a PR) must be exactly the version bump in Cargo.toml and Cargo.lock from its merge base; extra content is refused even when already tagged.

If it's interrupted, run it again with the same version: it picks up the existing branch or pull request and carries on from the first step not yet done. With --review, it asks about a new summary before opening a pull request for a branch already pushed. It opens a PR for a pushed branch, waits for its checks, tags the bump head, waits for publication, then merges the PR. A PR already merged is still tagged at its bump head. Once the tag is on that head, it says the release is already tagged and resumes the release workflow: it waits for a run still going, or merges the open bump PR immediately when the workflow succeeded or the GitHub Release already exists, then prints the Release URL and exits 0.

An untagged pushed branch or open bump PR is re-cut if its content base does diff --git a/site/how-it-works/index.html b/site/how-it-works/index.html index fe0c84579..b29176e70 100644 --- a/site/how-it-works/index.html +++ b/site/how-it-works/index.html @@ -317,6 +317,7 @@

How thirdshift works

A Verdict stays waiting on rechecks until you have checked every borderline Score and every decisive Score, whose flip would change the Verdict. The pending list gives Trial IDs for thirdshift recheck <id> <trial> pass|fail --reason '…'. The JSON view names each decisive flip and the provisional Verdict it gives. thirdshift recheck <id> --pending lists all pending Scores and a fingerprint; thirdshift recheck <id> --batch <fingerprint> pass|fail --reason '…' records one decision and reason for that reviewed list in one locked save, refusing a changed list. Your latest decision feeds the calculation; the Grader's Score and your timestamped history remain. Each recheck updates the pending list and the latest invocation's Command outcome and Inbox group. A confirmed pass moves from Needs you to Done.

scripts/release.sh runs on Linux. It refuses other hosts before Git or GitHub work. On macOS, use an Ubuntu 24.04 Linux VM such as Lima; on Windows, use WSL2.

thirdshift experiment --release <commit> compares touched suites with the last release tag. The release script skips Experiments by default. Add --with-experiment to scripts/release.sh on each invocation to run one: it waits for rechecks before pushing and asks whether to release on a fail or inconclusive Verdict. Reruns with the flag reuse the Experiment, including before opening a pushed branch’s PR. It selects the newest main commit descending from the last release tag whose latest push CI passed, tags that content plus the version bump, waits for publication, then merges the bump PR. A saved Candidate is selected again if a new release tag on main invalidates its ancestry; the requested version must still be higher than main. An untagged branch or open bump PR predating that tag is re-cut from a green main descendant, with a new summary and version diff, honoring --review before force-pushing the branch and updating the open PR body. Tagged or merged releases keep their resume behavior. After a failed unpublished release, rerunning re-cuts from newer green main content or automatically reruns failed jobs; published releases are never moved. If the publication check cannot be completed, the script refuses and moves nothing. Recovery retains the prior release boundary for the summary and Experiment. Main can move during the release; a failed merge leaves the release standing and the PR for the Day shift. No touched suite or a last release with no Trial flag carries on with a note. Release checks.

+

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

Architect runs every few minutes and Security runs on a schedule, with fixing allowed, started by your operating system.

Next: inspect every prompt and skill, see a real Ticket run's recorded output, see the architecture in six figures, use the manual for installation, commands and configuration, or watch the explainer videos.

diff --git a/site/index.html b/site/index.html index a1a9cec9c..3b905ea76 100644 --- a/site/index.html +++ b/site/index.html @@ -523,6 +523,7 @@

Seven workflows

Each Trial removes its Harness caches when its session ends. After grading and all rechecks finish, the Experiment removes every attempt's repository copy. Paused Experiments keep repositories for resume; Session evidence, reports, packets and Scores remain readable.

scripts/release.sh runs on Linux. It refuses other hosts before Git or GitHub work. On macOS, use an Ubuntu 24.04 Linux VM such as Lima; on Windows, use WSL2.

thirdshift experiment --release <commit> compares touched suites with the last release tag. The release script skips Experiments by default. Add --with-experiment to scripts/release.sh on each invocation to run one: it waits for rechecks before pushing and asks whether to release on a fail or inconclusive Verdict. Reruns with the flag reuse the Experiment, including before opening a pushed branch’s PR. It selects the newest main commit descending from the last release tag whose latest push CI passed, tags that content plus the version bump, waits for publication, then merges the bump PR. A saved Candidate is selected again if a new release tag on main invalidates its ancestry; the requested version must still be higher than main. An untagged branch or open bump PR predating that tag is re-cut from a green main descendant, with a new summary and version diff, honoring --review before force-pushing the branch and updating the open PR body. Tagged or merged releases keep their resume behavior. After a failed unpublished release, rerunning re-cuts from newer green main content or automatically reruns failed jobs; published releases are never moved. If the publication check cannot be completed, the script refuses and moves nothing. Recovery retains the prior release boundary for the summary and Experiment. Main can move during the release; a failed merge leaves the release standing and the PR for the Day shift. No touched suite or a last release with no Trial flag carries on with a note. Release checks.

+

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

diff --git a/tests/release.rs b/tests/release.rs index 475e796bf..dce35b040 100644 --- a/tests/release.rs +++ b/tests/release.rs @@ -21,6 +21,9 @@ mod recovery; #[path = "release/ancestry.rs"] mod ancestry; +#[path = "release/checks.rs"] +mod checks; + use std::fs; use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; @@ -230,7 +233,19 @@ impl Release { let cargo = release.bin().join("cargo"); write_executable( &cargo, - "#!/bin/sh\nwhile [ \"$1\" != -- ]; do shift; done\nshift\nexec \"$THIRDSHIFT_TEST_BINARY\" \"$@\"\n", + r#"#!/bin/sh +while [ "$1" != -- ]; do shift; done +shift +if [ "$1" = fix ]; then + printf '%s\n' "$*" >> "$FAKE_GH_STATE.fix-calls" + if [ -f "$FAKE_GH_STATE.fix" ]; then + exec bash "$FAKE_GH_STATE.fix" "$@" + fi + echo 'No simulated Fix repair configured' >&2 + exit 1 +fi +exec "$THIRDSHIFT_TEST_BINARY" "$@" +"#, ); fs::create_dir(root.join("home")).unwrap(); fs::write( diff --git a/tests/release/checks.rs b/tests/release/checks.rs new file mode 100644 index 000000000..d239e5061 --- /dev/null +++ b/tests/release/checks.rs @@ -0,0 +1,424 @@ +use super::*; + +fn set_pr(release: &Release, field: &str, value: Value) { + let mut state = release.gh_state(); + let pr = state["prs"] + .as_array_mut() + .unwrap() + .iter_mut() + .find(|pr| pr["head"] == "release-0.2.0") + .unwrap(); + pr[field] = value; + fs::write(release.root().join("gh-state.json"), state.to_string()).unwrap(); +} + +fn fix_runs(release: &Release, script: &str) { + fs::write(release.root().join("gh-state.json.fix"), script).unwrap(); +} + +fn fix_calls(release: &Release) -> Vec { + fs::read_to_string(release.root().join("gh-state.json.fix-calls")) + .unwrap_or_default() + .lines() + .map(str::to_owned) + .collect() +} + +fn run(release: &Release, args: &[&str], env: &[(&str, &str)]) -> Output { + // This bounds a broken orchestration loop too. The guardian also owns any + // fake Fix children; no real agent or publication can run in this fixture. + with_input( + release + .command(Path::new("timeout"), &release.maintainer()) + .arg("10s") + .arg(manifest_dir().join("scripts/release.sh")) + .args(args) + .envs(env.iter().copied()), + "", + ) +} + +/// Prepare the exact reported pattern: a bump based on older main, followed +/// by a competing version change. Keep a real conflict-resolution merge ready +/// for the fake Fix command to push, with the requested version retained. +fn conflicted_bump(release: &Release) -> (String, String) { + let old = release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + let contributor = release.contributor(); + for file in ["Cargo.toml", "Cargo.lock"] { + fs::write(contributor.join(file), with_version(file, "0.1.1")).unwrap(); + } + git( + &contributor, + &["commit", "-q", "-a", "-m", "Competing version"], + ); + git(&contributor, &["push", "-q", "origin", "HEAD:main"]); + git(&contributor, &["checkout", "-q", "release-0.2.0"]); + let conflict = isolated(&mut Command::new("git")) + .current_dir(&contributor) + .args(["merge", "--no-edit", "origin/main"]) + .output() + .unwrap(); + assert_eq!(conflict.status.code(), Some(1)); + assert!(stdout(&conflict).contains("CONFLICT")); + for file in ["Cargo.toml", "Cargo.lock"] { + fs::write(contributor.join(file), with_version(file, "0.2.0")).unwrap(); + } + git(&contributor, &["add", "Cargo.toml", "Cargo.lock"]); + git( + &contributor, + &["commit", "-q", "-m", "Resolve bump conflict"], + ); + let repaired = git(&contributor, &["rev-parse", "HEAD"]); + git( + &contributor, + &["push", "-q", "origin", "HEAD:prepared-repair"], + ); + set_pr(release, "mergeable", json!("CONFLICTING")); + (old, repaired) +} + +fn adopt_script(repaired: &str) -> String { + format!( + r#"set -euo pipefail +git --git-dir="${{FAKE_GH_STATE%/*}}/origin.git" update-ref refs/heads/release-0.2.0 {repaired} +python3 - <<'PY' +import json, os, pathlib +path = pathlib.Path(os.environ['FAKE_GH_STATE']) +state = json.loads(path.read_text()) +for pr in state['prs']: + if pr['head'] == 'release-0.2.0': pr['mergeable'] = 'MERGEABLE' +path.write_text(json.dumps(state)) +PY +gh fake checks {repaired} '{GREEN}' +"# + ) +} + +#[test] +fn a_conflict_without_workflows_runs_one_targeted_fix_and_releases_the_repaired_head() { + let release = Release::new(); + fs::create_dir_all(release.root().join("home/.thirdshift")).unwrap(); + fs::write( + release.root().join("home/.thirdshift/config.toml"), + "[merge]\nalways = true\n[base]\nfix = true\n", + ) + .unwrap(); + let (old, repaired) = conflicted_bump(&release); + fix_runs(&release, &adopt_script(&repaired)); + + let output = run(&release, &["0.2.0"], &[]); + + assert!(output.status.success(), "{}", stderr(&output)); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired.clone())); + assert_ne!(repaired, old); + assert_eq!(release.pr("release-0.2.0")["state"], "MERGED"); + assert_eq!( + fix_calls(&release), + [format!( + "fix {} --no-merge --no-base-fix", + release.pr("release-0.2.0")["url"].as_str().unwrap() + )] + ); + assert!( + release + .pr_body("release-0.2.0") + .contains("-version = \"0.1.1\"") + ); +} + +#[test] +fn missing_workflows_expire_with_the_pr_head_and_recovery_command() { + let release = Release::new(); + let head = release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + + let output = run( + &release, + &["0.2.0"], + &[("RELEASE_WORKFLOW_STARTUP_SECONDS", "0")], + ); + + assert_eq!(output.status.code(), Some(1), "{}", stderr(&output)); + for text in [ + "no workflows appeared", + &head, + "pull/4", + "scripts/release.sh 0.2.0", + ] { + assert!(stderr(&output).contains(text), "{}", stderr(&output)); + } + assert!(fix_calls(&release).is_empty()); + assert_eq!(release.origin_tag("v0.2.0"), None); +} + +#[test] +fn pending_and_healthy_ci_never_start_a_fix_even_after_startup_grace() { + for resume in [false, true] { + let release = Release::new(); + if resume { + release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + } + release.ci_reports(GREEN); + + let output = run( + &release, + &["0.2.0"], + &[("RELEASE_WORKFLOW_STARTUP_SECONDS", "0")], + ); + + assert!(output.status.success(), "{}", stderr(&output)); + assert!(fix_calls(&release).is_empty()); + } +} + +#[test] +fn completed_failing_ci_uses_the_same_fix_path() { + let release = Release::new(); + release.ci_reports(RED); + fix_runs( + &release, + &format!( + "gh fake checks \"$(git ls-remote origin refs/heads/release-0.2.0 | cut -f1)\" '{GREEN}'" + ), + ); + + let output = run(&release, &["0.2.0"], &[]); + + assert!(output.status.success(), "{}", stderr(&output)); + assert_eq!(fix_calls(&release).len(), 1); + assert!(stderr(&output).contains("CI failed (test)")); +} + +#[test] +fn failed_interrupted_or_ineffective_fixes_leave_no_tag_and_are_not_retried() { + for script in ["exit 1", "exit 130", "exit 0"] { + let release = Release::new(); + conflicted_bump(&release); + fix_runs(&release, script); + + let output = run(&release, &["0.2.0"], &[]); + + assert_eq!(output.status.code(), Some(1), "{}", stderr(&output)); + assert_eq!(fix_calls(&release).len(), 1); + assert_eq!(release.origin_tag("v0.2.0"), None); + assert!(stderr(&output).contains("thirdshift fix")); + assert!(stderr(&output).contains("scripts/release.sh 0.2.0")); + } +} + +#[test] +fn a_second_ci_failure_after_a_successful_fix_stops_without_another_fix() { + let release = Release::new(); + let (_, repaired) = conflicted_bump(&release); + fix_runs(&release, &adopt_script(&repaired).replace(GREEN, RED)); + + let output = run(&release, &["0.2.0"], &[]); + + assert_eq!(output.status.code(), Some(1), "{}", stderr(&output)); + assert!(stderr(&output).contains("CI failed (test) after the automatic Fix run")); + assert_eq!(fix_calls(&release).len(), 1); + assert_eq!(release.origin_tag("v0.2.0"), None); +} + +#[test] +fn an_external_head_change_during_a_green_read_is_adopted_without_a_fix() { + let release = Release::new(); + let (old, repaired) = conflicted_bump(&release); + set_pr(&release, "mergeable", json!("MERGEABLE")); + release.gh(&[ + "fake", + "on-ci-read", + "1", + &format!( + "{}\ngh fake checks {old} '{GREEN}'", + adopt_script(&repaired) + ), + ]); + + let output = run(&release, &["0.2.0"], &[]); + + assert!(output.status.success(), "{}", stderr(&output)); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); + assert!(fix_calls(&release).is_empty()); +} + +#[test] +fn identity_mutations_after_fix_or_external_updates_refuse_publication() { + for external in [false, true] { + for (field, value) in [ + ("base", json!("elsewhere")), + ("head", json!("elsewhere")), + ("state", json!("CLOSED")), + ("state", json!("MERGED")), + ("url", json!("https://github.com/other/repo/pull/4")), + ("isCrossRepository", json!(true)), + ("isDraft", json!(true)), + ] { + let release = Release::new(); + release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + let mutation = format!("gh fake pr release-0.2.0 {field} '{value}'"); + if external { + release.gh(&["fake", "on-ci-read", "1", &mutation]); + } else { + set_pr(&release, "mergeable", json!("CONFLICTING")); + fix_runs(&release, &mutation); + } + + let output = run(&release, &["0.2.0"], &[]); + + assert_eq!( + output.status.code(), + Some(1), + "{field}: {}", + stderr(&output) + ); + assert_eq!(release.origin_tag("v0.2.0"), None, "{field}"); + assert!( + stderr(&output).contains("identity/state/branch"), + "{field}: {}", + stderr(&output) + ); + assert_eq!(fix_calls(&release).len(), usize::from(!external)); + } + } +} + +#[test] +fn non_bump_changes_after_fix_or_external_updates_refuse_publication() { + for external in [false, true] { + for change in [ + BumpChange::WrongVersion, + BumpChange::AddedFile, + BumpChange::AppendComment("Cargo.lock"), + BumpChange::FileMode, + ] { + let release = Release::new(); + let old = release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + let changed = release.push_change_to_bump("0.2.0", change); + release.origin(&["update-ref", "refs/heads/release-0.2.0", &old]); + if external { + release.gh(&["fake", "on-ci-read", "1", &adopt_script(&changed)]); + } else { + set_pr(&release, "mergeable", json!("CONFLICTING")); + fix_runs(&release, &adopt_script(&changed)); + } + + let output = run(&release, &["0.2.0"], &[]); + + assert_eq!( + output.status.code(), + Some(1), + "{change:?}: {}", + stderr(&output) + ); + assert_eq!(release.origin_tag("v0.2.0"), None); + assert!( + stderr(&output).contains("changes more than the version bump"), + "{change:?}: {}", + stderr(&output) + ); + } + } +} + +#[test] +fn a_resumed_repaired_merge_checks_the_experiment_on_its_content_base() { + let release = Release::new(); + let (_, repaired) = conflicted_bump(&release); + let base = release.origin(&["rev-parse", "main"]); + release.origin(&["update-ref", "refs/heads/release-0.2.0", &repaired]); + set_pr(&release, "mergeable", json!("MERGEABLE")); + release.ci_reports(GREEN); + experiments::cargo_reports(&release, &[experiments::verdict("pass")]); + + let output = run(&release, &["--with-experiment", "0.2.0"], &[]); + + assert!(output.status.success(), "{}", stderr(&output)); + assert_eq!(experiments::calls(&release)[0]["args"][2], base); +} + +#[test] +fn a_repair_refreshes_the_experiment_summary_and_diff_for_its_new_content_base() { + let release = Release::new(); + let (old, repaired) = conflicted_bump(&release); + let base = release.origin(&["rev-parse", "main"]); + let old_base = release.origin(&["rev-parse", &format!("{old}^")]); + experiments::cargo_reports( + &release, + &[experiments::verdict("pass"), experiments::verdict("pass")], + ); + fix_runs(&release, &adopt_script(&repaired)); + + let output = run(&release, &["--with-experiment", "0.2.0"], &[]); + + assert!(output.status.success(), "{}", stderr(&output)); + let calls = experiments::calls(&release); + assert_eq!(calls.len(), 2); + assert_eq!(calls[0]["args"][2], old_base); + assert_eq!(calls[1]["args"][2], base); + assert!( + release + .pr_body("release-0.2.0") + .contains("-version = \"0.1.1\"") + ); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); +} + +#[test] +fn a_new_content_base_cannot_publish_under_an_obsolete_experiment() { + let release = Release::new(); + let (_, repaired) = conflicted_bump(&release); + experiments::cargo_reports( + &release, + &[experiments::verdict("pass"), experiments::verdict("fail")], + ); + fix_runs(&release, &adopt_script(&repaired)); + + let output = run(&release, &["--with-experiment", "0.2.0"], &[]); + + assert_eq!(output.status.code(), Some(1), "{}", stderr(&output)); + assert!(stderr(&output).contains("Experiment fail; no release override given")); + assert_eq!(experiments::calls(&release).len(), 2); + assert_eq!(release.origin_tag("v0.2.0"), None); +} + +#[test] +fn continued_external_head_churn_is_bounded_without_starting_fix_runs() { + let release = Release::new(); + let old = release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + let contributor = release.contributor(); + git(&contributor, &["checkout", "-q", "release-0.2.0"]); + let mut heads = vec![old.clone()]; + for _ in 0..7 { + git( + &contributor, + &["commit", "-q", "--allow-empty", "-m", "Another head"], + ); + heads.push(git(&contributor, &["rev-parse", "HEAD"])); + } + git( + &contributor, + &["push", "-q", "origin", "HEAD:prepared-churn"], + ); + let mut script = String::from("case $FAKE_CI_SHA in\n"); + for pair in heads.windows(2) { + script.push_str(&format!( + "{}) git --git-dir=\"${{FAKE_GH_STATE%/*}}/origin.git\" update-ref refs/heads/release-0.2.0 {} ;;\n", + pair[0], pair[1] + )); + } + script.push_str("esac\n"); + release.gh(&["fake", "on-ci-read", "10", &script]); + + let output = run(&release, &["0.2.0"], &[]); + + assert_eq!(output.status.code(), Some(1), "{}", stderr(&output)); + assert!(stderr(&output).contains("head changed more than five times")); + assert_eq!(release.origin_tag("v0.2.0"), None); + assert!(fix_calls(&release).is_empty()); +} diff --git a/tests/release/experiments.rs b/tests/release/experiments.rs index 116dbb4cf..434189171 100644 --- a/tests/release/experiments.rs +++ b/tests/release/experiments.rs @@ -2,7 +2,7 @@ use super::*; const ID: &str = "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"; -fn verdict(outcome: &str) -> Value { +pub(super) fn verdict(outcome: &str) -> Value { json!({ "id": ID, "state": "completed", "pending_rechecks": [], "verdict": {"verdict": outcome, "suites": {"code-review": { @@ -12,7 +12,11 @@ fn verdict(outcome: &str) -> Value { }) } -fn cargo_reports(release: &Release, views: &[Value]) { +pub(super) fn cargo_reports(release: &Release, views: &[Value]) { + let driver = release.bin().join("cargo-driver"); + if !driver.exists() { + fs::copy(release.bin().join("cargo"), driver).unwrap(); + } fs::write( release.root().join("experiment-views.json"), json!(views).to_string(), @@ -21,9 +25,10 @@ fn cargo_reports(release: &Release, views: &[Value]) { fs::write(release.bin().join("cargo"), r#"#!/usr/bin/env python3 import json, os, pathlib, subprocess, sys args = sys.argv[sys.argv.index('--') + 1:] -if args == ['--release-summary']: - os.execv(os.environ['THIRDSHIFT_TEST_BINARY'], [os.environ['THIRDSHIFT_TEST_BINARY'], *args]) root = pathlib.Path(os.environ['FAKE_GH_STATE']).parent +if args == ['--release-summary'] or args[0] == 'fix': + driver = str(root / 'bin/cargo-driver') + os.execv(driver, [driver, *sys.argv[1:]]) branch = subprocess.check_output(['git', 'ls-remote', 'origin', 'refs/heads/release-0.2.0'], text=True).strip() with (root / 'experiment-calls.jsonl').open('a') as calls: calls.write(json.dumps(dict(args=args, branch=branch)) + '\n') @@ -38,7 +43,7 @@ print(json.dumps(view)) "#).unwrap(); } -fn calls(release: &Release) -> Vec { +pub(super) fn calls(release: &Release) -> Vec { fs::read_to_string(release.root().join("experiment-calls.jsonl")) .unwrap_or_default() .lines() From 1608e50420ac07fe549375f0a3ea67a82293844d Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 08:34:14 -0500 Subject: [PATCH 02/12] Preserve release notes identity across interrupted Fix runs --- scripts/release.sh | 39 ++++++++++++----- tests/release.rs | 13 +++++- tests/release/checks.rs | 97 +++++++++++++++++++++++++++++------------ 3 files changed, 108 insertions(+), 41 deletions(-) diff --git a/scripts/release.sh b/scripts/release.sh index 341907e38..2d2e7e2a5 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -82,13 +82,13 @@ # Exits 0 once the release workflow run for the tag has succeeded: the GitHub # Release is published with its final body and the crate is on crates.io. It # exits 1 with nothing more pushed if it refuses or the review says no, exits -# 1 leaving the PR open if automatic CI recovery fails, and exits 1 leaving the tag in -# place and the PR open if the release workflow run fails, naming its failed +# 1 leaving the PR open if automatic CI recovery fails, and exits 1 leaving +# the tag in place and the PR open if the release workflow run fails, naming its failed # jobs and run URL. Recovery is only attempted on a subsequent invocation. # A failed merge after publication leaves the release standing, names # the PR for the Day shift and still exits 0. -# Running CI and publication wait with no overall timeout, so Ctrl-C and running it again -# with the same version carries on waiting. +# Running CI and publication wait with no overall timeout, so Ctrl-C and +# running it again with the same version carries on waiting. set -euo pipefail # The prompt the summary agent gets ahead of the release's input. @@ -500,6 +500,7 @@ experiment_status() { write_summary() { local base base=$(git merge-base "$head" origin/main) + summary_base=$base progress "writing the summary with the configured Harness" diff=$(version_diff "$base" "$head") summary=$(release_summary "$base" "$tag" "$diff") @@ -531,14 +532,24 @@ edit_pr() ( # budget; a second blockage or continued head churn returns to the operator. wait_for_pr_checks() { local runs failed content_base next_base startup=$SECONDS + local notes_ref=refs/thirdshift/release-notes/$version [[ "$workflow_startup_seconds" =~ ^[0-9]+$ ]] || refuse "RELEASE_WORKFLOW_STARTUP_SECONDS must be a nonnegative integer" - content_base=$(git merge-base "$head" origin/main) + observe_release_pr + content_base=$observed_notes_base + if [ "$content_base" = - ]; then + # Older PR bodies lack the marker. Remember their starting content + # before Fix can push and then fail, so rerunning still detects it. + content_base=$(git rev-parse --quiet --verify "$notes_ref^{commit}" || release_content_base) + fi + git update-ref "$notes_ref" "$content_base" progress "waiting for CI on $url" while :; do observe_release_pr - if [ "$observed_head" != "$head" ]; then - head_updates=$((head_updates + 1)) - [ "$head_updates" -le 5 ] || release_blocked "the PR head changed more than five times" + if [ "$observed_head" != "$head" ] || [ "$(release_content_base)" != "$content_base" ]; then + if [ "$observed_head" != "$head" ]; then + head_updates=$((head_updates + 1)) + [ "$head_updates" -le 5 ] || release_blocked "the PR head changed more than five times" + fi head=$observed_head fetch_main validate_bump_pr @@ -553,6 +564,7 @@ wait_for_pr_checks() { edit_pr fi content_base=$next_base + git update-ref "$notes_ref" "$content_base" startup=$SECONDS progress "adopted $url at ${head:0:7}" fi @@ -595,19 +607,20 @@ wait_for_pr_checks() { observe_release_pr() { local observation origin_head [ "$(git remote get-url origin)" = "$release_origin" ] || release_blocked "origin changed" - if ! observation=$(gh pr view "$url" --json url,state,headRefName,baseRefName,headRefOid,isCrossRepository,isDraft,mergeable | + if ! observation=$(gh pr view "$url" --json url,state,headRefName,baseRefName,headRefOid,isCrossRepository,isDraft,mergeable,body | python3 -c ' -import json, sys +import json, re, sys pr = json.load(sys.stdin) url, branch = sys.argv[1:] if (pr["url"] != url or pr["state"] != "OPEN" or pr["headRefName"] != branch or pr["baseRefName"] != "main" or pr["isCrossRepository"] or pr["isDraft"]): sys.exit("release PR identity/state/branch changed or PR is draft") -print(pr["headRefOid"], pr["mergeable"]) +notes = re.search(r"^$", pr["body"], re.M) +print(pr["headRefOid"], pr["mergeable"], notes[1] if notes else "-") ' "$url" "$branch"); then release_blocked "could not validate the release PR identity/state/branch" fi - read -r observed_head observed_mergeable <<<"$observation" + read -r observed_head observed_mergeable observed_notes_base <<<"$observation" origin_head=$(git ls-remote origin "refs/heads/$branch" | cut -f 1) [ -n "$origin_head" ] && [ "$observed_head" = "$origin_head" ] || release_blocked "PR head does not match refs/heads/$branch on origin" } @@ -823,6 +836,8 @@ pr_body() { $2 + + EOF } diff --git a/tests/release.rs b/tests/release.rs index dce35b040..4133b56d1 100644 --- a/tests/release.rs +++ b/tests/release.rs @@ -562,7 +562,17 @@ fi /// Runs the script with `args`, with `stdin` on its standard input and /// `env` added to its environment. fn run_script_with(&self, args: &[&str], stdin: &str, env: &[(&str, &str)]) -> Output { - with_input( + self.run_script_with_bound(args, stdin, env, support::WAIT_BOUND) + } + + fn run_script_with_bound( + &self, + args: &[&str], + stdin: &str, + env: &[(&str, &str)], + bound: std::time::Duration, + ) -> Output { + support::child_lifetime::FixtureChild::spawn( self.command( &manifest_dir().join("scripts/release.sh"), &self.maintainer(), @@ -571,6 +581,7 @@ fi .envs(env.iter().copied()), stdin, ) + .wait(bound) } /// Sources the script in the maintainer's clone, with what `run_script` diff --git a/tests/release/checks.rs b/tests/release/checks.rs index d239e5061..b6eba4079 100644 --- a/tests/release/checks.rs +++ b/tests/release/checks.rs @@ -1,15 +1,7 @@ use super::*; fn set_pr(release: &Release, field: &str, value: Value) { - let mut state = release.gh_state(); - let pr = state["prs"] - .as_array_mut() - .unwrap() - .iter_mut() - .find(|pr| pr["head"] == "release-0.2.0") - .unwrap(); - pr[field] = value; - fs::write(release.root().join("gh-state.json"), state.to_string()).unwrap(); + release.gh(&["fake", "pr", "release-0.2.0", field, &value.to_string()]); } fn fix_runs(release: &Release, script: &str) { @@ -25,17 +17,10 @@ fn fix_calls(release: &Release) -> Vec { } fn run(release: &Release, args: &[&str], env: &[(&str, &str)]) -> Output { - // This bounds a broken orchestration loop too. The guardian also owns any - // fake Fix children; no real agent or publication can run in this fixture. - with_input( - release - .command(Path::new("timeout"), &release.maintainer()) - .arg("10s") - .arg(manifest_dir().join("scripts/release.sh")) - .args(args) - .envs(env.iter().copied()), - "", - ) + // Ten seconds covers script startup, Git/GitHub reads, fake Fix/summary/ + // Experiment work, publication/merge and output collection. Fixture setup + // happens before this bound; the guardian owns all fake Fix children too. + release.run_script_with_bound(args, "", env, std::time::Duration::from_secs(10)) } /// Prepare the exact reported pattern: a bump based on older main, followed @@ -82,14 +67,7 @@ fn adopt_script(repaired: &str) -> String { format!( r#"set -euo pipefail git --git-dir="${{FAKE_GH_STATE%/*}}/origin.git" update-ref refs/heads/release-0.2.0 {repaired} -python3 - <<'PY' -import json, os, pathlib -path = pathlib.Path(os.environ['FAKE_GH_STATE']) -state = json.loads(path.read_text()) -for pr in state['prs']: - if pr['head'] == 'release-0.2.0': pr['mergeable'] = 'MERGEABLE' -path.write_text(json.dumps(state)) -PY +gh fake pr release-0.2.0 mergeable '"MERGEABLE"' gh fake checks {repaired} '{GREEN}' "# ) @@ -422,3 +400,66 @@ fn continued_external_head_churn_is_bounded_without_starting_fix_runs() { assert_eq!(release.origin_tag("v0.2.0"), None); assert!(fix_calls(&release).is_empty()); } + +#[test] +fn spec_resume_after_interrupted_fix_refreshes_release_notes() { + let release = Release::new(); + let (_, repaired) = conflicted_bump(&release); + set_pr( + &release, + "body", + json!("Old release notes: version diff starts at 0.1.0"), + ); + fix_runs(&release, &(adopt_script(&repaired) + "\nexit 130\n")); + let interrupted = run(&release, &["0.2.0"], &[]); + assert_eq!( + interrupted.status.code(), + Some(1), + "{}", + stderr(&interrupted) + ); + assert_eq!(release.origin_tag("v0.2.0"), None); + assert_eq!( + release.pr_body("release-0.2.0"), + "Old release notes: version diff starts at 0.1.0" + ); + + let resumed = run(&release, &["0.2.0"], &[]); + assert!(resumed.status.success(), "{}", stderr(&resumed)); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); + assert!( + release + .pr_body("release-0.2.0") + .contains("-version = \"0.1.1\""), + "Published repaired content with stale release notes: {}", + release.pr_body("release-0.2.0") + ); +} + +#[test] +fn a_notes_marker_detects_an_external_repair_from_another_checkout() { + let release = Release::new(); + let (old, repaired) = conflicted_bump(&release); + let old_base = release.origin(&["rev-parse", &format!("{old}^")]); + set_pr( + &release, + "body", + json!(format!( + "Old release notes\n" + )), + ); + release.origin(&["update-ref", "refs/heads/release-0.2.0", &repaired]); + set_pr(&release, "mergeable", json!("MERGEABLE")); + release.ci_reports(GREEN); + + let output = run(&release, &["0.2.0"], &[]); + + assert!(output.status.success(), "{}", stderr(&output)); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); + assert!(fix_calls(&release).is_empty()); + assert!( + release + .pr_body("release-0.2.0") + .contains("-version = \"0.1.1\"") + ); +} From 943e6d8099606faedebfeeb6cad5fefad15a5f5e Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 08:44:21 -0500 Subject: [PATCH 03/12] Resume failed publication after a release Fix merge --- scripts/release.sh | 18 ++++++++++++------ tests/release/checks.rs | 23 +++++++++++++++++++++++ 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/scripts/release.sh b/scripts/release.sh index 2d2e7e2a5..69adc761f 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -354,7 +354,7 @@ bump_tree() ( # left behind by an interrupted recovery. With no earlier run, # check_can_release refuses an existing tag in its turn. find_existing_tag() { - local old_base base run id run_url status conclusion attempt + local old_base= parent base run id run_url status conclusion attempt tagged= if [ -z "$pr_state" ] && [ -z "$head" ]; then return @@ -371,13 +371,19 @@ find_existing_tag() { # tag. Accept only two exact bumps, a strictly newer content base, and an # unpublished failed workflow on the old tag; unrelated tags still refuse. git fetch --quiet origin "$tagged" - old_base=$(git rev-parse "$tagged^1") - base=$(git rev-parse "$head^1") + base=$(release_content_base) + # Fix can leave a merge with the old bump first and main second. Accept + # only a parent on the new content's history whose exact bump is the tag. + for parent in $(git show -s --format=%P "$tagged"); do + if [ "$parent" != "$base" ] && git merge-base --is-ancestor "$parent" "$base" && + [ "$(bump_tree "$parent")" = "$(git rev-parse "$tagged^{tree}")" ]; then + old_base=$parent + break + fi + done run=$(release_run "$tagged") read -r id run_url status conclusion attempt <<<"$run" - if [ "$pr_state" = OPEN ] && [ "$old_base" != "$base" ] && - git merge-base --is-ancestor "$old_base" "$base" && - [ "$(bump_tree "$old_base")" = "$(git rev-parse "$tagged^{tree}")" ] && + if [ "$pr_state" = OPEN ] && [ -n "$old_base" ] && [ "$status" = completed ] && [ "$conclusion" != success ] && release_is_absent; then progress "resuming re-cut of unpublished $tag on ${head:0:7}" diff --git a/tests/release/checks.rs b/tests/release/checks.rs index b6eba4079..4f3dfbd42 100644 --- a/tests/release/checks.rs +++ b/tests/release/checks.rs @@ -463,3 +463,26 @@ fn a_notes_marker_detects_an_external_repair_from_another_checkout() { .contains("-version = \"0.1.1\"") ); } + +#[test] +fn recovery_after_a_repaired_tag_resumes_between_branch_and_tag_pushes() { + let release = Release::new(); + let (_, repaired) = conflicted_bump(&release); + fix_runs(&release, &adopt_script(&repaired)); + release.release_workflow(failed_release()); + let failed = run(&release, &["0.2.0"], &[]); + assert_release_failed(&release, &failed, &repaired); + release.merge_pr("newer-content", "Recovery content", ""); + release.main_ci("completed", "success"); + release.ci_reports(RED); + fix_runs(&release, "exit 1"); + let stopped = run(&release, &["0.2.0"], &[]); + assert_eq!(stopped.status.code(), Some(1), "{}", stderr(&stopped)); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); + let head = release.pr_head("release-0.2.0"); + release.gh(&["fake", "checks", &head, GREEN]); + release.release_workflow(json!({})); + let resumed = run(&release, &["0.2.0"], &[]); + assert!(resumed.status.success(), "{}", stderr(&resumed)); + assert_eq!(release.origin_tag("v0.2.0"), Some(head)); +} From 864c03fe6d76f35b8f1f309669bc509e6ff4be08 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 08:50:50 -0500 Subject: [PATCH 04/12] Refresh legacy release notes before tagging repaired content --- scripts/release.sh | 24 ++++++++++++++++-------- tests/release.rs | 9 ++++++++- tests/release/checks.rs | 24 ++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 9 deletions(-) diff --git a/scripts/release.sh b/scripts/release.sh index 69adc761f..38a0cf9c0 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -537,7 +537,7 @@ edit_pr() ( # Observe this PR throughout CI. One targeted Fix run shares Delivery's Repair # budget; a second blockage or continued head churn returns to the operator. wait_for_pr_checks() { - local runs failed content_base next_base startup=$SECONDS + local runs failed content_base next_base refresh_notes= startup=$SECONDS local notes_ref=refs/thirdshift/release-notes/$version [[ "$workflow_startup_seconds" =~ ^[0-9]+$ ]] || refuse "RELEASE_WORKFLOW_STARTUP_SECONDS must be a nonnegative integer" observe_release_pr @@ -546,6 +546,9 @@ wait_for_pr_checks() { # Older PR bodies lack the marker. Remember their starting content # before Fix can push and then fail, so rerunning still detects it. content_base=$(git rev-parse --quiet --verify "$notes_ref^{commit}" || release_content_base) + # Without a marker the notes may predate this script/checkout. The + # current head alone cannot establish what their summary describes. + refresh_notes=1 fi git update-ref "$notes_ref" "$content_base" progress "waiting for CI on $url" @@ -562,15 +565,9 @@ wait_for_pr_checks() { next_base=$(git merge-base "$head" origin/main) if [ "$next_base" != "$content_base" ]; then wait_for_experiment "$next_base" - write_summary "no new tag was pushed" - # A Harness or Experiment can take time. Check identity again - # before editing, and never publish stale notes on a newer head. - observe_release_pr - if [ "$observed_head" != "$head" ]; then continue; fi - edit_pr + refresh_notes=1 fi content_base=$next_base - git update-ref "$notes_ref" "$content_base" startup=$SECONDS progress "adopted $url at ${head:0:7}" fi @@ -597,6 +594,17 @@ wait_for_pr_checks() { if [ -n "$previous" ] && ! git merge-base --is-ancestor "$previous" "$content_base"; then release_blocked "the content base predates the latest release $previous" fi + if [ -n "$refresh_notes" ]; then + write_summary "no new tag was pushed" + # A Harness can take time; never edit a changed PR or + # publish these notes on a newer head. + observe_release_pr + if [ "$observed_head" != "$head" ]; then continue; fi + edit_pr + git update-ref "$notes_ref" "$content_base" + refresh_notes= + continue + fi observe_release_pr if [ "$observed_head" = "$head" ] && [ "$observed_mergeable" = MERGEABLE ]; then return; fi continue diff --git a/tests/release.rs b/tests/release.rs index 4133b56d1..300dfc6d6 100644 --- a/tests/release.rs +++ b/tests/release.rs @@ -2335,7 +2335,14 @@ fn a_rerun_after_the_pr_was_opened_merges_and_tags_without_a_second_pr_or_bump() assert_eq!(release.prs_from("release-0.2.0").len(), 1); let pr = release.pr("release-0.2.0"); assert_eq!(pr["state"], "MERGED"); - assert_eq!(pr["body"], "Summary from the first run"); + // A legacy PR without a notes-base marker gets a verified summary once. + assert!(pr["body"].as_str().unwrap().contains(AGENT_SUMMARY)); + assert!( + pr["body"] + .as_str() + .unwrap() + .contains("release-content-base:") + ); assert_eq!(release.pr_head("release-0.2.0"), head); let merge = release.origin(&["rev-parse", "main"]); assert_eq!(release.origin(&["rev-parse", &format!("{merge}^2")]), head); diff --git a/tests/release/checks.rs b/tests/release/checks.rs index 4f3dfbd42..7ab12c3bd 100644 --- a/tests/release/checks.rs +++ b/tests/release/checks.rs @@ -486,3 +486,27 @@ fn recovery_after_a_repaired_tag_resumes_between_branch_and_tag_pushes() { assert!(resumed.status.success(), "{}", stderr(&resumed)); assert_eq!(release.origin_tag("v0.2.0"), Some(head)); } + +#[test] +fn an_unmarked_legacy_pr_repaired_before_resume_refreshes_its_notes() { + let release = Release::new(); + let (_, repaired) = conflicted_bump(&release); + set_pr( + &release, + "body", + json!("Legacy notes: version diff starts at 0.1.0"), + ); + release.origin(&["update-ref", "refs/heads/release-0.2.0", &repaired]); + set_pr(&release, "mergeable", json!("MERGEABLE")); + release.ci_reports(GREEN); + let output = run(&release, &["0.2.0"], &[]); + assert!(output.status.success(), "{}", stderr(&output)); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); + assert!( + release + .pr_body("release-0.2.0") + .contains("-version = \"0.1.1\""), + "Published repaired legacy content with stale release notes: {}", + release.pr_body("release-0.2.0") + ); +} From a64916a70e11d1c4933ed009a5a6a3e4ed619df7 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 10:04:09 -0500 Subject: [PATCH 05/12] Re-measure a release base changed during its Experiment The notes marker is read after the Experiment returns. A PR edit during that yield can name the new content base, so the script tags that head under the old result. Snapshot the marker before the yield and treat Experiment coverage as the base this process measured. --- scripts/release.sh | 78 ++++++++++++++++++++++----- tests/release/checks.rs | 114 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 180 insertions(+), 12 deletions(-) diff --git a/scripts/release.sh b/scripts/release.sh index 38a0cf9c0..12a2e0e0b 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -105,6 +105,12 @@ main() { with_experiment= fix_attempted= head_updates=0 + # Notes claim captured before a yield, and the base this process measured. + # A PR body edit must not count as an Experiment of a base we have not run. + release_notes_known= + release_notes_base= + release_notes_refresh= + experimented_base= while :; do case ${1-} in --review) review=1 ;; @@ -157,7 +163,12 @@ main() { OPEN) progress "resuming: $url is open" recut_untagged_bump - if [ -z "$tagged" ] && [ -z "$recut" ]; then wait_for_experiment "$(release_content_base)"; fi + if [ -z "$tagged" ] && [ -z "$recut" ]; then + # The Experiment yields. Snapshot the notes claim first so a body + # edit during it cannot name a base this process has not measured. + remember_notes_claim + wait_for_experiment "$(release_content_base)" + fi ;; MERGED) progress "resuming: $url was merged as ${merge:0:7}" @@ -442,6 +453,8 @@ saved_candidate_is_current() { # wait_for_experiment # Use this checkout's runner and private Eval suites, as for summary_agent. wait_for_experiment() { + # Including the skip path: adoption must not treat an unmeasured base as covered. + experimented_base=$1 local snapshot status outcome id answer previous local baseline_args=() if [ -z "$with_experiment" ]; then @@ -522,10 +535,16 @@ open_pr() { url=$(gh pr create --base main --head "$branch" --title "Release $version" \ --body "$(pr_body "$summary" "$diff")" | tail -n 1) progress "opened $url" + remember_written_notes } # The subshell owns the temporary PR body even if gh fails. -edit_pr() ( +edit_pr() { + edit_pr_file + remember_written_notes +} + +edit_pr_file() ( local body_file body_file=$(mktemp) trap 'rm -f "$body_file"' EXIT @@ -533,6 +552,29 @@ edit_pr() ( gh pr edit "$url" --body-file "$body_file" >/dev/null ) +# Snapshot the PR's notes claim before a yield. Reading the marker again +# after the Experiment returns would let a body edit name an unmeasured base. +remember_notes_claim() { + local notes_ref=refs/thirdshift/release-notes/$version + observe_release_pr + if [ "$observed_notes_base" = - ]; then + release_notes_base=$(git rev-parse --quiet --verify "$notes_ref^{commit}" || release_content_base) + # Without a marker the notes may predate this script/checkout. + release_notes_refresh=1 + else + release_notes_base=$observed_notes_base + release_notes_refresh= + fi + release_notes_known=1 +} + +# Notes this process just wrote. Do not re-read the body for that claim. +remember_written_notes() { + release_notes_known=1 + release_notes_base=$summary_base + release_notes_refresh= +} + # wait_for_pr_checks # Observe this PR throughout CI. One targeted Fix run shares Delivery's Repair # budget; a second blockage or continued head churn returns to the operator. @@ -540,16 +582,23 @@ wait_for_pr_checks() { local runs failed content_base next_base refresh_notes= startup=$SECONDS local notes_ref=refs/thirdshift/release-notes/$version [[ "$workflow_startup_seconds" =~ ^[0-9]+$ ]] || refuse "RELEASE_WORKFLOW_STARTUP_SECONDS must be a nonnegative integer" - observe_release_pr - content_base=$observed_notes_base - if [ "$content_base" = - ]; then - # Older PR bodies lack the marker. Remember their starting content - # before Fix can push and then fail, so rerunning still detects it. - content_base=$(git rev-parse --quiet --verify "$notes_ref^{commit}" || release_content_base) - # Without a marker the notes may predate this script/checkout. The - # current head alone cannot establish what their summary describes. - refresh_notes=1 + if [ -n "$release_notes_known" ]; then + content_base=$release_notes_base + refresh_notes=$release_notes_refresh + else + observe_release_pr + content_base=$observed_notes_base + if [ "$content_base" = - ]; then + # Older PR bodies lack the marker. Remember their starting content + # before Fix can push and then fail, so rerunning still detects it. + content_base=$(git rev-parse --quiet --verify "$notes_ref^{commit}" || release_content_base) + # Without a marker the notes may predate this script/checkout. The + # current head alone cannot establish what their summary describes. + refresh_notes=1 + fi fi + # A path that has not called wait_for_experiment has not measured another base. + if [ -z "$experimented_base" ]; then experimented_base=$content_base; fi git update-ref "$notes_ref" "$content_base" progress "waiting for CI on $url" while :; do @@ -563,8 +612,13 @@ wait_for_pr_checks() { fetch_main validate_bump_pr next_base=$(git merge-base "$head" origin/main) - if [ "$next_base" != "$content_base" ]; then + # Measure a base this process has not run. Refresh notes when the + # snapshotted claim is a different base, even if a body edit now + # names the one we are adopting. + if [ "$next_base" != "$experimented_base" ]; then wait_for_experiment "$next_base" + fi + if [ "$next_base" != "$content_base" ]; then refresh_notes=1 fi content_base=$next_base diff --git a/tests/release/checks.rs b/tests/release/checks.rs index 7ab12c3bd..c4376338b 100644 --- a/tests/release/checks.rs +++ b/tests/release/checks.rs @@ -487,6 +487,120 @@ fn recovery_after_a_repaired_tag_resumes_between_branch_and_tag_pushes() { assert_eq!(release.origin_tag("v0.2.0"), Some(head)); } +/// The Experiment is a yield: the cargo stand-in moves the release branch to +/// an exact bump of newer main and rewrites the notes marker to that base +/// before the script observes the pull request again. +fn forge_new_base_during_experiment(release: &Release, repaired: &str, new_base: &str) { + let driver = release.bin().join("cargo-driver"); + if !driver.exists() { + fs::copy(release.bin().join("cargo"), &driver).unwrap(); + } + let body = format!("Old release notes\n"); + fs::write( + release.root().join("during-experiment.sh"), + format!( + r#"set -euo pipefail +git --git-dir="${{FAKE_GH_STATE%/*}}/origin.git" update-ref refs/heads/release-0.2.0 {repaired} +gh fake pr release-0.2.0 mergeable '"MERGEABLE"' +gh fake pr release-0.2.0 body '{body}' +gh fake checks {repaired} '[{{"name": "test", "conclusion": "success"}}]' +"#, + repaired = repaired, + body = serde_json::to_string(&body).unwrap(), + ), + ) + .unwrap(); + fs::write( + release.root().join("experiment-views.json"), + json!([experiments::verdict("pass"), experiments::verdict("pass")]).to_string(), + ) + .unwrap(); + fs::write( + release.bin().join("cargo"), + r#"#!/usr/bin/env python3 +import json, os, pathlib, subprocess, sys +args = sys.argv[sys.argv.index('--') + 1:] +root = pathlib.Path(os.environ['FAKE_GH_STATE']).parent +if args == ['--release-summary'] or args[:1] == ['fix']: + driver = str(root / 'bin/cargo-driver') + os.execv(driver, [driver, *sys.argv[1:]]) +branch = subprocess.check_output(['git', 'ls-remote', 'origin', 'refs/heads/release-0.2.0'], text=True).strip() +with (root / 'experiment-calls.jsonl').open('a') as calls: + calls.write(json.dumps(dict(args=args, branch=branch)) + '\n') +marker = root / 'experiment-mutated' +if not marker.exists(): + marker.write_text('1') + subprocess.check_call(['bash', str(root / 'during-experiment.sh')]) +views = root / 'experiment-views.json' +remaining = json.loads(views.read_text()) +assert remaining, 'unexpected Experiment call' +view = remaining.pop(0) +views.write_text(json.dumps(remaining)) +print(json.dumps(view)) +"#, + ) + .unwrap(); +} + +#[test] +fn a_base_change_during_the_experiment_is_measured_before_tagging() { + let release = Release::new(); + let (old, repaired) = conflicted_bump(&release); + let old_base = release.origin(&["rev-parse", &format!("{old}^")]); + let new_base = release.origin(&["rev-parse", "main"]); + assert_ne!(old_base, new_base); + set_pr( + &release, + "body", + json!(format!( + "Old release notes\n" + )), + ); + set_pr(&release, "mergeable", json!("MERGEABLE")); + release.gh(&[ + "fake", + "checks", + &old, + r#"[{"name": "test", "conclusion": "success"}]"#, + ]); + forge_new_base_during_experiment(&release, &repaired, &new_base); + + let output = run(&release, &["--with-experiment", "0.2.0"], &[]); + + let tag = release.origin_tag("v0.2.0"); + let calls = experiments::calls(&release); + let measured: Vec<&str> = calls + .iter() + .filter_map(|call| call["args"].get(2).and_then(Value::as_str)) + .collect(); + // The hook rewrote the marker. Publishing that head anyway requires a + // measurement of its base and notes that describe the new diff. + assert!( + release + .pr_body("release-0.2.0") + .contains(&format!("release-content-base: {new_base}")) + || measured.contains(&new_base.as_str()), + "experiment hook did not run: {}\n{}", + release.pr_body("release-0.2.0"), + stderr(&output) + ); + if tag.as_deref() == Some(repaired.as_str()) { + assert!(output.status.success(), "{}", stderr(&output)); + assert!( + measured.contains(&new_base.as_str()), + "tagged {repaired} under an experiment for {old_base}, not {new_base}: {measured:?}\n{}", + stderr(&output) + ); + assert!( + release + .pr_body("release-0.2.0") + .contains("-version = \"0.1.1\""), + "published repaired content with the pre-change notes: {}", + release.pr_body("release-0.2.0") + ); + } +} + #[test] fn an_unmarked_legacy_pr_repaired_before_resume_refreshes_its_notes() { let release = Release::new(); From 63be9d165db1b3c1b9cdd366eda00b0a9a87a137 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 10:32:40 -0500 Subject: [PATCH 06/12] Bound mergeability waits and stop non-fixable release PRs without Fix advice Review of #2021: green CI with mergeability stuck UNKNOWN now stops after the startup grace instead of waiting forever. A closed, retargeted or moved PR, a changed origin, or a stale content base no longer recommends a Fix run. Release check tests use the shared 120-second fixture bound, the base-change Experiment test asserts unconditionally, and the legacy notes refresh is documented. --- docs/guide.md | 9 ++- scripts/release.sh | 56 +++++++++-------- site/architecture/index.html | 2 +- site/docs/index.html | 9 ++- site/how-it-works/index.html | 2 +- site/index.html | 2 +- tests/release.rs | 12 +--- tests/release/checks.rs | 114 +++++++++++++++++++++++++---------- 8 files changed, 133 insertions(+), 73 deletions(-) diff --git a/docs/guide.md b/docs/guide.md index 4114144fc..aa0f17bfc 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -4247,7 +4247,8 @@ Self-merge and Base fix defaults: the release script still tags, waits for publication, then merges. Healthy, queued and running CI start no Fix run. Missing workflows without a conflict have a five-minute startup grace period (`RELEASE_WORKFLOW_STARTUP_SECONDS=300`, adjustable for testing), then stop with -the PR, head and recovery command. This is not a deadline on CI runtime. +the PR, head and recovery command. After CI passes, GitHub gets the same grace +to report the PR mergeable. Neither is a deadline on CI runtime. After a Fix run or external push, the script fetches and validates the current PR identity, open state, readiness, origin, branches, head and exact version-bump @@ -4256,7 +4257,11 @@ before tagging. A changed content base reruns the Experiment when requested and regenerates the summary and version diff, honoring `--review`. More than five head updates, a failed or interrupted Fix, or another conflict or CI failure after the automatic Fix stops with recovery instructions and no new tag pushed. -Ctrl-C and rerunning the same version use the usual resume behavior. +A closed, retargeted or draft PR, a changed origin, or a content base older than +the latest release stops without suggesting a Fix run: inspect the PR, then rerun. +Ctrl-C and rerunning the same version use the usual resume behavior. The PR body +records the content base its notes describe; a resumed PR from before that marker +has its summary and version diff regenerated once, honoring `--review`. Before it pushes anything, it refuses a release that can't be cut and says why: a version that isn't plain `X.Y.Z` (no leading `v`), a version that isn't higher than the one in `Cargo.toml` on `origin/main`, or a `v` tag that already exists locally or on origin. A resumed PR must belong to origin and match its branch tip while open. Its tree (or a pushed branch without a PR) must be exactly the version bump in `Cargo.toml` and `Cargo.lock` from its merge base; extra content is refused even when already tagged. diff --git a/scripts/release.sh b/scripts/release.sh index 12a2e0e0b..ea5cdaecd 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -552,19 +552,27 @@ edit_pr_file() ( gh pr edit "$url" --body-file "$body_file" >/dev/null ) -# Snapshot the PR's notes claim before a yield. Reading the marker again -# after the Experiment returns would let a body edit name an unmeasured base. -remember_notes_claim() { +# Read the content base the PR's notes describe, and whether they need a refresh. +read_notes_claim() { local notes_ref=refs/thirdshift/release-notes/$version observe_release_pr if [ "$observed_notes_base" = - ]; then + # Older PR bodies lack the marker. Remember their starting content + # before Fix can push and then fail, so rerunning still detects it. release_notes_base=$(git rev-parse --quiet --verify "$notes_ref^{commit}" || release_content_base) - # Without a marker the notes may predate this script/checkout. + # Without a marker the notes may predate this script/checkout. The + # current head alone cannot establish what their summary describes. release_notes_refresh=1 else release_notes_base=$observed_notes_base release_notes_refresh= fi +} + +# Snapshot the PR's notes claim before a yield. Reading the marker again +# after the Experiment returns would let a body edit name an unmeasured base. +remember_notes_claim() { + read_notes_claim release_notes_known=1 } @@ -579,24 +587,12 @@ remember_written_notes() { # Observe this PR throughout CI. One targeted Fix run shares Delivery's Repair # budget; a second blockage or continued head churn returns to the operator. wait_for_pr_checks() { - local runs failed content_base next_base refresh_notes= startup=$SECONDS + local runs failed content_base next_base refresh_notes= startup=$SECONDS settling= local notes_ref=refs/thirdshift/release-notes/$version [[ "$workflow_startup_seconds" =~ ^[0-9]+$ ]] || refuse "RELEASE_WORKFLOW_STARTUP_SECONDS must be a nonnegative integer" - if [ -n "$release_notes_known" ]; then - content_base=$release_notes_base - refresh_notes=$release_notes_refresh - else - observe_release_pr - content_base=$observed_notes_base - if [ "$content_base" = - ]; then - # Older PR bodies lack the marker. Remember their starting content - # before Fix can push and then fail, so rerunning still detects it. - content_base=$(git rev-parse --quiet --verify "$notes_ref^{commit}" || release_content_base) - # Without a marker the notes may predate this script/checkout. The - # current head alone cannot establish what their summary describes. - refresh_notes=1 - fi - fi + [ -n "$release_notes_known" ] || read_notes_claim + content_base=$release_notes_base + refresh_notes=$release_notes_refresh # A path that has not called wait_for_experiment has not measured another base. if [ -z "$experimented_base" ]; then experimented_base=$content_base; fi git update-ref "$notes_ref" "$content_base" @@ -623,6 +619,7 @@ wait_for_pr_checks() { fi content_base=$next_base startup=$SECONDS + settling= progress "adopted $url at ${head:0:7}" fi if [ "$observed_mergeable" = CONFLICTING ]; then @@ -646,7 +643,7 @@ wait_for_pr_checks() { local previous previous=$(last_release_tag) if [ -n "$previous" ] && ! git merge-base --is-ancestor "$previous" "$content_base"; then - release_blocked "the content base predates the latest release $previous" + release_stopped "the content base predates the latest release $previous" fi if [ -n "$refresh_notes" ]; then write_summary "no new tag was pushed" @@ -663,6 +660,12 @@ wait_for_pr_checks() { if [ "$observed_head" = "$head" ] && [ "$observed_mergeable" = MERGEABLE ]; then return; fi continue fi + # Green CI on a head GitHub has not reported mergeable gets the + # same grace as workflow startup, never an unbounded wait. + settling=${settling:-$SECONDS} + if [ "$((SECONDS - settling))" -ge "$workflow_startup_seconds" ]; then + release_stopped "GitHub did not report the PR mergeable within ${workflow_startup_seconds}s of its checks passing" + fi elif [ -z "$runs" ] && [ "$((SECONDS - startup))" -ge "$workflow_startup_seconds" ]; then release_blocked "no workflows appeared within ${workflow_startup_seconds}s; inspect GitHub Actions" fi @@ -674,7 +677,7 @@ wait_for_pr_checks() { # state, origin or branch changes before adopting a head or publishing a tag. observe_release_pr() { local observation origin_head - [ "$(git remote get-url origin)" = "$release_origin" ] || release_blocked "origin changed" + [ "$(git remote get-url origin)" = "$release_origin" ] || release_stopped "origin changed" if ! observation=$(gh pr view "$url" --json url,state,headRefName,baseRefName,headRefOid,isCrossRepository,isDraft,mergeable,body | python3 -c ' import json, re, sys @@ -686,11 +689,11 @@ if (pr["url"] != url or pr["state"] != "OPEN" or pr["headRefName"] != branch notes = re.search(r"^$", pr["body"], re.M) print(pr["headRefOid"], pr["mergeable"], notes[1] if notes else "-") ' "$url" "$branch"); then - release_blocked "could not validate the release PR identity/state/branch" + release_stopped "could not validate the release PR identity/state/branch" fi read -r observed_head observed_mergeable observed_notes_base <<<"$observation" origin_head=$(git ls-remote origin "refs/heads/$branch" | cut -f 1) - [ -n "$origin_head" ] && [ "$observed_head" = "$origin_head" ] || release_blocked "PR head does not match refs/heads/$branch on origin" + [ -n "$origin_head" ] && [ "$observed_head" = "$origin_head" ] || release_stopped "PR head does not match refs/heads/$branch on origin" } fix_release_pr() { @@ -706,6 +709,11 @@ release_blocked() { refuse "$url at $head: $1; the PR is left open for human review and no new tag was pushed. Recover with thirdshift fix $url --no-merge --no-base-fix, then scripts/release.sh${with_experiment:+ --with-experiment} $version" } +# A changed PR, origin or release boundary is not something a Fix run repairs. +release_stopped() { + refuse "$url at $head: $1; no new tag was pushed. Inspect the PR, then rerun scripts/release.sh${with_experiment:+ --with-experiment} $version" +} + # merge_bump_pr # The tag is already public. A failed merge leaves the release standing and # names the PR for the Day shift; publication has already succeeded. diff --git a/site/architecture/index.html b/site/architecture/index.html index a24ae1290..b273f9b27 100644 --- a/site/architecture/index.html +++ b/site/architecture/index.html @@ -281,7 +281,7 @@

Architecture

The Experiment owns recheck history beside its immutable Grader Scores. thirdshift experiments <id> --json exposes pending borderline and decisive Scores, naming each decisive flip and its resulting provisional Verdict. thirdshift recheck <id> <trial> pass|fail --reason '…' takes the Experiment's exclusive lock and atomically saves the history, recalculated Verdict and pending list. thirdshift recheck <id> --pending lists the pending Scores and their fingerprint; thirdshift recheck <id> --batch <fingerprint> pass|fail --reason '…' checks that list under the same lock and records every listed Score in one atomic save. A stale fingerprint changes nothing. Rechecks stay in the CLI. They also update the latest invocation's Command outcome and Inbox group, preserving its completion time. The Verdict waits until the list is empty; flipping either pass/fail or a wrong-finding flag can make a Score decisive. Waiting rechecks place that invocation in Needs you; a confirmed pass moves it to Done.

scripts/release.sh runs on Linux. It refuses other hosts before Git or GitHub work. On macOS, use an Ubuntu 24.04 Linux VM such as Lima; on Windows, use WSL2.

thirdshift experiment --release <commit> compares the content commit with the last release tag on touched suites. The release script skips Experiments by default. Add --with-experiment to scripts/release.sh on each invocation to run one: it waits for rechecks and asks permission for a fail or inconclusive Verdict before pushing. Reruns with the flag reuse the Experiment, including before opening a pushed branch's PR. It selects the newest main commit descending from the last release tag whose latest push CI passed, tags that content plus the version bump, waits for publication, then merges the bump PR. A saved Candidate is selected again if a new release tag on main invalidates its ancestry; the requested version must still be higher than main. An untagged branch or open bump PR predating that tag is re-cut from a green main descendant, with a new summary and version diff, honoring --review before force-pushing the branch and updating the open PR body. Tagged or merged releases keep their resume behavior. After a failed unpublished release, rerunning re-cuts from newer green main content or automatically reruns failed jobs; published releases are never moved. If the publication check cannot be completed, the script refuses and moves nothing. Recovery retains the prior release boundary for the summary and Experiment. Main can move during the release; a failed merge leaves the release standing and the PR for the Day shift. No touched suite or a last release without the Trial flag carries on with a note. Releasing.

-

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

+

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period, as does GitHub's mergeability report after checks pass; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

diff --git a/site/docs/index.html b/site/docs/index.html index d179b9d1e..1cc365eb9 100644 --- a/site/docs/index.html +++ b/site/docs/index.html @@ -3932,7 +3932,8 @@ publication, then merges. Healthy, queued and running CI start no Fix run. Missing workflows without a conflict have a five-minute startup grace period (RELEASE_WORKFLOW_STARTUP_SECONDS=300, adjustable for testing), then stop with -the PR, head and recovery command. This is not a deadline on CI runtime.

+the PR, head and recovery command. After CI passes, GitHub gets the same grace +to report the PR mergeable. Neither is a deadline on CI runtime.

After a Fix run or external push, the script fetches and validates the current PR identity, open state, readiness, origin, branches, head and exact version-bump tree against main. Extra source changes or a wrong version require human review @@ -3940,7 +3941,11 @@ regenerates the summary and version diff, honoring --review. More than five head updates, a failed or interrupted Fix, or another conflict or CI failure after the automatic Fix stops with recovery instructions and no new tag pushed. -Ctrl-C and rerunning the same version use the usual resume behavior.

+A closed, retargeted or draft PR, a changed origin, or a content base older than +the latest release stops without suggesting a Fix run: inspect the PR, then rerun. +Ctrl-C and rerunning the same version use the usual resume behavior. The PR body +records the content base its notes describe; a resumed PR from before that marker +has its summary and version diff regenerated once, honoring --review.

Before it pushes anything, it refuses a release that can't be cut and says why: a version that isn't plain X.Y.Z (no leading v), a version that isn't higher than the one in Cargo.toml on origin/main, or a v<version> tag that already exists locally or on origin. A resumed PR must belong to origin and match its branch tip while open. Its tree (or a pushed branch without a PR) must be exactly the version bump in Cargo.toml and Cargo.lock from its merge base; extra content is refused even when already tagged.

If it's interrupted, run it again with the same version: it picks up the existing branch or pull request and carries on from the first step not yet done. With --review, it asks about a new summary before opening a pull request for a branch already pushed. It opens a PR for a pushed branch, waits for its checks, tags the bump head, waits for publication, then merges the PR. A PR already merged is still tagged at its bump head. Once the tag is on that head, it says the release is already tagged and resumes the release workflow: it waits for a run still going, or merges the open bump PR immediately when the workflow succeeded or the GitHub Release already exists, then prints the Release URL and exits 0.

An untagged pushed branch or open bump PR is re-cut if its content base does diff --git a/site/how-it-works/index.html b/site/how-it-works/index.html index b29176e70..546b758a4 100644 --- a/site/how-it-works/index.html +++ b/site/how-it-works/index.html @@ -317,7 +317,7 @@

How thirdshift works

A Verdict stays waiting on rechecks until you have checked every borderline Score and every decisive Score, whose flip would change the Verdict. The pending list gives Trial IDs for thirdshift recheck <id> <trial> pass|fail --reason '…'. The JSON view names each decisive flip and the provisional Verdict it gives. thirdshift recheck <id> --pending lists all pending Scores and a fingerprint; thirdshift recheck <id> --batch <fingerprint> pass|fail --reason '…' records one decision and reason for that reviewed list in one locked save, refusing a changed list. Your latest decision feeds the calculation; the Grader's Score and your timestamped history remain. Each recheck updates the pending list and the latest invocation's Command outcome and Inbox group. A confirmed pass moves from Needs you to Done.

scripts/release.sh runs on Linux. It refuses other hosts before Git or GitHub work. On macOS, use an Ubuntu 24.04 Linux VM such as Lima; on Windows, use WSL2.

thirdshift experiment --release <commit> compares touched suites with the last release tag. The release script skips Experiments by default. Add --with-experiment to scripts/release.sh on each invocation to run one: it waits for rechecks before pushing and asks whether to release on a fail or inconclusive Verdict. Reruns with the flag reuse the Experiment, including before opening a pushed branch’s PR. It selects the newest main commit descending from the last release tag whose latest push CI passed, tags that content plus the version bump, waits for publication, then merges the bump PR. A saved Candidate is selected again if a new release tag on main invalidates its ancestry; the requested version must still be higher than main. An untagged branch or open bump PR predating that tag is re-cut from a green main descendant, with a new summary and version diff, honoring --review before force-pushing the branch and updating the open PR body. Tagged or merged releases keep their resume behavior. After a failed unpublished release, rerunning re-cuts from newer green main content or automatically reruns failed jobs; published releases are never moved. If the publication check cannot be completed, the script refuses and moves nothing. Recovery retains the prior release boundary for the summary and Experiment. Main can move during the release; a failed merge leaves the release standing and the PR for the Day shift. No touched suite or a last release with no Trial flag carries on with a note. Release checks.

-

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

+

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period, as does GitHub's mergeability report after checks pass; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

Architect runs every few minutes and Security runs on a schedule, with fixing allowed, started by your operating system.

Next: inspect every prompt and skill, see a real Ticket run's recorded output, see the architecture in six figures, use the manual for installation, commands and configuration, or watch the explainer videos.

diff --git a/site/index.html b/site/index.html index 3b905ea76..22dffe5f5 100644 --- a/site/index.html +++ b/site/index.html @@ -523,7 +523,7 @@

Seven workflows

Each Trial removes its Harness caches when its session ends. After grading and all rechecks finish, the Experiment removes every attempt's repository copy. Paused Experiments keep repositories for resume; Session evidence, reports, packets and Scores remain readable.

scripts/release.sh runs on Linux. It refuses other hosts before Git or GitHub work. On macOS, use an Ubuntu 24.04 Linux VM such as Lima; on Windows, use WSL2.

thirdshift experiment --release <commit> compares touched suites with the last release tag. The release script skips Experiments by default. Add --with-experiment to scripts/release.sh on each invocation to run one: it waits for rechecks before pushing and asks whether to release on a fail or inconclusive Verdict. Reruns with the flag reuse the Experiment, including before opening a pushed branch’s PR. It selects the newest main commit descending from the last release tag whose latest push CI passed, tags that content plus the version bump, waits for publication, then merges the bump PR. A saved Candidate is selected again if a new release tag on main invalidates its ancestry; the requested version must still be higher than main. An untagged branch or open bump PR predating that tag is re-cut from a green main descendant, with a new summary and version diff, honoring --review before force-pushing the branch and updating the open PR body. Tagged or merged releases keep their resume behavior. After a failed unpublished release, rerunning re-cuts from newer green main content or automatically reruns failed jobs; published releases are never moved. If the publication check cannot be completed, the script refuses and moves nothing. Recovery retains the prior release boundary for the summary and Experiment. Main can move during the release; a failed merge leaves the release standing and the PR for the Day shift. No touched suite or a last release with no Trial flag carries on with a note. Release checks.

-

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

+

While waiting for release PR checks, a Merge conflict or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix. Missing workflows have a five-minute startup grace period, as does GitHub's mergeability report after checks pass; queued and running CI keep waiting. A repaired or externally updated head must still be an exact version bump. A changed content base refreshes the requested Experiment, summary and version diff. A failed Fix, another blockage or more than five head changes stops before tagging with recovery instructions. The release script still tags, waits for publication, then merges.

diff --git a/tests/release.rs b/tests/release.rs index 300dfc6d6..0acfa32af 100644 --- a/tests/release.rs +++ b/tests/release.rs @@ -562,16 +562,6 @@ fi /// Runs the script with `args`, with `stdin` on its standard input and /// `env` added to its environment. fn run_script_with(&self, args: &[&str], stdin: &str, env: &[(&str, &str)]) -> Output { - self.run_script_with_bound(args, stdin, env, support::WAIT_BOUND) - } - - fn run_script_with_bound( - &self, - args: &[&str], - stdin: &str, - env: &[(&str, &str)], - bound: std::time::Duration, - ) -> Output { support::child_lifetime::FixtureChild::spawn( self.command( &manifest_dir().join("scripts/release.sh"), @@ -581,7 +571,7 @@ fi .envs(env.iter().copied()), stdin, ) - .wait(bound) + .wait(support::WAIT_BOUND) } /// Sources the script in the maintainer's clone, with what `run_script` diff --git a/tests/release/checks.rs b/tests/release/checks.rs index c4376338b..bf96a7f64 100644 --- a/tests/release/checks.rs +++ b/tests/release/checks.rs @@ -4,7 +4,7 @@ fn set_pr(release: &Release, field: &str, value: Value) { release.gh(&["fake", "pr", "release-0.2.0", field, &value.to_string()]); } -fn fix_runs(release: &Release, script: &str) { +fn simulate_fix(release: &Release, script: &str) { fs::write(release.root().join("gh-state.json.fix"), script).unwrap(); } @@ -17,10 +17,10 @@ fn fix_calls(release: &Release) -> Vec { } fn run(release: &Release, args: &[&str], env: &[(&str, &str)]) -> Output { - // Ten seconds covers script startup, Git/GitHub reads, fake Fix/summary/ - // Experiment work, publication/merge and output collection. Fixture setup - // happens before this bound; the guardian owns all fake Fix children too. - release.run_script_with_bound(args, "", env, std::time::Duration::from_secs(10)) + // The shared release bound covers script startup, Git/GitHub reads, fake + // Fix/summary/Experiment work, publication/merge and output collection on + // a loaded runner; the guardian owns all fake Fix children too. + release.run_script_with(args, "", env) } /// Prepare the exact reported pattern: a bump based on older main, followed @@ -83,7 +83,7 @@ fn a_conflict_without_workflows_runs_one_targeted_fix_and_releases_the_repaired_ ) .unwrap(); let (old, repaired) = conflicted_bump(&release); - fix_runs(&release, &adopt_script(&repaired)); + simulate_fix(&release, &adopt_script(&repaired)); let output = run(&release, &["0.2.0"], &[]); @@ -155,7 +155,7 @@ fn pending_and_healthy_ci_never_start_a_fix_even_after_startup_grace() { fn completed_failing_ci_uses_the_same_fix_path() { let release = Release::new(); release.ci_reports(RED); - fix_runs( + simulate_fix( &release, &format!( "gh fake checks \"$(git ls-remote origin refs/heads/release-0.2.0 | cut -f1)\" '{GREEN}'" @@ -174,7 +174,7 @@ fn failed_interrupted_or_ineffective_fixes_leave_no_tag_and_are_not_retried() { for script in ["exit 1", "exit 130", "exit 0"] { let release = Release::new(); conflicted_bump(&release); - fix_runs(&release, script); + simulate_fix(&release, script); let output = run(&release, &["0.2.0"], &[]); @@ -190,7 +190,7 @@ fn failed_interrupted_or_ineffective_fixes_leave_no_tag_and_are_not_retried() { fn a_second_ci_failure_after_a_successful_fix_stops_without_another_fix() { let release = Release::new(); let (_, repaired) = conflicted_bump(&release); - fix_runs(&release, &adopt_script(&repaired).replace(GREEN, RED)); + simulate_fix(&release, &adopt_script(&repaired).replace(GREEN, RED)); let output = run(&release, &["0.2.0"], &[]); @@ -242,7 +242,7 @@ fn identity_mutations_after_fix_or_external_updates_refuse_publication() { release.gh(&["fake", "on-ci-read", "1", &mutation]); } else { set_pr(&release, "mergeable", json!("CONFLICTING")); - fix_runs(&release, &mutation); + simulate_fix(&release, &mutation); } let output = run(&release, &["0.2.0"], &[]); @@ -282,7 +282,7 @@ fn non_bump_changes_after_fix_or_external_updates_refuse_publication() { release.gh(&["fake", "on-ci-read", "1", &adopt_script(&changed)]); } else { set_pr(&release, "mergeable", json!("CONFLICTING")); - fix_runs(&release, &adopt_script(&changed)); + simulate_fix(&release, &adopt_script(&changed)); } let output = run(&release, &["0.2.0"], &[]); @@ -329,7 +329,7 @@ fn a_repair_refreshes_the_experiment_summary_and_diff_for_its_new_content_base() &release, &[experiments::verdict("pass"), experiments::verdict("pass")], ); - fix_runs(&release, &adopt_script(&repaired)); + simulate_fix(&release, &adopt_script(&repaired)); let output = run(&release, &["--with-experiment", "0.2.0"], &[]); @@ -354,7 +354,7 @@ fn a_new_content_base_cannot_publish_under_an_obsolete_experiment() { &release, &[experiments::verdict("pass"), experiments::verdict("fail")], ); - fix_runs(&release, &adopt_script(&repaired)); + simulate_fix(&release, &adopt_script(&repaired)); let output = run(&release, &["--with-experiment", "0.2.0"], &[]); @@ -410,7 +410,7 @@ fn spec_resume_after_interrupted_fix_refreshes_release_notes() { "body", json!("Old release notes: version diff starts at 0.1.0"), ); - fix_runs(&release, &(adopt_script(&repaired) + "\nexit 130\n")); + simulate_fix(&release, &(adopt_script(&repaired) + "\nexit 130\n")); let interrupted = run(&release, &["0.2.0"], &[]); assert_eq!( interrupted.status.code(), @@ -468,14 +468,14 @@ fn a_notes_marker_detects_an_external_repair_from_another_checkout() { fn recovery_after_a_repaired_tag_resumes_between_branch_and_tag_pushes() { let release = Release::new(); let (_, repaired) = conflicted_bump(&release); - fix_runs(&release, &adopt_script(&repaired)); + simulate_fix(&release, &adopt_script(&repaired)); release.release_workflow(failed_release()); let failed = run(&release, &["0.2.0"], &[]); assert_release_failed(&release, &failed, &repaired); release.merge_pr("newer-content", "Recovery content", ""); release.main_ci("completed", "success"); release.ci_reports(RED); - fix_runs(&release, "exit 1"); + simulate_fix(&release, "exit 1"); let stopped = run(&release, &["0.2.0"], &[]); assert_eq!(stopped.status.code(), Some(1), "{}", stderr(&stopped)); assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); @@ -584,21 +584,20 @@ fn a_base_change_during_the_experiment_is_measured_before_tagging() { release.pr_body("release-0.2.0"), stderr(&output) ); - if tag.as_deref() == Some(repaired.as_str()) { - assert!(output.status.success(), "{}", stderr(&output)); - assert!( - measured.contains(&new_base.as_str()), - "tagged {repaired} under an experiment for {old_base}, not {new_base}: {measured:?}\n{}", - stderr(&output) - ); - assert!( - release - .pr_body("release-0.2.0") - .contains("-version = \"0.1.1\""), - "published repaired content with the pre-change notes: {}", - release.pr_body("release-0.2.0") - ); - } + assert!(output.status.success(), "{}", stderr(&output)); + assert_eq!(tag.as_deref(), Some(repaired.as_str())); + assert!( + measured.contains(&new_base.as_str()), + "tagged {repaired} under an experiment for {old_base}, not {new_base}: {measured:?}\n{}", + stderr(&output) + ); + assert!( + release + .pr_body("release-0.2.0") + .contains("-version = \"0.1.1\""), + "published repaired content with the pre-change notes: {}", + release.pr_body("release-0.2.0") + ); } #[test] @@ -624,3 +623,56 @@ fn an_unmarked_legacy_pr_repaired_before_resume_refreshes_its_notes() { release.pr_body("release-0.2.0") ); } + +#[test] +fn green_ci_with_mergeability_stuck_unknown_stops_instead_of_waiting_forever() { + let release = Release::new(); + release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + set_pr(&release, "mergeable", json!("UNKNOWN")); + release.ci_reports(GREEN); + let output = run( + &release, + &["0.2.0"], + &[("RELEASE_WORKFLOW_STARTUP_SECONDS", "0")], + ); + assert_eq!(output.status.code(), Some(1), "{}", stderr(&output)); + assert!( + stderr(&output).contains("did not report the PR mergeable"), + "{}", + stderr(&output) + ); + assert!(fix_calls(&release).is_empty()); + assert_eq!(release.origin_tag("v0.2.0"), None); +} + +#[test] +fn a_pr_closed_during_ci_does_not_recommend_a_fix_run() { + let release = Release::new(); + release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + release.gh(&[ + "fake", + "on-ci-read", + "1", + "gh fake pr release-0.2.0 state '\"CLOSED\"'", + ]); + let output = run(&release, &["0.2.0"], &[]); + assert_eq!(output.status.code(), Some(1), "{}", stderr(&output)); + assert!( + !stderr(&output).contains("left open"), + "{}", + stderr(&output) + ); + assert!( + !stderr(&output).contains("thirdshift fix"), + "{}", + stderr(&output) + ); + assert!( + stderr(&output).contains("rerun scripts/release.sh 0.2.0"), + "{}", + stderr(&output) + ); + assert_eq!(release.origin_tag("v0.2.0"), None); +} From fc42a9865c4083088391d0d18fdfd769dd028947 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 11:17:11 -0500 Subject: [PATCH 07/12] thirdshift: failed ticket run (cannot synchronize acquired checkout /home/jacob/repos/thirdshift-issue-2016 on expected Issue branch issue-2016: cannot inspect every registration in /home/jacob/repos/thirdshift/.git/worktrees; checkout path and head unknown) 2026-11-11T16:17:11Z, host stephens. Uncommitted work at the time of failure is included in this commit. From f9d0599ecab57777e379b0b33e63d7428f49b76b Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 11:49:38 -0500 Subject: [PATCH 08/12] Name the ready step when a draft bump PR blocks the merge after publication (#2016) --- docs/guide.md | 2 +- scripts/release.sh | 12 +++++++++--- site/docs/index.html | 2 +- tests/release.rs | 31 +++++++++++++++++++++++++++++++ 4 files changed, 42 insertions(+), 5 deletions(-) diff --git a/docs/guide.md b/docs/guide.md index aa0f17bfc..55449ed3d 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -4237,7 +4237,7 @@ From a clone of this repo, signed in to `gh` and the User config's default Harne scripts/release.sh 0.4.0 ``` -It works from `origin/main` in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from `release-` that bumps the version in `Cargo.toml` and `Cargo.lock` and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in [`scripts/release-summary.md`](../scripts/release-summary.md). The script uses `cargo run` in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With `--review` (`scripts/release.sh --review 0.4.0`), it then prints the summary and asks `[y]es / [e]dit / [n]o`: `y` carries on, `e` opens the summary in `$EDITOR` and carries on with what you save, and `n` stops with no branch, pull request or tag pushed. Without `--review`, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of `release-` as `v` and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into `main` with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it names the open PR for the Day shift to merge and still exits `0`. CI's run on `main` for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as `HH:MM:SS` as thirdshift's own progress lines are, so a long wait can be told from a stalled one. A Merge conflict or completed failing PR checks can start the bounded Fix run described below. If the release workflow fails, it exits `1`, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Running CI and publication have no overall timeout. +It works from `origin/main` in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from `release-` that bumps the version in `Cargo.toml` and `Cargo.lock` and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in [`scripts/release-summary.md`](../scripts/release-summary.md). The script uses `cargo run` in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With `--review` (`scripts/release.sh --review 0.4.0`), it then prints the summary and asks `[y]es / [e]dit / [n]o`: `y` carries on, `e` opens the summary in `$EDITOR` and carries on with what you save, and `n` stops with no branch, pull request or tag pushed. Without `--review`, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of `release-` as `v` and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into `main` with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it warns that the bump PR is not merged, names the open PR for the Day shift to merge (with `gh pr ready ` first when the PR has become a draft) and still exits `0`. CI's run on `main` for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as `HH:MM:SS` as thirdshift's own progress lines are, so a long wait can be told from a stalled one. A Merge conflict or completed failing PR checks can start the bounded Fix run described below. If the release workflow fails, it exits `1`, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Running CI and publication have no overall timeout. While waiting for PR CI, the script observes that PR's current head and mergeability. A Merge conflict, including one with no workflows, or completed failing CI starts diff --git a/scripts/release.sh b/scripts/release.sh index ea5cdaecd..5f533d8d4 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -716,13 +716,19 @@ release_stopped() { # merge_bump_pr # The tag is already public. A failed merge leaves the release standing and -# names the PR for the Day shift; publication has already succeeded. +# names the PR for the Day shift; publication has already succeeded. A PR +# made draft after its last validation names the ready step it needs. merge_bump_pr() { + local action="merge $url" if gh pr merge "$url" --merge --match-head-commit "$head" >/dev/null; then progress "merged $url" - else - progress "warning: $tag is left in place and the release still stands; the Day shift must merge $url" + return + fi + if gh pr view "$url" --json isDraft 2>/dev/null | + python3 -c 'import json, sys; sys.exit(not json.load(sys.stdin)["isDraft"])' 2>/dev/null; then + action="merge $url; $url is a draft, so first mark it ready with gh pr ready $url" fi + progress "warning: $tag is left in place and the release still stands, but the bump PR is not merged; the Day shift must $action" } # wait_for_release diff --git a/site/docs/index.html b/site/docs/index.html index 1cc365eb9..47e70a199 100644 --- a/site/docs/index.html +++ b/site/docs/index.html @@ -3923,7 +3923,7 @@

From a clone of this repo, signed in to gh and the User config's default Harness, run the release script with the new version:

scripts/release.sh 0.4.0
-

It works from origin/main in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from release-<version> that bumps the version in Cargo.toml and Cargo.lock and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in scripts/release-summary.md. The script uses cargo run in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With --review (scripts/release.sh --review 0.4.0), it then prints the summary and asks [y]es / [e]dit / [n]o: y carries on, e opens the summary in $EDITOR and carries on with what you save, and n stops with no branch, pull request or tag pushed. Without --review, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of release-<version> as v<version> and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into main with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it names the open PR for the Day shift to merge and still exits 0. CI's run on main for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as HH:MM:SS as thirdshift's own progress lines are, so a long wait can be told from a stalled one. A Merge conflict or completed failing PR checks can start the bounded Fix run described below. If the release workflow fails, it exits 1, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Running CI and publication have no overall timeout.

+

It works from origin/main in a temporary worktree, so your checked-out branch and uncommitted changes don't matter and aren't touched. It opens a pull request from release-<version> that bumps the version in Cargo.toml and Cargo.lock and nothing else. Its body is a summary of the release, then the version diff. The User config’s default Harness, with its configured Model and Effort (Claude Code when no default is set), writes the summary from the version diff and the title, number and body of each pull request merged between the last release tag and the release base, following the prompt in scripts/release-summary.md. The script uses cargo run in this checkout to share thirdshift’s config parser and Harness adapters, and runs the summary in a temporary directory without Factory skills. Because pull request titles and bodies are contributor-editable, no Harness gets its session permission bypass, and each runs with as few tools as its CLI allows. Claude runs with no tools. Codex runs read-only and sandboxed, with its shell, MCP, web, app, plugin, browser and hook tools off. Grok gets only its to-do tool. OpenCode refuses every tool call. Muse has no shell, writes or web, and one model step. agy denies every tool that needs a permission, though it can still read files. Kimi Code runs an agent whose tool list is empty, with the release input as its instructions in a file beside the prompt. A summary that tries a refused tool usually comes back empty, so the script falls back as below. If the config is invalid, the chosen Harness fails, or it returns no summary, the script warns and uses GitHub's generated notes as the summary instead, with a note saying so. It writes the summary before it pushes anything. With --review (scripts/release.sh --review 0.4.0), it then prints the summary and asks [y]es / [e]dit / [n]o: y carries on, e opens the summary in $EDITOR and carries on with what you save, and n stops with no branch, pull request or tag pushed. Without --review, only a failed or inconclusive Experiment prompts. It waits for the pull request's checks, tags the bump commit at the head of release-<version> as v<version> and pushes the tag. Pushing the tag starts the release workflow (below). The script waits for that workflow on the bump commit, printing the run's URL so you can watch it. Only after the workflow succeeds does it merge the bump PR into main with a merge commit, preserving the tagged commit in main's history even if other content has arrived. If that merge fails, the published release stands: it warns that the bump PR is not merged, names the open PR for the Day shift to merge (with gh pr ready <url> first when the PR has become a draft) and still exits 0. CI's run on main for the merge isn't waited for. The script prints the GitHub Release's URL last. It prints a line on stderr for each step, stamped with the local time as HH:MM:SS as thirdshift's own progress lines are, so a long wait can be told from a stalled one. A Merge conflict or completed failing PR checks can start the bounded Fix run described below. If the release workflow fails, it exits 1, naming the failed jobs and run URL, leaving the tag in place and the PR open and unmerged. Recovery happens when you run the script again with the same version. Running CI and publication have no overall timeout.

While waiting for PR CI, the script observes that PR's current head and mergeability. A Merge conflict, including one with no workflows, or completed failing CI starts at most one targeted thirdshift fix <PR URL> --no-merge --no-base-fix per release diff --git a/tests/release.rs b/tests/release.rs index 0acfa32af..44ccaec45 100644 --- a/tests/release.rs +++ b/tests/release.rs @@ -936,6 +936,37 @@ fn a_failed_bump_merge_keeps_the_tag_and_finishes_publishing() { assert!(err.contains(&release_url()), "{err}"); } +/// The 0.29.0 recovery: the PR turned draft after its head was validated, +/// so the merge after publication failed. The warning names that blocker. +#[test] +fn a_draft_bump_pr_after_publication_names_the_ready_step() { + let release = Release::new(); + release.ci_reports(GREEN); + release.gh(&[ + "fake", + "on-merge", + "1", + "gh fake pr release-0.2.0 isDraft true", + ]); + release.gh(&[ + "fake", + "refuse-merges", + "1", + "GraphQL: Pull Request is still a draft (mergePullRequest)", + ]); + + let output = release.run_script("0.2.0"); + + assert!(output.status.success(), "{}", stderr(&output)); + let pr = release.pr("release-0.2.0"); + assert_eq!(pr["state"], "OPEN"); + let url = pr["url"].as_str().unwrap(); + let err = stderr(&output); + assert!(err.contains("release still stands"), "{err}"); + assert!(err.contains(&format!("{url} is a draft")), "{err}"); + assert!(err.contains(&format!("gh pr ready {url}")), "{err}"); +} + #[test] fn main_moving_during_bump_checks_does_not_change_the_tagged_content() { let release = Release::new(); From 821f337ca75b069963fab734c3c7b8d73862e006 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 12:05:51 -0500 Subject: [PATCH 09/12] Name the ready step when a Fix-drafted release PR is rerun (#2016) --- docs/guide.md | 2 ++ scripts/release.sh | 17 +++++++++++------ site/docs/index.html | 2 ++ tests/release/checks.rs | 38 +++++++++++++++++++++++++++++++++++++- 4 files changed, 52 insertions(+), 7 deletions(-) diff --git a/docs/guide.md b/docs/guide.md index 55449ed3d..ce7422d5c 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -4259,6 +4259,8 @@ head updates, a failed or interrupted Fix, or another conflict or CI failure after the automatic Fix stops with recovery instructions and no new tag pushed. A closed, retargeted or draft PR, a changed origin, or a content base older than the latest release stops without suggesting a Fix run: inspect the PR, then rerun. +A failed or interrupted Fix run leaves the PR draft, so the rerun names +`gh pr ready `: mark it ready after review, then rerun. Ctrl-C and rerunning the same version use the usual resume behavior. The PR body records the content base its notes describe; a resumed PR from before that marker has its summary and version diff regenerated once, honoring `--review`. diff --git a/scripts/release.sh b/scripts/release.sh index 5f533d8d4..01bb7df43 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -74,7 +74,9 @@ # changed content base refreshes the requested Experiment, summary and diff. # More than five head changes, an unsuccessful Fix, or another blockage stops # before tagging with recovery instructions. Workflows have 300 seconds to -# appear (RELEASE_WORKFLOW_STARTUP_SECONDS); queued/running CI has no deadline. +# appear (RELEASE_WORKFLOW_STARTUP_SECONDS), and the same grace bounds GitHub +# reporting green checks as mergeable; queued/running CI has no deadline. A +# draft PR stops before tagging, naming gh pr ready. # # Prints a progress line on stderr for each step, stamped with the local time # as HH:MM:SS, with the release workflow run's URL once it waits for that run, @@ -676,7 +678,7 @@ wait_for_pr_checks() { # Re-read the captured PR, never search for a replacement. Refuse identity, # state, origin or branch changes before adopting a head or publishing a tag. observe_release_pr() { - local observation origin_head + local observation origin_head draft [ "$(git remote get-url origin)" = "$release_origin" ] || release_stopped "origin changed" if ! observation=$(gh pr view "$url" --json url,state,headRefName,baseRefName,headRefOid,isCrossRepository,isDraft,mergeable,body | python3 -c ' @@ -684,14 +686,17 @@ import json, re, sys pr = json.load(sys.stdin) url, branch = sys.argv[1:] if (pr["url"] != url or pr["state"] != "OPEN" or pr["headRefName"] != branch - or pr["baseRefName"] != "main" or pr["isCrossRepository"] or pr["isDraft"]): - sys.exit("release PR identity/state/branch changed or PR is draft") + or pr["baseRefName"] != "main" or pr["isCrossRepository"]): + sys.exit("release PR identity/state/branch changed") notes = re.search(r"^$", pr["body"], re.M) -print(pr["headRefOid"], pr["mergeable"], notes[1] if notes else "-") +print(pr["headRefOid"], pr["mergeable"], notes[1] if notes else "-", pr["isDraft"]) ' "$url" "$branch"); then release_stopped "could not validate the release PR identity/state/branch" fi - read -r observed_head observed_mergeable observed_notes_base <<<"$observation" + read -r observed_head observed_mergeable observed_notes_base draft <<<"$observation" + # A failed or interrupted Fix run drafts the PR; publishing it would leave + # the merge blocked, so a human reviews it and marks it ready first. + [ "$draft" = False ] || release_stopped "the PR is a draft, as a failed or interrupted Fix run leaves it; review it and mark it ready with gh pr ready $url" origin_head=$(git ls-remote origin "refs/heads/$branch" | cut -f 1) [ -n "$origin_head" ] && [ "$observed_head" = "$origin_head" ] || release_stopped "PR head does not match refs/heads/$branch on origin" } diff --git a/site/docs/index.html b/site/docs/index.html index 47e70a199..a3acd3b8e 100644 --- a/site/docs/index.html +++ b/site/docs/index.html @@ -3943,6 +3943,8 @@ after the automatic Fix stops with recovery instructions and no new tag pushed. A closed, retargeted or draft PR, a changed origin, or a content base older than the latest release stops without suggesting a Fix run: inspect the PR, then rerun. +A failed or interrupted Fix run leaves the PR draft, so the rerun names +gh pr ready <url>: mark it ready after review, then rerun. Ctrl-C and rerunning the same version use the usual resume behavior. The PR body records the content base its notes describe; a resumed PR from before that marker has its summary and version diff regenerated once, honoring --review.

diff --git a/tests/release/checks.rs b/tests/release/checks.rs index bf96a7f64..a3ee13297 100644 --- a/tests/release/checks.rs +++ b/tests/release/checks.rs @@ -254,8 +254,14 @@ fn identity_mutations_after_fix_or_external_updates_refuse_publication() { stderr(&output) ); assert_eq!(release.origin_tag("v0.2.0"), None, "{field}"); + // A draft names the ready step instead of a changed identity. + let refusal = if field == "isDraft" { + "the PR is a draft" + } else { + "identity/state/branch" + }; assert!( - stderr(&output).contains("identity/state/branch"), + stderr(&output).contains(refusal), "{field}: {}", stderr(&output) ); @@ -676,3 +682,33 @@ fn a_pr_closed_during_ci_does_not_recommend_a_fix_run() { ); assert_eq!(release.origin_tag("v0.2.0"), None); } + +/// A real Fix run drafts the PR when it fails or is interrupted. The rerun +/// names that blocker and resumes once the PR is marked ready. +#[test] +fn a_rerun_after_an_interrupted_fix_that_drafted_the_pr_names_the_ready_step() { + let release = Release::new(); + let (_, repaired) = conflicted_bump(&release); + simulate_fix( + &release, + &(adopt_script(&repaired) + "\ngh fake pr release-0.2.0 isDraft true\nexit 130\n"), + ); + let interrupted = run(&release, &["0.2.0"], &[]); + assert_eq!( + interrupted.status.code(), + Some(1), + "{}", + stderr(&interrupted) + ); + + let resumed = run(&release, &["0.2.0"], &[]); + let err = stderr(&resumed); + assert_eq!(resumed.status.code(), Some(1), "{err}"); + assert!(err.contains("the PR is a draft"), "{err}"); + assert!(err.contains("gh pr ready https://"), "{err}"); + assert_eq!(release.origin_tag("v0.2.0"), None); + release.gh(&["pr", "ready", "release-0.2.0"]); + let ready = run(&release, &["0.2.0"], &[]); + assert!(ready.status.success(), "{}", stderr(&ready)); + assert_eq!(release.origin_tag("v0.2.0"), Some(repaired)); +} From 62d814eee72eec41c4c213e9787ebca68e83d434 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 12:06:29 -0500 Subject: [PATCH 10/12] Reuse with_input for the release script runner (#2016) --- tests/release.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/release.rs b/tests/release.rs index 44ccaec45..1a2e31534 100644 --- a/tests/release.rs +++ b/tests/release.rs @@ -562,7 +562,7 @@ fi /// Runs the script with `args`, with `stdin` on its standard input and /// `env` added to its environment. fn run_script_with(&self, args: &[&str], stdin: &str, env: &[(&str, &str)]) -> Output { - support::child_lifetime::FixtureChild::spawn( + with_input( self.command( &manifest_dir().join("scripts/release.sh"), &self.maintainer(), @@ -571,7 +571,6 @@ fi .envs(env.iter().copied()), stdin, ) - .wait(support::WAIT_BOUND) } /// Sources the script in the maintainer's clone, with what `run_script` From c972a3bc4180ca1a668bdf11dd94051210e87fa8 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 12:06:36 -0500 Subject: [PATCH 11/12] Preserve release recovery diagnostics for interrupted Fix and draft PRs (#2016) --- docs/guide.md | 4 ++ scripts/release.sh | 4 ++ site/docs/index.html | 3 ++ tests/release.rs | 3 +- tests/release/checks.rs | 86 ++++++++++++++++++++++-------------- tests/release/experiments.rs | 6 +++ 6 files changed, 70 insertions(+), 36 deletions(-) diff --git a/docs/guide.md b/docs/guide.md index aa0f17bfc..3eb1284dd 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -4263,6 +4263,10 @@ Ctrl-C and rerunning the same version use the usual resume behavior. The PR body records the content base its notes describe; a resumed PR from before that marker has its summary and version diff regenerated once, honoring `--review`. +After publication, a failed merge leaves the release standing and names the PR +for the Day shift. If the PR is draft, mark it ready with `gh pr ready ` +before merging it. Rerunning the release command keeps the published tag in place. + Before it pushes anything, it refuses a release that can't be cut and says why: a version that isn't plain `X.Y.Z` (no leading `v`), a version that isn't higher than the one in `Cargo.toml` on `origin/main`, or a `v` tag that already exists locally or on origin. A resumed PR must belong to origin and match its branch tip while open. Its tree (or a pushed branch without a PR) must be exactly the version bump in `Cargo.toml` and `Cargo.lock` from its merge base; extra content is refused even when already tagged. If it's interrupted, run it again with the same version: it picks up the existing branch or pull request and carries on from the first step not yet done. With `--review`, it asks about a new summary before opening a pull request for a branch already pushed. It opens a PR for a pushed branch, waits for its checks, tags the bump head, waits for publication, then merges the PR. A PR already merged is still tagged at its bump head. Once the tag is on that head, it says the release is already tagged and resumes the release workflow: it waits for a run still going, or merges the open bump PR immediately when the workflow succeeded or the GitHub Release already exists, then prints the Release URL and exits `0`. diff --git a/scripts/release.sh b/scripts/release.sh index ea5cdaecd..7870ec8af 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -700,9 +700,12 @@ fix_release_pr() { [ -z "$fix_attempted" ] || release_blocked "$1 after the automatic Fix run" fix_attempted=1 progress "$1 on $url; starting one targeted Fix run" + # Cargo itself can be interrupted before thirdshift installs its handler. + trap 'release_blocked "the targeted Fix run was interrupted"' INT TERM HUP if ! cargo run --quiet --manifest-path "$summary_manifest" -- fix "$url" --no-merge --no-base-fix; then release_blocked "$1; the targeted Fix run failed or was interrupted" fi + trap - INT TERM HUP } release_blocked() { @@ -722,6 +725,7 @@ merge_bump_pr() { progress "merged $url" else progress "warning: $tag is left in place and the release still stands; the Day shift must merge $url" + progress "if the PR is draft, mark it ready with gh pr ready $url before merging" fi } diff --git a/site/docs/index.html b/site/docs/index.html index 1cc365eb9..8a62a28cc 100644 --- a/site/docs/index.html +++ b/site/docs/index.html @@ -3946,6 +3946,9 @@ Ctrl-C and rerunning the same version use the usual resume behavior. The PR body records the content base its notes describe; a resumed PR from before that marker has its summary and version diff regenerated once, honoring --review.

+

After publication, a failed merge leaves the release standing and names the PR +for the Day shift. If the PR is draft, mark it ready with gh pr ready <PR URL> +before merging it. Rerunning the release command keeps the published tag in place.

Before it pushes anything, it refuses a release that can't be cut and says why: a version that isn't plain X.Y.Z (no leading v), a version that isn't higher than the one in Cargo.toml on origin/main, or a v<version> tag that already exists locally or on origin. A resumed PR must belong to origin and match its branch tip while open. Its tree (or a pushed branch without a PR) must be exactly the version bump in Cargo.toml and Cargo.lock from its merge base; extra content is refused even when already tagged.

If it's interrupted, run it again with the same version: it picks up the existing branch or pull request and carries on from the first step not yet done. With --review, it asks about a new summary before opening a pull request for a branch already pushed. It opens a PR for a pushed branch, waits for its checks, tags the bump head, waits for publication, then merges the PR. A PR already merged is still tagged at its bump head. Once the tag is on that head, it says the release is already tagged and resumes the release workflow: it waits for a run still going, or merges the open bump PR immediately when the workflow succeeded or the GitHub Release already exists, then prints the Release URL and exits 0.

An untagged pushed branch or open bump PR is re-cut if its content base does diff --git a/tests/release.rs b/tests/release.rs index 0acfa32af..a1c9a0d40 100644 --- a/tests/release.rs +++ b/tests/release.rs @@ -562,7 +562,7 @@ fi /// Runs the script with `args`, with `stdin` on its standard input and /// `env` added to its environment. fn run_script_with(&self, args: &[&str], stdin: &str, env: &[(&str, &str)]) -> Output { - support::child_lifetime::FixtureChild::spawn( + with_input( self.command( &manifest_dir().join("scripts/release.sh"), &self.maintainer(), @@ -571,7 +571,6 @@ fi .envs(env.iter().copied()), stdin, ) - .wait(support::WAIT_BOUND) } /// Sources the script in the maintainer's clone, with what `run_script` diff --git a/tests/release/checks.rs b/tests/release/checks.rs index bf96a7f64..7d587bd0f 100644 --- a/tests/release/checks.rs +++ b/tests/release/checks.rs @@ -186,6 +186,54 @@ fn failed_interrupted_or_ineffective_fixes_leave_no_tag_and_are_not_retried() { } } +#[test] +fn a_signal_interrupted_fix_reports_recovery_without_tagging() { + let release = Release::new(); + let (head, _) = conflicted_bump(&release); + // Signal the release shell while its foreground Fix also dies by signal, + // as Ctrl-C can do during Cargo startup before thirdshift's handler runs. + // Name only these two fixture processes, keeping the guardian alive. + simulate_fix(&release, "kill -INT \"$PPID\"\nkill -INT \"$$\"\n"); + + let output = run(&release, &["0.2.0"], &[]); + + let err = stderr(&output); + assert_eq!(output.status.code(), Some(1), "{err}"); + assert!(err.contains(&head), "{err}"); + assert!(err.contains("Recover with thirdshift fix"), "{err}"); + assert!(err.contains("scripts/release.sh 0.2.0"), "{err}"); + assert_eq!(fix_calls(&release).len(), 1); + assert_eq!(release.origin_tag("v0.2.0"), None); +} + +#[test] +fn a_published_draft_pr_explains_how_to_finish_merging_without_moving_the_tag() { + let release = Release::new(); + let head = release.push_bump("0.2.0"); + release.open_bump_pr("0.2.0"); + release.origin(&["tag", "v0.2.0", &head]); + set_pr(&release, "isDraft", json!(true)); + release.gh(&["fake", "fails", "pr merge"]); + let refs = release.origin_refs(); + + let output = run(&release, &["0.2.0"], &[]); + + let err = stderr(&output); + assert!(output.status.success(), "{err}"); + assert!(err.contains("release still stands"), "{err}"); + assert!(err.contains("draft"), "{err}"); + assert!( + err.contains(&format!( + "gh pr ready {}", + release.pr("release-0.2.0")["url"].as_str().unwrap() + )), + "{err}" + ); + assert_eq!(release.origin_refs(), refs); + assert_eq!(release.pr("release-0.2.0")["state"], "OPEN"); + assert!(fix_calls(&release).is_empty()); +} + #[test] fn a_second_ci_failure_after_a_successful_fix_stops_without_another_fix() { let release = Release::new(); @@ -491,10 +539,10 @@ fn recovery_after_a_repaired_tag_resumes_between_branch_and_tag_pushes() { /// an exact bump of newer main and rewrites the notes marker to that base /// before the script observes the pull request again. fn forge_new_base_during_experiment(release: &Release, repaired: &str, new_base: &str) { - let driver = release.bin().join("cargo-driver"); - if !driver.exists() { - fs::copy(release.bin().join("cargo"), &driver).unwrap(); - } + experiments::cargo_reports( + release, + &[experiments::verdict("pass"), experiments::verdict("pass")], + ); let body = format!("Old release notes\n"); fs::write( release.root().join("during-experiment.sh"), @@ -510,36 +558,6 @@ gh fake checks {repaired} '[{{"name": "test", "conclusion": "success"}}]' ), ) .unwrap(); - fs::write( - release.root().join("experiment-views.json"), - json!([experiments::verdict("pass"), experiments::verdict("pass")]).to_string(), - ) - .unwrap(); - fs::write( - release.bin().join("cargo"), - r#"#!/usr/bin/env python3 -import json, os, pathlib, subprocess, sys -args = sys.argv[sys.argv.index('--') + 1:] -root = pathlib.Path(os.environ['FAKE_GH_STATE']).parent -if args == ['--release-summary'] or args[:1] == ['fix']: - driver = str(root / 'bin/cargo-driver') - os.execv(driver, [driver, *sys.argv[1:]]) -branch = subprocess.check_output(['git', 'ls-remote', 'origin', 'refs/heads/release-0.2.0'], text=True).strip() -with (root / 'experiment-calls.jsonl').open('a') as calls: - calls.write(json.dumps(dict(args=args, branch=branch)) + '\n') -marker = root / 'experiment-mutated' -if not marker.exists(): - marker.write_text('1') - subprocess.check_call(['bash', str(root / 'during-experiment.sh')]) -views = root / 'experiment-views.json' -remaining = json.loads(views.read_text()) -assert remaining, 'unexpected Experiment call' -view = remaining.pop(0) -views.write_text(json.dumps(remaining)) -print(json.dumps(view)) -"#, - ) - .unwrap(); } #[test] diff --git a/tests/release/experiments.rs b/tests/release/experiments.rs index 434189171..f5a5e82e8 100644 --- a/tests/release/experiments.rs +++ b/tests/release/experiments.rs @@ -32,6 +32,12 @@ if args == ['--release-summary'] or args[0] == 'fix': branch = subprocess.check_output(['git', 'ls-remote', 'origin', 'refs/heads/release-0.2.0'], text=True).strip() with (root / 'experiment-calls.jsonl').open('a') as calls: calls.write(json.dumps(dict(args=args, branch=branch)) + '\n') +# A concrete release-race fixture mutates the PR during the first Experiment. +action = root / 'during-experiment.sh' +if action.exists(): + consumed = action.with_suffix('.ran') + action.rename(consumed) + subprocess.check_call(['bash', str(consumed)]) views = root / 'experiment-views.json' remaining = json.loads(views.read_text()) assert remaining, 'unexpected Experiment call' From 25d92bc3f0dd260bcfe0e1ef05a3df9f8cd99fd7 Mon Sep 17 00:00:00 2001 From: Jacob Stephens Date: Wed, 11 Nov 2026 13:05:32 -0500 Subject: [PATCH 12/12] thirdshift: failed ticket run (PR https://github.com/JacobStephens2/thirdshift/pull/2021 is a draft) 2026-11-11T18:05:32Z, host lima-thirdshift. Uncommitted work at the time of failure is included in this commit.