Skip to content

Re-run failed checks once before a Declined CI fix fails the Run - #267

Merged
JacobStephens2 merged 9 commits into
mainfrom
issue-262
Oct 1, 2026
Merged

JacobStephens2 merged 9 commits into
mainfrom
issue-262

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

A CI-fix Repair that leaves the head unchanged no longer fails the Run straight away. The head first gets one Check re-run, so a check that failed on a flaky test or a runner fault can pass.

 Repair loop, after the Base branch is merged in and pushed
   head = worktree head
-  if head is the one the last CI-fix Repair was given
-    fail: Declined CI fix
-  ci = watch CI on head
+  if head is the one the last CI-fix Repair was given
+    ci = Check re-run of head            # ci::rerun
+      any own failed check not a GitHub Actions job  -> none
+      gh run rerun <run> --failed, once per workflow run
+        the request fails (GitHub refuses it)         -> none
+      wait (grace period) until none of the failed check runs is listed
+        still listed                                  -> none
+      watch CI on head to its end
+    if none, or an own check is red again
+      fail: Declined CI fix               # same cause string as before
+  else
+    ci = watch CI on head
   green / absent     -> carry on as for any green head
   only Inherited red -> Base branch moved? Base fix? fail "fix <base> first"
   own red            -> CI-fix Repair, remember (head, failed checks)
  • The trigger is the unchanged head and nothing else. What the Repair concluded, and its "CI notes", are never read (ADR 0010, in line with ADR 0008).
  • A Check re-run starts no Repair and touches no budget. MAX_REPAIRS and the CI-fix Repair prompt are unchanged.
  • Check gains job: Option<ActionsJob> (check run id, workflow run id), read from the check run's id and details URL. Ci::has_own_failures tells the Repair loop whether the re-run left a check of the branch's own red.
  • No stale result: a re-run job gets a new check run id and GitHub lists only the latest check run of a name. I checked this against a re-run commit in rust-lang/cargo (29 check runs with the default filter, 58 with filter=all).
  • Docs: CONTEXT.md (the Declined CI fix entry, new Check re-run term), README step 6, Failed runs and the Spec PR, the help text, and docs/adr/0010-one-check-re-run-before-a-declined-ci-fix.md.
  • main is merged in. Its ADR 0009 took the number this branch had used, so the ADR here is 0010.

Evidence

  • Before: CI fails, the Repair commits nothing, the Run ends as CI red on <short sha> and the Repair found nothing to fix on the branch although the same commit would pass (One module for how a Run's ending shows and how a child Run's is read back #258).
    After: cargo test passes on the branch with main merged in: 29 test binaries, 0 failed, including 11 new tests. cargo fmt --check and cargo clippy --all-targets -- -D warnings are clean.
tests/ci.rs
  a_ci_fix_repair_that_makes_no_commit_gets_one_check_re_run_and_green_is_success
      3 failed checks over 2 workflow runs -> 1 Repair, re-run requests [900, 901], PR ready
  a_check_re_run_that_fails_again_is_a_failed_run_with_no_second_repair_or_re_run
  a_failed_commit_status_cant_be_re_run_so_nothing_is
  a_check_re_run_github_refuses_is_a_declined_ci_fix_that_shows_the_refusal
      stderr: the failed checks on <short sha> were not re-run: gh run rerun 900 --failed … HTTP 502
  a_check_re_run_leaves_inherited_failures_alone
  an_inherited_failure_re_run_along_with_the_branchs_own_is_waited_for_too
  a_ci_fix_repair_that_commits_gets_no_check_re_run
  the_attempt_before_a_check_re_run_is_never_taken_for_its_result
  a_check_re_run_that_never_appears_is_a_declined_ci_fix
tests/merge_run.rs
  a_check_re_run_that_passes_after_a_ci_fix_repair_made_no_commit_lets_the_merge_run_merge
tests/spec_run.rs
  a_ci_fix_repair_on_the_spec_pr_that_makes_no_commit_gets_a_check_re_run
src/github.rs
  only_a_check_run_whose_details_are_a_jobs_page_is_an_actions_job

The existing Declined CI fix tests pass unedited: their checks are not GitHub Actions jobs, so nothing is re-run. The fake gh gives check runs ids and answers gh run rerun <run> --failed, with a rerun field on a check saying what the new attempt comes to.

Not tested against real GitHub: the re-run request itself, and how soon the new attempt is listed. Only the listing behaviour above was checked live.

Merge Danger

Door: two-way

Reverting the commits restores the old behaviour. Nothing is stored or migrated.

Blast Radius: Repair-loop

  • A Run now makes a write request to GitHub Actions (gh run rerun), which needs a token that may re-run workflows. A request that fails ends the Run as before, with the error on stderr.
  • A flaky failure costs one more CI cycle before the Run ends either way.
  • The request is per workflow run, so an Inherited failure that shares a workflow run with one of the branch's own failed checks is re-run along with it. The watch waits for its new attempt too.
  • A job that was skipped because the failed job it needs failed is re-run by GitHub as well, but only gets its new check run once that job finishes. A poll that lands in between can read the old skipped and call CI green before the dependent job has run (see Unaddressed findings).

Unaddressed findings

Addressed from this review: the progress line for a failed request no longer says "GitHub refused" for every error (it now reads the failed checks on <short sha> were not re-run: <error>), has_own_failures moved onto Ci, before_rerun renamed previous_attempt, and two duplications in the tests and the fake gh removed.

Standards

  • tests/merge_run.rs and tests/spec_run.rs inline the check JSON and the gh run rerun call that tests/ci.rs has helpers for. Skipped: test files here keep their own copies of small helpers (checks_on_head, commits_fix), and each uses them once.
  • handed_to_repair is an unnamed (String, FailedChecks) pair. Skipped: it is one local with two uses, and a named type would be more to read than the pair.
  • ci::rerun does the eligibility check, the request, the wait and the watch in one body, with three None exits, and the Repair loop still reads Option<Ci>. Skipped: each exit says why in a progress line, and what Inherited failures mean after a re-run is the Repair loop's to decide, so ci reports what CI came to.
  • ADR 0010 writes "ADR 0008" where the README writes "ADR-0008". Skipped: the ADRs already use both forms (ADR 0005 uses the spaced one).

Spec

  • A dependent job's old skipped result can be read as passing. --failed also re-runs jobs that depend on a failed one, and their new check runs appear only after it finishes. Skipped: a first CI watch has the same gap (a job waiting on needs has no check run yet), so closing it means watching workflow runs instead of check runs everywhere, which is a change to the CI watch as a whole. The issue's rule, never to take the failed result of the old attempt for the re-run's, holds.
  • The wait for the new attempt is capped at the grace period (60s). A failed job that needs another failed job, or waits on a concurrency group or an approval, gets its new check run later than that, and the Run then fails as a Declined CI fix while the re-run goes on. Skipped: it is the same rule as checks appearing after a push, which is what "the same waiting rules as any other CI watch" asks for, and it fails the way the Run failed before this change.
  • no re-run appeared on <sha> within Ns is an end state the issue does not name. Kept: without it the watch would report the old attempt's result, which the issue forbids.
  • Inherited failures in the same workflow run as an own failure are re-run too. Skipped: the issue asks for one request per workflow run, and GitHub's "re-run failed jobs" has no narrower form short of one request per job. ADR 0010 records it.
  • A request that fails on the second of two workflow runs leaves the first re-running unwatched, and no test covers it. Skipped: the Run fails either way, the progress line quotes the gh command that failed, and the loop returns on the first error, which the single-run test covers.
  • A check run of another app (not a GitHub Actions job) has only a unit test, only_a_check_run_whose_details_are_a_jobs_page_is_an_actions_job. Skipped: the acceptance criterion allows a commit status for the Run-level test, and both take the same job: None path.
  • no re-run appeared on <sha> within 0s shows 0s for a sub-second grace period. Skipped: only tests set one, and waiting up to 0s for CI already reads the same way.

Closes #262

🤖 Generated with Claude Code

JacobStephens2 and others added 4 commits October 1, 2026 12:51
When a CI-fix Repair leaves the head unchanged, ask GitHub to re-run the
branch's own failed checks, once per workflow run, and watch CI on that
head again from the new attempt on. Green carries on; red again, a check
that can't be re-run, or a refused re-run is a Declined CI fix as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on main at 362b9ca; fix main first)

2026-10-01T13:02:36Z, host thirdshift. Uncommitted work at the time of failure is included in this commit.
JacobStephens2 and others added 3 commits October 1, 2026 17:22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hared test helpers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JacobStephens2
JacobStephens2 marked this pull request as ready for review October 1, 2026 17:50
@JacobStephens2
JacobStephens2 merged commit af22210 into main Oct 1, 2026
13 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-262 branch October 1, 2026 18:11
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-run failed checks once before a Declined CI fix fails the Run

1 participant