Repository navigation
A GitHub API rate limit fails the run instead of waiting for the reset #1973
Description
This was generated by AI during triage.
What happened
A Ticket run's CI watch that hits GitHub's REST rate limit fails the whole Ticket run. It does not wait for the quota to reset. On 2026-11-10 at 15:02 EST, #1968 and #1895 both ended as Failed ticket runs with:
gh api repos/JacobStephens2/thirdshift/actions/runs?head_sha=<sha>&per_page=100 failed: gh: API rate limit exceeded for user ID 53905376.
#1895 had already failed the same way on 2026-11-09 at 19:10 EST, along with #1898. A Review run on JacobStephens2/stephens.page failed the same way at 15:01 (gh api repos/JacobStephens2/stephens.page/pulls/30). Each run had to be handed back by hand. Their work survived only because it was already in draft PRs (#1970, #1910).
Why
All five machines (the Mac/lima VM, stephens, thirdshift, orchestrate and dev) run their scheduled Pickup, Review, Fix and Triage Passes as one GitHub user. That user gets 5,000 REST calls an hour. When the runs overlap, they use up the hour's calls.
In the code, check_read::is_transient (src/github/check_read.rs:46) deliberately treats rate-limit errors as non-transient: "Quota exhaustion belongs to quota-aware deferral, not a short retry." But read_checks in src/ci.rs:272 has no such deferral. It returns the error, and the Ticket run fails. operational_cause (src/github.rs:73) already classifies the error as GitHub rate-limit exhaustion, so the information is there. src/github/quota.rs can already read both quota clocks and their reset times.
Expected
When a rate limit is the cause, the run waits for the reported reset, or pauses and later continues on its own, instead of becoming a Failed ticket run. Maybe the same applies to other gh reads during a run (the Review run above failed reading a PR). It should not need the Day shift to hand the issue back.
Open questions for triage
- Should the CI watch wait in place until the reset (≤1 h), or should the run end as something like Paused and be continued by Pickup?
- Should Passes check
rate_limitbefore starting and skip when the remaining quota is low, like the Inbox's fixed reserve inquota.rs? - Is the shared account across machines worth addressing directly, for example with a token per machine or a GitHub App?
Activity
JacobStephens2 commentedon Nov 11, 2026
Recommend closing #1973 as a duplicate of open Spec #1994, specifically its Ticket #1997: #1997. The defect remains present on the selected Base; this recommendation consolidates the work rather than rejecting the reported problem.
Confirmed locally: src/github/check_read.rs:46 excludes primary quota exhaustion from transient retries; src/ci.rs:273 immediately propagates such errors from CI watch; src/github.rs:1246 does likewise for other adapter reads. src/github/quota.rs exposes reset clocks and a 500-unit reserve only for Inbox reads. Two existing execution tests passed, confirming that quota failures receive no retry and that check reads classify primary exhaustion as non-transient.
#1994 and #1997 already specify waiting within the running Command, automatic continuation, bounded waiting, interruption handling, and coverage for CI watch and Review runs. The related Pass admission work is #1996, shared quota observations are #1995, and GitHub App authentication is #1993. No additional split or implementation issue is needed.