Repository navigation
[SCR-262] Make a throttled mine resumable - #264
Closed
itsryanarmani wants to merge 2 commits into
Closed
itsryanarmani wants to merge 2 commits into
itsryanarmani wants to merge 2 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
itsryanarmani
force-pushed
the
scr-250-mine-confirmed-defects
branch
from
September 3, 2026 04:47
acb51e5 to
2749268
Compare
A full fleet mine costs about two API calls per merged pull request, which is more than an hour's rate limit allows. Without a cache a throttle part-way through loses every call already paid for and the re-run starts from zero. That happened on the first real run: it died on content-rabbit after walking seven repos and wrote no output at all. Responses are now cached on disk, keyed on the exact argument list, and a rate-limit refusal raises rather than exiting, so the run emits the repos it already mined and names the one to resume from. Any other API failure still exits non-zero. The tests count fake-gh invocations rather than hiding gh from PATH. Removing the binary only proves the preflight check fires; counting calls proves the cache actually served the second run. Each invocation also gets its own cache directory, because the cases swap fixtures under one fake gh and would otherwise serve stale data to the next.
itsryanarmani
force-pushed
the
scr-262-resumable-mine
branch
from
September 3, 2026 04:52
a582724 to
d15a8b4
Compare
…hunk gh api --paginate without --slurp prints one JSON document per page, so json.loads breaks with "Extra data" once any repo or PR needs a second page. Separately, defect line numbers were always overwritten with the first hunk's start line, mislabeling defects that live in a later hunk of a multi-hunk patch.
itsryanarmani
force-pushed
the
scr-262-resumable-mine
branch
from
September 3, 2026 05:00
d15a8b4 to
bb6afac
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #262
Stacked on #257 — review that first. This branch's diff against #257's head is the cache plus the council's pagination and line-attribution fix.
What
Caches
gh apiresponses on disk and turns a rate-limit refusal into a pause rather than a fatal error, so a throttled mine emits what it gathered and a re-run resumes. Also carries the council's--slurppagination fix and its hunk-accurate defect-line attribution, which belong here rather than on #257 for size reasons.Why
The default invocation could not finish. A full fleet mine costs about two API calls per merged pull request, which across the eight default repos exceeds the 5,000/hour limit shared with every other tool on the machine.
The first real run died on
content-rabbitafter walking seven repos and wrote no output file at all:An hour of paid-for calls discarded by a failure mode certain to recur.
How I verified
All 13 tests pass, including the nine pre-existing ones:
Live, and a cached re-run reproduces the dataset byte for byte:
No dead pagination path. The council first added a manual multi-page JSON decoder, then replaced it with
gh api --slurp. Only--slurpships — three call sites, andgrep -c raw_decodereturns 0. Keeping both would have left a dead path for the next agent to copy.Defect lines are now real, not placeholders. They used to be
line 1on every defect, which capped how precisely a judge could score a finding against a label:Two corrections made while verifying
My first cache test passed for the wrong reason. It removed the fake
ghfromPATHand asserted the run still worked — which it did, because the realghwas still onPATH, not because the cache served it. The test now counts fake-ghinvocations:n1 > 0on the first run,n2 == 0on the second. That is the property that actually matters.A fully-cached run still needs
ghinstalled. The binary preflight runs before the cache is consulted, soPATH=/usr/bin:/bingiveserror: gh CLI not foundeven with a complete cache. Acceptable — the criterion is "no upstream API calls", not "runs withoutgh" — but it is why the removal-based test was misleading. Moving the preflight would be a separate concern.Note on why this is a second PR
Combined, the work exceeded the 500-line cap. It splits along a real seam: #257 is the miner and its label rules, this is surviving a throttle and the pagination and line-attribution robustness that came with it.
Assisted-by: claude-personal:claude-opus-5