Repository navigation
fix(mtg): never wait longer than 30 seconds for a Retry-After - #76
Merged
Merged
Conversation
On a 429 the Scryfall client slept for whatever Retry-After said, so a long value held a worker and stalled the refresh or the art build without any progress recorded. A wait of up to 30 seconds (MAX_BACK_OFF, the time Scryfall's rate limit page says a 429 limits access for) is still slept exactly. A longer one ends the call at once with Catalog::Sources::RateLimitedError, a TransientError whose message says how long the server asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) Integer() took "-5", "0x1F" and "1_0" as numbers: a negative value reached sleep, which raises ArgumentError on one, and the other two were waited as 31 and 10 seconds. An HTTP date, the header's other form, was ignored, so the retry came after 1 or 2 seconds however far off the date was. Retry-After is now a run of digits or an HTTP date. A date is waited for to the next whole second and is held to the same 30-second bound. Anything else, and a date already past, counts as no header: the usual 1 and 2 seconds. Zero still retries at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te limited (#74) get_json slept after its third 429 and then raised, so a call that was going to fail anyway held its worker for one more Retry-After. fetch_image already skipped that wait. Both now go through one back_off: it waits only when a retry follows, so a call waits 60 seconds at most (2 waits of up to 30). A call that ends on a 429, or on a wait too long to sit out (a 5xx's Retry-After included, on any attempt), raises Catalog::Sources::RateLimitedError. An image whose three attempts end on a 5xx still raises a plain TransientError: the image failed, and nothing asked the client to stay away. Callers can now tell the two apart. The class and method comments describe the bounded back-off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build counted every Catalog::Sources::Error from an image as that image's failure and went on to the next. While the client slept through each Retry-After that was slow but polite. Now that the client ends a call on a wait longer than 30 seconds, going on would send a request for every image left, 100 ms apart, to a host that asked the build to stay away, and record each as failed. A Catalog::Sources::RateLimitedError is no longer an image's failure: the build stops, its run is recorded as failed with the client's message, and no image after it is requested. Images already fetched stay cached, so the next build carries on. An image that answers 5xx to its last attempt, or another status, is still recorded as failed and passed over (AC-3.6). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither job changes. The refresh job already retries Catalog::Sources::TransientError, and RateLimitedError is one, so a rate limit the client won't sit out is held by the queue instead of a worker. The art build job retries only a busy database, so a build the image host rate limited fails and is started again by an admin or the next refresh; its comment now says why. Both examples pass as written. Each was seen to fail against a mutation: RateLimitedError made a plain Error, and TransientError added to the build job's retry_on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spec 002 1.2.0: FR-3 says what the adapter waits before a retry (what Retry-After asks for, at most 30 seconds, nothing after the last attempt, 60 seconds a call) and that a longer wait asked for ends the call as a transient rate-limit error. Error Scenarios gain that case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… limit (#74) Spec 011 1.2.0: AC-3.4 gives the image request's waits and when it ends as rate limited; AC-3.6 says a rate limit is not an image's failure and stops the build, whose job isn't retried; Reliability and Error Scenarios say the same, with the 429 and 5xx rows now separate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or (#74) Catalog::Sources::RateLimitedError#retry_after is the last wait asked for, in seconds: the one too long to sit out, or the last 429's when the attempts ran out, or nil when the source never said. A job needs it to come back no sooner than asked. Also covers a wait over the bound on a middle attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#74) With the client no longer sleeping through a long Retry-After, the refresh job took the error as any transient one: it retried about 3 and 18 seconds later, sooner than the source asked, and then failed for good. A refresh that used to sleep and succeed became a quick permanent failure. Catalog::RateLimitRetry handles Catalog::Sources::RateLimitedError ahead of the job's transient rule. The job runs again after the wait the source asked for, and 30 seconds at least (a source that never said, or asked for less). A wait of more than an hour isn't scheduled, and a job is retried this way 3 times at most: beyond either the job fails and shows under Failed. The job examples assert the time the retry is scheduled for, and one runs a real refresh against a stubbed 429 to check the run's message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) Stopping the build on a rate limit was right; failing its job without a retry was not. Three 429s with no Retry-After land within 3 seconds, and a single 503 asking for 120 seconds is enough too: either ended an hours-long build until an admin or the next refresh started it again. The build job now includes Catalog::RateLimitRetry, as the refresh job does: it runs again after the wait the host asked for, 30 seconds at least, an hour at most, 3 times at most. The retry keeps the job id, and the new run carries on from the cached images; one example runs the build, its stop and the queued retry end to end. The job's comment no longer says a retry can't know the host's wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Specs 002 and 011, still 1.2.0 (unreleased on this branch): the refresh job and the art build job run again after the wait the source asked for, 30 seconds at least, an hour at most, 3 such retries at most, and fail beyond that. 002's "retried with back-off by the job system" read as honouring the wait when it didn't; FR-4 now states the rule. 011's AC-3.6, Reliability and Error Scenarios no longer say the build job isn't retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three examples the review asked for: the retry count travels in the queued job's data, so the fourth run from the queue fails (refresh job and art build job), and rate limits and other transient errors are counted each on their own in the refresh job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#74) Catalog::RateLimitRetry's comment and spec 002 said the source isn't asked again early. That is true of the job that waits; it holds no lock meanwhile, so the schedule's refresh or one started by hand can start and ask at once. No behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: - app/models/mtg/art/build.rb: the class comment keeps both sentences, the rate limit that stops the build (#74) and the heartbeat between images (#73). The code merged by itself: #73's BEAT_EVERY and beat_if_due, and #74's re-raise in fetch. - docs/specs/011-card-scanner-art-matching/spec.md: one version history, #73's 1.1.4 row and then #74's 1.2.0; the spec is at 1.2.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#73, #74) Where the two fixes meet: a build the image host rate limited has closed its run as failed and its job is scheduled in the queue. The art index panel reads "Queued.", not interrupted and not failed, however old the build's heartbeat, and the retry's MTG::ArtBuild.start! opens a new run and leaves the failed one as it ended. No production code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cond cap (#73, #74) Issue #73's statements counted "the waits a rate limit or a server error asks for" as open-ended, as they were on that branch. With #74 they are 30 seconds each and 60 in all for one image, and a longer one stops the build. So the waits alone leave at most 90 seconds between heartbeats (30 before the image, 60 within it), inside the 2-minute grace; slow requests or a slow decode on top of them can still pass it. Says so in MTG::ArtBuild::JOB_ENDED_GRACE's and MTG::Art::Build::BEAT_EVERY's comments, spec 015's residual row (1.2.4) and CLAUDE.md, which now also names the cap, RateLimitedError and Catalog::RateLimitRetry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Scryfall client never waits longer than 30 seconds for a
Retry-After. When the source asks for more, the call ends with a rate-limit error and the job comes back when the source asked, instead of a worker sleeping for as long as the header says.Closes #74
Stacked on #75 (the fix for #73): the two conflict in
app/models/mtg/art/build.rband spec 011, so this branch merges that one. Merge #75 first; this PR's diff is against its branch.What changed
MTG::Scryfall::Client::MAX_BACK_OFFis 30 seconds per wait, the time Scryfall's rate-limits page says a 429 limits access for. One call sleeps at most 60 seconds (two waits between three attempts). A longer ask ends the call at once, with no sleep and no further request, raisingCatalog::Sources::RateLimitedError(aTransientError) that carries the wait asked for.Retry-After. Whole seconds or an HTTP date, nothing else. The oldInteger()parse had live bugs:-5reachedsleepand raisedArgumentError,0x1Fwas waited as 31 seconds, and an HTTP date was ignored. Anything unusable falls back to 1 s, then 2 s, as before.get_jsonslept after its third 429 and then raised.Catalog::RateLimitRetry, inCatalog::RefreshJobandMTG::Art::BuildJob): after the wait the source asked for, 30 seconds at least. A wait over 1 hour isn't scheduled, and a job gets at most 3 such retries; beyond either the job fails and shows under Failed. Without this the refresh retried at about 3 s and 21 s and then failed for good, wheremainslept and succeeded (found by the first review).CLAUDE.mdmatch.Decisions for you
What the admin page shows while a job waits
Verification
bin/cion7d843b9(merged with fix(admin): show a crashed art build as interrupted at once #75's branch): passed. RSpec 1347 examples, 0 failures, 1 pending (the art agreement spec, unrelated).retry_ononly when included after it (documented in the job, and the refresh examples catch a reorder).Retry-After.Known limits and follow-ups
catalog:refreshstarts another refresh that asks at once. "No sooner than asked" holds per job.*.scryfall.io, the image host, has no rate limits.Retry-After, the client's in-call retries are still 1 s and 2 s, inside Scryfall's 30 seconds; only the job's return is floored at 30 s.docs/roadmap.mddoesn't list Scryfall client sleeps for Retry-After with no upper bound #74 yet.🤖 Generated with Claude Code