Skip to content

fix(mtg): never wait longer than 30 seconds for a Retry-After - #76

Merged
plainprogrammer merged 16 commits into
mainfrom
074-scryfall-retry-after-cap
Oct 13, 2026
Merged

plainprogrammer merged 16 commits into
mainfrom
074-scryfall-retry-after-cap

Conversation

@plainprogrammer

Copy link
Copy Markdown
Owner

The Scryfall client never waits longer than 30 seconds for a Retry-After. When the source asks for more, the call ends with a rate-limit error and the job comes back when the source asked, instead of a worker sleeping for as long as the header says.

Closes #74

Stacked on #75 (the fix for #73): the two conflict in app/models/mtg/art/build.rb and spec 011, so this branch merges that one. Merge #75 first; this PR's diff is against its branch.

What changed

  • The bound. MTG::Scryfall::Client::MAX_BACK_OFF is 30 seconds per wait, the time Scryfall's rate-limits page says a 429 limits access for. One call sleeps at most 60 seconds (two waits between three attempts). A longer ask ends the call at once, with no sleep and no further request, raising Catalog::Sources::RateLimitedError (a TransientError) that carries the wait asked for.
  • Reading Retry-After. Whole seconds or an HTTP date, nothing else. The old Integer() parse had live bugs: -5 reached sleep and raised ArgumentError, 0x1F was waited as 31 seconds, and an HTTP date was ignored. Anything unusable falls back to 1 s, then 2 s, as before.
  • No wait after the last attempt. get_json slept after its third 429 and then raised.
  • The art build stops on a rate limit rather than counting a failed image. With the cap alone it would have sent one request per remaining image, 100 ms apart, to a host asking it to stay away, and recorded each as failed. Fetched images stay cached, so the next build carries on. A 404, a network error or a 5xx to the last attempt is still one failed image.
  • Both jobs come back when asked (Catalog::RateLimitRetry, in Catalog::RefreshJob and MTG::Art::BuildJob): after the wait the source asked for, 30 seconds at least. A wait over 1 hour isn't scheduled, and a job gets at most 3 such retries; beyond either the job fails and shows under Failed. Without this the refresh retried at about 3 s and 21 s and then failed for good, where main slept and succeeded (found by the first review).
  • Specs 002 and 011 (both 1.2.0), spec 015 (1.2.4, the art build's worst-case quiet time with the cap) and CLAUDE.md match.

Decisions for you

  1. The bounds on the job's retry: a 1-hour ceiling and 3 retries per job. Both are our numbers, derived from nothing.
  2. Stopping the art build is wider than the issue's text. Both reviews judged it necessary once the wait is capped.

What the admin page shows while a job waits

  • Refresh: "Queued to retry. The last run failed while downloading: …" with a "Retrying at" fact. Never interrupted or stalled: the run is closed as failed before the retry is scheduled.
  • Art build: plain "Queued.", with no reason and no retry time; the reason is on the failed run until the retry starts.

Verification

  • bin/ci on 7d843b9 (merged with fix(admin): show a crashed art build as interrupted at once #75's branch): passed. RSpec 1347 examples, 0 failures, 1 pending (the art agreement spec, unrelated).
  • New examples failed before their code, with the exceptions the commits record (the job examples that pin "leave it alone" were checked against a temporary mutation).
  • Two independent read-only reviews (Opus). The first found the two job defects above, now fixed. The second ran a probe against real Solid Queue rows: the retry count survives the round trip, a job beyond the bounds gets a failed execution, the refresh's lock is released while a retry waits, and the handler takes precedence over the generic retry_on only when included after it (documented in the job, and the refresh examples catch a reorder).
  • Not verified: a real 429 from Scryfall. Their page doesn't say whether one carries Retry-After.

Known limits and follow-ups

  • A second refresh can ask early. The waiting job holds no lock, so the weekly job or catalog:refresh starts another refresh that asks at once. "No sooner than asked" holds per job.
  • No override while a retry waits. "Refresh now" and "Build art index" are refused while a retry is scheduled, and the jobs page discards only failed jobs: up to about 3 hours in the worst case.
  • The 3 retries are per job lifetime. A multi-hour first build fails on its fourth rate limit even when they are hours apart. Scryfall says *.scryfall.io, the image host, has no rate limits.
  • The art build's panel hides the reason while its retry waits, unlike the refresh's.
  • Without a Retry-After, the client's in-call retries are still 1 s and 2 s, inside Scryfall's 30 seconds; only the job's return is floored at 30 s.
  • HTTP-date values are measured against the local clock.
  • Spec 011's Reliability line says the build job retries transient errors; it retries a busy database and a rate limit.
  • docs/roadmap.md doesn't list Scryfall client sleeps for Retry-After with no upper bound #74 yet.

🤖 Generated with Claude Code

plainprogrammer and others added 16 commits October 13, 2026 16:44
On a 429 the Scryfall client slept for whatever Retry-After said, so a long
value held a worker and stalled the refresh or the art build without any
progress recorded.

A wait of up to 30 seconds (MAX_BACK_OFF, the time Scryfall's rate limit
page says a 429 limits access for) is still slept exactly. A longer one ends
the call at once with Catalog::Sources::RateLimitedError, a TransientError
whose message says how long the server asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Integer() took "-5", "0x1F" and "1_0" as numbers: a negative value reached
sleep, which raises ArgumentError on one, and the other two were waited as
31 and 10 seconds. An HTTP date, the header's other form, was ignored, so
the retry came after 1 or 2 seconds however far off the date was.

Retry-After is now a run of digits or an HTTP date. A date is waited for to
the next whole second and is held to the same 30-second bound. Anything
else, and a date already past, counts as no header: the usual 1 and 2
seconds. Zero still retries at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te limited (#74)

get_json slept after its third 429 and then raised, so a call that was going
to fail anyway held its worker for one more Retry-After. fetch_image already
skipped that wait. Both now go through one back_off: it waits only when a
retry follows, so a call waits 60 seconds at most (2 waits of up to 30).

A call that ends on a 429, or on a wait too long to sit out (a 5xx's
Retry-After included, on any attempt), raises
Catalog::Sources::RateLimitedError. An image whose three attempts end on a
5xx still raises a plain TransientError: the image failed, and nothing asked
the client to stay away. Callers can now tell the two apart.

The class and method comments describe the bounded back-off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build counted every Catalog::Sources::Error from an image as that
image's failure and went on to the next. While the client slept through
each Retry-After that was slow but polite. Now that the client ends a call
on a wait longer than 30 seconds, going on would send a request for every
image left, 100 ms apart, to a host that asked the build to stay away, and
record each as failed.

A Catalog::Sources::RateLimitedError is no longer an image's failure: the
build stops, its run is recorded as failed with the client's message, and
no image after it is requested. Images already fetched stay cached, so the
next build carries on. An image that answers 5xx to its last attempt, or
another status, is still recorded as failed and passed over (AC-3.6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither job changes. The refresh job already retries
Catalog::Sources::TransientError, and RateLimitedError is one, so a rate
limit the client won't sit out is held by the queue instead of a worker.
The art build job retries only a busy database, so a build the image host
rate limited fails and is started again by an admin or the next refresh;
its comment now says why.

Both examples pass as written. Each was seen to fail against a mutation:
RateLimitedError made a plain Error, and TransientError added to the build
job's retry_on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spec 002 1.2.0: FR-3 says what the adapter waits before a retry (what
Retry-After asks for, at most 30 seconds, nothing after the last attempt,
60 seconds a call) and that a longer wait asked for ends the call as a
transient rate-limit error. Error Scenarios gain that case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… limit (#74)

Spec 011 1.2.0: AC-3.4 gives the image request's waits and when it ends as
rate limited; AC-3.6 says a rate limit is not an image's failure and stops
the build, whose job isn't retried; Reliability and Error Scenarios say the
same, with the 429 and 5xx rows now separate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or (#74)

Catalog::Sources::RateLimitedError#retry_after is the last wait asked for,
in seconds: the one too long to sit out, or the last 429's when the
attempts ran out, or nil when the source never said. A job needs it to
come back no sooner than asked.

Also covers a wait over the bound on a middle attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#74)

With the client no longer sleeping through a long Retry-After, the refresh
job took the error as any transient one: it retried about 3 and 18 seconds
later, sooner than the source asked, and then failed for good. A refresh
that used to sleep and succeed became a quick permanent failure.

Catalog::RateLimitRetry handles Catalog::Sources::RateLimitedError ahead of
the job's transient rule. The job runs again after the wait the source
asked for, and 30 seconds at least (a source that never said, or asked for
less). A wait of more than an hour isn't scheduled, and a job is retried
this way 3 times at most: beyond either the job fails and shows under
Failed.

The job examples assert the time the retry is scheduled for, and one runs a
real refresh against a stubbed 429 to check the run's message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Stopping the build on a rate limit was right; failing its job without a
retry was not. Three 429s with no Retry-After land within 3 seconds, and a
single 503 asking for 120 seconds is enough too: either ended an
hours-long build until an admin or the next refresh started it again.

The build job now includes Catalog::RateLimitRetry, as the refresh job
does: it runs again after the wait the host asked for, 30 seconds at
least, an hour at most, 3 times at most. The retry keeps the job id, and
the new run carries on from the cached images; one example runs the build,
its stop and the queued retry end to end. The job's comment no longer says
a retry can't know the host's wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Specs 002 and 011, still 1.2.0 (unreleased on this branch): the refresh job
and the art build job run again after the wait the source asked for, 30
seconds at least, an hour at most, 3 such retries at most, and fail beyond
that. 002's "retried with back-off by the job system" read as honouring the
wait when it didn't; FR-4 now states the rule. 011's AC-3.6, Reliability
and Error Scenarios no longer say the build job isn't retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three examples the review asked for: the retry count travels in the queued
job's data, so the fourth run from the queue fails (refresh job and art
build job), and rate limits and other transient errors are counted each on
their own in the refresh job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#74)

Catalog::RateLimitRetry's comment and spec 002 said the source isn't asked
again early. That is true of the job that waits; it holds no lock
meanwhile, so the schedule's refresh or one started by hand can start and
ask at once. No behaviour changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- app/models/mtg/art/build.rb: the class comment keeps both sentences, the
  rate limit that stops the build (#74) and the heartbeat between images
  (#73). The code merged by itself: #73's BEAT_EVERY and beat_if_due, and
  #74's re-raise in fetch.
- docs/specs/011-card-scanner-art-matching/spec.md: one version history,
  #73's 1.1.4 row and then #74's 1.2.0; the spec is at 1.2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#73, #74)

Where the two fixes meet: a build the image host rate limited has closed
its run as failed and its job is scheduled in the queue. The art index
panel reads "Queued.", not interrupted and not failed, however old the
build's heartbeat, and the retry's MTG::ArtBuild.start! opens a new run and
leaves the failed one as it ended. No production code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cond cap (#73, #74)

Issue #73's statements counted "the waits a rate limit or a server error
asks for" as open-ended, as they were on that branch. With #74 they are 30
seconds each and 60 in all for one image, and a longer one stops the build.
So the waits alone leave at most 90 seconds between heartbeats (30 before
the image, 60 within it), inside the 2-minute grace; slow requests or a
slow decode on top of them can still pass it.

Says so in MTG::ArtBuild::JOB_ENDED_GRACE's and MTG::Art::Build::BEAT_EVERY's
comments, spec 015's residual row (1.2.4) and CLAUDE.md, which now also
names the cap, RateLimitedError and Catalog::RateLimitRetry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scryfall client sleeps for Retry-After with no upper bound

1 participant