Skip to content

fix(mtg): don't ask again inside the 30 seconds a 429 limits access for - #87

Merged
plainprogrammer merged 11 commits into
mainfrom
081-scryfall-429-fallback-wait
Oct 14, 2026
Merged

plainprogrammer merged 11 commits into
mainfrom
081-scryfall-429-fallback-wait

Conversation

@plainprogrammer

Copy link
Copy Markdown
Owner

After a 429 with no usable Retry-After, the Scryfall client no longer asks again inside the 30 seconds Scryfall says a 429 limits access for. The call ends at once and the job comes back 45 seconds later.

Closes #81

Stacked on #85 (the fix for #79 and #80): both edit spec 011's version history and the same CLAUDE.md paragraph, so this branch merges that one. Merge #85 first.

What changed

  • A 429 with no usable Retry-After ends the call at once with Catalog::Sources::RateLimitedError carrying no wait, on any attempt, for get_json and fetch_image. Before, the client retried after 1 and 2 seconds, so all three attempts landed inside the window. "No usable" covers an absent, empty, whitespace-only, negative, non-numeric or past-dated header, and now a zero: obeying Retry-After: 0 allowed three requests in about 0.2 seconds.
  • The job's floor is 45 seconds (Catalog::RateLimitRetry::SHORTEST_WAIT, was 30). The retry is scheduled from when the error is handled, so at 30 the next request cleared the window only by the dispatcher's polling delay.
  • Unchanged: a 5xx without the header still retries after 1 and 2 seconds; a usable Retry-After of 1 to 30 seconds is slept exactly; a longer one ends the call carrying the wait; the worst-case sleep per call stays 60 seconds, so the art build's 2-minute rule holds.
  • Specs 002 and 011 (both 1.3.0), CLAUDE.md and the comments in client.rb, art_build.rb and art/build.rb state the new figures.

What Scryfall's pages say

Their rate-limits, API overview and errors pages state that a 429 limits access for 30 seconds, that continuing may lead to a ban, and that *.scryfall.io has no rate limits. None mentions Retry-After. What a real 429 carries is still unobserved (no rate limit was provoked to find out), so both cases are specified.

Decisions for you

  1. Raise at once, not wait 30 seconds in the call. No worker sleeps and the client has one rule for every caller. The cost is on the art build: one header-less 429 from the image host stops the build and costs one of the job's 3 rate-limit retries, where it used to cost about 3 seconds. The review agreed with the choice and asked for Rate-limit retries count over a job's whole life, so a long build can run out #86 to cover the retry budget.
  2. 45 seconds for the floor. The review suggested 45 to 60.

Verification

  • bin/ci: passed, 1405 examples, 0 failures, 1 pending (the art agreement spec, unrelated).
  • New client, refresh-job, build-job, build and admin-page examples, each seen failing first.
  • One independent read-only review (Opus): no must-fix. With the old back-off patched back in, 25 examples fail. It checked the quotes against the fetched Scryfall pages. Its follow-ups (the zero header, the floor's margin, the CLAUDE.md sentence, one admin-page example, one spec 011 row) are the last five commits.
  • Not verified: a real 429 from Scryfall.

Known limits and follow-ups

🤖 Generated with Claude Code

plainprogrammer and others added 11 commits October 13, 2026 20:39
…#81)

Scryfall limits access for 30 seconds after a 429. Without a usable
Retry-After the client retried after 1 and then 2 seconds, so all three
attempts landed inside that window. It now ends the call at once with a
RateLimitedError that carries no wait, whichever attempt the 429 answers;
the job's retry, 30 seconds later at the soonest, is the next request.

A 5xx without the header keeps its 1 and 2 seconds, and a usable
Retry-After is honoured as before. Negative, non-numeric, empty and past
values count as none.

Chosen over waiting 30 seconds in the call: a retry at the 30th second is
the earliest one that could work and the likeliest to be turned away, and
a second 429 there is the overload the page warns about. Raising keeps the
wait in the queue, as #74 did for a long Retry-After.

The art build therefore stops at one such 429 from the image host, which
its specs now pin.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
End to end through the job: one request to Scryfall, the run closed as
failed with the reason, and the job queued to run again 30 seconds later.
Seen failing against the client before its fix (3 requests, the old
message).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The grace and beat comments gave one figure for an image's waits. They
now say which case it is: 60 seconds when each Retry-After asks for 30,
3 seconds after server errors without one, and none after a 429 without
one, which stops the build. The rate-limit error's comment said the
source "kept answering"; one answer is now enough.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
FR-3, FR-4 and the error scenarios, version 1.3.0. Also records what
Scryfall's documentation says of a 429 (30 seconds of limited access)
and what it doesn't (whether one carries Retry-After).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
AC-3.4, AC-3.6, Reliability and the error scenarios, version 1.3.0. A 5xx
without the header keeps its 1 and 2 seconds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
PR #85 merges first. Spec 011's version history keeps its 1.2.1 row
(#79), with 1.3.0 (#81) after it, the table running oldest first; the
header stays at 1.3.0. Everything else merged without conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#81)

The floor was 30 seconds, exactly the time Scryfall says a 429 limits
access for. The wait runs from when the error is handled, so the next
request cleared that window only by the dispatcher's polling delay. 45 is
the documented 30 plus a margin. The client's own cap stays at 30.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Obeyed, a zero put all three requests about 0.2 seconds apart inside the
30 seconds a 429 limits access for. A 429 that asks for zero (or an HTTP
date that is now) ends the call at once, like one that names no wait. A
5xx that asks for zero is still retried at once. A whitespace-only header
was already no wait; it is now pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ued to retry (#81)

End to end: the client's error, the job's retry through Solid Queue and
the page, with the retry's time 45 seconds on. Seen failing on the time
against the 30-second floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ps the build (#81)

Specs 002 and 011, still 1.3.0 (unreleased): the job's shortest wait is
45 seconds, a Retry-After of zero on a 429 counts as none, and spec 011's
5xx row says a 5xx asking for more than 30 seconds stops the build. The
1.2.0 history rows keep the 30 seconds they recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…floor is 45 seconds (#81)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from 079-art-index-stalled-and-retry-reason to main October 14, 2026 02:14
@plainprogrammer plainprogrammer self-assigned this Oct 14, 2026
@plainprogrammer
plainprogrammer merged commit 53f04b0 into main Oct 14, 2026
4 checks passed
@plainprogrammer
plainprogrammer deleted the 081-scryfall-429-fallback-wait branch October 14, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scryfall client retries inside the 30 seconds a 429 limits access for

1 participant