Skip to content

feat(plugin): two-lane retry for never-suppress failures — fast lease retries for blips, cadence backoff for persistent ones; v0.24.0 - #61

Merged
harper-joseph merged 3 commits into
mainfrom
feat/failure-retry-backoff
Aug 3, 2026
Merged

harper-joseph merged 3 commits into
mainfrom
feat/failure-retry-backoff

Conversation

@harper-joseph

@harper-joseph harper-joseph commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

What

Follow-up to #59. The never-suppress failures (401/403 auth-shaped, 408/429/5xx transient) now retry in two lanes instead of a flat +renderInterval, keyed on the target's strike count (render.failureRetry.fastRetries, default 2):

  • Fast lane (strikes ≤ fastRetries): leave the schedule holding its claim lease, so the retry comes on lease expiry (queue.jobLeaseTime = 10 min). An origin blip recovers in minutes, and the cached page's swrTtl window covers bots throughout.
  • Slow lane (after fastRetries consecutive failures): reschedule at the target's normal cadence — a persistently failing page must not hot-loop 144 renders/day.

The kept page's expiry is deliberately not extended in the slow lane (an earlier revision did; reviewed out): page.swrTtl is the product bound on how stale we serve as if fresh. Past it, bots fall through to the origin on purpose — a live page for auth-shaped failures, an honest 5xx for transient ones — because serving arbitrarily old snapshots while users get errors would break bot/user parity. If serve-stale-on-origin-error is ever wanted, it belongs in the serving path as explicit RFC 5861 stale-if-error semantics, not as stored-expiry mutation.

Strikes are the target's existing shared counter; any successful render already clears it (#59), which is what snaps a recovered page back to the fast lane. Shared countedStrikes() helper (coerce-before-finite, BigInt-safe — review finding) now backs all three strike sites (suppress, recordRedirectStrike, retryAfterFailure).

Why

#59 fixed what happens to the target on these failures (nothing destructive) but left the when wrong: a flat +24 h retry meant a one-minute origin blip cost a full day of staleness. The old lease-retry model (leave the schedule alone) has the opposite failure: no backoff, so a permanently-500 page renders every 10 minutes forever. Two lanes give blips fast recovery and persistent failures a bounded cost.

Notes for review

  • retryAfterFailure replaces rescheduleAtTargetCadence at all three never-suppress call sites (auth, transient, redirect-landing guard).
  • Worst case per persistently-failing URL: ~2 fast retries then daily. Versus: 144/day hot loop (pure lease) or 24 h staleness per blip (flat cadence).
  • Per-device: only the failing device's schedule is touched, as elsewhere.
  • Tests: two-lane transitions, BigInt strikes coercion, slow-lane leaves page expiry untouched (sentinel-pinned); 338/338, lint + format clean.

🤖 Generated with Claude Code

… for blips, cadence + extended page expiry for persistent ones; v0.24.0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a two-lane retry mechanism (failureRetry) for non-suppressing HTTP failures (such as 401/403 and transient 408/429/5xx errors). The first few failures are retried quickly on the job's claim lease, while subsequent failures drop to the target's normal cadence to prevent hot-looping. The changes include updates to the configuration, RenderQueue scheduling logic, and corresponding test coverage. Feedback on the implementation highlights a critical bug in RenderQueue.js where Number.isFinite is called on renderTarget.strikes without prior coercion; if this value is a BigInt, the check will fail, preventing the strike count from incrementing past 1 and trapping the target in the fast retry lane.

await RenderSchedule.delete(cacheKey);
return;
}
const strikes = (Number.isFinite(renderTarget.strikes) ? Number(renderTarget.strikes) : 0) + 1;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The renderTarget.strikes column can be surfaced as a BigInt (e.g., if it is a Long column in the database). Since Number.isFinite returns false for BigInt values, this check will fail and default the strike count to 1 on every subsequent failure. This prevents the target from ever incrementing past 1 and transitioning to the slow lane, causing persistently failing pages to hot-loop renders on lease expiry.

Coerce the value to a Number before performing the Number.isFinite check to ensure robust handling of BigInt values.

Suggested change
const strikes = (Number.isFinite(renderTarget.strikes) ? Number(renderTarget.strikes) : 0) + 1;
const strikes = (Number.isFinite(Number(renderTarget.strikes)) ? Number(renderTarget.strikes) : 0) + 1;
References
  1. Coerce before the finite check: Long columns can surface the stored interval as a BigInt, which Number.isFinite rejects outright — without this, every such target would silently fall back to the default cadence.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4b767aa. strikes is an Int column (the documented BigInt surfacing is for Long), so this is unlikely to fire in practice — but the repo already treats coerce-before-finite as the standard guard (the resolveRenderInterval precedent you cited), and the consequence here (permanent fast-lane hot loop) is exactly the failure mode this PR exists to prevent, so it's worth being defensive.

Went one step further than the suggestion: the same vulnerable pattern existed at three sites (Target.suppress and recordRedirectStrike from #59, plus this one), so they now share a single countedStrikes() helper exported from Target.js — coerces before the finite check, and treats null/undefined/negative as 0. Added a test seeding strikes as a BigInt and asserting the slow-lane transition still happens. 339/339 passing.

(agent reply — Claude)

harper-joseph and others added 2 commits August 3, 2026 14:26
…, review) — shared countedStrikes helper across suppress/redirect/failure paths

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…taleness bound; past it, origin fallthrough is the honest answer

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@harper-joseph harper-joseph changed the title feat(plugin): two-lane retry for never-suppress failures — fast lease retries for blips, cadence + extended page expiry for persistent ones; v0.24.0 feat(plugin): two-lane retry for never-suppress failures — fast lease retries for blips, cadence backoff for persistent ones; v0.24.0 Aug 3, 2026
@harper-joseph
harper-joseph merged commit 03a0afb into main Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant