[harper-joseph]

Why

A customer deployment had a two-hour origin outage: the origin answered most pass-through fetches with a 504, slowly. Cache serving held, and no failed render was stored, so correctness was fine. Our own traffic did not back off, though:

• render claims kept flowing, and every page that came due took failure strikes it had not earned, pushing its retry out to a full cadence or longer after the origin was back;
• every crawler miss was held for as long as the origin took to give up (the plugin's undici Agent waits up to 300 s for headers).

Design: each client backs off on its own requests

There is no node-wide breaker. The outage was narrower than the origin: one kind of page failed while product pages answered normally. A breaker fed by crawler misses (origin_fetch) would have tripped hard and stopped the change probe and the renders that were still succeeding, giving up price-change detection for a small cut in our own load. So every source of origin traffic judges the errors of its own requests:

Origin traffic	Signal	Response
Renders (each render worker)	its own navigations: threw, or 408/429/5xx	halve render slots and claim size per failing window; add a quarter back per healthy one (new)
Render failures (the queue)	failure share of a route's results on this node	a failure on a broadly failing route counts no strike and retries after a flat 30 min (new)
Crawler misses	none: a crawler's request is not ours to drop	an optional deadline on the wait for the origin's headers (new, off by default)
Change probe	429/502/503/504/timeouts on its own calls	unchanged: already halves its rate and ends a pass after abortAfterDistress (50) distressed calls

Changes

Plugin v0.109.0

origin.passthroughDeadline (timeout 0 = off, status 504 | 503, retryAfter 2 min). A crawler request that goes to the origin waits at most timeout for response headers, then gets status, empty and cache-control: no-store; a 503 carries Retry-After. The clock runs from the start of the fetch (connecting included) to the headers, through an AbortController cleared on headers, so a body already streaming is never cut. The bot handler writes one summary warning a minute instead of an error per request.

Off by default, and the status is configurable: a CDN that fails over to the origin on Harper's 5xx makes the crawler wait the deadline plus the origin's own time, and the right status depends on what the CDN does with each. Health line: originDeadline, node and per route, a subset of originUnreachable (never own5xx).

render.failureRetry.broadFailure (on; window 10 min, minResults 20, failureShare 0.5, retryAfter 30 min, maxDuration 6 h). New util/broadFailure.js keeps a per-route ring of one-minute buckets in node shared memory (getSab), so every worker's results land in one window. While a route is broad, a failed render on it:

• counts no strike (strikes already counted stay; any success still clears them);
• is filed min(retryAfter, cadence) out with its change mark kept, and releases its lease, unlike the fast lane, which would hold a lease per failing page for the whole outage;
• a one-device row beside the rotation keeps its lease instead (no row of its own to back off; dropping it would lose a render someone asked for).

Judged per route because outages often hit one kind of page: judged node-wide, a failing route that is a minority of renders never crosses the threshold. maxDuration caps a spell because pages retried every 30 min come back far more often than healthy ones on their cadence, so pages broken for good could otherwise hold a route's share up by themselves. Past the cap, strikes count again until the share falls.

Observability: a warning when a spell starts and when it is capped, a notice when it ends; health line failures section (window results/failed, broad, capped, per-route counts); queue_health series broad_failure_retry.

Browser v1.43.0

originBackoff (on; windowMs 60 s, minNavigations 10, failureShare 0.5, recoverStep 0.25). New OriginBackoff.ts, one per process, read by the worker's slot wait and the consumer's claim size. A failed navigation is one page.goto threw for, or whose document arrived 408, 429 or 5xx; a recorded status decides before any error (the renderer aborts its own signal on a 4xx document). Per closed window:

• at least minNavigations and at least failureShare failed: halve the share of concurrency in use, never below one slot (the trickle that notices recovery);
• a thinner sample still failing at that share: hold (a worker on one slot sees few navigations);
• otherwise add back recoverStep; an empty window counts as healthy.

awaitSlot now loops, since one render finishing need not bring the in-flight count under a cut-down limit. Stats line: originBackoff (answered, failed, share, slots). Invalid settings throw at startup.

Deliberately not changed

• No "serve stale on origin error" mode. Serving past expiresAt + page.swrTtl stays refused (bot/user parity, see failureRetry); whether to offer it is a customer decision still open.
• Crawler pass-through load. A crawler's request is proxied; with a CDN that fails over on our 5xx, nothing on our side removes that load. The deadline only bounds how long each one is held here.

Rollout

• The two packages release independently (prerender-v0.109.0, v1.43.0); neither needs the other.
• broadFailure and originBackoff are on by default; passthroughDeadline is off until the CDN question is settled per deployment.
• The health line gains fields, which changes its shared-buffer layout fingerprint: a new buffer on the first start, as with any field change.

Tests

• Plugin: test/botServePath.test.js (real sockets: 504 at the deadline with the origin request torn down; 503 with Retry-After rounded up; a body streaming past the deadline arrives whole; off by default waits as before; health filing); test/broadFailure.test.js (threshold needs both counts; routes judged apart; window forgets; ring reuse; maxDuration cap and reset; health section; off); test/renderQueueVariants.test.js (no strike, lease released, retryAfter due time; no escalation past fastRetries; change mark kept; one-device row keeps its lease; under the threshold strikes as before).
• Browser: test/originBackoff.test.ts (verdicts; halve/hold/recover curve on a fake clock; idle recovery; off; settings validation; a real worker.run whose peak concurrency drops from 4 to 2 to 1 and ramps back).
• Results: plugin 2,130 of 2,132; browser 453 of 453; console 527 of 529.
• Known failures, not from this change:
  • missCause.test.js waits a fixed 5 ms for detached work and fails intermittently under full-suite load. Run alone, 36 times in parallel batches of 12: main failed 2, this branch 1.
  • Console: client → proxy → plugin (the sampling admin route is not exposed) and a probe-view assertion that expects 05:05 UTC, broken by the daylight-saving change. Both fail identically on main at af55ca9.
• The console's queue_health contract test waives broad_failure_retry with a reason; a Queue-view tile is a follow-up.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces robust origin backoff and broad render failure handling mechanisms to gracefully manage origin outages. In the browser package, an OriginBackoff class is added to scale down render slots and job claim limits when a worker detects high navigation failure rates. In the plugin package, a broadFailure utility is implemented to track failures per route using shared memory, allowing failed renders during broad outages to retry without counting strikes. Additionally, a passthroughDeadline option is introduced to bound crawler wait times. The review feedback suggests improving the robustness of elapsed time comparisons against potential NaN values in both broadFailure.js and OriginBackoff.ts by using the !(elapsed < interval) pattern.

[gemini-code-assist]

When performing elapsed time comparisons where a timestamp might be null, undefined, or unparseable (resulting in NaN), use the pattern !(elapsed < interval) instead of elapsed >= interval to ensure NaN values fail the inequality and correctly fall through to the fallback/unconditional branch (failing closed).

Suggested change

	if ((minute - brokeAt) * MINUTE_MS >= maxDuration) {
	if (!((minute - brokeAt) * MINUTE_MS < maxDuration)) {

References

1. When performing elapsed time comparisons where a timestamp might be null, undefined, or unparseable (resulting in NaN), use the pattern !(elapsed < interval) instead of elapsed >= interval to ensure NaN values fail the inequality and correctly fall through to the fallback/unconditional branch (failing closed).

[harper-joseph]

Adopted in d696582: the cap check is now !((minute - brokeAt) * MINUTE_MS < maxDuration), so a reading that is not a number caps the spell (strikes count, the pre-existing behavior) instead of excusing failures on the route for good.

🤖 Addressed by Claude Code

[gemini-code-assist]

When performing elapsed time comparisons where a timestamp might be null, undefined, or unparseable (resulting in NaN), use the pattern !(elapsed < interval) instead of elapsed >= interval to ensure NaN values fail the inequality and correctly fall through to the fallback/unconditional branch (failing closed).

		if (!(elapsed < windowMs)) {

References

1. When performing elapsed time comparisons where a timestamp might be null, undefined, or unparseable (resulting in NaN), use the pattern !(elapsed < interval) instead of elapsed >= interval to ensure NaN values fail the inequality and correctly fall through to the fallback/unconditional branch (failing closed).

[harper-joseph]

Changed in d696582, differently: if (elapsed < windowMs) return; already let NaN fall through, and falling through was the problem (windowStart became NaN for good). A non-finite elapsed now restarts the window at the current reading, and a new test covers it.

🤖 Addressed by Claude Code

[harper-joseph]

Review fixes pushed in d696582:

• broadFailure.js: the maxDuration check now fails toward capped on a reading that is not a number (strikes count, the pre-existing behavior).
• OriginBackoff.ts: a non-finite elapsed restarts the window at the current reading instead of leaving windowStart NaN; new test in test/originBackoff.test.ts.

Both inline threads have a reply; left open for the reviewer to resolve.

🤖 Addressed by Claude Code