[harper-joseph]

What

On a large catalog the crawlable URL space is far bigger than the corpus worth rendering. Facet and parameter combinations a bot invents by following links own no Target, are never scheduled, and therefore miss on every request — measured at 89.8% of requests to one deployment's listing route. Every one of those is an origin fetch for a document the origin had already served, minutes earlier, to a different crawler.

So the fetch is not the thing to optimise away; the repeat of it is. On a miss for a route that opts in, the body the crawler is already reading is teed, captured, and stored — and the next crawler asking for that URL is answered from storage. Marginal cost is one write: no render capacity, no second origin request, no scheduling.

Off by default (render.raw.enabled: false) and off for every route until it opts in with rawCache: true. Unset config is byte-identical to today.

render:
  raw:
    enabled: true
    maxBytes: 1048576
    expiry: midnight            # or a number of ms
    expiryTimezone: America/Chicago
    contentTypes: ['text/html']

ingress:
  routes:
    - { match: prefix, path: '/catalog/', rawCache: true, discoverTargets: false }

What it must never do

These are the invariants, and each one is pinned by a test:

• It only ever replaces an origin proxy. A stored document is read when PrerenderedPage held nothing at all — a true miss. stale and invalidated found a page row, which means a Target, which means a render is coming; for those the live origin is the better answer. Widening this to stale would trade a live document for one up to a day old on exactly the URLs that have a render scheduled.
• It is not a prerendered snapshot, and nothing downstream may blur the two. Raw serves carry their own cache status, are excluded from page_age (there is no render to be old relative to), and record no demand — a RawPage owns no Target, and feeding the ladder's Bloom filter the combinatorial facet space is exactly what saturates it, at which point it answers "visited" for everything.
• A bulk invalidation still reaches them. The epoch is read only when a stored row exists, and compared with the same !(a > b) shape so an unreadable timestamp fails closed. No verification lookup: a verification certifies a render's claims against the origin, and nothing rendered this.
• A personalized document is refused. sanitizeOriginResponseHeaders drops set-cookie, so fetchOriginResource now surfaces hadSetCookie — the only surviving evidence that the origin tried to personalize the response.

Its own table, not a PrerenderedPage row

Three measured reasons, the same ones Invalidation and PageVerification already give for their own databases:

• PrerenderedPage has no expiration, and this population is unbounded and long-tailed by construction — a facet nobody visits again would keep its bytes forever. @table(expiration:) makes the tail collect itself.
• It is not residency-pinned, so every row replicates to every node. This is a much larger key space than the render corpus.
• page_cache is the database whose transaction log has been measured pegging worker threads at 85–95% of CPU on the receive side of a copy. A new blob stream of this shape does not belong in it.

Cost on a raw miss is one extra point read. On a raw hit it is three: the RawPage.get, an epoch read (LKG-cached), and the bounded blob read. All only on requests that already missed the rendered cache and were headed to the origin anyway.

The encoding rule

Bytes are stored exactly as the origin sent them, with the origin's own content-encoding (gzip), because response.js re-encodes from the stored header at serve time. content-length is dropped on the way in: a re-encoded body has a different length, and a stored one would contradict the bytes actually written — the class of mismatch that makes a crawler read a truncated document under a 200.

Consequence worth stating: maxBytes therefore caps compressed bytes. It bounds memory and storage; it is not a test against a search engine's decompressed size limit. That is deliberately out of scope here.

expiry: midnight

Not a convenience spelling of 24h. It exists for origins whose content steps at a fixed hour rather than drifting: there an interval is the wrong shape — a document fetched at 23:00 with a 6h TTL serves post-change content for five hours, while one fetched at 01:00 expires long before anything about it has changed. There is no herd to spread, because raw pages refill on demand at the rate bots actually arrive.

On tee()

A tee() buffers for whichever branch is slower. The capture reads in a tight loop, so it is always the faster branch and can never hold the crawler's response back — but that is the smaller half of the story, and an adversarial review measured the rest.

Retention is ~2x maxBytes per capture, not 1x: one copy in chunks, plus one the tee holds for the branch the crawler has not read. Measured with a stalled reader, a 768 KB document held 1.5 MB.

And capturing removes origin→crawler backpressure. Without it a slow client costs socket buffers — the reader stops, the TCP window closes, the origin holds the data. The capture drains the origin at full speed however slowly the client reads, so this worker's heap becomes the buffer. On a route that is ~90% misses that was a heap lever the client controlled.

So the number of simultaneous captures is capped (maxConcurrentCaptures, default 16); past it a response is served without being captured. Worst case is ~2 x maxBytes x maxConcurrentCaptures per worker. Degrading to "this one is not stored" costs nothing — the next request stores it.

Cancelling one branch does not cancel the source, so abandoning an oversize capture leaves the response untouched — but the cancel is not awaited, because a tee branch's cancel() returns the shared cancel promise and never settles when the crawler's branch goes undrained (disconnect, HEAD, local 304). The store itself is detached, never awaited on the response path.

Fixed after adversarial review

• A bulk invalidation was defeated by a conditional request. A refused raw page left cacheStatus at 'miss', and both 304 defences key on that value — so a crawler replaying the stored etag got a 304 from the origin and kept the pre-invalidation bytes, while every signal reported success. Now reports invalidated.
• A zero-length 200 was stored and replayed — Buffer.concat([], 0) is truthy.
• expiry: <ms> was unreachable: the config layer type-checks against a string default, so a numeric value was rejected and the default silently kept. Split into expiry: midnight|interval + expiryMs.
• An unreadable raw blob was silent while costing the full read budget per request; now serve_error raw-blob-timeout/raw-blob-unreadable.
• An authorized missHeader: origin no longer gets a stored document.
• raw is declared in CACHE_STATUSES/SERVE_SOURCES and added to the console's CACHE_SERVED — omitting it there reads as a smaller hit rate, not as missing.
• x-harper-raw dropped (no reader, and it shipped to every crawler ungated).

Tests

test/rawCache.test.js (25) plus two route-compiler cases. 1132 pass, lint and format clean.

The ones worth reading: an oversize document is still delivered in full while its capture is abandoned; a truncated origin body yields no capture rather than a partial document; a garbage expiry falls back to the boundary and never to NaN (which would store everything and serve none of it); and content-length is dropped while content-encoding survives.

Not in this PR

Decompressed-size post-processing (the search-engine size limit). That needs a decode and belongs in the browser package, where a parser dependency is cheap.

Relates to #180 — this implements the raw-document half of it; the probe-gating half is already achievable with a route renderInterval plus the existing probe trigger.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a raw-document caching feature to cache origin documents for URLs outside the render rotation, reducing repeat origin fetches for long-tail crawlable spaces. This includes adding configuration options (rawCache and render.raw), database schemas (RawPage in raw_cache), metrics tracking, and integration into the bot request handler. Feedback on the changes suggests adding a truthiness check on stored.lastCached before passing it to epochMsOf to prevent unexpected date coercion bugs, and appending a .catch() block to the floating promise chain in captureForRawCache to safely handle any unhandled promise rejections.

[gemini-code-assist]

Perform a truthiness check on stored.lastCached before passing it to epochMsOf. Since lastCached is a nullable Date field in the RawPage schema, a missing or null value could evaluate to 0 (epoch 0) or cause unexpected behavior, potentially bypassing invalidation checks when no raw epoch is active.

Suggested change

	const servable = stored && !(rawEpoch && !(epochMsOf(stored.lastCached) > rawEpoch.at));
	const servable = stored && stored.lastCached && !(rawEpoch && !(epochMsOf(stored.lastCached) > rawEpoch.at));

References

1. When parsing or coercing date values in JavaScript, always perform a truthiness check first (e.g., ensuring the value is not null or undefined) before passing it to new Date(). This avoids bugs where new Date(null).getTime() evaluates to 0 (epoch 0) instead of NaN, which can corrupt metrics or calculations expecting a missing date to result in NaN.

[harper-joseph]

Declining this one — the premise does not hold here, and the suggested change would narrow the behaviour.

epochMsOf already returns NaN for a missing value; that is the entire reason it exists rather than a bare new Date(). From its own docstring:

The empty check is not decoration: new Date(null).getTime() is 0, not NaN, so a missing column would otherwise read as 1970 — an infinitely old timestamp — rather than as absent.

Verified against the real implementation:

lastCached	epochMsOf	servable vs an active epoch?
null / undefined / ''	NaN	no
'garbage'	NaN	no
0	0	no

Every unreadable value already fails closed, which is the direction that matters — the comparison is !(a > b) precisely so NaN refuses rather than serves. It is the same shape pageFreshness.js uses on the rendered-page path.

The suggestion would also change behaviour in a way I do not want: adding stored.lastCached && makes a row with no lastCached unservable even when no invalidation is active. For a raw page lastCached is not the freshness field — expiresAt is, and readRawPage has already enforced it before this line. So that row is perfectly servable absent an epoch, and refusing it would be a silent, unnecessary miss.

The second note on this PR (the floating promise) was right and is fixed in 2d16d09 — the catch now logs with e?.message ?? String(e) rather than swallowing.

🤖 Generated with Claude Code

[gemini-code-assist]

Add a .catch() block to the floating promise chain to handle any potential synchronous or asynchronous rejections thrown during the capture or storage process (e.g., if metrics.rawCache or storeRawPage throws). This prevents unhandled promise rejections in Node.js, and ensures we safely handle non-standard exceptions using e?.message ?? String(e).

Suggested change

	captured.then((result) => {
	if (!result.bytes) {
	metrics.rawCache(result.outcome);
	return;
	}
	return storeRawPage({ cacheKey, resource, bytes: result.bytes, policy });
	});
	captured.then((result) => {
	if (!result.bytes) {
	metrics.rawCache(result.outcome);
	return;
	}
	return storeRawPage({ cacheKey, resource, bytes: result.bytes, policy });
	}).catch((e) => {
	logger.error?.("[prerender] raw cache capture failed: " + (e?.message ?? String(e)));
	});

References

1. Ensure that error handling in critical boot-path operations (such as database subscriptions during startup) safely handles non-standard exceptions using e?.message ?? String(e). A TypeError escaping a catch block on the boot path can cause the entire component or application to fail to load, whereas a safely handled error allows the system to fall back to a degraded but functional state (e.g., falling back to a poll instead of a subscription).

[harper-joseph]

Intended kohls rollout, decided 2026-09-18: catalog routes only — rawCache stays OFF on /product/prd-.

Recording it here because the config PR can't be written until this merges and the release is cut, and the reasoning is worth keeping with the feature.

A raw-cached PDP is an unprobed Merchant Center surface: it carries prices, nothing renders it, and the change probe never sees it, so its TTL would be the only bound on price staleness. Catalog is the opposite case — the origin's SSR document already carries the whole SEO surface (title, canonical, the full CollectionPage → ItemList → 15 Product/Offer JSON-LD with prices, the 120-product grid), and 89.8% of listing-route requests are misses today.

We considered the more elaborate option — blanket-invalidate non-sitemap PDPs and reactivate them on probe evidence, i.e. the PageVerification pattern — and it collapses to "don't cache them":

• Invalidation scopes are all or one compiled route. There is no membership axis, and the schema deliberately refuses free-text scopes because an unvalidatable scope reads as applied and matches nothing.
• The probe sweep walks the Target registry. Under discoverTargets: false these URLs have no Target by definition, so they are never probed, no baseline is ever stored, and the reactivating evidence can never arrive.
• An invalidated raw doc falls back to the origin proxy — which is exactly the pre-raw-cache state.

So invalidate-by-default plus evidence-that-cannot-arrive produces the same serving behaviour as not caching, at the cost of a new scope axis and a probe extension.

If PDP raw caching is ever wanted, the lever is a short TTL (expiry: <ms>, already supported), not invalidate-plus-verify.

🤖 Generated with Claude Code