[harper-joseph]

Implements option A from #138, with the design questions there answered by measurement rather than assumption.

The gap

The probe compares the origin to its own previous answer. That bounds drift, but it is structurally blind to a value that changes and changes back between two passes — the signatures compare equal and nothing triggers. If a render landed inside that window, the cached page keeps the transient value until its render interval expires.

Measured on a production commerce deployment (n=143 cached product pages, 112 inside the serve window):

• price mismatches among served pages: 0/112 — drift detection is working.
• availability mismatches among served pages: 3/112 (2.7%), every one the same shape: the page reads OutOfStock, the origin says available, and the price matches exactly.

One traced end to end: the probe stored [39.99,35.99,35.99,true] at 11:04:37Z; the page rendered at 11:10:33Z — six minutes later — capturing OutOfStock; the origin now returns exactly the stored signature again. The next pass compares equal and triggers nothing.

Four competing explanations were tested and eliminated before writing any code: the endpoint is stable (5/5 identical), store-independent (6 store identifiers, including a nonsense one), the page agrees with it at the same instant (34/34 on price and availability), and the cached page carries the same SKU as the origin's default — so it is not variant selection either.

What this adds

A rule may declare a mapping from its own extract array to what the page renders:

pageCheck:
  enabled: true
  priceFrom: 2        # index into `extract`
  availableFrom: 3

• Render path records what each page claims (ProbeState.pageSignature) — one node-local write per render on matching URLs, no extra origin traffic. The claim comes only from the renderer, which reads it off its live DOM and posts it with the result (structuredOffers, browser >= 1.20.0 — companion PR feat(browser): post the page's structured Product offers with the result; v1.20.0 #140).

There is deliberately NO fallback to parsing the stored HTML. Recovering the offers here means a regex scan plus a JSON parse of a ~1MB commerce PDP on the hottest write path in this process, to reconstruct what the browser already had structured in front of it. So pageCheck is inert against a renderer older than 1.20.0 — and says so, hourly, rather than passing silently, because a config that looks enabled and protects nothing is the worst of the three outcomes. The wire contract makes that diagnosis honest: an absent structuredOffers means the renderer predates the feature (warn); null means a >=1.20.0 renderer ran the extraction and the page declared no Product offers (silent, nothing to record). Neither package blocks the other on deploy; pageCheck simply does nothing until both are in place.

• Probe pass compares the origin against that claim before the unchanged early-return, because "unchanged" is precisely the case this exists to catch.
• On a disagreement it triggers the normal revalidate, then clears the claim so the next pass does not re-detect the same one and re-spend the trigger budget on a page that is already expired and filed.

A dimension compares only when BOTH sides actually claim it. The one failure mode this feature must not have is a systematic false disagreement: it would hard-expire the page, the re-render would produce the same claim, and the next pass would expire it again — a permanent churn loop across every matched page, triggered fleet-wide by nothing more than a site-side markup change. So every reduction fails toward "no claim, incomparable" rather than toward a guess:

• Page availability reduces to true only for a recognized in-stock vocabulary (Google's own reading: InStock, InStoreOnly, OnlineOnly, LimitedAvailability), false only when every stated availability is an unmixed definitive negative (OutOfStock, SoldOut, Discontinued), and null for anything else — PreOrder, BackOrder, a private vocabulary. URL forms normalize defensively even though the renderer already strips them.
• Endpoint availability is a claim only for an unambiguous boolean (true/'true'/false/'false'); a count or status string is null, never a guess.
• Price compares only when the page printed at least one parseable price (an unreadable price format or an AggregateOffer reduces to "no price claim") and the endpoint projected one.
• Availability with a null on either side never disagrees; price still compares independently, so an unrecognized vocabulary costs only the availability protection, not the price protection.

Kept separate from the drift signature, not folded into it. A page can only ever express (price, availability), so making the stored signature mean "what the page says" would silently narrow drift detection and stop catching regularPrice / salePrice moves — the class that produced 19 measured price mismatches. That is why this costs a column.

ProbeState now has two writers, and the verbs are load-bearing. patch does not create a missing record (verified against the engine: "updated 0 of 1 records", key in skipped_hashes) and put replaces the whole row. So an existing row only ever takes a patch naming that writer's own columns — the probe patches signature/probedAt (plus an explicit pageSignature: null on an acted trip), the render path patches pageSignature — and the other writer's column survives structurally rather than by being copied through a read that may be a full probe-request stale. A missing row takes a put (patch cannot create; that verb mistake was caught in review and would have made every seed a silent no-op). Both verb choices are pinned by direct unit tests. The one remaining race — the render path creating a row between the probe's read and its seeding put — costs at most one baseline, re-seeded on the next pass, and is documented at the write.

Design answers from #138

1. Device types. Compared desktop vs mobile cached pages for 40 URLs on full (sku, price, availability) sets: 39/40 byte-identical. The one exception was two renders 41 hours apart, and the stale side had already expired and was unservable. Structured data is device-independent, so one row per URL is correct and last-writer-wins is safe.
2. Failure semantics. A page yielding no Product offers records nothing — the same rule a failed probe already follows, so a markup change cannot make every page look like a disagreement (and per above, neither can a vocabulary or price-format change).
3. Replace vs supplement. Supplement, per above.
4. Seeding. A page disagreement triggers even with no stored signature: the page provably disagrees with reality now, whatever the probe's history.

Deliberate limitation

A status-signal literal (e.g. sold-out expressed as a 4xx via statusSignals) carries no price/availability to project, so pageCheck skips those responses rather than guessing. That leaves the sold-out-page-says-in-stock case to the existing signature comparison, which does catch it as a change. Worth revisiting with an implies: clause on statusSignals if it proves material.

Verification

• 872/872 unit tests, 20 new. The pass-level tests pin the exact production case (origin matches its baseline, page disagrees → trigger, claim cleared), agreement staying unchanged, inertness with no stored claim, the status-signal skip, triggering without a baseline, seeding a claim-only row via patch, EVERY acted trip clearing the claim (a drift trip too — a preserved claim disagrees with the new baseline by construction and would re-trip an already-expired page each pass), and dry-run preserving it.
• Stats bucket by signature outcome alone (probed = seeded + unchanged + changed + failed holds; the canary's changed + unchanged denominator keeps page-mismatch rows) with probe_page_mismatch as an overlay — pinned by tests.
• Spec tests cover mapping compile/bounds/source rejection, the tri-state availability vocabulary (including mixed and unrecognized), both-sides-claimed gating on each dimension, the price-absent-from-page asymmetry, multi-variant pages not reading as disagreements, and corrupted stored claims never ending a sweep.
• recordPageClaim has direct coverage with a fake table pinning the verb choice, the absent-vs-null wire contract including the hourly warn throttle, the changeProbe.enabled master switch gating storage ("Off = … nothing stored"), rule selection matching the sweep's (first match of ALL rules — a "first match with pageCheck" search would let a pageCheck-less rule shadow the URL on the sweep side and silently never compare), and never throwing into the render path. writeSignature's verb/column choice has its own direct test.
• Two bugs caught by this PR's own tests during development: canonicalPrice(null) returned "0.00" (Number(null) === 0), and an intermediate writeSignature put→patch change that would have made every seed a silent no-op — patch cannot create.
• pageCheck.enabled that is truthy but not boolean true (a YAML "true" string) warns instead of silently disabling.
• Also fixed while here: a pre-existing millisecond-tick flake in the trip-expiry test (observed failing once during review).
• Lint and format clean. No repo CI on PRs — verified locally per repo policy.

For the human reviewer

• The mapping surface is the judgment call. Indices into extract are terse; named paths would be friendlier but would duplicate extraction config. Dropped whole on any out-of-bounds index, because a half-applied mapping silently compares the wrong column.
• A wrong-but-parseable priceFrom index is the one remaining mass-churn footgun this code cannot detect (e.g. mapping regularPrice when the page prints the sale price would disagree on every discounted product). The mitigation is operational: validate the mapping empirically against a sample of live pages before enabling, and watch probe_page_mismatch after.
• recordPageClaim is awaited on the render path. It is one node-local point read plus one write on a small table, wrapped so it can never fail the render.
• Removing the fallback exposed a real integration bug: the call site read result.offers while the renderer posts result.structuredOffers. With the fallback present, the renderer path would never have been taken and every claim would have come from the expensive path — silently. Evidence that the fallback was actively harmful rather than merely costly.
• pageCheck being inert on an old renderer is a deliberate failure mode. The hourly warning is the mitigation; if that is judged too quiet for something this load-bearing, a config warning at startup would be the alternative.
• In dry-run the claim is deliberately not cleared (nothing was expired, so the disagreement still stands): probe_page_mismatch reads as a standing gauge of disagreeing pages per pass in dry-run, and as a detection rate when armed.

Review coverage

Authored by Claude (Fable). Two self-review rounds plus a fresh-context Claude adversarial pass over both PRs' diffs with full repo access (partial mitigation for the missing outside model — it reproduced its blockers executably). Outside-model legs were attempted and are unavailable on this machine (Codex CLI: burned refresh token, needs codex logout && codex login; Gemini CLI: no API key configured), so model coverage is Claude-only and degraded, flagged per policy.

The adversarial pass raised 8 findings; 6 were real and are fixed in the two fix(plugin)/fix(browser) review commits (stats accounting, write-verb TOCTOU, rule-selection symmetry, master switch, the browser cap trio) plus the test flake. Declined with rationale: a per-URL repeat-trigger damper — the overflow-refusal and both-sides-claimed gating remove every identified systematic-churn class, the measured device-divergence rate is 0/39 servable pages, and maxTriggersPerSweep bounds anything residual; worth revisiting only if probe_page_mismatch shows the same URLs repeating.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a "PAGE CHECK" (pageCheck) feature to resolve "round-trip blindness" in the prerender plugin, allowing the system to detect when a cached page's content (such as price or availability) disagrees with the origin API even if the origin's overall signature remains unchanged. This is achieved by recording page claims during renders, comparing them against API claims during probe passes, and tracking mismatches with a new probe_page_mismatch metric. The feedback suggests two improvements: first, wrapping the array destructuring in claimsDisagree inside the try...catch block to prevent potential TypeError crashes on malformed database records; second, parsing the URL once outside the .find() callback in recordPageClaim using URL.parse to avoid redundant allocations on the hot render path.

[gemini-code-assist]

The claimsDisagree function parses pageClaim and apiClaim inside a try...catch block, but performs array destructuring outside of it. If the parsed JSON is not an array (e.g., if the database contains a corrupted string, a primitive, or an object), the destructuring will throw a TypeError (e.g., TypeError: page is not iterable), which will propagate and crash the entire probe sweep. Wrapping the entire destructuring and validation logic inside the try...catch block ensures robust error handling.

Suggested change

	export const claimsDisagree = (pageClaim, apiClaim) => {
	if (!pageClaim || !apiClaim) return false;
	let page, api;
	try {
	page = JSON.parse(pageClaim);
	api = JSON.parse(apiClaim);
	} catch {
	return false;
	}
	const [pagePrices, pageInStock] = page;
	const [apiPrices, apiInStock] = api;
	if (apiInStock !== pageInStock) return true;
	if (apiPrices.length && !apiPrices.every((p) => pagePrices.includes(p))) return true;
	return false;
	};
	export const claimsDisagree = (pageClaim, apiClaim) => {
	if (!pageClaim || !apiClaim) return false;
	try {
	const page = JSON.parse(pageClaim);
	const api = JSON.parse(apiClaim);
	if (!Array.isArray(page) || !Array.isArray(api)) return false;
	const [pagePrices, pageInStock] = page;
	const [apiPrices, apiInStock] = api;
	if (!Array.isArray(pagePrices) || !Array.isArray(apiPrices)) return false;
	if (apiInStock !== pageInStock) return true;
	if (apiPrices.length && !apiPrices.every((p) => pagePrices.includes(p))) return true;
	return false;
	} catch {
	return false;
	}
	};

[gemini-code-assist]

Constructing a new URL(url) inside the .find() callback is inefficient because it parses the URL repeatedly for every rule being evaluated. Since url is constant for the duration of recordPageClaim, parsing it once outside the .find() call avoids redundant object allocations and C++ boundary crossings on the hot render path. Additionally, prefer using URL.parse to maintain consistency with the repository's established idioms.

Suggested change

	const rule = probeRules().find((r) => r.pageCheck && r.pathPattern.test(new URL(url).pathname));
	const pathname = URL.parse(url)?.pathname;
	const rule = probeRules().find((r) => r.pageCheck && r.pathPattern.test(pathname));

References

1. Prefer using URL.parse for parsing URLs to maintain consistency with the repository's established idioms, especially when the effective runtime floor is Node 22+ and compatibility with older Node versions is not required.

[harper-joseph]

Good call on the extraction — taken, and it made the design better.

Pushed a second commit: recordPageClaim now prefers result.structuredOffers from the renderer and only falls back to parsing the stored HTML for results from renderers that don't send it. pageClaimFromOffers is split out of pageClaimOf so both paths share one normalization, keeping the comparison contract in the plugin.

Renderer side is #140 (browser v1.20.0): one page.evaluate against the settled DOM, sorted, capped at 200 triples, best-effort. That removes the ~1MB regex scan and JSON parse from this repo's hottest write path, which was the cost I'd flagged for a second opinion in the original description.

No deploy ordering constraint — the fallback means either package can go first and a mixed fleet works throughout. 859/859 plugin tests, 150/150 browser tests.

[harper-joseph]

Fallback removed entirely per review — agreed it's too expensive to keep even as a backstop.

pageCheck now takes the claim only from result.structuredOffers and is inert against a renderer older than 1.20.0, logged hourly rather than silently. pageClaimOf is gone with its last caller.

Removing it surfaced a real bug it had been masking: the call site read result.offers while the renderer posts result.structuredOffers. With the fallback in place the renderer path would never have been taken — every claim would have come from the expensive HTML parse, silently, and the integration would have looked fine. Fixed.

858–859/859 tests (one flaky test unrelated to this change — passes on rerun and in isolation).

[harper-joseph]

Both findings were real — fixed and pushed.

claimsDisagree destructuring outside the try/catch is the more serious one. A stored claim is data from a previous release or a damaged row, so it can be any JSON; destructuring a non-array throws a TypeError from inside the sweep's per-URL path, which would end the entire pass rather than skipping one URL. Shape-check now happens inside the try, and anything unrecognisable reads as "no comparable claim". Added a regression test covering seven corrupted shapes (null, a number, a string, an object, [], [null,null], [{},true]).

new URL() inside the find() predicate re-parsed the same URL once per rule on every render. Parsed once now, via URL.parse per repo idiom.

860/860 tests.

[harper-joseph]

Your question about where ProbeState rows come from caught a serious bug I introduced in this branch — thank you, this would have shipped.

patch does not create a missing record. Verified against the engine directly: an update on a missing primary key returns "updated 0 of 1 records" with the key in skipped_hashes, and no row appears.

Earlier in this branch I switched writeSignature from put to patch so it wouldn't clobber the new pageSignature field. But the seeding write is by definition to a URL with no row yet, so that would have made every first observation a silent no-op — the probe would re-seed forever and never detect a single change. That breaks the deployed feature, not just the new one, and no test caught it because the test harness stubs the table.

Fixed: back to put, carrying pageSignature through explicitly so a signature write cannot erase the claim the render path owns. The trigger's claim-clear folds into that same write, so the separate clearPageClaim port is gone. recordPageClaim has the same hazard in reverse (patch can't create, put would clobber the probe's fields), so it reads once and chooses.

Added a test pinning that a drift write carries the claim through. 861/861.

To answer the question directly: yes — ProbeState rows are PDP-only, from both paths. The sweep writes them for rule-matched URLs, and recordPageClaim returns early unless a pageCheck rule matches the path. With the rule's pathPattern being ^/product/prd-, catalog and homepage renders do nothing beyond one regex test.

[harper-joseph]

Second self-review round, plus a fresh-context adversarial pass over the final diff (outside-model CLIs are still dead on this machine — details in the body's coverage section). Three commits since the last review:

c681e96 — comparison hardening. The claim reductions could produce a confident wrong verdict: any availability vocabulary outside literal InStock (PreOrder, LimitedAvailability, a private API status string) or an unparseable price format reduced to a claim that disagrees with the other side on every pass — hard-expiring every matched page after every render, forever, and a site-side vocabulary change would have fired that fleet-wide. Availability is now tri-state on both sides (true / false / null-no-claim, Google's own in-stock set), price compares only when the page printed a parseable one, and a dimension compares only when both sides claim it. Also: null structuredOffers no longer takes the 'renderer too old' warn (only an absent field does), enabled: "true" warns instead of silently disabling, and recordPageClaim got direct verb-pinning tests.

9c0a442 — trips clear the claim unconditionally. A drift trip preserved the old page claim, which disagrees with the freshly written baseline by construction on any price move — re-tripping the same already-expired, already-filed page every pass until its re-render lands.

74cc652 — adversarial-pass findings. The probe's whole-row put could clobber a claim a render wrote mid-probe (now: patch naming only the probe's columns on existing rows); page-mismatch rows silently left the seeded/unchanged/changed buckets, distorting the canary's denominator and breaking the documented probed = invariant (now: buckets count by signature outcome, mismatch overlays); recordPageClaim selected rules differently from the sweep, so a shadowing pageCheck-less rule meant claims written but never compared (now: identical first-match selection); and the master switch now gates storage too.

872/872 tests. The body is rewritten to describe the current state.