Repository navigation
Change probe is blind to a value that changes and changes back: compare the origin against the rendered page, not against the previous probe #138
Description
The gap
The change probe compares the origin's answer to its own previous answer. That bounds drift — a value changing after a render — but it is structurally blind to a value that changes and changes back between two probes. If a render lands inside that window, the cached page keeps the transient state and the probe never notices, because the signature it reads still equals the one it stored.
Measured on a production commerce deployment (n=143 sampled cached product pages, 112 of them inside the serve window):
- price mismatches among served pages: 0/112 (0.0%) — the probe is doing its job on drift.
- availability mismatches among served pages: 3/112 (2.7%), every one of them the same shape: the cached page says
OutOfStock, the origin says available, and the price matches exactly.
One case traced end to end: the probe stored [39.99,35.99,35.99,true] (available) at 11:04:37Z; the page rendered at 11:10:33Z — six minutes later — capturing OutOfStock; the origin now returns exactly the stored signature again. The probe's next pass will compare equal and trigger nothing, so the page serves a wrong availability until its render interval expires (48h in that deployment).
Four alternative explanations were tested and eliminated:
| hypothesis | test | result |
|---|---|---|
| endpoint is flaky | 5 consecutive calls | byte-identical signatures |
| answer varies per store | 6 store identifiers incl. a nonsense one | identical for every probed field |
| render captured a different variant | compared SKUs | cached page carries the same SKU as the origin's default |
| systematic render defect | control pages | controls render the correct availability |
At ~180k product-page serves/hour on that deployment, 2.7% is ~5k serves/hour carrying a stale availability. A shorter sweepInterval shrinks the window a round trip must fit inside but cannot close the class, and the exposure scales with render interval — so it gets worse exactly when you stretch intervals on the strength of the probe.
Proposal: compare the origin against what the PAGE says, not against the last probe
Make the stored signature mean "what the cached page currently claims" rather than "what the origin said last time". Then the existing comparison — origin-now vs stored — becomes origin-now vs page, which is the question that actually matters for structured-data correctness, and any disagreement triggers regardless of history.
Mechanism, reusing what already exists:
- On render completion (
RenderQueue.processJobResult), when a probe rule matches the URL and the new option is enabled, extract the rendered page's own structured-data values with the document-mode extractor already inchangeProbeSpec(extractJsonLdOffers), normalize, and write that as the URL'sProbeStatesignature. - The probe pass is then unchanged. It fetches the origin, compares to the stored signature, and triggers on difference — which now means "the page disagrees with reality".
This needs no schema change and no new storage: ProbeState already holds exactly one signature per URL, and the render result is processed on the owner node (claims are owner-scoped), which is the same node the owner-scoped sweep uses.
The one genuinely new piece: field mapping
The request-mode signature is an ordered extract array (e.g. [regularPrice, salePrice, lowestApplicablePrice, isAvailableForShip]); the page's structured data yields (price, availability) per offer. They are not directly comparable, and the correspondence is site-specific — so the rule must declare it:
pageCheck:
enabled: true
priceFrom: 2 # index into `extract` holding the price the page renders
availableFrom: 3 # index into `extract` holding the availability verdictComparison: mismatch when the origin's price is absent from the page's offer prices, or when the origin's availability disagrees with whether any offer on the page is in stock (a page whose every SKU is out of stock while the origin says available is a mismatch — that is one of the three measured cases).
Cost
One structured-data extraction per render on pages matching a pageCheck rule — a bounded regex scan plus a JSON parse, order milliseconds against a render that already stores a ~1MB blob. Nothing added to the serve path, and nothing added to the probe path.
Design questions worth settling before implementation
- Device types. A URL renders per device and each render could capture different content, but
ProbeStateis one row per URL. Last-writer-wins, or only trust one device, or store per device? - Failure semantics. If the page yields no structured data (extraction returns null), it must leave the stored signature untouched rather than write a null — same rule the probe already applies to failed probes, so a markup change cannot mass-trigger.
- Does this replace or supplement drift detection? Making the signature mean "what the page says" is strictly better for correctness, but it loses the ability to report a pure change rate from probe-to-probe deltas. If that metric matters, it needs its own counter.
- Interaction with seeding. A first observation currently seeds without triggering. Under this design a freshly rendered page always has a signature, so seeding should become rarer — worth confirming it does not mask a real disagreement on the first pass after a render.
Happy to implement once the shape is agreed — particularly the mapping surface in (the pageCheck block) and the device-type answer in (1).
Activity
harper-joseph commentedon Aug 25, 2026
Measured answers to the design questions, plus one fork the measurement exposed that changes the cost of the proposal.
(1) Device types — ANSWERED: structured data is device-independent
Compared the
desktopandmobilecached pages of the same URL for 40 URLs that had both, on full(sku, price, availability)offer sets:2026-08-23T18:12, mobile2026-08-25T11:42) and the price moved between them. The stale side had already expired (2026-08-24T18:12) and was therefore not servable.So last-writer-wins is correct and safe: both devices produce the same claim given the same moment, and a device page that lags far enough to hold a different claim is by definition past its expiry and not being served. One row per URL in
ProbeStateis the right shape; no per-device storage needed.(2) Failure semantics — leave the signature untouched
If extraction yields nothing, write nothing — mirroring the existing rule that a failed probe changes no state. That keeps a markup change from mass-triggering, which is the same reasoning already documented for all-null extraction.
(4) Seeding — no masking
A freshly rendered page always writes a signature, so the pass after a render compares instead of seeding. That is the desired behaviour and cannot hide a disagreement.
(3) Replace vs supplement — this is the real fork, and it costs a column
I understated this in the issue. The two signatures are not the same shape, and that matters more than I credited:
extractarray (e.g.[regularPrice, salePrice, lowestApplicablePrice, isAvailableForShip]);(price, availability)only.So a page-derived signature can only ever be compared on the projected
(price, available)pair. Making the stored signature mean "what the page says" therefore silently narrows drift detection: changes toregularPrice/salePrice— the strikethrough pair a shopper sees — would stop triggering, because they are not represented in the page-shaped signature. That is a real regression in exchange for closing round-trip blindness, and it is not a trade worth making silently.Two honest options:
A. Supplement (recommended). Keep
signatureas the drift baseline and add a nullablepageSignaturecolumn toProbeState, written at render completion. The pass then makes two comparisons:api-now vs signature(drift, unchanged) andproject(api-now) vs pageSignature(page disagreement, new). Costs one nullable column on a node-local, non-replicated table — the cheapest schema change available here — and loses nothing.B. Replace. No schema change, but drift detection narrows to
(price, available)as described. Only acceptable if the deployment genuinely does not care about regular/sale price changes, which this one does — an earlier audit found 19 price mismatches driven by exactly those markdowns.I would build A. It needs a nullable-column addition to
ProbeState, which is why I am confirming before writing it rather than after.harper-joseph commentedon Aug 25, 2026
Implemented as option A in #139 (v0.58.0, default off).
All four design questions are answered in the PR — the load-bearing one empirically: desktop vs mobile structured data is 39/40 byte-identical across 40 URLs, and the single exception was two renders 41 hours apart with the stale side already expired, so one row per URL is correct and last-writer-wins is safe.
859/859 tests, 9 new, including the exact traced production case (origin matches its own baseline, page disagrees, trigger fires, claim cleared so it does not re-detect forever).
harper-joseph commentedon Sep 16, 2026
Shipped. Closing.
#139 (
prerender-v0.58.0) implemented option A from the comment above — supplement, not replace:ProbeState.pageSignature+pageClaimAt(nullable, node-local, non-replicated), written at render completion from the page's own structured data.origin-now vs signature(drift, unchanged, still the full orderedextractarray) andproject(origin-now) vs pageSignature(page disagreement, new). Nothing narrowed.pageCheck: { enabled, priceFrom, availableFrom }per rule carries the site-specific field mapping.The traced production case in this issue — origin matches its own baseline, page disagrees, trigger fires, claim cleared so it cannot re-detect forever — is pinned as a test.
Since then: console v0.10.0 (#142) reads the page-mismatch signal, and v0.65.0 (#159) widened
pageCheckto availability words (In Stock,IN_STOCK, schema.org forms) with per-ruleavailableValues/unavailableValues, plus[*]per-variant lists.