[harper-joseph]

Stacked on #206 (plugin 0.86.0, not yet merged). The base is feat/probe-append-extract; retarget to main after #206 merges. It consumes the renderer contract from #208 (browser 1.37.0, pageFacts).

TL;DR

• The render path now stores a page record: what each cached page claims for every field it visibly shows (canonical, title, meta description, h1, product name/brand/image/rating/offers, breadcrumbs). It comes from the renderer's pageFacts and is written as ProbeState.pageFacts in the same patch as the existing claim.
• A rule maps extract slots to page facts (pageCheck.fields) and chooses a comparator for each. The probe then:
  • re-renders a page that disagrees with the origin on any mapped field, even when the origin has not changed;
  • does not re-render a page that already caught up with an origin change (counted as caughtUp);
  • ignores changes confined to pageCheck.ignoreChanges slots, the fields a page cannot show (counted as ignored). The canary does not count those as changes either.
• A mapping-defect guard disarms a mapped field that disagrees with most witnessed pages, so one wrong mapping cannot re-render a whole corpus on every pass.
• Pass records split changed by slot (slotChanges) and page mismatches by field (fieldMismatch). No new metric series, so no console release is needed.
• Nothing changes until a rule maps fields or sets ignoreChanges. Rules without them behave exactly as before, byte for byte. Adding a mapping re-baselines nothing, because mappings are not part of the rule fingerprint.

Problem

The probe compares each observation with the last origin observation. That answers "did the origin change since I last looked?", which is not the same question as "is the cached page wrong?". It fails in two ways:

• It re-renders pages that are already right. A cadence render that lands after an origin change already shows the new value, but the probe sees the signature move and re-renders the page again.
• It re-renders on fields no page shows. An endpoint returns fields the page cannot display (an inventory counter, say). The rule has to watch them because it watches the whole endpoint response, and every change to them costs a render.

pageCheck (priceFrom/availableFrom) already compares price and availability against what the page claims. This PR extends that comparison to every field the page visibly shows.

Design

1. The record (render path)

recordPageClaim(url, structuredOffers, cachedAt, { pageFacts, complete }) in util/changeProbe.js, called from processJobResult:

• When it is written. Only for a rule that maps fields. It reuses the claim's read and patch, so it adds no extra read and no extra write. It sits behind the same master switch as the claim.

• Which variant it comes from. The same way the claim chooses: the first stored variant that ran the extraction.

• Canonical JSON. Fixed key order, and only the contract's keys and types, so equal facts store equal bytes.

• It always describes one render. A rule with fields writes the whole record on every render, nulls included. It stores null when:

  • pageFacts is absent (the renderer predates 1.37.0; a warning is logged at most hourly);
  • pageFacts is null (the extraction ran and found nothing);
  • the record is over 16 KB (refused, never truncated; see below);
  • the render did not store every default device (complete: false, a partial failure or a one-device render).

  Leaving an older render's record in place beside a newer pageClaimAt would compare a page that is no longer served.

• Why partial renders store null. The record is one per URL. If desktop re-rendered and mobile failed, the mobile page is older than the record would claim. A record that "caught up" with a change would then skip a mobile page that did not.

• Why 16 KB. The record is written on every render and read on every probe, in a node-local table with one row per probed URL. A typical product record is an estimated 1–3 KB (the renderer measures well under 2 KB; 50 SKU offers add about 2 KB). A record at the renderer's own cap of 200 offers fits (tested). The renderer's worst case is about 125 KB, which would make one row cost what fifty normal rows cost.

• Cleared on trigger. When a trigger hard-expires the page, pageFacts is cleared together with the claim (writeSignature clearClaim).

• Schema. New column ProbeState.pageFacts: String. ProbeState is @sealed, so the column loads with the component's workers on deploy, not on a config reload.

2. The mapping (pageCheck.fields, compiled in changeProbeSpec.js)

{ slot, fact, compare, ...options }. slot indexes extract. A bad entry is dropped with a warning; the rest of the rule is kept. An entry is refused at compile time if the comparator does not apply to the fact's kind (a priceSet against a title, for example).

priceFrom/availableFrom compile exactly as before. They become optional once fields or ignoreChanges is given. The vocabulary (availableValues/unavailableValues) serves both the pair and skus.

Every comparator fails toward no claim: null, absent, empty or unparseable on either side means "not compared". It is never counted as a disagreement, and never as evidence that a page caught up.

compare	facts	rule	measured quirk it encodes
text	title, metaDescription, h1, product.name/brand, canonical, image	exact after NFC, whitespace collapse and trim; no case folding, no HTML stripping	the endpoint's meta description can contain <br>/<li><a …>, and the page's <meta name=description> holds that raw string exactly (stripping would turn 100% agreement into disagreement)
path	canonical, product.image	URL pathname only; relative values resolve against the page URL	the image URL differed on every page, only by size query parameters
number	product.rating.value / .count	numeric equality, optional tolerance	the endpoint sends "4.0", the page 4
priceSet	product.offers	vs the set of page offer prices, 2-decimal: a single endpoint price must be among them; for a list, every price the page prints must still be in the endpoint's set (NOT equality); any null element means no claim	the endpoint's single lowest price is null exactly when the product is out of stock; its price set matched the page's offers 100%; a page's offers can stop at 50 while the endpoint lists every variant
names	breadcrumbs	endpoint list (strings, or objects with nameKey) vs the page list's tail	the page's trail starts with a home crumb the endpoint omits
skus	product.offers	tuples keyed by SKU, over the intersection only; availability verdicts (tri-state, rule vocabulary) and, when both sides state one, price	the page lists at most 50 SKU offers where the endpoint lists more

3. Tuple projection in extract paths

A trailing [*].{a,b.c} projects each element to [a, b.c]: dotted inner paths, no brackets, no nesting. It sorts like any [*] projection. For example, payload.products[0].variants[*].{sku,availability,price.current} keeps each SKU's state attached to that SKU. Two separate [*] projections are each sorted independently and lose the pairing. Malformed brace syntax drops the rule with a warning, because it would otherwise extract null forever. The path is part of the fingerprint.

4. Decisions in processOne (#206's comparable/extended flow unchanged)

situation	outcome
an armed mapped field disagrees (signature unchanged, changed, seeded, or an extended row)	counts as a page mismatch and triggers, exactly like a claim-pair disagreement
the origin changed; every changed slot is ignored	baseline written, nothing triggered; bucketed unchanged, so the canary's verdict does not count it; ignored overlays
the origin changed; every non-ignored changed slot is caught up	baseline written, nothing triggered; still changed for the canary (the rest of the corpus has not caught up); caughtUp overlays
any other change: an unmapped slot, a slot with no record, a status-signal literal	triggers, exactly as before

When a slot counts as caught up. Only on positive evidence:

• at least one armed mapped field on the slot compared;
• every armed field on it that compared agrees with the new value and not the old one. A change the comparator cannot see is not evidence: a crumb URL inside a names object, a query parameter under path, or a whitespace edit all leave the page agreeing with both values;
• for skus, every SKU whose tuple changed is proven individually. A changed SKU the page does not list is never caught up.

Verification (PageVerification) keeps its original gate and additionally requires that no armed mapped field disagrees. A rule with fields but no claim pair verifies only when an armed mapped field agreed. A row that caught up is verified on the next pass, not this one.

5. The mapping-defect guard (changeProbe.mappingGuard)

A wrong mapping disagrees on nearly every page. A correct one disagrees on a witnessed page only after a genuine round trip, which was measured at well under 1%. A witnessed page is one where the record was stamped after the baseline's probedAt and the origin has been unchanged since.

• What it counts. Per field: witnessed comparisons and witnessed disagreements. Both the sweep and the canary feed it, and it counts in dry run too.
• When it disarms. When a field's disagreement rate reaches threshold (default 0.2) over at least minWitnessed comparisons (default 200). At a 1% true rate you would expect about 2 disagreements in 200, and the threshold needs 40.
• What disarmed means. The field stops triggering and stops counting as evidence for caught-up and verification. It is still counted in fieldMismatch. A warning names the rule, the slot and its extract path. It stays disarmed until that field's mapping changes or the process restarts.
• The rate is recent. Both counts halve each time the sample reaches 10 × minWitnessed. Without that, a mapping that was right for a week and then broke (for example, the site changed its title template) would first have to outweigh the week of agreements.
• It never suppresses an individual disagreement. A single witnessed disagreement is exactly the round trip the page check exists to catch.

6. Stats

These are new fields on the pass record, returned by GET /prerender_admin/change-probe and printed in the pass log line: caughtUp, ignored, slotChanges ({ rule: { slot | "signal": n } }), fieldMismatch ({ rule: { "<slot>:<fact>": n } }), and fieldGuard ({ rule: { field: { witnessed, disagreed, armed } } }).

pageMismatch now also counts armed mapped mismatches. probed = seeded + rebaselined + unchanged + changed + failed still holds. The console's cluster merge sums only the counters it already knows, so the new fields are visible per node. Showing them in the console is a follow-up.

Example rule

changeProbe:
  rules:
    - label: pdp
      pathPattern: '^/product/prd-([^/]+)'
      source: request
      request:
        urlTemplate: 'https://api.example.com/products/$1'
        headers: { accept: application/json }
      extract:
        - payload.products[0].title                                                  # 0
        - payload.products[0].seoUrl                                                 # 1
        - payload.products[0].images[0].url                                          # 2
        - payload.products[0].variants[*].{sku,availability,price.current}           # 3
        - payload.products[0].breadcrumbs                                            # 4
        - payload.products[0].rating.average                                         # 5
        - payload.products[0].inventory.total                                        # 6
      pageCheck:
        enabled: true
        fields:
          - { slot: 0, fact: title, compare: text }
          - { slot: 0, fact: product.name, compare: text }
          - { slot: 1, fact: canonical, compare: path }
          - { slot: 2, fact: product.image, compare: path }
          - { slot: 3, fact: product.offers, compare: skus }   # tuple defaults to [sku, availability, price]
          - { slot: 4, fact: breadcrumbs, compare: names }      # objects: nameKey defaults to "name"
          - { slot: 5, fact: product.rating.value, compare: number }
        ignoreChanges: [6]

Adversarial review: where this could re-render a corpus or swallow a change

path	what stops it
a wrong mapping disagrees on every page	the guard disarms it after ~minWitnessed witnessed disagreements per node; before that, triggers are bounded by maxTriggersPerSweep as always
a mapping that broke mid-life (the site's template changed)	the recent-rate window: disarmed within a few thousand comparisons, not after outweighing the history
a render that stored only one device is taken as the whole URL's record	complete: false stores null, so no caught-up
a change the comparator cannot see (crumb URL, query parameter, skipped tuple position)	the page must disagree with the old value, so these trigger
a changed SKU the page does not list	per-SKU proof; not caught up
a stale record from an older render survives a newer render	records are always written whole (nulls included) for mapped rules, and cleared on trigger
an older renderer in the fleet	absent pageFacts stores null (warned hourly): no claim, no suppression
an ignored-slot edit across the whole site	not a change for the canary, so no bulk invalidation
a disarmed mapping hides an origin change	it cannot: disarming only stops mismatch triggers and caught-up evidence; signature changes trigger as always

Tests

• Plugin suite: 1282 pass, 0 fail (1228 on feat(plugin): appending extract paths to a probe rule keeps detecting on the existing slots; v0.86.0 #206, so 54 new tests).
• Console suite: 277 pass (not touched).
• npm run lint and npm run format:check exit 0.

New tests by file:

• changeProbeSpec.test.js:
  • tuple projection: sorting, null elements, nested [*], reorder vs. a moved price, extractPathProblem, malformed-rule drop, fingerprint;
  • field, pair and ignore compilation, including each bad-entry case dropping alone;
  • mappings are not in the fingerprint;
  • one test per comparator, each with its measured quirk;
  • fieldCaughtUp: an off-page SKU, a removed SKU, invisible changes, and a visible change plus an off-page change together;
  • changedSlots;
  • corrupted records never throw;
  • canonical serialization and the 16 KB refusal.
• changeProbe.test.js (runProbePass):
  • a mapped disagreement triggers;
  • agreement means nothing written;
  • a caught-up change does not trigger;
  • a change that has not caught up triggers;
  • an unmapped change triggers;
  • no record, or a record that cannot compare, triggers;
  • an off-page SKU is not caught up;
  • an ignored-only change does not trigger and does not trip the canary (with a control that does trip);
  • ignored plus caught-up, and ignored plus unmapped;
  • literal transitions;
  • feat(plugin): appending extract paths to a probe rule keeps detecting on the existing slots; v0.86.0 #206 extended rows;
  • guard: disarms a broken mapping (9 renders, then silent), never disarms on 2% round trips, counts only witnessed comparisons, a disarmed field gives no caught-up evidence, stays disarmed on reload and re-arms on a mapping change, the recent-rate window, dry run, the real warning text;
  • verification gating for pair-plus-fields and fields-only rules;
  • dry-run writes;
  • slotChanges on a rule with no pageCheck.
  • recordPageClaim: same-write storage, absent vs null pageFacts, oversize refusal, complete: false, fields-only and pair-only rules, and status output.
• renderQueueVariants.test.js: the record is stored with the pages' own lastCached; a partial render retires the record.

Revert checks

Each mutation was applied to the source, the three test files were run, and the source was restored:

reverted / mutated	fails
changeProbe.js restored to #206	33
changeProbeSpec.js restored to #206	all (import)
RenderQueue.js restored to #206	1
caught-up without the "must not agree with the old value" check	2
skus caught-up without the per-SKU proof	1
caught-up disabled	4
mapped disagreement does not trigger	7
ignoreChanges disabled / ignored bucketed as changed	2 / 1
witnessed ignores pageClaimAt > probedAt	1
guard never disarms / unbounded memory / disarmed still vouches	4 / 1 / 1
verification: fields-only proof always true / caught-up verified	1 / 1
record: partial render stores facts / oversize accepted / absent leaves old record	2 / 2 / 1
RenderQueue passes complete: true always	1
clearClaim leaves pageFacts	1
tuple projection disabled	3
names prefix instead of suffix / path compares the full URL / text strips HTML	2 / 15 / 1
priceSet drops nulls / skus union instead of intersection / number ignores strings	1 / 4 / 1

Rollout

1. Render fleet on @harperfast/prerender-browser ≥ 1.37.0 (#208). Older plugins ignore the new field.

2. Deploy plugin 0.88.0 with a worker restart. The ProbeState.pageFacts column is new on a @sealed table and loads with the workers, not on a config reload. This step is inert until a rule maps fields.

3. Add fields / ignoreChanges to the rule. No re-baseline. Records fill as pages render: mapped fields compare as no claim until each URL's next render stores one.

4. Watch:

   • fieldMismatch per field; a field whose count tracks the matched count is mis-mapped;
   • caughtUp / changed, the renders saved;
   • ignored;
   • fieldGuard, and any … is DISARMED warning.

   Do this in dry run first if you can: the guard counts and disarms in dry run too.

Open items and limits

• The guard only sees witnessed rows. Mismatches on rows whose record predates the baseline are not witnessed: a brand-new rule's first passes, baselines written in dry run, and rule-edit re-baselines. So a wrong mapping on a new rule can spend up to maxTriggersPerSweep per node for a pass or two before the guard has evidence. Adding mappings to an existing rule (the rollout above) is witnessed almost immediately.
• A burst of stale renders can disarm a correct price mapping. If renders systematically capture a stale origin document right after a mass change, witnessed disagreements can exceed 20% of the recent window and disarm a correct skus/priceSet mapping until restart. That never hides an origin change; it only stops correcting stale pages. Untested in production.
• The guard is per process. A pass started by hand on a worker other than 0 starts with an empty guard.
• Removals always trigger. A SKU removed from the endpoint is never caught up (the page agrees with the old list too). This is conservative; it matches today's behaviour.
• priceSet is a subset test, not equality (changed in review, 210f99a). Equality disagreed forever with a page whose offer list is truncated (a price only on an unlisted variant), re-rendering that product every pass where the guard could not see it. The subset test gives up one case — a new price on a listed variant while its old price survives on another — which is what skus covers.
• For rules that map fields, a render with no offers now clears the claim-pair claim. Before, it left the older render's claim in place, and it still does for rules without fields.
• Console. No panel yet for slotChanges/fieldMismatch/caughtUp/fieldGuard; they are per node in the status endpoint. The cluster merge (PROBE_COUNTERS) could sum caughtUp/ignored in a follow-up console release.
• Next version is taken. #209 (0.89.0) also edits the change probe; expect a rebase for whichever merges second.

Version: plugin 0.88.0 (0.85.0 is #205, 0.86.0 is #206, 0.87.0 is #207).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request upgrades the prerender plugin to version 0.88.0 and introduces a page record feature (pageCheck.fields) along with a mapping-defect guard (mappingGuard) to compare extracted page facts against cached page states, preventing unnecessary re-renders and disarming faulty mappings. It also adds support for trailing tuple projections ([*].{a,b.c}) to extract nested array values. Feedback on the changes points out a compatibility issue in changeProbeSpec.js where URL.parse is used; since this method is only supported in Node.js v22.1.0+, it should be replaced with new URL() inside a try...catch block to prevent runtime errors on older Node.js LTS versions.

[gemini-code-assist]

Avoid using URL.parse as it is only supported in Node.js v22.1.0+ and is not backported to Node.js v20. Using it will cause runtime errors on older Node.js LTS versions. Use new URL() wrapped in a try...catch block instead.

	let url;
	try {
		url = new URL(value.trim(), base ?? undefined);
	} catch {
		return null;
	}

References

1. Avoid using URL.parse if the target Node.js version is below v22.1.0, as it is not backported to Node 20. Use new URL() with appropriate error handling or pre-validation instead.

[harper-joseph]

Not changing: URL.parse is the repo idiom (it returns null instead of throwing on a malformed value — see recordPageClaim, ruleForUrl, buildProbeRequest), it exists in every Node line this plugin runs on (added in 22.1.0 and backported to 20.18.0; deployments run Node 24), and the same finding was dismissed on #127 for that reason.

[harper-joseph]

Self-review (author side) — one defect fixed in 210f99a, the rest checked.

• Fixed — priceSet equality could re-render a product forever. A page's structured data can list fewer offers than the endpoint has variants (measured: one origin stops at 50). Set equality then disagrees on every pass for any product with a price only on an unlisted variant; the re-render reproduces the truncation, and the guard can't see it because such products are a small minority of witnessed comparisons. Now a single endpoint price must be among the page's prices, and for a list every printed price must still be offered. Revert check: equality fails the new truncated-subset case. Body updated.
• Checked, no change: existing extract paths can't be refused by the new tuple syntax check (it only looks at paths containing braces); non-tuple extraction is byte-identical; ignored-only changes count as unchanged for the canary; caught-up needs the page to show the new value AND not the old one; the guard only learns from witnessed comparisons and never suppresses a single disagreement; acted trips clear the page record; partial renders store no record; verification still requires proof.
• Minor, not changed: every probe now selects pageFacts even for rules without fields (~1-3 KB per read on a node-local table). Worth a follow-up only if the read shows up.