[harper-joseph]

Merged after #240 and #241, which it builds on. Needs browser 1.40.0 (#239) for one of its observers.

On a site where every spelling of a product is one document, every fetch from the origin says where the product lives now. This PR makes every such fetch tell the entity registry, adopts a canonical no target holds from any of them, and makes the entity serve trust the newest observation. The result is more misses answered from a rendered page, and a re-slug handled within minutes instead of a day.

Why

Measured on one production deployment (2026-10-01 and 10-02, read-only):

• Products re-slug ~100 times a day. The product sitemap changes once a day and never lists an out-of-stock product, so it cannot be relied on to say where a product moved.
• One crawler made 10,993 product-page misses a day on spellings whose own target is suppressed as canonical-mismatch. #240's entity serve refused every one of them as has-target.
• 93% of the product documents the raw cache stored were a spelling other than their own canonical, always the same product. The origin answered every spelling with the same document (68 of 70 identical; the other 2 changed between fetches).

Before this PR, only the nightly probe and stored renders wrote the registry. A daytime re-slug left the old page "confirmed" until its next check (up to 12h on the measured deployment). Meanwhile the entity serve handed that page, which names the old slug, to other spellings: gap 1 in the review of #240/#241. The render that first saw the re-slug was a canonical-mismatch verdict, which carried no canonical back: gap 2.

For the human reviewer

• The registry now has four observers. probe (the nightly pass), render (a stored page's canonical, and the canonical a canonical verdict declares), check (a serve-time check, from the endpoint's slot or the origin document) and origin (a document proxied from the origin on an entityServe route).
  • Each is the origin's own answer for the product id, on a route that has declared every spelling one document. A crawler-invented spelling cannot make one invent a canonical.
  • The origin observer only runs on entityServe routes for exactly that reason.
  • Crawler traffic cannot create rows. An origin observation moves a row the registry already holds (from the probe or a render), and an entity with no row reads untracked.
  • No query strings. A canonical with one is foreign, and a URL with one is not tapped.
• Adoption from any observer. Same rules as #241: another URL of the same entity, no target holding it in rotation in any spelling, a canonical-verdict suppression reactivated, anything else left alone.
  • It is bounded by entities.adopt.maxPerHour per node across every worker thread, and by retryAfter per entity. It is dry run by default.
• The entity serve changes in four ways.
  • It answers spellings suppressed as a canonical verdict. Their own render found that the page names its canonical elsewhere, which is the case the serve exists for.
    • The verdict is the spelling's own newest word. So when it is newer than the candidate's last confirmation, the spelling is answered only if the verdict named that candidate. That is the new Target.suppressedCanonical, written from browser 1.40.0's declaredCanonical. Otherwise it falls through as moved. This does not depend on the registry.
    • A target in rotation, or one suppressed about the URL itself (a 404, a noindex), is still has-target.
  • moved: when the registry names another document, first heard after the candidate was last confirmed (rendered, or checked and agreed), the candidate predates a re-slug and is not served. An unreadable instant vetoes.
    • The registry stamps observations with when the origin was read. A page row carries only its store time, and a check only its write time. So a confirmation is taken as made CONFIRMATION_READ_SLACK_MS (2 min) earlier, and an observation in that gap vetoes.
    • The same observation that moved the registry also adopted the new canonical (filed due now), which serves once it renders.
  • A tie between two servable spellings goes to the one the registry names. Otherwise it is ambiguous, as before.
  • has-query: a spelling with a query string falls through. A route that keys a param says it can change the document.
• The origin tap is on the bot path, so it is built not to cost anything.
  • The proxied body is teed. The tap's branch reads only until the canonical (identity) or at most 128 KiB (compressed, prefix inflated), then is cancelled without being awaited.
  • At most 32 taps run at once per worker; past that a document goes by unread.
  • The crawler's branch is untouched, and nothing the response sends waits on the tap.
  • An unsent response (a local 304, a client gone) cancels its branch, and the source is cancelled once the tap has its prefix. So nothing is drained, and a stalled origin pins nothing.
  • It stops with ingress.entityServe.enabled, the one switch an operator stops the entity serve's bot-path work with.

What

• Browser (#239, 1.40.0, commit 8739270): a non-indexable render whose canonical names another URL posts declaredCanonical, resolved as the verdict resolved it, without its fragment. It is absent on every other result, so an older plugin never sees the key.
• util/entity.js:
  • observeRenderedCanonical(url, canonical, atMs) now resolves: it records and adopts.
  • canonical_adopt carries the observer as its context.
  • The header documents the four observers.
• resources/RenderQueue.js:
  • A stored render resolves pageFacts.canonical, falling back to its declaredCanonical. A sitemap target is stored even when its canonical moved, and its page-record extraction can fail.
  • A verdict passes declaredCanonical to Target.suppress, which records it as suppressedCanonical, and then resolves it, observed at the result time less the job's renders.
  • Both never reject.
• util/serveCheck.js:
  • The endpoint check reports the rule's mapped canonical slot, unless the mapping guard disarmed it. The document check reports the document's canonical.
  • runCheck resolves it, whatever the verdict, observed when the origin was asked. A throw costs the check nothing.
  • canonicalFieldOf is now exported from changeProbe.js.
• util/originCanonical.js (new): tapOriginCanonical and tapsOriginCanonical, wired into bot_request.js's miss proxy before the raw-cache capture. The tap applies only to a GET with a 200 HTML response, on an entityServe route, with the registry on.
• util/entityServe.js:
  • New outcomes has-query and moved.
  • The read projects suppressedReason, and a spelling suppressed as a canonical verdict proceeds.
  • Servable candidates are collected and the registry breaks a tie.
  • One Entity point read, made only once there is a page to serve and only with the registry on.
  • The veto, and prerender_ops / entity_serve_ms per evaluation.
• prerender_ops / served_wrong (new, 32c1ec1): every cached copy found to differ from the origin while it was being served, at the moment a detector found it.
  • The value is an upper bound on how long it was out: ms since it was last known right. That is its render, or the last check that agreed with that very copy; for a stored 404, its last confirmation.
  • Detail is the detector: check, check-raw, sweep, render-check, negative-recheck, negative-fetch. The canary is left out: its cohort repeats every half hour.
  • Context is what disagreed: the pageCheck field label, claim, or 404.
  • Cost: one numeric sample per detection, from timestamps the detector already holds. No per-serve work, and no extra reads.
  • Serves per wrong copy are not counted: that needs a per-key serve counter, which the plugin deliberately does not keep.
• Config descriptions, README.md, METRICS.md and the metrics catalog are updated.
• Console 0.26.0:
  • The Entity serve panel names moved and has-query and shows the cost per evaluation (median, ≈p95).
  • A new Entity registry panel on Traffic: re-slugs seen, by the observer that saw them first; new entities; adopted, or would-adopt plus capped in a dry run; not adopted, by reason.
  • The two metric-coverage waivers #241 added are removed.
  • A Served while wrong card on Traffic: per detector, the count and how long copies were out (median, ≈p95); and what was most often wrong.

Verification

• test/entityServe.test.js, end to end through handleBotRequest:

  • A spelling suppressed as canonical-mismatch or canonical-variant is served. One in rotation, or suppressed as noindex, http-gone or with no reason, is has-target.
  • moved: the registry's word after the render vetoes, and before it does not. A spelling of the page itself, in any decodeURI-equal form, does not veto. An unreadable instant vetoes. No row, or no canonical, does not.
  • Against a check-confirmed page, the later of the render and the check decides.
  • The tie-break: off, it is ambiguous; on, the named spelling is served; a registry naming neither is ambiguous.
  • The registry is read only when it is on, and only once there is a page.
  • has-query, and entity_serve_ms on every outcome.
  • A miss that reaches the origin writes the registry (origin) and adopts the canonical its document declares, while the crawler gets every byte the origin sent. There is no tap with the registry off, off an opted-in route, or for a HEAD.

• test/originCanonical.test.js (new, 7):

  • The bytes are unchanged.
  • Identity and gzip bodies are read.
  • The tap stops within its bound, so a 4 MB body is not pulled through the tee.
  • Nothing is tapped that is not a 200 HTML document.
  • A relative canonical says nothing.
  • The in-flight bound holds, a body that errors returns its slot, and a resolver that throws costs nothing.

• test/serveCheck.test.js:

  • An agreeing check and a re-slug both report the endpoint's canonical; a disarmed field reports nothing.
  • A throwing registry still records the check.
  • A document check reports the document's canonical.

• test/renderQueueVariants.test.js:

  • A canonical verdict writes the registry from render and files the new canonical due now and urgent.
  • A verdict from an older renderer writes nothing.

• test/entity.test.js: a render, a check and an origin observation each adopt, and both series name the observer.

• Console: the newer fall-throughs, the cost stat, the registry panel's dry-run read and its "off" state.

• Mutation checks. Each of these fails at least one test:

  • removing the veto;
  • ignoring the check in the veto;
  • an unreadable instant serving;
  • treating every own row as has-target;
  • removing the tie-break or the query guard;
  • reading the registry while it is off;
  • feeding a disarmed canonical, or silencing the document check;
  • not feeding the verdict;
  • the tap reading to the end, or tapping a HEAD;
  • unwiring the tap;
  • dropping the observer from canonical_adopt.

• Review fixes, each with tests:

  • a suppressed spelling: its verdict named the page, named another, predates the field, is older than the confirmation, or has an unreadable instant;
  • the read-to-store slack;
  • untracked, and a query canonical as foreign;
  • the tap: a query URL or the master switch off means no tap; an unsent body is cancelled and not drained; stacked under the raw-cache capture, both copies are whole;
  • the stored path's fallback, for a sitemap render;
  • suppressedCanonical, and the verdict's read instant.

  Each of those 10 mutations is caught too.

• served_wrong, each detection point with a test that fails without it:

  • a serve-time check measures from the render, or from an earlier check that agreed with the same copy;
  • the sweep measures from the render, or from the last pass when the origin moved since;
  • a confirmed render check;
  • a stored 404 found live by the re-check, or by an armed proxied request, but not under a dry run;
  • agreements and held disagreements report nothing.

• Plugin 2018/2018, console 529/529, browser 428/428; lint and format clean. With the registry on by default, the tap tests now switch it off explicitly; a tap that ignored the switch fails them.

• A flaky test, not this PR's: documentFactsOfYielding: the same result as documentFactsOf, with the event loop let in between rounds (test/documentFacts.test.js, on main since 0.98.0) failed about once in a dozen full runs, on this branch and on #241's. It asserts the scan yielded by counting setInterval(0) ticks, and a 0 ms interval is clamped to 1 ms, so a gzip scan that finishes inside a millisecond records none. This PR's timing-sensitive files ran 15 times under parallel load with no failure.

• Review: a fresh-context Claude subagent reviewed 5cbabf7, #241's rework and #239's declaredCanonical adversarially. It found no bot-path throw, hang or slot leak, and no corruption of the response bytes. What it did find:

  1. High: a spelling suppressed as a canonical verdict was answered even when that verdict was newer than the candidate's confirmation. Only the registry could catch that, and the registry is off by default. Fixed: suppressedCanonical and the rule above (cb1289d).
  2. The dry-run adoption number counted observations, not entities, and a caller's dry run spent an armed node's real budget. Fixed in #241 (ea67ce5): dry-run memory, and a lane of its own.
  3. The tap's release drained an unsent body without limit. Fixed: it cancels.
  4. Timestamps were compared across read and store instants. Fixed: the slack, observations stamped when the origin was asked, and a verdict stamped less the whole job's renders.
  5. Query strings and crawler-chosen ids reached the observers. Fixed: foreign for a query canonical, no tap for a query URL, untracked.
  6. Adoption could ping-pong between two spellings naming each other. Fixed in feat(plugin): an Entity registry holding each product's current canonical, and the change probe adopts a canonical no target holds; v0.101.0 #241: retryAfter is per entity.
  7. declaredCanonical was posted for noindex pages, any scheme and any length. Fixed in fix(browser): CSS url() self-references are fragment-only, and a canonical verdict reports the canonical it declared; v1.40.0 #239 (a4f96b7): canonical verdicts only, http(s), keyable.
  8. There was no off switch for the tap short of the registry's. Fixed: it obeys ingress.entityServe.enabled.

  Plus doc corrections: moved can persist under the adoption dry run; the schema and docstrings now list every observer; the console says "Canonical moves", since a move is not always a re-slug. No outside-model review is available, so cross-model coverage is degraded.

• Gemini's code-assist bot left three comments here, each on code from an earlier PR, so each is fixed there and merged in. All three threads are answered:

  • the budget's hour rollover race: one 64-bit cell per lane, #241 789375c;
  • sameDocument with a non-string, also feat(plugin): an Entity registry holding each product's current canonical, and the change probe adopts a canonical no target holds; v0.101.0 #241;
  • an entity with exactly 8 rows read as ambiguous: #240 d745752.

• Live renders (#239, measured on one production origin) with 1.38.0 and 1.40.0, three product pages on two devices: 0 unexpected diffs; the stars fill at another URL; a misspelled product URL reports its real canonical.

Deploy notes

The order matters:

1. Browser 1.40.0 to the render fleet (#239). It is needed for the render observer's canonical verdicts and for the star fills at other spellings. Snapshots rendered before it keep their absolute self-references until their next render.
2. Plugin 0.102.0, console 0.26.0. Schema changes, all added on deploy, which restarts the workers:
   • PageCheck gains a column (#240);
   • Entity is a new table (#241);
   • Target gains suppressedCanonical, written from the next verdict of each spelling. Older suppressions are answered only when the canonical's page is confirmed after them.
3. The registry is on at deploy (on by default since #241's ae02917), with adoption in dry run. Renders and checks fill it from the start; the first nightly pass writes a row per probed product; proxied misses then move rows. Read the Entity registry panel: "Canonical moves", by observer, and "Would adopt" (would-adopt plus capped, which is what arming files).
4. entities.adopt.dryRun: false.
5. The entity serve, in dry run, on the product route, with the entity gate armed (ingress.entityGate.dryRun: false). Read would-serve, moved and has-target on the Entity serve panel, and entity_serve_ms.
6. ingress.entityServe.dryRun: false.

Steps 3 and 4 come before step 6 because, with the gate armed, a re-slugged unlisted product's new canonical is filed only by adoption. And it is the registry that tells the entity serve the old page has moved.

Not in this PR

• The probe walking Entity instead of every target, through a host-supplied entity source: #242, and Entity table: one row per product with its current canonical — the probe checks entities, a re-slug becomes a retarget #166 phase 2.
• Feeding freshness (title, price) from a proxied miss's document into the canonical's page check. A miss's document is the canonical's current document on such a site, so this would be a free serve-time check. A possible follow-up.

Release: plugin 0.102.0 and console 0.26.0 (0.99.0 is #238's, 0.100.0 and console 0.25.0 #240's, 0.101.0 #241's). Config: no new options; the existing ones gain readers. Schema: Target.suppressedCanonical.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces the 'Entity Serve' and 'Entity Registry' features, allowing the prerender plugin to serve a cached canonical render for variant spellings of an entity and track canonical URLs observed from the origin. The changes span the plugin's request handling, metrics, admin API, change probe, and the console's traffic view. Feedback on the implementation highlights a race condition in the hourly adoption budget reset, a potential type coercion bug in safeDecodeURI for non-string values, and a false-negative ambiguity check in readEntityRows when the number of rows exactly matches the query limit.

[gemini-code-assist]

There is a brief race condition at the boundary of a new hour. If thread A successfully updates B_HOUR via compareExchange but has not yet executed Atomics.store(cell, B_COUNT, 0), another thread B can read the new hour, skip the reset block, and read the stale high B_COUNT from the previous hour, leading to spurious capped rejections. Resetting B_COUNT to 0 before updating B_HOUR ensures that any thread observing the new hour is guaranteed to see the reset count.

Suggested change

	const observed = Atomics.load(cell, B_HOUR);
	if (hour > observed && Atomics.compareExchange(cell, B_HOUR, observed, hour) === observed) {
	Atomics.store(cell, B_COUNT, 0);
	}
	if (hour > observed) {
	Atomics.store(cell, B_COUNT, 0);
	Atomics.compareExchange(cell, B_HOUR, observed, hour);
	}

[harper-joseph]

Fixed in #241 (789375c), merged here. The budget moved to util/hourlyBudget.js when it gained a dry-run lane. Each lane is now one 64-bit cell holding hour << 32 | count, so rolling the hour and zeroing the count is a single compare-and-swap and no thread can see one without the other. Resetting the count before the hour (the suggestion) would only move the race: a second thread could zero a count the new hour had already spent. A thread carrying an older hour is now refused instead of rolling the cell back. Tested across four real worker threads released together from an Atomics.wait gate: one hour grants exactly its limit, and an hour rolled under contention grants the next hour its full limit. With the CAS replaced by a plain store, the tests fail.

[gemini-code-assist]

If value is not a string (e.g., null or undefined), decodeURI will implicitly coerce it to a string (returning "null" or "undefined"), which can cause sameDocument(null, "null") to incorrectly evaluate to true. Adding a type check ensures robust and strictly correct behavior.

Suggested change

	const safeDecodeURI = (value) => {
	try {
	return decodeURI(value);
	} catch {
	return null;
	}
	};
	const safeDecodeURI = (value) => {
	if (typeof value !== 'string') return null;
	try {
	return decodeURI(value);
	} catch {
	return null;
	}
	};

References

1. Guard exported functions against invalid inputs (e.g., non-string types) to ensure the robustness of the module's public API.

[harper-joseph]

Fixed in #241 (789375c), merged here. No current caller passes a non-string (each guards first), but sameDocument now returns false unless both are strings, and safeDecodeURI returns null for a non-string. Tested with null/"null" and undefined/"undefined".

[gemini-code-assist]

When querying a database with a limit, using a 'limit + 1' pattern is a standard way to determine if there are more results beyond the limit. Currently, if there are exactly limit rows under the prefix, complete is marked as false and the request falls through as AMBIGUOUS, even if those were the only rows. Querying with limit + 1 and checking if read <= limit avoids this false negative.

export const readEntityRows = async ({ table, prefix, url, limit = ENTITY_READ_LIMIT }) => {
	let own = null;
	let read = 0;
	let unreadable = false;
	let ended = false;
	const keys = [];
	for await (const row of table.search(
		{
			conditions: [{ attribute: 'url', comparator: 'greater_than_equal', value: prefix }],
			sort: { attribute: 'url' },
			select: [...ENTITY_ROW_SELECT],
			limit: limit + 1,
		},
		{ replicateFrom: false }
	)) {
		read++;
		const key = row?.url;
		if (typeof key !== 'string') {
			unreadable = true;
			continue;
		}
		if (!key.startsWith(prefix)) {
			ended = true;
			break;
		}
		if (read > limit) {
			break;
		}
		if (key === url) own = { state: row.state ?? null, suppressedReason: row.suppressedReason ?? null };
		else if (inRotation(row)) keys.push(key);
	}
	return { own, inRotation: keys, complete: !unreadable && (ended || read <= limit) };
};

[harper-joseph]

Fixed in #240 (d745752), where readEntityRows lives, and merged here. The read asks for limit + 1 rows and is complete when it ended on a key outside the prefix, or read at most limit. Tested: an entity with exactly 8 rows is served at the end of the table and with another product after it; one more row under the prefix is ambiguous. With the old read < limit test, it fails.