Answer a variant spelling of an entity URL from the canonical's cached render, instead of proxying or raw-caching each spelling #237

On one production origin, most misses on the product route are bots asking for another spelling of a product we already have rendered: /product/prd-<id>/<any slug>. The origin answers every such spelling with the same document, declaring the canonical. This issue answers those misses from the canonical URL's cached render, instead of proxying them or storing a raw copy per spelling. It builds on the entity prefix of #166, which the entity discovery gate already uses.

Status: built, in review, not released.

• #239: browser 1.40.0, fragment-only url() self-references.
• #240: plugin 0.100.0 (entityServe, PageCheck.canonicalAgreed) and console 0.25.0.
• #243: plugin 0.102.0 and console 0.26.0. Every fetch from the origin writes the entity registry (#166, #241). The entity serve then answers spellings suppressed as a canonical verdict, refuses a page the registry has since heard re-slugged, and lets the registry break a tie.

Evidence

Everything below was measured on 2026-10-01 against one production origin. Product pages there were raw-cached for about 25 minutes as a test (rawCache + rawFacts on the product route).

• Most product misses are variant spellings (observed, from the raw rows' stored facts):
  • First read: 584 of 626 stored product documents (93%) sat at a spelling other than their own canonical, all of them the same product. Later: 1,869 of 1,984.
  • 0 named a different product.
  • Examples: /prd-N/email.jsp, /prd-N/product.jsp, capitalised or truncated slugs.
  • Canonical spellings rarely reach the raw cache because they are render targets, so they are cache hits.
• The origin serves one document per product (verified):
  • 70 products: the stored variant document (fetched through the normal miss path) against the origin's canonical document, both through documentFactsOf.
  • Identical canonical, title, description, name, brand, image, every offer (sku, price, currency, availability) and breadcrumbs for 68 of 70.
  • The 2 differences were the origin changing between fetches: a rename that added a SKU, and a description that flapped. Refetching every spelling back to back gave identical documents.
  • The CDN in front keeps one copy per product, ignoring both the slug and the query string: a random cache-busting query changed nothing.
  • The only place the spelling shows is an analytics pageUrl prop, and it carries whichever spelling filled the edge cache, so the canonical's own copy can name a variant.
  • <link rel=canonical>, og:url and JSON-LD url/@id always name the canonical.
  • Documents differ only in one per-response timing script line, which also differs between two fetches of the same URL.
• The canonical's render is almost always cached (measured): 199 of 200 sampled variant spellings, each a different product, had a fresh, indexable, 200 render of their canonical cached for both devices.
  • Render age: median 28h, p90 40h, max 48h, on a 96h render interval.
  • The one miss was a product whose canonical slug had just changed. No spelling of it had a servable page, so it would fall through.
• The choice is never ambiguous (measured): for all 199, exactly one spelling of the product has a servable render, and it is the canonical the variant document declares.
• The origin never 404s one spelling of a product it serves at another (measured over stored 404 bodies):
  • 450 product ids sampled from three nodes: 449 have no render at all for that id (retired or never-registered products).
  • One has an old render at the same URL that is no longer fresh, which the "fresh" guard below excludes.
  • None is a 404 spelling of a product with a live render. A 404 is per product, not per spelling.
• Every product spelling in real traffic is a .jsp path, and none redirects (measured):
  • 1,932 stored 200 variants: lowercase slugs 1,605, one-word slugs (email.jsp, product.jsp) 302, capitals 20, collection ids 5.
  • 16,000 stored 404s: all .jsp.
  • On the product route, bot_miss redirect was 0 of 361k misses in 24h across four nodes.
  • Empty slugs, missing slashes, extra segments and encoded characters never occur. Synthetic requests for those shapes cannot be tested from our side: the origin's bot mitigation answers a non-crawler 403.
• Collection pages behave the same (measured, n=5): each variant, often a member product's slug under the collection id, declared the collection's canonical. That canonical had a cached render on both devices, and the facts were identical.

Design (as built in #240)

A route option entityServe: true beside its entityPrefix, governed by ingress.entityServe (enabled, dryRun default true, maxConfirmAge). On a true miss, before the raw and negative caches, the targets under the URL's entity prefix are read. That is one one-sided primary-key range over Target: at most 8 rows, node-local because Target is not residency-pinned, with replicateFrom: false. The miss is served from another target's page only when all of these hold. Anything else falls through to the existing miss path and is counted on prerender_ops / entity_serve:

It is served as bot_serve source and status entity, with the canonical's own stored headers and validators. A serve-time check of it checks the canonical's key. Nothing is stored and nothing is minted for the spelling.

Confirmation is the canonical's, and the probes record it

The risk is a slug re-spell. The origin re-spells slugs: one wave moved canonical-mismatch from 4% to 45% of render outcomes in 2.5h. Until the old page re-renders, it names its old slug while the origin already declares the new one at every spelling. Serving it at more spellings would spread that contradiction.

What counts as confirmation. A candidate counts as confirmed only if either:

• it was rendered at or after the threshold (the render read the origin's canonical), or
• a check at or after the threshold compared its canonical with the origin's and found them the same.

The threshold is the last anchor in anchored mode, tightened by maxConfirmAge. Outside anchored mode, nothing is confirmed unless maxConfirmAge is set.

Why a new column. An overall check outcome cannot carry this. agree means "nothing compared disagreed", and that is also true when the mapping guard disarmed the canonical field or the endpoint's slot was null. So PageCheck gains canonicalAgreed, written by both the serve-time check and the probe sweep from their per-field verdicts. With serve-time checks off, the sweep records checks for entity-serve routes.

What a deployment needs:

• The probe rule must map canonical (pageCheck.fields). On the measured origin that is slot 6, seoURL, compared by path.
• That slot must stay out of ignoreChanges, so a re-spell is a change the sweep acts on: it expires and re-renders the old page.

Reach. An unconfirmed candidate is offered to the serve-time check, under that check's own switches and budget. The request that finds it unconfirmed is what gets it confirmed.

A re-spell during the day (#243). The confirmation alone left a gap: a page confirmed at 06:00 stays confirmed after a 10:00 re-slug, until its next check. With the registry on, the first fetch from the origin to see the new slug moves the registry, whether a crawler's proxied miss, a render, a check or the probe. From then on, moved refuses the old page, and the same observation files the new canonical due now.

What it needs beside it

• The entity gate, armed. A spelling the entity serve does not answer (every one, in a dry run) goes to the origin, and a minting crawler's miss mints it. From then on it reads has-target. With the gate in dry run, only each spelling's first request can be served.
• The entity registry on, and its adoption armed (#243's deploy notes give the order). With the gate armed, a re-slugged out-of-stock product's new canonical is filed only by adoption.
• Browser 1.40.0 on the fleet first. Snapshots rendered before it keep absolute self-referencing star fills, which paint nothing at another spelling, until their next render (see the next section).

The known difference, fixed in the renderer

The snapshot is rendered at the canonical URL, so anything a script builds from location carries the canonical. On two examined snapshots, every product URL was the canonical and there were no share links. The exception is a third-party reviews widget:

• SVG fills, fill="url(<page URL>#rating_star_filled)". Served at another URL, an absolute self-reference becomes an external paint server. Measured in Chrome 148 (#239), it paints no fill, and the browser fetches the render URL while trying to resolve it.
• Review pagination links, ?bvstate=pg:2/ct:r. These pointing at the canonical is harmless.

This already happens for every query-string variant of a route with queryParams: []. #239 rewrites every CSS url(<this page's URL>#id) in the serialized snapshot to url(#id), which refers to the current document at any URL.

Still open

• Site-specific. The measurements are from one origin. Another site may serve different content per spelling, which is why the option is off by default. Its docs carry the experiment that settles it: two spellings fetched from the origin must agree on every fact and name the same canonical.
• The dry-run census has not been read on a live deployment. That is the next step once both releases are deployed.

Done when

• A route-level option serves a variant spelling from the canonical's cached render behind the guards above, and is tested for each fall-through: a Target exists, the page is stale or non-indexable, the canonical is unconfirmed since the anchor, or there is no render. (#240)
• The post-processing rewrite makes absolute self-references in url() fragment-only, with a test. (#239)
• bot_serve reports the new source, so its share of misses is visible. (#240, with the console panel)

🤖 Generated with Claude Code