Skip to content

Commit cb1289d

Browse files
trialclaude
andcommitted
fix(plugin): review of 0.102.0 -- a suppressed spelling is answered only with the page its verdict named, confirmations allow for the read-to-store gap, crawler misses create no registry rows, no query strings, the tap obeys the master switch and lets go of an unsent body, the stored path falls back to declaredCanonical; v0.102.0
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent d9b757a commit cb1289d

18 files changed

Lines changed: 311 additions & 61 deletions

‎packages/console/src/admin/views/traffic.js‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2422,7 +2422,8 @@ function entityServe(ctx, data, filter) {
24222422
}
24232423

24242424
// Who told the entity registry (plugin `entity_canonical` / `canonical_adopt` context), in the order a re-slug is
2425-
// usually seen: the origin documents crawlers pull, the renders, the serve-time checks, the nightly probe.
2425+
// usually seen: the origin documents crawlers pull, the renders, the serve-time checks, the nightly probe. A move
2426+
// is a re-slug, or one observer correcting another (an endpoint and a page that disagree flap between them).
24262427
const ENTITY_OBSERVERS = [
24272428
['origin', 'proxied origin documents'],
24282429
['render', 'renders'],
@@ -2490,7 +2491,7 @@ function entityRegistry(ctx, data) {
24902491
body: [
24912492
stats([
24922493
stat(
2493-
'Re-slugs seen',
2494+
'Canonical moves',
24942495
fmtCount(seen('moved')),
24952496
byObserver((from) => seen('moved', from))
24962497
),

‎packages/console/test/trafficView.test.js‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1408,7 +1408,7 @@ test('the entity-serve panel names the newer fall-throughs and what an evaluatio
14081408
assert.match(text, /median · p95/);
14091409
});
14101410

1411-
test('the registry panel: re-slugs seen by the observer that saw them, and the adoption dry-run number', async () => {
1411+
test('the registry panel: canonical moves by the observer that saw them, and the adoption dry-run number', async () => {
14121412
const analytics = {
14131413
...ANALYTICS,
14141414
series: [
@@ -1432,7 +1432,7 @@ test('the registry panel: re-slugs seen by the observer that saw them, and the a
14321432
const text = everything(ctx);
14331433
assert.match(text, /Entity registry/);
14341434
assert.match(text, /adoption dry run/);
1435-
assert.match(text, /Re-slugs seen/);
1435+
assert.match(text, /Canonical moves/);
14361436
assert.match(text, /60 proxied origin documents · 25 renders · 15 the nightly probe/);
14371437
assert.match(text, /Would adopt/);
14381438
assert.match(text, /70 under the hourly budget · 5 past it/);

‎packages/plugin/METRICS.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -196,13 +196,17 @@ Notes that bite:
196196
a target by then (since v0.102.0 a spelling suppressed as a canonical verdict is answered, not
197197
`has-target`). `unconfirmed` climbs right after the anchor and falls as the pass and the serve-time
198198
checks confirm canonicals. If a route stays there, no check compares its `canonical`. `moved`
199-
(v0.102.0, `entities.enabled`) is a page the registry has since heard re-slugged: it should be rare,
200-
and should not persist for an entity, because the observation that moved the registry also adopted
201-
the new canonical. `entity_serve_ms` (v0.102.0) is each evaluation's duration, every outcome: a
199+
(v0.102.0) is a page the origin has since been heard naming another canonical: by the registry
200+
(`entities.enabled`), or by the spelling's own canonical verdict when it is newer than the page's last
201+
confirmation and did not name it. It should be rare. With adoption armed it should not persist for an
202+
entity, because the observation that moved the registry also filed the new canonical; under the
203+
adoption dry run, or past `maxPerHour`, it lasts until the new canonical renders by another path. `entity_serve_ms` (v0.102.0) is each evaluation's duration, every outcome: a
202204
duration, so read its percentiles.
203205
- **The entity registry is `prerender_ops` / `entity_canonical` and `canonical_adopt`** (v0.101.0,
204206
`entities.enabled`). `entity_canonical`: one emit per observation of an entity's canonical, detail =
205-
what it did to the registry (`new`, `moved`, `same`, `older`, `foreign`, `unreadable`, `error`),
207+
what it did to the registry (`new`, `moved`, `same`, `older`, `foreign` — another entity's prefix, or a
208+
query string — `untracked` — a crawler's proxied miss of an entity with no row, which creates none —
209+
`unreadable`, `error`),
206210
context = the observer: `probe`, `render` (a stored page, or a canonical verdict's declared canonical),
207211
`check` (a serve-time check) or `origin` (a proxied origin document on an `entityServe` route; the last
208212
two since v0.102.0). `moved` per day is the re-slug rate, and which observer moves it first says how

‎packages/plugin/README.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -341,8 +341,10 @@ origin, exactly as before:
341341
- **The spelling has no target the render path keeps.** A spelling with no row of its own is answered,
342342
and so is one whose own target was suppressed as a canonical verdict (`canonical-mismatch`,
343343
`canonical-variant`): its own render found that the page names its canonical elsewhere. Measured on
344-
one deployment, one crawler made ~11k such misses a day. A row in rotation (a new canonical arriving
345-
from the sitemap) or one suppressed about the URL itself (a 404, a noindex) is `has-target`.
344+
one deployment, one crawler made ~11k such misses a day. Such a spelling is answered only with the page
345+
its own verdict named (`Target.suppressedCanonical`, browser ≥ 1.40.0), or with a page confirmed after
346+
that verdict; otherwise it is `moved`. A row in rotation (a new canonical arriving from the sitemap) or
347+
one suppressed about the URL itself (a 404, a noindex) is `has-target`.
346348
- **One candidate.** One target of the entity in rotation has a page for this device that is a 200,
347349
indexable, inside its own expiry (not SWR), and not covered by an invalidation it predates. When more
348350
than one does, the [entity registry](#the-entity-registry-and-adopting-the-canonical-the-origin-names-entities)
@@ -425,15 +427,17 @@ entities:
425427
- `check`: a serve-time check, from the endpoint's canonical slot or the origin document's canonical.
426428
- `origin`: a miss proxied to the origin on an `entityServe` route. Its canonical is read off the head
427429
as the crawler's bytes stream by (at most 128 KiB, and at most 32 at once per worker), so it costs no
428-
second request and nothing the response waits on.
430+
second request and nothing the response waits on. It moves a row the registry already holds and never
431+
creates one (`untracked`), so a crawler asking for invented product ids cannot create rows; a URL with a
432+
query string is not read, and the tap stops with `ingress.entityServe.enabled`.
429433
- When two observations disagree, the newer wins. A render's instant is when it read the origin
430434
(store time less the sum of its renders), so a render claimed before a re-slug can't undo the probe
431435
that saw it.
432436
- An unchanged observation writes nothing, and neither does the same canonical spelled otherwise
433437
(`%27` for an apostrophe). That is one document, as the probe's own `path` comparator already
434438
treats it.
435-
- A canonical under another entity's prefix is ignored, and so is a relative path (only an absolute
436-
URL or a `/`-rooted path counts).
439+
- A canonical under another entity's prefix is ignored, and so is one with a query string, and a relative
440+
path (only an absolute URL or a `/`-rooted path counts).
437441
- **Adoption.** When any observer names a canonical that is another URL of the same entity than the one
438442
it observed, and no target holds it in rotation, its target is filed due now and urgent, as redirect
439443
adoption does.

‎packages/plugin/src/http_handlers/bot_request.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -620,7 +620,7 @@ async function resolveResource({ request, url, cacheUrl, deviceType, routeClass,
620620
// entityServe route every spelling is one document, so every fetch that reaches here — a miss, a stale or an
621621
// invalidated page — says where the product lives now. Read off the head as the bytes stream by: no second
622622
// origin request, and nothing this response waits on.
623-
const observed = tapsOriginCanonical(info.route, request.method)
623+
const observed = tapsOriginCanonical(info.route, request.method, cacheUrl)
624624
? tapOriginCanonical(resource, { url: cacheUrl })
625625
: resource;
626626
// Keep what we just fetched, for the next crawler asking the same question. The capture rides the

‎packages/plugin/src/metrics.js‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -731,7 +731,9 @@ export const METRICS = Object.freeze({
731731
'util/entity.js), by what it did to the registry: new (the entity\u2019s first row), moved (the canonical ' +
732732
'changed — a re-slug, or one observation correcting another), same (nothing written), older (a ' +
733733
'disagreeing observation made before the stored one, ignored), foreign (a canonical under another ' +
734-
'entity\u2019s prefix, ignored), unreadable, error; context = the observer: probe (the change probe\u2019s ' +
734+
'entity\u2019s prefix, or carrying a query string, ignored), untracked (a crawler\u2019s proxied miss of an ' +
735+
'entity with no row: no row is created from crawler traffic), unreadable, error; context = the observer: ' +
736+
'probe (the change probe\u2019s ' +
735737
'mapped canonical), render (a stored page\u2019s canonical, or the one a canonical verdict declared), ' +
736738
'check (a serve-time check\u2019s endpoint or document) or origin (a proxied origin document on an ' +
737739
'entityServe route). moved per day is the re-slug rate; which observer moves it first says how fast a ' +
@@ -933,7 +935,7 @@ export const METRICS = Object.freeze({
933935
'(gated, would-gate, suppressed-only, no-siblings, no-prefix, error). entity_serve: the outcome ' +
934936
'(served, would-serve, no-prefix, has-query, has-target, no-sibling, no-page, not-indexable, stale, ' +
935937
'invalidated, ambiguous, unconfirmed, moved, not-self-canonical, unreadable, error). entity_canonical: the outcome ' +
936-
'(new, moved, same, older, foreign, unreadable, error). canonical_adopt: the outcome (adopted, ' +
938+
'(new, moved, same, older, foreign, untracked, unreadable, error). canonical_adopt: the outcome (adopted, ' +
937939
'reactivated, would-adopt, exists, suppressed, recent, capped, refused, error). raw_cache: THE OUTCOME — stored, ' +
938940
'stored-unshared, or the refusal name; this is the slot the console reads that panel from. ' +
939941
'negative_cache: the outcome — stored/stored-unshared; a refusal (has-cookie, private, no-store, ' +
@@ -1171,7 +1173,7 @@ export const metrics = Object.freeze({
11711173

11721174
/**
11731175
* One observation of an entity's canonical (util/entity.js) — a prerender_ops series. `outcome` is what it
1174-
* did to the registry (new, moved, same, older, foreign, unreadable, error); `from` is the observer: 'probe',
1176+
* did to the registry (new, moved, same, older, foreign, untracked, unreadable, error); `from` is the observer: 'probe',
11751177
* 'render', 'check' or 'origin'.
11761178
*/
11771179
entityCanonical: (outcome, from) =>

‎packages/plugin/src/resources/RenderQueue.js‎

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -880,17 +880,16 @@ export class RenderQueue extends Resource {
880880
const { deleted, absent, held } = await Target.suppress(url, {
881881
reason: verdict.reason,
882882
statusCode: verdict.statusCode,
883+
canonical: verdict.declaredCanonical ?? null,
883884
});
884885
// THE CANONICAL THE VERDICT NAMED (browser >= 1.40.0 `declaredCanonical`), into the entity registry: on a
885886
// site where every spelling of a product is one document, the render of an old spelling after a
886887
// re-slug is often the first fetch from the origin to see the new canonical, and this is where the
887888
// product lives now. Recorded and, when no target holds it, adopted — so the new canonical renders now
888889
// rather than at the next nightly probe. Observed when the origin was read. Never rejects.
889-
await observeRenderedCanonical(
890-
url,
891-
verdict.declaredCanonical,
892-
Date.now() - (Number(verdict.renderTime) || 0)
893-
).catch(() => {});
890+
await observeRenderedCanonical(url, verdict.declaredCanonical, Date.now() - renderSpanMs(variants)).catch(
891+
() => {}
892+
);
894893
// The other half of `suppression_lifted` below: a suppressed target rendered, and the verdict
895894
// stood. Together they are the precision of every early recheck — a gone target rendered
896895
// before its `recheckInterval` was filed by a reopen (util/goneReopen.js), and this is where
@@ -1085,9 +1084,14 @@ export class RenderQueue extends Resource {
10851084
// was READ — the store time less the stored renders, which run one after another — so a render
10861085
// claimed before a re-slug cannot outrank the probe that saw it. A point read, and a write only
10871086
// when the canonical moves; a canonical no target holds is adopted. Never rejects.
1088-
observeRenderedCanonical(scheduleUrl, describing.pageFacts?.canonical, cachedAt - renderSpanMs(stored)).catch(
1089-
() => {}
1090-
),
1087+
// A sitemap target is stored even when its canonical names another URL: its page record says so,
1088+
// and so does the verdict's `declaredCanonical`, which stands in when the record's extraction failed.
1089+
observeRenderedCanonical(
1090+
scheduleUrl,
1091+
describing.pageFacts?.canonical ??
1092+
stored.find((variant) => typeof variant.declaredCanonical === 'string')?.declaredCanonical,
1093+
cachedAt - renderSpanMs(stored)
1094+
).catch(() => {}),
10911095
...observed.map((variant) =>
10921096
recordReadinessExpectation({ url: scheduleUrl, deviceType: variant.deviceType }, variant.readiness)
10931097
),
@@ -1352,7 +1356,11 @@ export class RenderQueue extends Resource {
13521356
// suppression is a genuine content/gone verdict about the destination. A destination too long
13531357
// to key has no row to suppress, and needs none: nothing can ever schedule it.
13541358
if (keyable !== false && (!config.domains.length || config.domains.includes(domain))) {
1355-
await Target.suppress(destinationUrl, { reason: variant.reason, statusCode: variant.statusCode });
1359+
await Target.suppress(destinationUrl, {
1360+
reason: variant.reason,
1361+
statusCode: variant.statusCode,
1362+
canonical: variant.declaredCanonical ?? null,
1363+
});
13561364
}
13571365
return;
13581366
}

‎packages/plugin/src/resources/Target.js‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -220,7 +220,7 @@ export class Target extends TargetTable {
220220
* pages either way — stale content of a page that said "don't index me" must not keep
221221
* serving.
222222
*/
223-
static async suppress(url, { reason, statusCode } = {}) {
223+
static async suppress(url, { reason, statusCode, canonical = null } = {}) {
224224
const existing = await Target.get({
225225
id: url,
226226
select: [
@@ -294,6 +294,8 @@ export class Target extends TargetTable {
294294
state: 'suppressed',
295295
suppressedReason: storedReason,
296296
suppressedAt: Date.now(),
297+
// What the page named, for a canonical verdict: the entity serve's evidence (util/entityServe.js).
298+
suppressedCanonical: typeof canonical === 'string' && canonical !== '' ? canonical : null,
297299
strikes,
298300
});
299301

‎packages/plugin/src/schemas/schema.graphql‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,10 @@ type Target @table(database: "render_service") @export {
134134
# Why the last verdict suppressed it: noindex | canonical-mismatch | redirect-loop | ...
135135
suppressedReason: String
136136
suppressedAt: Date
137+
# The canonical a CANONICAL VERDICT's page named (browser >= 1.40.0 `declaredCanonical`), null otherwise.
138+
# The entity serve answers this spelling only with the page it named, or one confirmed after the verdict
139+
# (util/entityServe.js): the verdict is this spelling's own, newest word on where its product lives.
140+
suppressedCanonical: String
137141
# Consecutive non-indexable verdicts; render.suppression.maxStrikes deletes the target.
138142
strikes: Int
139143
# Current demand-ladder rung (ms), when render.demand is enabled. Absent = the target has
@@ -157,10 +161,11 @@ type Target @table(database: "render_service") @export {
157161
# two unrelated rows; this is where "which of them does the origin call canonical, now" lives.
158162
#
159163
# DERIVED, NEVER AUTHORITATIVE: `Target` stays the render registry. Written only by observations of the
160-
# origin — the change probe's mapped `canonical` slot and a stored render's declared canonical — and only
161-
# when the canonical MOVES (an unchanged observation writes nothing). Replicated and NOT residency-pinned,
162-
# like `Target`: writes are rare, and every node answers a point read locally. Not @export: internal state,
163-
# read through the admin API. Inert unless `entities.enabled`.
164+
# origin — the change probe's mapped `canonical` slot, a render's declared canonical, a serve-time check, and
165+
# a document proxied from the origin (which moves a row, never creates one) — and only when the canonical
166+
# MOVES or a dry run remembers an adoption (an unchanged observation writes nothing). Replicated and NOT
167+
# residency-pinned, like `Target`: a row per entity plus ~one write per re-slug, and every node answers a point
168+
# read locally. Not @export: internal state, read through the admin API. Inert unless `entities.enabled`.
164169
type Entity @table(database: "render_service") {
165170
id: String @primaryKey
166171
# The canonical URL (a Target key) the newest observation named.

‎packages/plugin/src/util/entity.js‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,17 @@ export const CanonicalOutcome = Object.freeze({
8585
SAME: 'same',
8686
/** A different canonical, observed before the one stored: ignored. */
8787
OLDER: 'older',
88-
/** A canonical outside the entity's own prefix: another entity's URL, ignored. */
88+
/**
89+
* Not this entity's canonical to record: under another entity's prefix, or carrying a query string the route
90+
* keys (one document per entity has no query to vary by). Ignored.
91+
*/
8992
FOREIGN: 'foreign',
93+
/**
94+
* An `origin` observation (a crawler's proxied miss) of an entity the registry has no row for: not recorded,
95+
* so a crawler asking for invented product ids cannot create rows. Every entity this plugin renders gets its
96+
* row from the probe or a render.
97+
*/
98+
UNTRACKED: 'untracked',
9099
/** A canonical that is not a URL this can key. */
91100
UNREADABLE: 'unreadable',
92101
/** A read or a write threw. Logged. */
@@ -161,7 +170,7 @@ const safeDecodeURI = (value) => {
161170
};
162171

163172
/**
164-
* Record that the origin, observed at `atMs` by `from` ('probe' | 'render'), names `canonical` as the
173+
* Record that the origin, observed at `atMs` by `from` ('probe' | 'render' | 'check' | 'origin'), names `canonical` as the
165174
* canonical of the entity `url` belongs to. Returns `{ outcome, key, canonical, row }`: `canonical` is the
166175
* keyed form the registry holds for it, `row` the entity as it stands after this observation (null when there
167176
* is no entity). Never throws.
@@ -174,7 +183,7 @@ const safeDecodeURI = (value) => {
174183
* two first observations racing — a render and a probe, or two nodes probing two spellings — then merge,
175184
* and neither can erase what the adoption recorded on the row.
176185
*/
177-
export async function observeCanonical({ url, canonical, from, atMs }) {
186+
export async function observeCanonical({ url, canonical, from, atMs, createIfMissing = true }) {
178187
if (!entitiesOn()) return { outcome: null, key: null, canonical: null, row: null };
179188
const entity = entityOf(url);
180189
if (!entity) return { outcome: null, key: null, canonical: null, row: null };
@@ -184,10 +193,12 @@ export async function observeCanonical({ url, canonical, from, atMs }) {
184193
};
185194
const target = typeof canonical === 'string' && canonical !== '' ? canonicalFormOf(canonical) : null;
186195
if (!target) return decided(CanonicalOutcome.UNREADABLE);
187-
// The entity's own prefix, and nothing else: a canonical under another prefix names another entity.
188-
if (!target.startsWith(entity.key)) return decided(CanonicalOutcome.FOREIGN);
196+
// The entity's own prefix, and nothing else: a canonical under another prefix names another entity. And no query
197+
// string: a param the route keys can change the document, and a crawler can choose its value.
198+
if (!target.startsWith(entity.key) || URL.parse(target)?.search) return decided(CanonicalOutcome.FOREIGN);
189199
try {
190200
const row = await table().get({ id: entity.key, select: [...ENTITY_SELECT] });
201+
if (!row && !createIfMissing) return decided(CanonicalOutcome.UNTRACKED);
191202
if (!row) {
192203
const created = {
193204
id: entity.key,
@@ -346,7 +357,8 @@ export const createCanonicalResolver = ({
346357
if (!entitiesOn()) return null;
347358
const canonical = canonicalFromValue(value, url);
348359
if (!canonical) return null;
349-
const observed = await observe({ url, canonical, from, atMs: atMs ?? now() });
360+
// A crawler's proxied miss moves a row the registry already holds, and creates none (`untracked`).
361+
const observed = await observe({ url, canonical, from, atMs: atMs ?? now(), createIfMissing: from !== 'origin' });
350362
// The observed URL IS the canonical (in any spelling), or there is nothing to adopt.
351363
if (!observed.row || !observed.canonical || sameDocument(observed.canonical, url)) return observed;
352364
const adopt = typeof settings === 'function' ? settings() : settings;

0 commit comments

Comments
 (0)