fix(plugin): review of 0.101.0 -- a dry run counts each entity once and spends its own lane of the budget, retryAfter is per entity, the budget is its own module tested across worker threads; v0.101.0

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Author: trial

packages/plugin/METRICS.md

@@ -189,7 +189,8 @@ Notes that bite:
   disagree about the canonical. `canonical_adopt` (`entities.adopt`): one emit per adoption decision,
   made only when an observation names a canonical that is another URL than the one observed: `adopted`,
   `reactivated`, `would-adopt` (**the dry-run number**), `exists` (every duplicate spelling, nightly),
-  `suppressed`, `recent`, `capped` (past `maxPerHour` on this node), `refused`, `error`.
+  `suppressed`, `recent` (the entity was adopted, or in a dry run would have been, within `retryAfter`),
+  `capped` (past `maxPerHour` on this node), `refused`, `error`.
 - **The change probe's `probe_*` series changed shape in v0.97.0, and the table row above predates
   it.** (1) The pass counters are emitted **per probed batch as increments**, not once when a pass
   ends: a nine-hour pass is no longer one row that a dropped analytics window loses whole, and a pass

packages/plugin/README.md

@@ -331,7 +331,7 @@ entities:
   observer; it is the endpoint's own answer for the product id. A stored render's declared
   `pageFacts.canonical` is the other.
   - When two observations disagree, the newer wins. A render's instant is when it read the origin
-    (store time less its longest render), so a render claimed before a re-slug can't undo the probe
+    (store time less the sum of its renders), so a render claimed before a re-slug can't undo the probe
     that saw it.
   - An unchanged observation writes nothing, and neither does the same canonical spelled otherwise
     (`%27` for an apostrophe). That is one document, as the probe's own `path` comparator already
@@ -343,10 +343,13 @@ entities:
   - A target suppressed as a canonical verdict (`canonical-mismatch`, `canonical-variant`) is reactivated
     the same way. One suppressed for any other reason (a 404, a noindex) is left alone.
   - Bounded by `maxPerHour` per node, shared by every worker thread and every observer; by `retryAfter`
-    per entity (a canonical that did not take is filed once per window, not nightly, so it costs one render
-    a week for as long as the origin names it); and by both dry runs. A probe pass run as a dry run,
-    including an operator's measure-only sweep, files nothing.
-  - In a dry run, what arming would file is `would-adopt` plus `capped`.
+    per entity, whichever canonical (a canonical that did not take is filed once per window, not nightly,
+    so it costs one render a week for as long as the origin names it, and two spellings naming each other
+    cannot reactivate each other in turn); and by both dry runs. A probe pass run as a dry run, including
+    an operator's measure-only sweep, files nothing.
+  - In a dry run, what arming would file is `would-adopt` plus `capped`. A dry run remembers each entity
+    it would have adopted (`wouldAdoptAt`), so a repeat inside `retryAfter` reads `recent` as it would
+    armed, and it spends its own lane of the hourly budget, never an armed node's real slots.
 - **Why.** Measured on one deployment, products re-slug ~100 times a day and the product sitemap
   changes once a day. An out-of-stock product is not in the sitemap at all, so its new canonical
   arrived only by traffic discovery, with its first render jittered across the route's 96h interval.

packages/plugin/src/configSchema.js

@@ -3126,9 +3126,11 @@ export const configSchema = group('Prerender plugin configuration.', {
 					dryRun: option(
 						true,
 						'Count `would-adopt` and file nothing. The default, because the number to know first is how many ' +
-							'canonicals arming would file: `would-adopt` plus `capped` (the budget counts would-adopts too). ' +
-							'A change-probe pass run as a dry run \u2014 the probe\u2019s own `dryRun`, or an operator\u2019s ' +
-							'measure-only sweep \u2014 files nothing either.'
+							'canonicals arming would file: `would-adopt` plus `capped`. A dry run remembers each entity it ' +
+							'would have adopted, so a repeat inside `retryAfter` reads `recent` as it would armed, and counts ' +
+							'against its own lane of the hourly budget, never an armed node\u2019s real slots. A change-probe ' +
+							'pass run as a dry run \u2014 the probe\u2019s own `dryRun`, or an operator\u2019s measure-only ' +
+							'sweep \u2014 files nothing either.'
 					),
 					maxPerHour: option(
 						60,
@@ -3140,9 +3142,10 @@ export const configSchema = group('Prerender plugin configuration.', {
 					),
 					retryAfter: option(
 						7 * DAY,
-						'An entity whose adopted canonical did not take (it 404s, or its page names another canonical ' +
-							'after all) is not filed again for this long, so a bad canonical costs one render per window, ' +
-							'not one per observation.',
+						'An entity adopted once is not adopted again for this long, whichever canonical is named then: a ' +
+							'canonical that did not take (it 404s, or its page names another canonical after all) costs one ' +
+							'render per window, not one per observation, and two spellings that name each other cannot ' +
+							'reactivate each other in turn.',
 						{ unit: 'ms', min: HOUR }
 					),
 				}

packages/plugin/src/metrics.js

@@ -716,7 +716,8 @@ export const METRICS = Object.freeze({
 			'filed due now), reactivated (a canonical-verdict suppression lifted, due now), would-adopt (either, in ' +
 			'a dry run — THE DRY-RUN NUMBER), exists (the canonical has a target in rotation — every duplicate ' +
 			'spelling, nightly), suppressed (its target is suppressed for a reason the origin\u2019s word does not overturn), ' +
-			'recent (adopted within retryAfter), capped (past maxPerHour on this node), refused (unkeyable, off the domain ' +
+			'recent (the entity was adopted, or in a dry run would have been, within retryAfter), capped (past maxPerHour ' +
+			'on this node), refused (unkeyable, off the domain ' +
 			'allowlist, or not on a prerender route), error. ' +
 			'raw_cache = one emit per raw-document store attempt, split by outcome: `stored`, `stored-unshared`, ' +
 			'or the reason it was refused (not-200, staging, has-cookie, content-type, no-store, no-body, ' +

packages/plugin/src/resources/PrerenderAdmin.js

@@ -1778,6 +1778,8 @@ export class PrerenderAdmin extends Resource {
 							canonicalAt: entityRow?.canonicalAt ? new Date(entityRow.canonicalAt).getTime() : null,
 							adoptedCanonical: entityRow?.adoptedCanonical ?? null,
 							adoptedAt: entityRow?.adoptedAt ? new Date(entityRow.adoptedAt).getTime() : null,
+							wouldAdoptCanonical: entityRow?.wouldAdoptCanonical ?? null,
+							wouldAdoptAt: entityRow?.wouldAdoptAt ? new Date(entityRow.wouldAdoptAt).getTime() : null,
 						}
 					: null,
 				// Already described (locally or by the owner) — see above.

packages/plugin/src/schemas/schema.graphql

@@ -172,10 +172,14 @@ type Entity @table(database: "render_service") {
 	# render claimed before a re-slug, landing after the probe saw it) does not.
 	canonicalAt: Date
 	firstSeenAt: Date
-	# The last canonical a target was ADOPTED for, and when: a canonical that did not take is not re-filed
-	# until `entities.adopt.retryAfter` has passed.
+	# The last canonical a target was ADOPTED for, and when: the entity is not adopted again until
+	# `entities.adopt.retryAfter` has passed, whichever canonical is named then.
 	adoptedCanonical: String
 	adoptedAt: Date
+	# The same, for an adoption a DRY RUN would have made: so the dry-run number counts entities, not
+	# observations, and arming starts from no memory at all (a dry run never blocks a real adoption).
+	wouldAdoptCanonical: String
+	wouldAdoptAt: Date
 }
 
 # The render queue: ONE ROW PER URL, due at `nextRenderTime`. A claim of a row is one job that

packages/plugin/src/util/entity.js

@@ -49,8 +49,9 @@ import { CacheKey } from './cacheKey.js';
 import { entityPrefixOf } from './entityGate.js';
 import { classifyPath, matchRoute, PRERENDER, queryAllowlistFor } from './routeClass.js';
 import { canonicalizeUrl } from './url.js';
-import { currentMinuteMs, dateColumnMs, HOUR } from './time.js';
+import { currentMinuteMs, dateColumnMs } from './time.js';
 import { getSab } from './coordination.js';
+import { createHourlyBudget, hourlyBudgetLength } from './hourlyBudget.js';
 import { Target } from '../resources/Target.js';
 
 const table = () => databases.render_service.Entity;
@@ -63,6 +64,8 @@ const ENTITY_SELECT = Object.freeze([
 	'canonicalAt',
 	'adoptedCanonical',
 	'adoptedAt',
+	'wouldAdoptCanonical',
+	'wouldAdoptAt',
 ]);
 
 /** Every outcome of observing a canonical (`prerender_ops` / `entity_canonical`). */
@@ -83,7 +86,7 @@ export const CanonicalOutcome = Object.freeze({
 	ERROR: 'error',
 });
 
-/** Every outcome of the probe's adoption check (`prerender_ops` / `canonical_adopt`). */
+/** Every outcome of an adoption check (`prerender_ops` / `canonical_adopt`). */
 export const AdoptOutcome = Object.freeze({
 	/** A target created for the canonical, due now. */
 	ADOPTED: 'adopted',
@@ -95,7 +98,7 @@ export const AdoptOutcome = Object.freeze({
 	EXISTS: 'exists',
 	/** Its target is suppressed for a reason the origin's word does not overturn (a 404, a noindex). */
 	SUPPRESSED: 'suppressed',
-	/** Adopted within `retryAfter` already: a canonical that did not take is not re-filed every night. */
+	/** The entity was adopted (or, in a dry run, would have been) within `retryAfter`, whichever canonical. */
 	RECENT: 'recent',
 	/** Past `entities.adopt.maxPerHour` on this node this hour. */
 	CAPPED: 'capped',
@@ -276,52 +279,22 @@ const canonicalFromValue = (value, probedUrl) => {
 	}
 };
 
-// THE ADOPTION BUDGET: adoptions this hour on this node, shared by every worker thread — [hourSinceEpoch, count]
-// in a shared buffer (util/coordination.js `getSab`). Every source can adopt (the probe on its worker, renders
-// and serve-time checks and proxied documents on any worker), so a per-thread counter would multiply the cap by
-// the thread count. Hours since the epoch fit an int32 for a quarter of a million years.
-//
-// The window rolls forward only, and whoever wins the roll zeroes the count — the same shape as
-// util/invalidationReenqueue.js `reserveSlot`, for the same reasons: a straggler carrying an older hour must not
-// zero a count the new hour has spent, and a loser of a race costs or grants at most one adoption. This bounds
-// render load from a burst (a site-wide re-spelling), it is not a safety property.
-const BUDGET_SAB_KEY = 'entity_adopt_budget_v1';
-const B_HOUR = 0;
-const B_COUNT = 1;
+// THE ADOPTION BUDGET: adoptions this hour on this node, shared by every worker thread (util/hourlyBudget.js, over
+// a shared buffer). Every source can adopt (the probe on its worker, renders and serve-time checks and proxied
+// documents on any worker), so a per-thread counter would multiply the cap by the thread count. Two lanes: real
+// adoptions, and dry-run ones, so a dry run measures what the cap would do without spending the slots an armed
+// node's real adoptions need (an operator's measure-only sweep on an armed node, for one).
+const BUDGET_SAB_KEY = 'entity_adopt_budget_v2';
+const REAL = 0;
+const DRY = 1;
 let budgetCell = null;
-const budgetI32 = () => (budgetCell ??= new Int32Array(getSab(BUDGET_SAB_KEY, 8)));
+const budgetI32 = () =>
+	(budgetCell ??= new Int32Array(getSab(BUDGET_SAB_KEY, hourlyBudgetLength(2) * Int32Array.BYTES_PER_ELEMENT)));
 
-/** The node's adoption budget: `reserve(limit)` takes a slot this hour (false past `limit`); `release()` returns one. */
-export const adoptionBudget = Object.freeze({
-	reserve(limit, nowMs = Date.now()) {
-		const cell = budgetI32();
-		const hour = Math.floor(nowMs / HOUR);
-		const observed = Atomics.load(cell, B_HOUR);
-		if (hour > observed && Atomics.compareExchange(cell, B_HOUR, observed, hour) === observed) {
-			Atomics.store(cell, B_COUNT, 0);
-		}
-		for (let attempt = 0; attempt < 8; attempt++) {
-			const used = Atomics.load(cell, B_COUNT);
-			if (used >= limit) return false;
-			if (Atomics.compareExchange(cell, B_COUNT, used, used + 1) === used) return true;
-		}
-		// Eight lost races: other workers are spending the budget right now. Refusing is the safe direction.
-		return false;
-	},
-	release() {
-		const cell = budgetI32();
-		for (let attempt = 0; attempt < 8; attempt++) {
-			const used = Atomics.load(cell, B_COUNT);
-			if (used <= 0 || Atomics.compareExchange(cell, B_COUNT, used, used - 1) === used) return;
-		}
-	},
-});
+/** The node's adoption budget (util/hourlyBudget.js): lane 0 real adoptions, lane 1 dry-run ones. */
+export const adoptionBudget = createHourlyBudget(budgetI32);
 /** Tests: an empty budget. */
-export const resetAdoptionCountsForTest = () => {
-	const cell = budgetI32();
-	Atomics.store(cell, B_HOUR, 0);
-	Atomics.store(cell, B_COUNT, 0);
-};
+export const resetAdoptionCountsForTest = () => adoptionBudget.reset();
 
 /**
  * RESOLVE A CANONICAL: record an observation of the origin naming `value` as the canonical of the entity `url`
@@ -337,13 +310,16 @@ export const resetAdoptionCountsForTest = () => {
  * Bounded three ways:
  *   - `entities.adopt.maxPerHour` per node, across every worker and every source (`adoptionBudget`). A put that
  *     fails gives its slot back.
- *   - `retryAfter` per entity, so a canonical that did not take (it 404s and is retired, or its page names
- *     another canonical and is suppressed) is filed at most once per window. Such an entity costs one render a
- *     window for as long as the origin keeps naming it.
+ *   - `retryAfter` per ENTITY, whichever canonical: an entity adopted within it is `recent`. So a canonical that
+ *     did not take (it 404s and is retired, or its page names another canonical and is suppressed) costs one
+ *     render a window for as long as the origin keeps naming it, and two spellings that name each other (an
+ *     origin re-slugging back, or serving two inconsistent copies) cannot reactivate each other in turn.
  *   - a dry run — `entities.adopt.dryRun`, or the caller's own (`dryRun`: the change probe passes its pass's
- *     effective dry run, which an operator's measure-only sweep sets) — that counts and files nothing. In a dry
- *     run the budget counts would-adopts too, so what arming files is `would-adopt + capped`, capped at
- *     `maxPerHour`. The OBSERVATION is written in a dry run too: it is an observation, not an action.
+ *     effective dry run, which an operator's measure-only sweep sets) — that counts and files nothing. A dry run
+ *     REMEMBERS what it would have filed (`wouldAdoptAt`), so a repeat observation within `retryAfter` reads
+ *     `recent` exactly as it would armed, and spends its own lane of the budget, so `would-adopt + capped` is
+ *     what arming files and a dry run never spends an armed node's slots. The OBSERVATION is written in a dry
+ *     run too: it is an observation, not an action.
  *
  * All I/O injectable for tests (`createCanonicalResolver`).
  */
@@ -381,13 +357,23 @@ export const createCanonicalResolver = ({
 			return decided(AdoptOutcome.ERROR);
 		}
 		const nowMs = now();
-		const last = dateColumnMs(observed.row.adoptedAt);
-		if (observed.row.adoptedCanonical === observed.canonical && nowMs - last < adopt.retryAfter) {
-			return decided(AdoptOutcome.RECENT);
-		}
+		const dry = Boolean(adopt.dryRun || dryRun);
+		// Per entity, whichever canonical: the last real adoption, or — in a dry run — the last one it would have made.
+		const last = dateColumnMs(dry ? observed.row.wouldAdoptAt : observed.row.adoptedAt);
+		if (nowMs - last < adopt.retryAfter) return decided(AdoptOutcome.RECENT);
 		if (!adoptable(observed.canonical)) return decided(AdoptOutcome.REFUSED);
-		if (!budget.reserve(adopt.maxPerHour, nowMs)) return decided(AdoptOutcome.CAPPED);
-		if (adopt.dryRun || dryRun) return decided(AdoptOutcome.WOULD_ADOPT);
+		if (!budget.reserve(adopt.maxPerHour, nowMs, dry ? DRY : REAL)) return decided(AdoptOutcome.CAPPED);
+		if (dry) {
+			// Remembered, so the dry-run number counts entities, not observations. Best-effort, like the real memory.
+			try {
+				await markAdopted(observed.key, { wouldAdoptCanonical: observed.canonical, wouldAdoptAt: new Date(nowMs) });
+			} catch (e) {
+				logger.warn?.(
+					`[prerender] entity: recording a would-adopt of ${observed.canonical} failed: ${e?.message ?? e}`
+				);
+			}
+			return decided(AdoptOutcome.WOULD_ADOPT);
+		}
 		// `put` REPLACES the row (and is the reactivation that clears a suppression and its strikes), so a
 		// suppressed row's own declarations ride along.
 		const interval = Number(target?.renderInterval);
@@ -400,7 +386,7 @@ export const createCanonicalResolver = ({
 				...(Number.isFinite(interval) && interval > 0 ? { renderInterval: interval } : {}),
 			});
 		} catch (e) {
-			budget.release();
+			budget.release(REAL);
 			logger.warn?.(`[prerender] entity: adopting ${observed.canonical} failed: ${e?.message ?? String(e)}`);
 			return decided(AdoptOutcome.ERROR);
 		}

packages/plugin/src/util/hourlyBudget.js

@@ -0,0 +1,62 @@
+/**
+ * AN HOURLY BUDGET SHARED BY EVERY WORKER THREAD ON A NODE, over an Int32Array the caller hands in (a shared
+ * buffer in production, util/coordination.js `getSab`). No Harper dependency, so it can be tested across real
+ * worker threads.
+ *
+ * Layout: [hourSinceEpoch, spent in lane 0, spent in lane 1, ...]. Hours since the epoch fit an int32 for a
+ * quarter of a million years. Each LANE is its own count against the same limit: the entity registry spends
+ * real adoptions in one and dry-run adoptions in the other, so a dry run measures what the cap would do without
+ * spending the slots a real adoption needs.
+ *
+ * The window rolls forward only, and whoever wins the roll zeroes every lane — the shape of
+ * util/invalidationReenqueue.js `reserveSlot`, for the same reasons: a straggler carrying an older hour must not
+ * zero a count the new hour has spent, and a loser of a race costs or grants at most one slot. This bounds load
+ * from a burst; it is not a safety property.
+ */
+
+const HOUR_MS = 3_600_000;
+const B_HOUR = 0;
+
+/** The Int32Array length a budget with `lanes` lanes needs. */
+export const hourlyBudgetLength = (lanes) => 1 + lanes;
+
+/**
+ * A budget over `cell()` (called on every use, so the caller can allocate lazily). `reserve(limit, nowMs, lane)`
+ * takes a slot in `lane` this hour, false past `limit`; `release(lane)` returns one.
+ */
+export const createHourlyBudget = (cell) => {
+	const roll = (i32, nowMs) => {
+		const hour = Math.floor(nowMs / HOUR_MS);
+		const observed = Atomics.load(i32, B_HOUR);
+		if (hour > observed && Atomics.compareExchange(i32, B_HOUR, observed, hour) === observed) {
+			for (let lane = 1; lane < i32.length; lane++) Atomics.store(i32, lane, 0);
+		}
+	};
+	return Object.freeze({
+		reserve(limit, nowMs = Date.now(), lane = 0) {
+			const i32 = cell();
+			roll(i32, nowMs);
+			const at = 1 + lane;
+			for (let attempt = 0; attempt < 8; attempt++) {
+				const used = Atomics.load(i32, at);
+				if (used >= limit) return false;
+				if (Atomics.compareExchange(i32, at, used, used + 1) === used) return true;
+			}
+			// Eight lost races: other workers are spending the budget right now. Refusing is the safe direction.
+			return false;
+		},
+		release(lane = 0) {
+			const i32 = cell();
+			const at = 1 + lane;
+			for (let attempt = 0; attempt < 8; attempt++) {
+				const used = Atomics.load(i32, at);
+				if (used <= 0 || Atomics.compareExchange(i32, at, used, used - 1) === used) return;
+			}
+		},
+		/** Tests: an empty budget. */
+		reset() {
+			const i32 = cell();
+			for (let i = 0; i < i32.length; i++) Atomics.store(i32, i, 0);
+		},
+	});
+};