feat(plugin): don't discover a URL whose product already has a target under another slug (per-route entityPrefix); v0.90.0

packages/console/test/adminAssets.test.js

@@ -278,6 +278,15 @@ test('a metric the plugin emits is charted by the console, or waived with a reas
 		['render_readiness.shortfall', 'report-only window; read raw from /prerender_admin/analytics — panel is #189'],
 		['render_readiness.rebaseline', 'report-only window; read raw from /prerender_admin/analytics — panel is #189'],
 		['render_readiness.satisfied_ms', 'report-only window; read raw from /prerender_admin/analytics — panel is #189'],
+		// The entity discovery gate (plugin v0.90.0) ships in dry run. Its evaluation census is read raw
+		// from the analytics endpoint during that window; the refusals that matter once it is ARMED are
+		// also emitted as `discovery_gated` with the gate name `entity`, which the Discovery gate panel
+		// already totals — so no console release is needed to see the armed gate's effect. A tile for
+		// the census (would-gate / suppressed-only / no-siblings / no-prefix) is the follow-up.
+		[
+			'prerender_ops.entity_gate',
+			'dry-run census; read raw from /prerender_admin/analytics — armed refusals land in discovery_gated',
+		],
 	]);
 
 	const client = [...clientSources.values()].join('\n');

packages/plugin/METRICS.md

@@ -185,6 +185,18 @@ Notes that bite:
   distinguishes a real crawler from a spoofed User-Agent: bot names come from the UA string alone,
   with no reverse-DNS or IP verification, so cross-check a search engine's own reporting before
   treating a per-bot breadth figure as that engine's crawl rate.
+- **The entity discovery gate is `prerender_ops` / `entity_gate`, and it is not in the table row
+  above** (v0.90.0). One emit per evaluation, detail = outcome, context = bot: `gated` (a sibling URL
+  of the same entity is in rotation; not minted), `would-gate` (the same verdict under
+  `ingress.entityGate.dryRun`; minted anyway), `suppressed-only` (every sibling suppressed; minted),
+  `no-siblings` (a new entity; minted), `no-prefix` (the route has an `entityPrefix` and this URL
+  produced no usable match; minted), `error` (the sibling read threw; minted). The outcomes sum to the
+  mints the gate evaluated. **`would-gate` is the dry-run number** — read it against `render` outcome
+  `suppressed`/`canonical-mismatch` before arming. A route that is nearly all `no-prefix` has a
+  pattern that does not match its URLs. Armed refusals are ALSO emitted as `discovery_gated` with the
+  gate name `entity` (never in a dry run), so every view of what the gates hold out includes them;
+  unlike the `route`/`bot` gates, `entity` is evaluated only for URLs with no target row, so it counts
+  refused mints rather than gated misses on known targets.
 - **`queue_health.overdue` includes in-flight renders** (a leased row keeps its past due time), so
   its healthy floor is the in-flight count, not zero — and it is not comparable with numbers from
   before v0.34.0. The scan is capped by `management.scanCap`: a backlog past the cap reports the cap.

packages/plugin/README.md

@@ -257,6 +257,61 @@ forwarding `/blog/*`"); passthrough is the coverage backlog ("we proxy this much
 on purpose"). The tally is in-process, so **every worker** flushes its own line — each carries
 `node=` and `worker=`, and a reader sums across them.
 
+### Discovery: one target per entity (`entityPrefix`)
+
+Traffic discovery mints a target for any unknown URL a bot gets a cacheable 200 for. `Target` is
+keyed by URL, so two spellings of one product are two unrelated rows — and crawlers keep asking for
+old or invented slugs of products whose correct URL is already tracked. Each one is minted, renders,
+finds its canonical pointing elsewhere, is suppressed as `canonical-mismatch`, re-renders on the
+suppression recheck until `maxStrikes` deletes it, and is minted again by the next crawler hit.
+Measured on one deployment: `canonical-mismatch` was 8.2% of all render outcomes, and 150 of 150
+sampled suppressed targets had another target for the same product id under the correct slug.
+
+A route that declares `entityPrefix` stops that at discovery:
+
+```yaml
+ingress:
+  routes:
+    - { match: prefix, path: '/product/prd-', queryParams: [], entityPrefix: '^/product/prd-[^/]+/' }
+  entityGate:
+    dryRun: true # the default: evaluate and count, mint anyway
+```
+
+- **The prefix.** The regex is anchored at the start of the URL path; its match plus the URL's origin
+  is the entity prefix (`https://www.example.com/product/prd-123/`). Every target whose URL starts
+  with it is a **sibling**.
+- **The match must end on `/`.** `…/prd-123` is a string prefix of `…/prd-1234/…`, so a prefix that
+  stops mid-segment would let product 1234 gate product 123 forever. A match that does not end in
+  `/` is ignored for that URL (it is discovered as before, counted `no-prefix`), and a pattern whose
+  source does not end in `/` is warned about at config time.
+- **The decision.** A sibling **in rotation** (not suppressed) means the entity already has a target
+  that will render, so the URL is not minted — it is still served (a miss proxies the origin). If
+  there are no siblings, or every sibling is **suppressed**, the URL is minted. Suppressed siblings
+  never block, which is what makes a genuine slug change a bounded delay rather than a permanent
+  block: the new slug is gated until the old slug's next render suppresses it (one render cycle of
+  the old target), then minted on its next crawl. `sitemapUrl` plays no part — a sitemap that lists
+  a non-canonical URL holds a listed-but-suppressed row, and the canonical URL must still be
+  discoverable.
+- **Discovery only.** Sitemap ingestion, redirect adoption and the REST API create targets without
+  consulting the gate; the declared corpus is always created.
+- **The read.** One one-sided primary-key range over `Target` (at most 3 rows, projecting `url` and
+  `state`, stopping at the first sibling in rotation), only for a URL with no target row, inside the
+  detached discovery step — never on the response path. `Target` is not residency-pinned, so the read
+  is node-local. Any failure mints. The 3 is fixed: measured on a production catalog, the first
+  sibling in rotation was within the first 3 keys for 150/150 sampled entities; past it the gate
+  mints, never refuses.
+
+Every URL under one prefix is the same entity, query variants the route keeps included — don't set
+`entityPrefix` on a route where several URLs per entity are distinct pages. The pattern runs against
+crawler-supplied paths, so keep it linear: a literal prefix plus `[^/]+` segments, no nested
+quantifiers.
+
+**Rollout.** Deploy with `dryRun: true` (the default), read `prerender_ops` / `entity_gate` outcome
+`would-gate` against `render` outcome `suppressed`/`canonical-mismatch`, then set `dryRun: false`.
+Existing suppressed rows are untouched: they age out through `maxStrikes` as before, and once the gate
+is armed they are not re-minted. Armed refusals are also counted on `discovery_gated` with the gate
+name `entity`.
+
 ### Sitemaps are filtered to prerender routes
 
 A sitemap is written for search engines: it lists every indexable URL on the site, which is routinely

packages/plugin/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender",
-	"version": "0.89.0",
+	"version": "0.90.0",
 	"type": "module",
 	"description": "Configurable Harper plugin for prerendering pages for bots and crawlers",
 	"license": "Apache-2.0",

packages/plugin/src/configSchema.js

@@ -106,7 +106,7 @@ export const configSchema = group('Prerender plugin configuration.', {
 					"{ match: 'exact' | 'prefix' | 'contains', path: string, mode?: 'prerender' | 'passthrough', " +
 					'queryParams?: string[], renderInterval?: number, discoverTargets?: boolean, demandFloor?: number, ' +
 					"departureAction?: 'none' | 'expire' | 'render', arrivalAction?: 'none' | 'render', " +
-					'rawCache?: boolean }.\n\n' +
+					'rawCache?: boolean, entityPrefix?: string }.\n\n' +
 					'FIRST MATCH WINS, so order most-specific first. That ordering is what lets a passthrough ' +
 					'carve-out sit inside a prerendered prefix (`/products/clearance/` above `/products/`) ' +
 					'without a second list and a precedence rule.\n\n' +
@@ -188,7 +188,25 @@ export const configSchema = group('Prerender plugin configuration.', {
 					'proxy, never a render: a stale or invalidated snapshot still proxies live. Pair it with ' +
 					'`discoverTargets: false` — gated URLs are exactly the population this is for, and gating ' +
 					'without it leaves them missing on every request forever. Enable it only on a route whose ' +
-					'server-rendered document already carries its SEO surface; see `render.raw`.',
+					'server-rendered document already carries its SEO surface; see `render.raw`.\n\n' +
+					'`entityPrefix` (prerender routes only; off when absent) — a regular expression, anchored at the ' +
+					'start of the URL PATH, whose match is the part of the URL that identifies ONE entity, e.g. ' +
+					"`'^/product/prd-[^/]+/'` on a `/product/prd-` route. Traffic discovery then refuses to mint a " +
+					'target for an unknown URL when another URL of the same entity (a target whose URL starts with ' +
+					'this URL\u2019s origin plus the match) is already in rotation: crawlers requesting an old or ' +
+					'invented slug of a product whose correct URL is tracked no longer cost a render that ends ' +
+					'suppressed as `canonical-mismatch`. The URL is still served (a miss proxies the origin). ' +
+					'Sitemap-listed URLs are always created. Siblings that are all SUPPRESSED do not block, so a ' +
+					'product whose slug really changed is discovered at its new URL once the old URL\u2019s next ' +
+					'render has suppressed it — a delay of one render cycle of the old target, never a permanent ' +
+					'block. THE MATCH MUST END ON `/`: a prefix that stops mid-segment is a string prefix of other ' +
+					"entities' URLs (`prd-123` of `prd-1234`), so a match that does not end in `/` is ignored for " +
+					'that URL (it is discovered as if no entityPrefix were set) and a pattern whose source does not ' +
+					'end in `/` is warned about. Every URL under one prefix counts as the same entity, query variants ' +
+					'the route keeps included, so do not set this on a route where several URLs per entity are ' +
+					'distinct pages. The pattern runs against crawler-supplied paths, so keep it linear — a literal ' +
+					'prefix plus `[^/]+` segments, no nested quantifiers. Governed by `ingress.entityGate` (dry run ' +
+					'by default).',
 				{ itemType: 'object' }
 			),
 			discoveryBots: option(
@@ -203,6 +221,39 @@ export const configSchema = group('Prerender plugin configuration.', {
 					'ingestion are all unaffected.',
 				{ itemType: 'string' }
 			),
+			entityGate: group(
+				'The entity discovery gate. Nothing here does anything until a route declares ' +
+					'`ingress.routes[].entityPrefix` — see that field for what an entity is.\n\n' +
+					'WHAT IT DOES. When a bot requests an unknown URL on such a route and discovery would mint a ' +
+					'target for it, the gate first reads the targets that share its entity prefix (one bounded ' +
+					'primary-key range read of at most 3 rows — fixed, not tunable: measured, the first sibling in ' +
+					'rotation was within the first 3 keys for every sampled entity — detached from the response and ' +
+					'node-local). If one of them is in rotation, the URL is not minted. If there are none, or every ' +
+					'one is suppressed, it is minted exactly as before. Any failure mints.\n\n' +
+					'WHAT IT SAVES. Each refused mint is a render (and an origin document fetch) that would have ' +
+					'ended `suppressed/canonical-mismatch`, then re-rendered on the suppression recheck until ' +
+					'`maxStrikes` deleted it and the next crawler hit minted it again. Existing suppressed rows are ' +
+					'left alone — they age out through `maxStrikes` as before, and once the gate is armed they are ' +
+					'not re-minted.\n\n' +
+					'Observed on `prerender_ops` / `entity_gate`, one emit per evaluation by outcome; armed ' +
+					'refusals are also counted on `discovery_gated` with the gate name `entity`.',
+				{
+					enabled: option(
+						true,
+						'Master switch. Routes still have to opt in with `entityPrefix`, so leaving this on costs ' +
+							'nothing until one does; it exists so an operator can stop the gate during an incident ' +
+							'without editing the route list. Off = no sibling read, nothing counted, every unknown URL ' +
+							'minted.'
+					),
+					dryRun: option(
+						true,
+						'Evaluate and count, but mint anyway. The default, because the number to know first is how ' +
+							'many mints the gate WOULD refuse — `entity_gate` outcome `would-gate` — which is read ' +
+							'against `render` outcome `suppressed`/`canonical-mismatch` before arming it. Turn it off ' +
+							'to arm the gate.'
+					),
+				}
+			),
 			excludePathPatterns: option(
 				['/search/'],
 				'Paths never auto-scheduled for rendering. Compiled into `routes` as ' +

packages/plugin/src/http_handlers/bot_request.js

@@ -34,6 +34,7 @@ import { recordVisit } from '../util/visitFilter.js';
 import { materializeCachedBody } from '../util/cachedBody.js';
 import { captureForRawCache, rawCachePolicy, rawKeyOf, readRawPage } from '../util/rawCache.js';
 import { rescueFromOwner } from '../util/peerRescue.js';
+import { evaluateEntityGate } from '../util/entityGate.js';
 import { deliverResource } from './response.js';
 
 export async function handleBotRequest(request) {
@@ -443,6 +444,11 @@ async function resolveResource({ request, url, cacheUrl, deviceType, routeClass,
 // Target.get entirely, which matters on a gated combinatorial route where misses are most of
 // the traffic. The gates stop target CREATION only — an existing target's miss was a no-op in
 // handlePageScheduling anyway — so `discovery_gated` counts gated misses, not denied mints.
+//
+// The ENTITY gate is the exception to "the gates sit here", and necessarily so: it asks whether
+// ANOTHER URL of the same entity has a target, which is a read, and it only means anything for a URL
+// with no target of its own. So it runs inside handlePageScheduling, after the existing-row check —
+// still detached, still off the response.
 function maybeSchedule(resource, routeClass, route, botName) {
 	if (routeClass !== PRERENDER || !resource.miss || resource.statusCode !== 200) return;
 	if (route && route.discoverTargets === false) {
@@ -453,7 +459,7 @@ function maybeSchedule(resource, routeClass, route, botName) {
 		metrics.discoveryGated('bot', botName);
 		return;
 	}
-	setImmediate(handlePageScheduling, resource);
+	setImmediate(handlePageScheduling, resource, route, botName);
 }
 
 // Cache statuses that never looked for a page row, so they can neither prove nor disprove that a
@@ -614,7 +620,8 @@ async function renderNow({ url, cacheUrl, deviceType, cacheKey, request, routeSc
 	};
 }
 
-async function handlePageScheduling(resource) {
+// Exported for tests, which drive the discovery mint end to end against a stubbed Target table.
+export async function handlePageScheduling(resource, route, botName) {
 	try {
 		if (isPrerenderCandidate(resource)) {
 			// resource.url is the origin-fetch URL built from the canonical half, so it is
@@ -628,6 +635,12 @@ async function handlePageScheduling(resource) {
 			// no row at all is genuinely new.
 			const existingTarget = await Target.get({ id: canonicalUrl, select: 'url' });
 			if (!existingTarget) {
+				// THE ENTITY GATE (util/entityGate.js): a URL whose product already has a target in rotation
+				// under another URL is not minted. Evaluated only here — traffic discovery — and only for a
+				// URL with no row, so a sitemap or any other creation path never meets it. Inert unless the
+				// route declares `entityPrefix`; under `ingress.entityGate.dryRun` it counts and mints anyway.
+				const { mint } = await evaluateEntityGate({ url: canonicalUrl, route, botName });
+				if (!mint) return;
 				// No explicit time → Target.put jitters the first render across the interval,
 				// so a crawl that discovers many URLs at once doesn't stampede.
 				// Deliberately NO renderInterval: cadence is resolved at schedule time