[harper-joseph]

Refs #166. This is the cheapest part of that issue: it recognises an entity (product) at discovery time. The entity table and turning a slug change into a retarget are not part of this PR, so #166 stays open.

Problem (measured on a production deployment)

Crawlers keep requesting old or invented slugs of products whose correct URL is already a target: /product/prd-1005777/old-spelling.jsp when /product/prd-1005777/new-spelling.jsp exists, or placeholders like /product/prd-1006030/product.jsp. Discovery mints a target for each one. The target renders, the render finds the canonical elsewhere, and it is suppressed (canonical-mismatch). It re-renders weekly on the suppression recheck. After maxStrikes it is deleted, and the next crawler hit mints it again.

• suppressed/canonical-mismatch = 8.2% of all render outcomes, about 1,500 URL jobs/hour.
• Sample of 150 such suppressed targets: 150/150 had another target for the same product id under the correct slug (65% sitemap-listed, 35% active but unlisted).
• Each one costs a render slot and an origin document fetch, and produces no page.

The suppressed rows are left alone. Today they are the only thing that stops these URLs being minted again, and they age out through maxStrikes as before.

What this adds

Per-route entityPrefix: a regular expression, anchored at the start of the URL path, whose match identifies one entity. The lookup prefix is <url origin> + <matched text>. Every Target whose key starts with it is a sibling.

ingress:
  routes:
    - { match: prefix, path: '/product/prd-', queryParams: [], entityPrefix: '^/product/prd-[^/]+/' }
  entityGate:
    dryRun: true      # default — evaluate + count, mint anyway
    # enabled: true   # default — master switch; routes must still opt in

The gate runs in handlePageScheduling only, for a URL with no target row:

siblings	armed	dryRun: true
at least one in rotation (state not suppressed)	not minted (still served; a miss proxies the origin)	minted, counted would-gate
all suppressed	minted (suppressed-only)	same
none	minted (no-siblings)	same
URL produces no usable prefix	minted (no-prefix)	same
read throws	minted (error)	same

• Sitemap-driven creation is never gated. Ingestion, redirect adoption, the REST API and suppression all create targets without calling the gate. A test checks that bot_request.js is its only caller.
• Validation works like departureAction / demandFloor. An invalid regex, a non-string or an empty string drops the field with a warning, never the route. Passthrough routes refuse the field. Setting it together with discoverTargets: false is warned as a no-op.

Deviation from the brief: sitemapUrl does not gate on its own

The brief said to gate when a sibling is sitemap-listed or in rotation. With that rule the gate can block a product's only correct URL forever. Suppose a sitemap lists a non-canonical URL S whose page canonicalizes to U. S's target is listed and suppressed, and every refresh keeps that row in place. U, the correct URL, would then never be discovered and would be served from the origin permanently. Before this PR, U would have been minted and rendered.

So a sibling gates only when it is in rotation. A listed sibling that is in rotation still gates, because it is in rotation. This rule covers the measured case, where the listed siblings were the correct slugs and so were in rotation. sitemapUrl is no longer read, and the projection is ['url', 'state'].

Prefix collision: the match must end on /

…/prd-123 is a string prefix of …/prd-1234/any.jsp. A prefix that stops mid-segment would let product 1234's target gate product 123 forever. How this is handled:

• Runtime: a match that does not end in / is ignored for that URL. The outcome is no-prefix and the URL is minted as before. This check is what makes it safe.
• Compile time: a pattern whose source does not end in / is warned about but kept. Some patterns like that still work. For example, ^/product/prd-\d+/? matches up to / whenever the URL continues past the id, and it is refused on the bare-id URL.
• A match no longer than a prefix route's own path, such as ^/product/ on /product/prd-, covers the whole route and is refused.
• The regex is compiled sticky, so it only matches at the start of the path even without ^.

The read: off the serve path, bounded, can't hang

• Where: inside the existing detached discovery step (setImmediate → handlePageScheduling), after the existing-row Target.get. It runs only for a would-be mint on a route that opted in, never on the response path.
• Residency: Target is not residency-pinned. Only RenderSchedule calls setResidencyById (resources/RenderSchedule.js:7), and util/pageOrphanSweep.js already relies on node-local Target reads. So the read is node-local. { replicateFrom: false } is passed anyway as the second argument (not a query field, which Harper would silently ignore). If Target is ever pinned, this read still won't turn into an untimed cross-node fetch.
• Shape: one one-sided PK range, url >= prefix, sorted, limit: 3 — fixed, not configuration: measured on 150 suppressed old-slug targets, the first sibling in rotation was at key-order position 1 for 135, 2 for 14 and 3 for 1 (1 row: 90%, 2: 99%, 3: 100%), and no entity had more than 4 siblings. Past it the gate mints. No rows are kept — only the first blocker's key and whether any sibling existed. The upper bound is enforced while reading: the first key outside the prefix ends the loop. This is the same shape as listPagesInner and walkPrefix, because a two-sided PK range falls back to a filtered intersection instead of a seek. The read stops at the first sibling in rotation.
• Cursor discipline: the loop body never awaits. The cursor is released by the break or by running out of rows before anything is recorded or written, and a test checks this.
• Fail-open: a read error, an unreadable row, or a URL the pattern doesn't match all mint, exactly as before this PR.

Observability

• New series prerender_ops / entity_gate: detail is the outcome (gated / would-gate / suppressed-only / no-siblings / no-prefix / error), context is the bot. Exactly one emit per evaluation. would-gate is the dry-run number.
• Armed refusals are also emitted as discovery_gated with gate entity. The console's Discovery gate panel already includes them in its "Gated misses" total with no console release. It has no dedicated "by entity" tile, so the total can exceed route + bot.
• Why not only a new label on discovery_gated? The console totals every discovery_gated row as "served from the origin, never scheduled". Counting dry-run would-gates or allowed outcomes there would make that total wrong. So the census gets its own series, and only armed refusals go into discovery_gated.
• Console coverage guard (console/test/adminAssets.test.js): it scans plugin metrics.js emit sites for literal series names and fails unless the console reads each one or waives it. I added prerender_ops.entity_gate to its NOT_CHARTED list with the reason. This is a test-only change, so no console release is needed. A census tile is follow-up work.
• METRICS.md: I added a bullet rather than editing the prerender_ops table row. That row is the widest cell, so any edit makes prettier re-pad the whole table, and that table is already triplicated on main (a pre-existing merge artifact that I left alone).

Rollout

1. Deploy with dryRun: true (the default) and set entityPrefix on the product route.
2. Read prerender_ops / entity_gate would-gate against render outcome suppressed/canonical-mismatch. If a route is mostly no-prefix, its pattern doesn't match its URLs.
3. Set ingress.entityGate.dryRun: false.
4. Existing suppressed canonical-mismatch rows age out through maxStrikes. Once the gate is armed, they are not minted again.

Adversarial re-read

• Can it permanently block an entity's only correct URL? Not through anything the gate decides:
  • Suppressed siblings never block.
  • sitemapUrl alone never blocks.
  • Prefix collision is refused at runtime.
  • Every failure mints.
• Remaining cases where a URL is held back for as long as its sibling stays in rotation:
  • A sibling whose renders keep failing without ever producing a verdict (failure backoff keeps it active). Its URLs are the same product page, so they usually fail together.
  • An origin that serves two self-canonical (or, with canonical.strict: false, variant) URLs for one entity. The first one minted renders and the other is served from the origin. Neither is more correct than the other from the origin's point of view.
• Bounded delay, stated: after a genuine re-slug of an unlisted product, the new URL is gated until the old URL's next render suppresses it. That is one render cycle of the old target: its cadence, or up to a full interval for a freshly minted target whose first render is jittered. After that, the new URL is minted on its next crawl. Listed re-slugs are created by the sitemap and never meet the gate. If the old URL 301s, redirect adoption creates the new target directly.
• Blocking read on the serve path? No. It runs detached, only for would-be mints, is node-local, and is limit-bounded.
• Operator regex on crawler-supplied paths: the docs ask for linear patterns (a literal prefix plus [^/]+ segments). The sticky anchor also stops the engine from trying later start positions.
• Query variants: every URL under a prefix counts as one entity, including query variants the route keeps. This is documented: don't set entityPrefix on a route where several URLs per entity are distinct pages.
• Merge conflicts: the route field is inserted after rawCache, so it should merge cleanly next to feat(plugin): probe only sitemap-listed targets (opt-in), and re-render a product that rejoins its sitemap; v0.89.0 #209's arrivalAction, which is inserted before rawCache.

Tests

test/entityGate.test.js, 33 tests:

• Compile and validation: valid pattern, absent or null, invalid regex, non-string or empty, passthrough, source not ending in /, discoverTargets: false.
• Prefix: origin and port, anchoring without ^, prd-123 vs prd-1234, optional trailing /, a match covering the whole route.
• Decisions: listed sibling in rotation → gated; active unlisted → gated; suppressed-only (including listed and suppressed) → minted; no siblings, with prd-1234 and prd-12 present → minted; the URL is not its own sibling; dry run mints and counts; no entityPrefix means no read and nothing recorded; master switch off; no-prefix; read throws → error and minted; unreadable row skipped.
• Read shape: one condition, greater_than_equal, sort, select ['url','state'], limit = 3 (SIBLING_READ_LIMIT), { replicateFrom: false } as the second argument; cursor closed before any metric is emitted (both on break and on running out of rows); stops at the first sibling in rotation; the limit binds in the suppressed case.
• End to end through handlePageScheduling with a stubbed Target: armed, dry run, no prefix, existing row, suppressed-only.
• Only caller: a source scan confirms evaluateEntityGate is referenced only in bot_request.js, at one call site after the existing-row check.
• Config defaults.

Results:

• Plugin: 1243 pass / 0 fail (1210 before this PR).
• Console: 277 pass / 0 fail.
• npm run lint and npm run format:check pass (exit 0).

Revert check

Each mutation was applied to the source with the tests unchanged:

mutation	result
remove if (!mint) return; from discovery	the ARMED discovery test fails
drop the runtime / rule	PREFIX COLLISION and optional-slash tests fail
restore the brief's rule (a listed sibling gates)	SUPPRESSED-ONLY and read-shape tests fail
ignore dryRun	both dry-run tests fail
drop { replicateFrom: false }	read-shape test fails
let the URL count as its own sibling	"never its own sibling" test fails

All reverted afterwards.

Not in this PR

• The id-keyed entity table and retargeting a slug change (Entity table: one row per product with its current canonical — the probe checks entities, a re-slug becomes a retarget #166 proper).
• A console tile for the entity_gate census (waived in the guard).
• Showing the resolved entity prefix in the explain admin view, so an operator can check a pattern against one URL. The no-prefix counts give the aggregate signal for now.

🤖 Generated with Claude Code

[harper-joseph]

Self-review (author side) — no defects found; no changes.

Checked: the sibling read runs only inside the already-detached handlePageScheduling, after the existing-row check, so it never touches the response; it is a one-sided primary-key range (url >= prefix, limit siblingLimit + 1, replicateFrom: false) that breaks at the first key outside the prefix, and Target is not residency-pinned; a match that does not end on / is ignored, so prd-123 cannot gate prd-1234; every failure mints as before; dry run (default) counts would-gate and mints. The deviation from the brief — only an IN-ROTATION sibling blocks, not a listed one on its own — is equivalent in practice (a sitemap-listed target's render posts as rendered, so it never reaches suppression) and strictly safer where it differs.

[gemini-code-assist]

Code Review

This pull request introduces an "entity discovery gate" feature (v0.90.0) to prevent redundant target minting for different URL variations of the same underlying entity. It adds a new entityPrefix configuration option for routes, implements the gate logic in packages/plugin/src/util/entityGate.js, integrates it into the page scheduling process, and adds corresponding metrics and tests. Regarding the feedback, a compatibility issue was identified in packages/plugin/src/util/entityGate.js where URL.parse is used. Since URL.parse is only supported in Node.js v22.1.0+ and not backported to Node 20 (LTS), it is recommended to use new URL() wrapped in a try...catch block instead to ensure compatibility with older Node.js versions.

[gemini-code-assist]

Avoid using URL.parse because it is only supported in Node.js v22.1.0+ and is not backported to Node 20 (LTS). To ensure compatibility with older Node.js versions, use new URL() wrapped in a try...catch block instead.

Suggested change

	const parsed = URL.parse(url);
	if (!parsed) return null;
	let parsed;
	try {
	parsed = new URL(url);
	} catch {
	return null;
	}

References

1. Avoid using URL.parse if the target Node.js version is below v22.1.0, as it is not backported to Node 20. Use new URL() with appropriate error handling or pre-validation instead.

[harper-joseph]

Not changing: URL.parse is the repo idiom (null instead of a throw on a malformed value, which is exactly the fail-open behaviour the gate wants), it exists on every supported Node line (22.1.0+, backported to 20.18.0; deployments run Node 24), and the same finding was dismissed on #127.

[harper-joseph]

Review change (91041b4): the sibling read is a fixed 3 rows and ingress.entityGate.siblingLimit is removed. Measured on 150 suppressed old-slug targets, the first sibling in rotation sat within the first 3 keys for 150/150 (1 row: 90%, 2: 99%), and no entity had more than 4 siblings; past the limit the gate still mints. readSiblings now keeps no rows ({ blocker, seen }). 1243/1243 plugin, 277/277 console, lint + format clean; a 6-row read fails the two updated tests. Body updated.