[harper-joseph]

Why

Traffic discovery treats every URL a crawler requests as corpus. On a site with combinatorial URL spaces (faceted navigation, filter permutations) that inverts control of the render budget: crawlers walking facet links mint every novel combination into a permanent render obligation. Measured on a production deployment, a catalog's discovered-facet corpus grew from ~224k to an estimated ~3M+ targets in eleven days (a 400-row sample was 100% discovered / 0% sitemap-listed, averaging 5.2 facet terms), putting theoretical demand far above the fleet and the queue ~49h behind — structurally unable to converge. The same mechanism at smaller volume mints malformed URLs invented by crawlers with broken link extractors (#81), which retry forever and permanently pollute the origin-auth alarm.

Route ceilings can't fix this: the corpus itself is the demand. The fix is to stop the minting (gate), then remove what was minted (purge) — in that order, or crawlers re-mint what the purge removes.

What

1. Per-route discovery gate — ingress.routes[].discoverTargets (default true).
A gated route's unknown URLs are still served (origin proxy on a miss, as today) but never create a target. Sitemap-declared URLs on the route schedule exactly as before — the gate distinguishes declared corpus from crawler-walked corpus. Compiled in routeClass.js with the renderInterval field rules (an invalid value drops the field, never the route; warned and ignored on passthrough).

2. Discovery bot allowlist — ingress.discoveryBots (default ['*']).
Only listed bots may mint targets, matched case-insensitively against the names getBotName resolves (registry, derived, or other). ['*'] preserves today's behavior; [] makes target creation sitemap-only site-wide. This ends the malformed-URL junk class (#81) at the source: the broken-extractor crawlers that invent those URLs are exactly the ones a deployment leaves off the list. Creation-only — serving, the demand ladder's visit signal, invalidation reenqueue, and sitemap ingestion are untouched. renderNow (explicit, operator-configured) is deliberately not gated.

Both gates sit in maybeSchedule, so a gated miss also skips the detached per-miss Target.get — on a gated combinatorial route, misses are most of the traffic. A new discovery_gated series (prerender_ops; dimensions: which gate, which bot) counts what's being held out.

3. Discovered-target purge — POST /prerender_admin/discovery-purge.
Detached, paced (ratePerSecond, default 200), owner-scoped (run on every node, like sweep-orphans — the in-flight lease check is node-local) bulk delete of targets under an absolute URL prefix whose sitemapUrl is null. dryRun defaults true (a bare start is a census). Progress on GET /prerender_admin/discovery-purge; { action: 'stop' } cancels. Two interlocks are refusals, not docs: a bare-origin prefix is always rejected, and a prefix whose matched route still allows discovery is rejected without force: true — gate first, then purge. The walk is a chunked one-sided keyset scan (the walkTargets shape) with the two-phase no-writes-under-an-open-cursor discipline from util/scan.js; deletes go through the resource class so the cascade (schedules, pages, probe baseline) holds.

4. Fix: orphan sweep now deletes through the resource class.
sweepKeyRuleOrphans destructured Target from the databases global — the raw table — so its deletes never cascaded, contrary to its own comment: schedule rows survived (each re-rendering once before self-retiring) and PrerenderedPage rows/blobs leaked permanently. Found while building the purge on the same pattern; never triggered in production (the sweep has only ever been run as a dry-run census).

5. Chore: repo lint is green again — no-unused-vars now ignores rest-sibling omissions (the console tests' fixture-downgrade idiom, red on main since v0.51.0).

Verification

• 820/820 unit tests (12 new): gate compile rules, allowlist semantics (['*']/list/[], case-insensitivity, live config), purge predicate / owner scope / lease deferral / dry-run / pacing / cancel, walkPrefix chunk+boundary behavior, both purge interlocks, and the detached start/finish lifecycle.
• Lint and format clean at root (this PR is what returns lint to green).
• No repo CI on PRs — verified locally per repo policy.

For the human reviewer

• discovery_gated counts gated misses, not denied mints (a miss on an already-known target counts too). Chosen so the gate check stays ahead of the per-miss Target.get; the metric doc says exactly this.
• The gate-first interlock refuses rather than warns. Purging an ungated prefix looks identical to success for a week, so the ordering is encoded in the API (force: true to override).
• In prefix (native) ingress mode there are no routes, so only the bot allowlist gates; the purge's route interlock passes with a null entry there.
• The orphan-sweep fix widens its blast per delete (cascade now real). Its maxDeletes/dryRun defaults are unchanged.

Review coverage

Authored by Claude (Fable). Self-review + domain pass only — no outside-model leg ran for this PR; flagging per policy so a reviewer weights the diff accordingly.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces traffic discovery gating and a discovery purge mechanism to prevent and clean up unbounded corpus growth from crawler traffic on combinatorial URL spaces. It adds route-level discoverTargets and global discoveryBots configuration options, gates target creation on bot requests, tracks gated misses with a new discovery_gated metric, and implements a paced, owner-scoped discovery-purge admin action. A compatibility issue was correctly identified in discoveredPurge.js where URL.parse is used; this should be replaced with new URL() wrapped in a try...catch block to ensure compatibility with Node 20.

[gemini-code-assist]

Avoid using URL.parse as it is not backported to Node 20 (only available in Node v22.1.0+). To ensure compatibility with Node 20, use new URL() wrapped in a try...catch block instead.

Suggested change

	const parsed = typeof urlPrefix === 'string' ? URL.parse(urlPrefix) : null;
	let parsed = null;
	if (typeof urlPrefix === 'string') {
	try {
	parsed = new URL(urlPrefix);
	} catch {}
	}

References

1. Avoid using URL.parse if the target Node.js version is below v22.1.0, as it is not backported to Node 20. Use new URL() with appropriate error handling or pre-validation instead.

[harper-joseph]

Keeping URL.parse deliberately — it is this repo's established idiom for exactly this shape (see routeClass.js#classifyUrl, which runs it on the hot serve path for every bot request, shipped many releases ago). A Node 20 runtime would already fail at ingress long before reaching this admin-only path, so the effective runtime floor here is ≥22 (the test suite's type-stripping already requires it); diverging to new URL()+try/catch in one file would trade consistency for a compatibility level the codebase doesn't actually have. Same conclusion as the identical finding on #126.