console v0.8.0: read the change probe and the discovery gate

packages/console/README.md

@@ -54,10 +54,16 @@ part of the contract:
 
 | Class      | Routes                                                                           | What happens                                                     |
 | ---------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
-| **merged** | `overview`, `analytics`, `unrouted`, `config`                                    | fanned out and summed (or, for config, compared)                 |
+| **merged** | `overview`, `analytics`, `unrouted`, `config`, `change-probe`, `discovery-purge` | fanned out and summed (or, for config, compared)                 |
 | **shared** | `pages`, `page-content`, `sitemaps`, `invalidations`, `crawl-breadth`, `metrics` | replicated data — **one** node answers, and the payload names it |
 | **single** | every POST                                                                       | writes are never fanned out                                      |
 
+The last two merged routes are owner-scoped passes rather than sums: a probe pass and a purge pass
+each cover the keys one node owns, so what the merge produces is every node's own slice side by
+side plus the running tally, and the nodes that have not run are named. One node's "deleted
+240,118" answered under a cluster label would read as done when three quarters of the keyspace has
+not been touched.
+
 A test pins every proxied GET to a class, so adding a route without deciding how it aggregates
 fails CI rather than silently answering from one node under an "all nodes" label.
 
@@ -74,7 +80,10 @@ Three properties worth knowing:
   the cluster by N.
 - **Actions are never fanned out.** A write under cluster scope lands on one node, which is
   correct for the replicated tables they touch. The routes that act on a single node's own state
-  — `reconcile`, `backlog`, `schedule` — refuse and ask for a node instead. A write whose result
+  — `reconcile`, `backlog`, `schedule`, `sweep-orphans`, `change-probe`, `discovery-purge` — refuse
+  and ask for a node instead. A probe pass refuses for a second reason on top of residency: its
+  rate cap is a promise made to whoever runs the origin, **per node**, and fanning the pass out
+  would spend it N times over. A write whose result
   is read back moments later by a panel that alarms on nodes disagreeing (today: `config-override`)
   additionally carries an envelope naming the node that accepted it and saying the rows replicate,
   so the console's own write path does not read as the failure that panel exists to raise.
@@ -146,14 +155,30 @@ browser ── same-origin (cookies, CSP 'self') ──▶ prerender-console com
 The view-by-view tour lives in the plugin README's
 [Management API](../plugin/README.md#management-api-prerender_admin) section alongside the
 API contract; the short version: **Overview** (scale, serve health, backlog shape, claim
-floor, schedule repair), **Traffic** (offload/hit-rate charts from one bounded analytics scan per
-node, freshness reported relative to each route's own render cadence, the non-hit verdicts broken
-out by what would fix them — coverage stated net of URLs the origin does not have — and a
-client-side bot filter), **Sitemaps**, **Page cache**, **Queue**
-(render/claim health and the backlog), **Nodes**, **Invalidations** (preview-first record/clear),
-**URL explainer**, **Metrics** (the live catalog), **Config**.
-
-Two of those changed shape when configuration became editable:
+floor, schedule repair, and the discovered-target purge), **Traffic** (offload/hit-rate charts from
+one bounded analytics scan per node, freshness reported relative to each route's own render cadence,
+the non-hit verdicts broken out by what would fix them — coverage stated net of URLs the origin does
+not have — the discovery gate, and a client-side bot filter), **Sitemaps**, **Page cache**,
+**Queue** (render/claim health and the backlog), **Nodes**, **Invalidations** (preview-first
+record/clear), **Change probe**, **URL explainer**, **Metrics** (the live catalog), **Config**.
+
+**Change probe is new** (plugin v0.53.0+), and it is the one freshness surface that does not measure
+pages against a cadence. It reads what the probe is actually detecting — the change rate against the
+probes that HAD a baseline, not against every probe, because a pass that is mostly seeding has
+compared nothing — beside the failure share, because those two numbers are only meaningful
+together: a probe whose endpoint has changed shape reports zero changes, triggers nothing, and is
+indistinguishable from a catalogue that is not moving. The canary's verdict is reported per node,
+since a trip is a threshold crossed against one node's own cohort, and a trip that recorded no
+invalidation says which of the three reasons it was.
+
+**The discovery gate** is split across the two views that own its halves: how much crawl traffic
+the gate is holding out of the render rotation is on Traffic (it carries a bot, so the bot filter
+applies), while the purge that removes what got in before the gate went on sits on Overview beside
+the key-rule orphan sweep — the console's other corpus-deleting action. The card states the
+gate-first interlock rather than leaving it to the plugin's 400: with the route still discovering,
+crawlers re-mint exactly what a purge removes.
+
+Two more changed shape when configuration became editable:
 
 - **Nodes is new**, and it exists because "is this node healthy" had four homes: liveness and the
   replication gap on Overview, observed status and pause intent on Queue, config divergence on
@@ -162,16 +187,16 @@ Two of those changed shape when configuration became editable:
   override layer adds — did my edit reach this node, and is this node's override subscription
   still live. A node whose subscription has died silently stops honouring every edit made from
   this console, so that is the headline of the panel rather than a footnote.
-- **Config** stopped being a JSON dump and became the searchable index of all 133 options: every
+- **Config** stopped being a JSON dump and became the searchable index of all 156 options: every
   option's default, deployed and override values, which layer won, and a filter for the questions
   operators actually arrive with (what is overridden, what differs from the repo, what is pending
   a restart). Divergence stays first, and stays the alarm it always was — a divergence at a path
   nobody overrides is still a deploy that skipped a node.
 
 Each domain view owns the options that govern the data it shows — `sitemap.*` under Sitemaps,
 `queue`/`render`/`scan` under Queue, `page`/`cacheKey` under Page cache, `analytics`/`crawlStats`
-under Traffic, `invalidation` under Invalidations — while Config remains exhaustive, so a setting
-can be found either by where it acts or by name.
+under Traffic, `invalidation` under Invalidations, `changeProbe` under Change probe — while Config
+remains exhaustive, so a setting can be found either by where it acts or by name.
 
 ## Development
 

packages/console/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender-console",
-	"version": "0.7.0",
+	"version": "0.8.0",
 	"type": "module",
 	"description": "Standalone Harper component serving the prerender management console UI, proxying to a prerender deployment's /prerender_admin API",
 	"license": "Apache-2.0",

packages/console/src/admin/app.js

@@ -19,6 +19,7 @@ import * as queue from './views/queue.js';
 import * as nodes from './views/nodes.js';
 import * as sitemaps from './views/sitemaps.js';
 import * as invalidations from './views/invalidations.js';
+import * as probe from './views/probe.js';
 import * as inspect from './views/inspect.js';
 import * as metricsref from './views/metricsref.js';
 import * as config from './views/config.js';
@@ -27,14 +28,17 @@ import { discardEdit } from './views/_configEdit.js';
 // Ordered as an operator triages: is it working (overview, traffic), what is it working on
 // (sitemaps), is the machinery healthy (queue, then the nodes running it), then actions and
 // reference. `inspect` sits below the divider because it is not a place you patrol — it is the
-// drill-down every other view hands a URL to.
+// drill-down every other view hands a URL to. The change probe sits after Invalidations because
+// that is what its canary produces: a mass change is answered with one invalidation row, not with
+// thousands of re-renders.
 const VIEWS = [
 	overview,
 	traffic,
 	sitemaps,
 	queue,
 	nodes,
 	invalidations,
+	probe,
 	null /* divider */,
 	inspect,
 	metricsref,

packages/console/src/admin/index.js

@@ -45,6 +45,7 @@ const ASSETS = new Map([
 	['views/nodes.js', ['views/nodes.js', JS]],
 	['views/sitemaps.js', ['views/sitemaps.js', JS]],
 	['views/invalidations.js', ['views/invalidations.js', JS]],
+	['views/probe.js', ['views/probe.js', JS]],
 	['views/inspect.js', ['views/inspect.js', JS]],
 	['views/metricsref.js', ['views/metricsref.js', JS]],
 	['views/config.js', ['views/config.js', JS]],

packages/console/src/admin/ui.js

@@ -88,6 +88,8 @@ export const ICONS = {
 	],
 	search: ['M13.5 13.5l3.5 3.5', 'M4 9a5 5 0 1010 0A5 5 0 004 9'],
 	refresh: ['M16.5 8A6.5 6.5 0 105 14.6', 'M16.5 3.5V8h-4.5'],
+	// A signal going out and coming back: the probe asks the origin whether anything moved.
+	probe: ['M10 12.5a1.5 1.5 0 100-3 1.5 1.5 0 100 3', 'M6.6 7.6a4.8 4.8 0 016.8 0M4 5a8.4 8.4 0 0112 0'],
 };
 
 /** The Harper mark: three stacked layers, drawn rather than shipped as a binary. */

packages/console/src/admin/views/config.js

@@ -93,6 +93,7 @@ const ELSEWHERE = {
 	sitemap: { id: 'sitemaps', label: 'Sitemaps' },
 	analytics: { id: 'traffic', label: 'Traffic' },
 	crawlStats: { id: 'traffic', label: 'Traffic' },
+	changeProbe: { id: 'probe', label: 'Change probe' },
 };
 
 /** Card titles. The dotted prefix is the fallback, and it is never wrong — only terse. */
@@ -114,6 +115,7 @@ const TITLES = {
 	queue: 'Render queue',
 	analytics: 'Analytics',
 	crawlStats: 'Crawl stats',
+	changeProbe: 'Change probe',
 };
 
 /**

packages/console/src/admin/views/overview.js

@@ -62,14 +62,19 @@ export const meta = { id: 'overview', label: 'Overview', crumb: 'overview', icon
 const DEEP_SCAN_CAP = 50_000;
 
 export async function load(ctx) {
-	const [res, analyticsRes] = await Promise.all([
+	const [res, analyticsRes, purgeRes] = await Promise.all([
 		ctx.get('overview'),
 		// The same range key Traffic and Queue default to, so all three views share one
 		// worker-cached scan instead of each paying their own.
 		ctx.get('analytics', { range: 3_600_000 }),
+		// In-memory state, no scan — the same argument the orphan sweep's last result rides on:
+		// a destructive pass that is RUNNING has to be visible on the page an operator lands on,
+		// not on one they would have to think to open.
+		ctx.get('discovery-purge'),
 	]);
 	ctx.data.overview = res.ok ? res.body : null;
 	ctx.data.analytics = analyticsRes.ok ? analyticsRes.body : null;
+	ctx.data.purge = purgeRes.ok ? purgeRes.body : null;
 	ctx.data.error = res.ok ? null : (res.body?.error ?? `Could not load the overview (${res.status})`);
 }
 
@@ -91,6 +96,7 @@ export function render(ctx) {
 		failures(ctx),
 		repair(ctx, data),
 		orphans(ctx, data),
+		discovered(ctx),
 	];
 }
 
@@ -594,6 +600,147 @@ function orphans(ctx, data) {
 	});
 }
 
+/**
+ * Targets that entered the corpus from TRAFFIC rather than from a sitemap, and the paced,
+ * owner-scoped pass that removes them.
+ *
+ * WHY THIS IS A DIFFERENT POPULATION FROM THE ORPHANS ABOVE. A key-rule orphan is a bookkeeping
+ * fault — a row nothing can ever ask for. A discovered target is a URL a crawler genuinely asked
+ * for, which the plugin then minted and has been re-rendering on cadence ever since. On a route
+ * whose URL space is combinatorial (facets, filter and sort permutations) that is unbounded: every
+ * novel combination a crawler walks becomes permanent render load, and the corpus grows faster
+ * than the fleet can keep it fresh.
+ *
+ * GATE BEFORE PURGING, and the plugin refuses the other order rather than trusting anyone to
+ * remember it: with `ingress.routes[].discoverTargets` still true on the matched route, crawlers
+ * re-mint exactly what the purge removed, so the pass is a rate-limited way to delete corpus and
+ * change nothing. The refusal is a 400 with that sentence in it, and `force` is the deliberate
+ * override.
+ */
+function discovered(ctx) {
+	const state = ctx.data.purge;
+	const clusterScope = isMerged(ctx.data.overview);
+	const busy = ctx.busy || !!state?.running;
+
+	// Kept in view scratch so the prefix survives the reload every action triggers.
+	const input = el('input', {
+		cls: 'mono',
+		type: 'text',
+		value: ctx.data.purgePrefix ?? '',
+		placeholder: 'https://www.example.com/catalog/',
+		style: { flex: '1', minWidth: '260px' },
+	});
+	const remember = () => {
+		ctx.data.purgePrefix = input.value.trim();
+		return ctx.data.purgePrefix;
+	};
+	const start = (dryRun) => {
+		const urlPrefix = remember();
+		if (!urlPrefix) return;
+		return ctx.run(() => ctx.post('discovery-purge', { urlPrefix, dryRun }));
+	};
+
+	const body = [
+		el('div', { cls: 'toolbar' }, [
+			input,
+			el('button', {
+				text: clusterScope ? 'Census (pick a node)' : 'Dry-run census',
+				disabled: busy || clusterScope,
+				title: clusterScope
+					? 'A purge walks the keys one node owns. Switch to a node to run it, and run every node to cover the keyspace.'
+					: 'Counts what a real pass would delete. Nothing is removed.',
+				onclick: () => start(true),
+			}),
+			el('button', {
+				cls: 'danger',
+				text: 'Purge discovered',
+				disabled: busy || clusterScope || !state?.startedAt || state?.dryRun === false,
+				title: !state?.startedAt
+					? 'Run the census first — this deletes what that census counted.'
+					: 'Deletes every discovered target under the prefix on this node.',
+				onclick: () => start(false),
+			}),
+			state?.running &&
+				el('button', {
+					text: 'Stop',
+					disabled: ctx.busy || clusterScope,
+					onclick: () => ctx.run(() => ctx.post('discovery-purge', { action: 'stop' })),
+				}),
+		]),
+	];
+
+	if (state?.error) body.push(el('div', { cls: 'note bad', text: `Last pass failed: ${state.error}` }));
+
+	// `startedAt` is what separates "has run" from "never run": a node that has never run answers
+	// `{ running: false }` and nothing else, and rendering that as a row of zeroes would read as a
+	// clean census.
+	const totals = state?.totals ?? state;
+	if (state?.startedAt) {
+		const stranded = (totals.discovered ?? 0) - (totals.leaseSkipped ?? 0) - (totals.deleted ?? 0);
+		body.push(
+			kv([
+				[
+					state.ranNodes > 1 ? `Oldest of ${state.ranNodes} passes` : state.running ? 'Started' : 'Last pass',
+					`${state.running ? ago(state.startedAt) + ' — still running' : state.finishedAt ? ago(state.finishedAt) : 'unknown'}` +
+						`${state.dryRun ? ' (census — nothing deleted)' : ''}${state.canceled ? ' · stopped early' : ''}`,
+				],
+				['Prefix', state.urlPrefix ?? (state.urlPrefixes?.length ? state.urlPrefixes.join(', ') : '—')],
+				['Rows examined', num(totals.examined)],
+				[state.ranNodes > 1 ? 'Owned across nodes' : 'Owned by this node', num(totals.owned)],
+				['Discovered (never in a sitemap)', totals.discovered ? pill(num(totals.discovered), 'warn') : pill('0', 'ok')],
+				['Deleted', state.dryRun ? muted('none — census') : num(totals.deleted)],
+				totals.leaseSkipped ? ['Deferred as in-flight', pill(num(totals.leaseSkipped), '')] : null,
+				state.canceled && stranded > 0
+					? ['Not reached', pill(`~${num(stranded)} left under this prefix`, 'warn')]
+					: null,
+			])
+		);
+	} else {
+		body.push(muted('No purge has run on this node since startup. It has no timer — it runs when you run it.'));
+	}
+
+	if (state?.unrunNodes?.length) {
+		body.push(
+			el('div', { cls: 'note warn' }, [
+				`Never run on ${state.unrunNodes.join(', ')} — those nodes' keys are not represented above, and their ` +
+					'discovered targets are still rendering.',
+			])
+		);
+	}
+
+	if (state?.urlPrefixes?.length > 1) {
+		body.push(
+			el('div', { cls: 'note warn' }, [
+				'The nodes last ran DIFFERENT prefixes, so the totals above add up two populations. The per-node ' +
+					'figures are the answer until every node has run the same one.',
+			])
+		);
+	}
+
+	body.push(
+		el('p', { cls: 'muted', style: { margin: '12px 0 0' } }, [
+			'Gate the route first — set ',
+			el('code', { text: 'discoverTargets: false' }),
+			' on it under Request ingestion — or crawlers re-mint what this removes; the plugin refuses an ' +
+				'ungated prefix for that reason, and refuses a bare origin always. A sitemap-declared URL is never ' +
+				'touched whatever its traffic looks like: the sitemap is the operator’s statement of what should ' +
+				'exist. ',
+			link('See how much traffic the gate is holding out →', () => ctx.go('traffic')),
+		])
+	);
+
+	return card('Discovered targets', {
+		head: [
+			pill('manual — no timer'),
+			state?.running &&
+				pill(state.runningOn?.length ? `running on ${state.runningOn.join(', ')}` : 'running now', 'warn'),
+			spacer(),
+			state?.ratePerSecond ? muted(`${num(state.ratePerSecond)}/s`) : null,
+		],
+		body,
+	});
+}
+
 /**
  * How long a node has HELD its current queue status — never "how long since it reported".
  *