diff --git a/package-lock.json b/package-lock.json index 52af2e2c..871ae3cb 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8556,7 +8556,7 @@ }, "packages/console": { "name": "@harperfast/prerender-console", - "version": "0.7.0", + "version": "0.8.0", "license": "Apache-2.0", "dependencies": { "undici": "^7.18.2" diff --git a/packages/console/README.md b/packages/console/README.md index ea8c913f..f100b496 100644 --- a/packages/console/README.md +++ b/packages/console/README.md @@ -54,10 +54,16 @@ part of the contract: | Class | Routes | What happens | | ---------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -| **merged** | `overview`, `analytics`, `unrouted`, `config` | fanned out and summed (or, for config, compared) | +| **merged** | `overview`, `analytics`, `unrouted`, `config`, `change-probe`, `discovery-purge` | fanned out and summed (or, for config, compared) | | **shared** | `pages`, `page-content`, `sitemaps`, `invalidations`, `crawl-breadth`, `metrics` | replicated data — **one** node answers, and the payload names it | | **single** | every POST | writes are never fanned out | +The last two merged routes are owner-scoped passes rather than sums: a probe pass and a purge pass +each cover the keys one node owns, so what the merge produces is every node's own slice side by +side plus the running tally, and the nodes that have not run are named. One node's "deleted +240,118" answered under a cluster label would read as done when three quarters of the keyspace has +not been touched. + A test pins every proxied GET to a class, so adding a route without deciding how it aggregates fails CI rather than silently answering from one node under an "all nodes" label. @@ -74,7 +80,10 @@ Three properties worth knowing: the cluster by N. - **Actions are never fanned out.** A write under cluster scope lands on one node, which is correct for the replicated tables they touch. The routes that act on a single node's own state - — `reconcile`, `backlog`, `schedule` — refuse and ask for a node instead. A write whose result + — `reconcile`, `backlog`, `schedule`, `sweep-orphans`, `change-probe`, `discovery-purge` — refuse + and ask for a node instead. A probe pass refuses for a second reason on top of residency: its + rate cap is a promise made to whoever runs the origin, **per node**, and fanning the pass out + would spend it N times over. A write whose result is read back moments later by a panel that alarms on nodes disagreeing (today: `config-override`) additionally carries an envelope naming the node that accepted it and saying the rows replicate, so the console's own write path does not read as the failure that panel exists to raise. @@ -146,14 +155,30 @@ browser ── same-origin (cookies, CSP 'self') ──▶ prerender-console com The view-by-view tour lives in the plugin README's [Management API](../plugin/README.md#management-api-prerender_admin) section alongside the API contract; the short version: **Overview** (scale, serve health, backlog shape, claim -floor, schedule repair), **Traffic** (offload/hit-rate charts from one bounded analytics scan per -node, freshness reported relative to each route's own render cadence, the non-hit verdicts broken -out by what would fix them — coverage stated net of URLs the origin does not have — and a -client-side bot filter), **Sitemaps**, **Page cache**, **Queue** -(render/claim health and the backlog), **Nodes**, **Invalidations** (preview-first record/clear), -**URL explainer**, **Metrics** (the live catalog), **Config**. - -Two of those changed shape when configuration became editable: +floor, schedule repair, and the discovered-target purge), **Traffic** (offload/hit-rate charts from +one bounded analytics scan per node, freshness reported relative to each route's own render cadence, +the non-hit verdicts broken out by what would fix them — coverage stated net of URLs the origin does +not have — the discovery gate, and a client-side bot filter), **Sitemaps**, **Page cache**, +**Queue** (render/claim health and the backlog), **Nodes**, **Invalidations** (preview-first +record/clear), **Change probe**, **URL explainer**, **Metrics** (the live catalog), **Config**. + +**Change probe is new** (plugin v0.53.0+), and it is the one freshness surface that does not measure +pages against a cadence. It reads what the probe is actually detecting — the change rate against the +probes that HAD a baseline, not against every probe, because a pass that is mostly seeding has +compared nothing — beside the failure share, because those two numbers are only meaningful +together: a probe whose endpoint has changed shape reports zero changes, triggers nothing, and is +indistinguishable from a catalogue that is not moving. The canary's verdict is reported per node, +since a trip is a threshold crossed against one node's own cohort, and a trip that recorded no +invalidation says which of the three reasons it was. + +**The discovery gate** is split across the two views that own its halves: how much crawl traffic +the gate is holding out of the render rotation is on Traffic (it carries a bot, so the bot filter +applies), while the purge that removes what got in before the gate went on sits on Overview beside +the key-rule orphan sweep — the console's other corpus-deleting action. The card states the +gate-first interlock rather than leaving it to the plugin's 400: with the route still discovering, +crawlers re-mint exactly what a purge removes. + +Two more changed shape when configuration became editable: - **Nodes is new**, and it exists because "is this node healthy" had four homes: liveness and the replication gap on Overview, observed status and pause intent on Queue, config divergence on @@ -162,7 +187,7 @@ Two of those changed shape when configuration became editable: override layer adds — did my edit reach this node, and is this node's override subscription still live. A node whose subscription has died silently stops honouring every edit made from this console, so that is the headline of the panel rather than a footnote. -- **Config** stopped being a JSON dump and became the searchable index of all 133 options: every +- **Config** stopped being a JSON dump and became the searchable index of all 156 options: every option's default, deployed and override values, which layer won, and a filter for the questions operators actually arrive with (what is overridden, what differs from the repo, what is pending a restart). Divergence stays first, and stays the alarm it always was — a divergence at a path @@ -170,8 +195,8 @@ Two of those changed shape when configuration became editable: Each domain view owns the options that govern the data it shows — `sitemap.*` under Sitemaps, `queue`/`render`/`scan` under Queue, `page`/`cacheKey` under Page cache, `analytics`/`crawlStats` -under Traffic, `invalidation` under Invalidations — while Config remains exhaustive, so a setting -can be found either by where it acts or by name. +under Traffic, `invalidation` under Invalidations, `changeProbe` under Change probe — while Config +remains exhaustive, so a setting can be found either by where it acts or by name. ## Development diff --git a/packages/console/package.json b/packages/console/package.json index ab258b16..da9b8016 100644 --- a/packages/console/package.json +++ b/packages/console/package.json @@ -1,6 +1,6 @@ { "name": "@harperfast/prerender-console", - "version": "0.7.0", + "version": "0.8.0", "type": "module", "description": "Standalone Harper component serving the prerender management console UI, proxying to a prerender deployment's /prerender_admin API", "license": "Apache-2.0", diff --git a/packages/console/src/admin/app.js b/packages/console/src/admin/app.js index fde55035..89e96515 100644 --- a/packages/console/src/admin/app.js +++ b/packages/console/src/admin/app.js @@ -19,6 +19,7 @@ import * as queue from './views/queue.js'; import * as nodes from './views/nodes.js'; import * as sitemaps from './views/sitemaps.js'; import * as invalidations from './views/invalidations.js'; +import * as probe from './views/probe.js'; import * as inspect from './views/inspect.js'; import * as metricsref from './views/metricsref.js'; import * as config from './views/config.js'; @@ -27,7 +28,9 @@ import { discardEdit } from './views/_configEdit.js'; // Ordered as an operator triages: is it working (overview, traffic), what is it working on // (sitemaps), is the machinery healthy (queue, then the nodes running it), then actions and // reference. `inspect` sits below the divider because it is not a place you patrol — it is the -// drill-down every other view hands a URL to. +// drill-down every other view hands a URL to. The change probe sits after Invalidations because +// that is what its canary produces: a mass change is answered with one invalidation row, not with +// thousands of re-renders. const VIEWS = [ overview, traffic, @@ -35,6 +38,7 @@ const VIEWS = [ queue, nodes, invalidations, + probe, null /* divider */, inspect, metricsref, diff --git a/packages/console/src/admin/index.js b/packages/console/src/admin/index.js index a6da2a67..aa990864 100644 --- a/packages/console/src/admin/index.js +++ b/packages/console/src/admin/index.js @@ -45,6 +45,7 @@ const ASSETS = new Map([ ['views/nodes.js', ['views/nodes.js', JS]], ['views/sitemaps.js', ['views/sitemaps.js', JS]], ['views/invalidations.js', ['views/invalidations.js', JS]], + ['views/probe.js', ['views/probe.js', JS]], ['views/inspect.js', ['views/inspect.js', JS]], ['views/metricsref.js', ['views/metricsref.js', JS]], ['views/config.js', ['views/config.js', JS]], diff --git a/packages/console/src/admin/ui.js b/packages/console/src/admin/ui.js index 8603a3c5..fc0dd877 100644 --- a/packages/console/src/admin/ui.js +++ b/packages/console/src/admin/ui.js @@ -88,6 +88,8 @@ export const ICONS = { ], search: ['M13.5 13.5l3.5 3.5', 'M4 9a5 5 0 1010 0A5 5 0 004 9'], refresh: ['M16.5 8A6.5 6.5 0 105 14.6', 'M16.5 3.5V8h-4.5'], + // A signal going out and coming back: the probe asks the origin whether anything moved. + probe: ['M10 12.5a1.5 1.5 0 100-3 1.5 1.5 0 100 3', 'M6.6 7.6a4.8 4.8 0 016.8 0M4 5a8.4 8.4 0 0112 0'], }; /** The Harper mark: three stacked layers, drawn rather than shipped as a binary. */ diff --git a/packages/console/src/admin/views/config.js b/packages/console/src/admin/views/config.js index 6c746a77..b8f98437 100644 --- a/packages/console/src/admin/views/config.js +++ b/packages/console/src/admin/views/config.js @@ -93,6 +93,7 @@ const ELSEWHERE = { sitemap: { id: 'sitemaps', label: 'Sitemaps' }, analytics: { id: 'traffic', label: 'Traffic' }, crawlStats: { id: 'traffic', label: 'Traffic' }, + changeProbe: { id: 'probe', label: 'Change probe' }, }; /** Card titles. The dotted prefix is the fallback, and it is never wrong — only terse. */ @@ -114,6 +115,7 @@ const TITLES = { queue: 'Render queue', analytics: 'Analytics', crawlStats: 'Crawl stats', + changeProbe: 'Change probe', }; /** diff --git a/packages/console/src/admin/views/overview.js b/packages/console/src/admin/views/overview.js index 0a262b2e..27acb838 100644 --- a/packages/console/src/admin/views/overview.js +++ b/packages/console/src/admin/views/overview.js @@ -62,14 +62,19 @@ export const meta = { id: 'overview', label: 'Overview', crumb: 'overview', icon const DEEP_SCAN_CAP = 50_000; export async function load(ctx) { - const [res, analyticsRes] = await Promise.all([ + const [res, analyticsRes, purgeRes] = await Promise.all([ ctx.get('overview'), // The same range key Traffic and Queue default to, so all three views share one // worker-cached scan instead of each paying their own. ctx.get('analytics', { range: 3_600_000 }), + // In-memory state, no scan — the same argument the orphan sweep's last result rides on: + // a destructive pass that is RUNNING has to be visible on the page an operator lands on, + // not on one they would have to think to open. + ctx.get('discovery-purge'), ]); ctx.data.overview = res.ok ? res.body : null; ctx.data.analytics = analyticsRes.ok ? analyticsRes.body : null; + ctx.data.purge = purgeRes.ok ? purgeRes.body : null; ctx.data.error = res.ok ? null : (res.body?.error ?? `Could not load the overview (${res.status})`); } @@ -91,6 +96,7 @@ export function render(ctx) { failures(ctx), repair(ctx, data), orphans(ctx, data), + discovered(ctx), ]; } @@ -594,6 +600,147 @@ function orphans(ctx, data) { }); } +/** + * Targets that entered the corpus from TRAFFIC rather than from a sitemap, and the paced, + * owner-scoped pass that removes them. + * + * WHY THIS IS A DIFFERENT POPULATION FROM THE ORPHANS ABOVE. A key-rule orphan is a bookkeeping + * fault — a row nothing can ever ask for. A discovered target is a URL a crawler genuinely asked + * for, which the plugin then minted and has been re-rendering on cadence ever since. On a route + * whose URL space is combinatorial (facets, filter and sort permutations) that is unbounded: every + * novel combination a crawler walks becomes permanent render load, and the corpus grows faster + * than the fleet can keep it fresh. + * + * GATE BEFORE PURGING, and the plugin refuses the other order rather than trusting anyone to + * remember it: with `ingress.routes[].discoverTargets` still true on the matched route, crawlers + * re-mint exactly what the purge removed, so the pass is a rate-limited way to delete corpus and + * change nothing. The refusal is a 400 with that sentence in it, and `force` is the deliberate + * override. + */ +function discovered(ctx) { + const state = ctx.data.purge; + const clusterScope = isMerged(ctx.data.overview); + const busy = ctx.busy || !!state?.running; + + // Kept in view scratch so the prefix survives the reload every action triggers. + const input = el('input', { + cls: 'mono', + type: 'text', + value: ctx.data.purgePrefix ?? '', + placeholder: 'https://www.example.com/catalog/', + style: { flex: '1', minWidth: '260px' }, + }); + const remember = () => { + ctx.data.purgePrefix = input.value.trim(); + return ctx.data.purgePrefix; + }; + const start = (dryRun) => { + const urlPrefix = remember(); + if (!urlPrefix) return; + return ctx.run(() => ctx.post('discovery-purge', { urlPrefix, dryRun })); + }; + + const body = [ + el('div', { cls: 'toolbar' }, [ + input, + el('button', { + text: clusterScope ? 'Census (pick a node)' : 'Dry-run census', + disabled: busy || clusterScope, + title: clusterScope + ? 'A purge walks the keys one node owns. Switch to a node to run it, and run every node to cover the keyspace.' + : 'Counts what a real pass would delete. Nothing is removed.', + onclick: () => start(true), + }), + el('button', { + cls: 'danger', + text: 'Purge discovered', + disabled: busy || clusterScope || !state?.startedAt || state?.dryRun === false, + title: !state?.startedAt + ? 'Run the census first — this deletes what that census counted.' + : 'Deletes every discovered target under the prefix on this node.', + onclick: () => start(false), + }), + state?.running && + el('button', { + text: 'Stop', + disabled: ctx.busy || clusterScope, + onclick: () => ctx.run(() => ctx.post('discovery-purge', { action: 'stop' })), + }), + ]), + ]; + + if (state?.error) body.push(el('div', { cls: 'note bad', text: `Last pass failed: ${state.error}` })); + + // `startedAt` is what separates "has run" from "never run": a node that has never run answers + // `{ running: false }` and nothing else, and rendering that as a row of zeroes would read as a + // clean census. + const totals = state?.totals ?? state; + if (state?.startedAt) { + const stranded = (totals.discovered ?? 0) - (totals.leaseSkipped ?? 0) - (totals.deleted ?? 0); + body.push( + kv([ + [ + state.ranNodes > 1 ? `Oldest of ${state.ranNodes} passes` : state.running ? 'Started' : 'Last pass', + `${state.running ? ago(state.startedAt) + ' — still running' : state.finishedAt ? ago(state.finishedAt) : 'unknown'}` + + `${state.dryRun ? ' (census — nothing deleted)' : ''}${state.canceled ? ' · stopped early' : ''}`, + ], + ['Prefix', state.urlPrefix ?? (state.urlPrefixes?.length ? state.urlPrefixes.join(', ') : '—')], + ['Rows examined', num(totals.examined)], + [state.ranNodes > 1 ? 'Owned across nodes' : 'Owned by this node', num(totals.owned)], + ['Discovered (never in a sitemap)', totals.discovered ? pill(num(totals.discovered), 'warn') : pill('0', 'ok')], + ['Deleted', state.dryRun ? muted('none — census') : num(totals.deleted)], + totals.leaseSkipped ? ['Deferred as in-flight', pill(num(totals.leaseSkipped), '')] : null, + state.canceled && stranded > 0 + ? ['Not reached', pill(`~${num(stranded)} left under this prefix`, 'warn')] + : null, + ]) + ); + } else { + body.push(muted('No purge has run on this node since startup. It has no timer — it runs when you run it.')); + } + + if (state?.unrunNodes?.length) { + body.push( + el('div', { cls: 'note warn' }, [ + `Never run on ${state.unrunNodes.join(', ')} — those nodes' keys are not represented above, and their ` + + 'discovered targets are still rendering.', + ]) + ); + } + + if (state?.urlPrefixes?.length > 1) { + body.push( + el('div', { cls: 'note warn' }, [ + 'The nodes last ran DIFFERENT prefixes, so the totals above add up two populations. The per-node ' + + 'figures are the answer until every node has run the same one.', + ]) + ); + } + + body.push( + el('p', { cls: 'muted', style: { margin: '12px 0 0' } }, [ + 'Gate the route first — set ', + el('code', { text: 'discoverTargets: false' }), + ' on it under Request ingestion — or crawlers re-mint what this removes; the plugin refuses an ' + + 'ungated prefix for that reason, and refuses a bare origin always. A sitemap-declared URL is never ' + + 'touched whatever its traffic looks like: the sitemap is the operator’s statement of what should ' + + 'exist. ', + link('See how much traffic the gate is holding out →', () => ctx.go('traffic')), + ]) + ); + + return card('Discovered targets', { + head: [ + pill('manual — no timer'), + state?.running && + pill(state.runningOn?.length ? `running on ${state.runningOn.join(', ')}` : 'running now', 'warn'), + spacer(), + state?.ratePerSecond ? muted(`${num(state.ratePerSecond)}/s`) : null, + ], + body, + }); +} + /** * How long a node has HELD its current queue status — never "how long since it reported". * diff --git a/packages/console/src/admin/views/probe.js b/packages/console/src/admin/views/probe.js new file mode 100644 index 00000000..d4dc1d2e --- /dev/null +++ b/packages/console/src/admin/views/probe.js @@ -0,0 +1,653 @@ +/** + * Change probe: whether re-rendering is being driven by what the origin actually says, and what + * it is costing to find out. + * + * WHY IT IS ITS OWN VIEW. Every other freshness surface in this console measures a page against a + * CADENCE — the interval someone guessed and the fleet then keeps. The probe replaces the guess: + * it asks a cheap endpoint (or the page's own JSON-LD offers) whether the fields bots care about + * moved, and files a re-render only when they did. That makes it the one subsystem whose health + * question is not "are we keeping up" but "is what we are detecting real" — a probe that quietly + * stopped extracting anything reports zero changes, triggers nothing, and looks exactly like a + * catalogue that is not moving. So the failure share sits beside the change rate on the same card, + * because one is only readable next to the other. + * + * TWO CADENCES, AND THE PANELS KEEP THEM APART. The SWEEP walks this node's whole owned slice on + * a long interval and catches per-URL drift; the CANARY probes a small fixed cohort every few + * minutes and exists for the event a sweep structurally cannot see in time — a promotion that + * reprices most of a catalogue at once. Their numbers are never added together: a sweep pass and a + * canary pass have different denominators, different populations, and different consequences (a + * per-URL revalidate versus one bulk invalidation row). + * + * EVERYTHING HERE IS OWNER-SCOPED. A node probes only the URLs it owns, so a cluster figure is + * the sum of every node's own slice and a node that has not swept contributes zero to it — which + * looks precisely like a node that swept and found nothing. The merge names the unswept nodes and + * this view prints them, for the same reason the orphan sweep does. + */ + +import { ago, card, duration, el, ICONS, kv, link, muted, note, num, pct, pill, spacer, stat, table } from '../ui.js'; +import { + emptyNote, + fmtCount, + isMerged, + legend, + pick, + rangePicker, + ratioOf, + scanFooter, + scopeLabel, + segmented, + stackBy, + stackedBars, + sumValues, + windowEmpty, +} from '../charts.js'; +import { appliedNote, editTray, loadConfig, settingsCard } from './_configEdit.js'; + +export const meta = { id: 'probe', label: 'Change probe', crumb: 'change probe', icon: ICONS.probe }; + +// A sweep interval's worth by default. The pass counters are emitted ONCE PER FINISHED PASS, so +// the 1h window every other view shares would usually contain canary passes and no sweep at all — +// and an empty panel on a healthy deployment is how a signal stops being read. +const RANGES = [ + { label: '1h', ms: 3_600_000 }, + { label: '6h', ms: 6 * 3_600_000 }, + { label: '24h', ms: 24 * 3_600_000 }, +]; + +/** The pass counters, in the order a pass produces them. */ +const OUTCOMES = [ + ['probed', 'Probes', '#3d8cff'], + ['seeded', 'Seeded', '#8a93a6'], + ['changed', 'Changed', '#f0a02a'], + ['triggered', 'Triggered', '#10a87e'], + ['deferred', 'Deferred', '#9d6bff'], + ['failed', 'Failed', '#e0566f'], +]; + +const OUTCOME_COLOR = Object.fromEntries(OUTCOMES.map(([key, , color]) => [`probe_${key}`, color])); +const OUTCOME_LABEL = Object.fromEntries(OUTCOMES.map(([key, label]) => [`probe_${key}`, label])); + +/** >50% of probes failing is the endpoint-changed-shape signature the plugin logs loudly about. */ +const FAILURE_ALARM = 0.5; + +export async function load(ctx) { + ctx.data.rangeMs ??= 24 * 3_600_000; + ctx.data.runMode ??= 'config'; + const [statusRes, analyticsRes] = await Promise.all([ + ctx.get('change-probe'), + ctx.get('analytics', { range: ctx.data.rangeMs }), + loadConfig(ctx), + ]); + ctx.data.status = statusRes.ok ? statusRes.body : null; + ctx.data.analytics = analyticsRes.ok ? analyticsRes.body : null; + ctx.data.error = statusRes.ok ? null : (statusRes.body?.error ?? `Could not read probe status (${statusRes.status})`); +} + +export function render(ctx) { + const status = ctx.data.status; + + const head = el('div', { cls: 'view-head' }, [ + el('span', { cls: 'eyebrow', text: 'Change probe' }), + spacer(), + rangePicker(RANGES, ctx.data.rangeMs, (ms) => { + ctx.data.rangeMs = ms; + ctx.reload(); + }), + el('button', { text: 'Refresh', disabled: ctx.busy, onclick: () => ctx.reload() }), + ]); + + // The settings ride along even when the status read failed: `changeProbe.enabled` is the + // likeliest reason this page has nothing on it, and the card that flips it belongs on the + // screen reporting the emptiness rather than a view away. + const knobs = [settings(ctx), editTray(ctx)]; + + if (!status) { + return [head, el('div', { cls: 'note bad', text: ctx.data.error ?? 'No probe status.' }), ...knobs]; + } + + return [ + head, + appliedNote(ctx), + state(ctx, status), + drift(ctx), + sweepCard(ctx, status), + canaryCard(ctx, status), + nodeTable(status), + ...knobs, + ]; +} + +// ---------------------------------------------------------------- state & actions + +/** + * What is armed, what it is probing, and the two buttons that run a pass by hand. + * + * THE RUN MODE IS EXPLICIT because `dryRun` is the difference between measuring and acting. The + * plugin defaults it ON and a POST that omits it inherits the configured value, so a button + * labelled only "Run sweep" means different things on two deployments. The picker names which of + * the two this click will be, and the pill beside it says what the config would do on its own — + * the timers use that value, not this one. + */ +function state(ctx, status) { + const clusterScope = isMerged(status); + const forcedDryRun = ctx.data.runMode === 'dry'; + const run = (action) => ctx.run(() => ctx.post('change-probe', forcedDryRun ? { action, dryRun: true } : { action })); + + const rules = status.rules ?? []; + const body = []; + + if (!status.enabled) { + body.push( + note('bad', [ + el('code', { text: 'changeProbe.enabled' }), + status.disabledOn?.length + ? ` is false on ${status.disabledOn.join(', ')}. The URLs those nodes own are never probed, so ` + + 'their pages fall back to interval-only freshness while every number on this page keeps ' + + 'looking healthy — the counters below simply do not include them.' + : ' is false. Nothing is probed and nothing here will move; the settings card below turns it on.', + ]) + ); + } else if (!rules.length) { + body.push( + note('warn', [ + 'The probe is enabled and no ', + el('code', { text: 'changeProbe.rules' }), + ' match anything, so no timer is armed. A rule names the path pattern to claim and where to read ' + + 'the fields that matter; without one there is nothing to compare.', + ]) + ); + } + + if (status.rulesDiverge) { + body.push( + note('bad', [ + 'The nodes do not agree on the rule list — this is a deploy or an override that did not reach every ' + + 'node, and the rules shown are one node’s. ', + link('Config →', () => ctx.go('config')), + ]) + ); + } + + body.push( + kv([ + ['Sweep', status.sweep?.armedInterval ? `every ${duration(status.sweep.armedInterval)}` : 'not armed'], + [ + 'Canary', + status.canary?.armedInterval + ? `every ${duration(status.canary.armedInterval)}` + : status.enabled + ? 'disabled (canary.interval is 0)' + : 'not armed', + ], + ['Rules', num(rules.length)], + ]) + ); + + if (rules.length) { + body.push( + table( + ['rule', 'path pattern', 'source', 'invalidate scope on a canary trip'], + rules.map((rule) => + el('tr', null, [ + el('td', { cls: 'mono', text: rule.label ?? '—' }), + el('td', { cls: 'mono truncate', text: rule.pathPattern ?? '—', title: rule.pathPattern ?? '' }), + el('td', null, [pill(rule.source ?? 'document', rule.source === 'request' ? 'info' : '')]), + el('td', null, [ + rule.invalidateScope ? el('span', { cls: 'mono', text: rule.invalidateScope }) : muted('detection only'), + ]), + ]) + ) + ) + ); + } + + body.push( + el('p', { cls: 'muted chart-note' }, [ + status.ownerScopeNote ?? + 'Probes only the URLs this node owns; every node sweeps its own slice. ' + + 'A probe is origin backend work — the rate cap is a promise to whoever runs it, per node, which ' + + 'is why a pass cannot be started for the whole cluster at once.', + ]) + ); + + return card(`Probe state — ${scopeLabel(status)}`, { + head: [ + status.enabled ? pill('enabled', 'ok') : pill('disabled', 'bad'), + status.dryRun + ? pill('dry run — detects, triggers nothing', 'warn') + : pill( + status.liveOn?.length && status.liveOn.length < (status.byNode?.length ?? 0) + ? 'live on some nodes' + : 'live', + 'ok' + ), + status.sweep?.running && pill('sweep running', 'info'), + status.canary?.running && pill('canary running', 'info'), + spacer(), + segmented( + [ + { label: 'as configured', value: 'config', title: 'Inherit changeProbe.dryRun — what the timers use.' }, + { label: 'force dry run', value: 'dry', title: 'Probe and re-baseline without triggering or invalidating.' }, + ], + ctx.data.runMode, + (value) => { + ctx.data.runMode = value; + ctx.render(); + } + ), + el('button', { + text: clusterScope ? 'Sweep (pick a node)' : status.sweep?.running ? 'Sweep running…' : 'Run sweep', + disabled: ctx.busy || clusterScope || !status.enabled || !!status.sweep?.running, + title: clusterScope + ? 'A pass covers the URLs one node owns, at that node’s probe rate. Switch to a node to run it there.' + : null, + onclick: () => run('sweep'), + }), + el('button', { + text: clusterScope ? 'Canary (pick a node)' : status.canary?.running ? 'Canary running…' : 'Run canary', + disabled: ctx.busy || clusterScope || !status.enabled || !!status.canary?.running, + title: clusterScope ? 'The cohort is built from the keys one node owns. Switch to a node.' : null, + onclick: () => run('canary'), + }), + ], + body, + }); +} + +// ---------------------------------------------------------------- the measured drift + +/** + * The change rate the probe is actually measuring, from the analytics window. + * + * THE DENOMINATOR IS `probed`, NOT the corpus. A pass probes what its rules matched among the + * rows this node owns, and `changed / probed` is the fraction of THOSE that moved — the number a + * dry-run week exists to produce and the one that argues for (or against) raising a route's + * render interval. `seeded` is excluded from nothing and included in `probed`: a first + * observation had nothing to compare against, so a pass that is mostly seeding reports a low + * change rate that means "no baseline yet", which is why it gets its own tile. + */ +function drift(ctx) { + const data = ctx.data.analytics; + if (!data) return card('Probe passes', { body: [note('bad', ['The analytics window did not load.'])] }); + if (data.available === false) { + return card('Probe passes', { + body: [ + note('', [ + 'Analytics is not available on this node, so the pass counters cannot be read. The last-pass ', + 'numbers below come from the probe’s own in-memory state and are still current.', + ]), + ], + }); + } + + const combos = pick(data, 'prerender_ops', (s) => typeof s.path === 'string' && s.path.startsWith('probe_')); + if (windowEmpty(data) || !combos.length) { + return card(`Probe passes — ${scopeLabel(data)}`, { + body: [ + emptyNote('change-probe', data), + note('', [ + 'These counters are emitted once per FINISHED pass. With a sweep every few hours and a window ', + 'of ', + duration(ctx.data.rangeMs), + ', an empty panel can simply mean no pass has completed inside it — widen the range before ', + 'reading it as "nothing is probing".', + ]), + ], + foot: [scanFooter(data)], + }); + } + + // `total` is COUNTER-ONLY in these rows and `count` is the number of PASSES, not of probes — + // the recorded value is what a pass counted, so the sum is Σ(mean × count). + const totalOf = (series) => sumValues(combos.filter((s) => s.path === `probe_${series}`)); + const probed = totalOf('probed'); + const changed = totalOf('changed'); + const failed = totalOf('failed'); + const seeded = totalOf('seeded'); + const deferred = totalOf('deferred'); + const triggered = totalOf('triggered'); + const trips = totalOf('canary_trip'); + const invalidated = totalOf('invalidated'); + + // Compared = probes that had a baseline to compare against. Seeds and failures had none, so + // including them in the denominator understates the drift rate by exactly the seeding backlog. + const compared = Math.max(0, probed - seeded - failed); + // Through `ratioOf` like every other ÷ figure in this console, and NOT because these two can be + // null — `sumValues` reduces from 0 over finite products, so they are always numbers. It is that + // the guard belongs at the site by convention rather than by an argument a reader has to + // reconstruct from the helper's contract. Compared explicitly, so nothing rests on `null > 0.5`. + const failureShare = ratioOf(failed, probed); + const failing = failureShare !== null && failureShare > FAILURE_ALARM; + + const bucketCount = data.bucketCount ?? 0; + const { keys, stacks } = stackBy( + combos.filter((s) => OUTCOME_LABEL[s.path]), + 'path', + bucketCount, + { values: true } + ); + + return card(`Probe passes — ${scopeLabel(data)}, last ${duration(data.rangeMs ?? ctx.data.rangeMs)}`, { + head: [ + failing ? pill('probe failures dominate', 'bad') : null, + spacer(), + legend(keys.map((key) => ({ label: OUTCOME_LABEL[key], color: OUTCOME_COLOR[key] }))), + ], + body: [ + failing && + note('bad', [ + `${pct(failed, probed)} of probes failed. That is the shape a replatformed origin makes: the ` + + 'endpoint or the markup a rule was written against has changed, every failed probe leaves the ' + + 'stored signature untouched, and those pages are silently back on interval-only freshness. A ' + + 'failure never triggers and never re-baselines, so nothing else in this console will move.', + ]), + el('div', { cls: 'stats' }, [ + stat('Probes', fmtCount(probed), 'attempts across every finished pass'), + stat('Changed', pct(changed, compared), `${fmtCount(changed)} of ${fmtCount(compared)} compared`), + stat( + 'Triggered', + fmtCount(triggered), + deferred ? `${fmtCount(deferred)} deferred past the cap` : 'per-URL re-renders filed' + ), + stat('Seeded', fmtCount(seeded), 'first observation — nothing to compare yet'), + stat('Failed', pct(failed, probed), `${fmtCount(failed)} of ${fmtCount(probed)}`, { warn: failing }), + stat('Canary trips', fmtCount(trips), `${fmtCount(invalidated)} recorded an invalidation`), + ]), + keys.length + ? stackedBars(data, keys, stacks, (key) => OUTCOME_COLOR[key] ?? '#8a93a6', { format: fmtCount }) + : null, + trips > invalidated && + note('warn', [ + `${fmtCount(trips - invalidated)} canary trip${trips - invalidated === 1 ? '' : 's'} recorded no ` + + 'invalidation. That is a dry run, a scope still inside its holdoff, or a rule whose ' + + 'invalidateScope names no configured route — the plugin log line says which of the three.', + ]), + el('p', { cls: 'muted chart-note' }, [ + 'One emit per finished pass, sweep and canary alike, so the bars are passes and not probes — a tall ', + 'bar is a pass that landed in that bucket, not a busier minute. “Changed” is measured against the ', + 'probes that HAD a baseline; seeds and failures are excluded from that denominator because neither ', + 'compared anything.', + ]), + ], + foot: [scanFooter(data)], + }); +} + +// ---------------------------------------------------------------- the two passes + +function sweepCard(ctx, status) { + const sweep = status.sweep ?? {}; + const last = sweep.lastRun; + const body = []; + + if (sweep.unsweptNodes?.length) { + body.push( + note('warn', [ + `No sweep has finished on ${sweep.unsweptNodes.join(', ')} since startup — the URLs those nodes own ` + + 'are not represented in the counters below, so every figure here covers less of the corpus than ' + + 'it appears to.', + ]) + ); + } + + // The merge takes the FIRST error it finds across nodes, so an error and a full set of counters + // can arrive together: three nodes swept and one threw. Treating the error as the whole answer + // would drop the three good slices, and treating the counters as the whole answer would hide + // that a quarter of the keyspace was not covered. Both are shown, and the error says so. + const hasCounters = Number.isFinite(last?.probed); + if (last?.error) { + body.push( + note('bad', [ + `Last sweep failed: ${last.error}`, + hasCounters ? ' — the counters below cover only the passes that did finish.' : '', + ]) + ); + } + if (hasCounters) { + const compared = Math.max(0, (last.probed ?? 0) - (last.seeded ?? 0) - (last.failed ?? 0)); + body.push( + kv([ + [ + last.nodes > 1 ? `Oldest of ${last.nodes} sweeps` : 'Last sweep', + `${last.finishedAt ? ago(last.finishedAt) : 'unknown'}${last.dryRun ? ' (dry run — nothing triggered)' : ''}`, + ], + ['Rows examined', num(last.examined)], + [last.nodes > 1 ? 'Owned across nodes' : 'Owned by this node', num(last.owned)], + ['Matched a rule', num(last.matched)], + ['Probed', num(last.probed)], + [ + 'Changed', + `${num(last.changed)}${compared ? ` of ${num(compared)} compared (${pct(last.changed, compared)})` : ''}`, + ], + ['Re-renders filed', last.triggered ? pill(num(last.triggered), 'ok') : pill('0', '')], + last.deferred ? ['Deferred past the trigger cap', pill(num(last.deferred), 'warn')] : null, + ['Failed probes', last.failed ? pill(num(last.failed), 'warn') : pill('0', 'ok')], + last.errors ? ['Trigger write errors', pill(num(last.errors), 'bad')] : null, + last.aborted ? ['Interrupted', pill('stood down for a reseed, or the probe was disabled', '')] : null, + ]) + ); + } else if (!last) { + body.push(muted('No sweep has finished since startup. The first one runs after changeProbe.startDelay.')); + } + + if (last?.failureSamples?.length) { + body.push( + el('p', { cls: 'muted chart-note' }, [ + 'The first few failures of that pass. A failure leaves the stored signature untouched and triggers ', + 'nothing, so these are pages back on interval-only freshness until the rule fits again.', + ]), + table( + ['url', 'rule', 'error'], + last.failureSamples.map((sample) => + el('tr', null, [ + el('td', { cls: 'mono truncate', text: sample.url ?? '—', title: sample.url ?? '' }), + el('td', { cls: 'mono', text: sample.rule ?? '—' }), + el('td', { cls: 'mono truncate', text: sample.error ?? '—', title: sample.error ?? '' }), + ]) + ) + ) + ); + } + + return card('Rolling sweep', { + head: [ + sweep.armedInterval ? pill(`every ${duration(sweep.armedInterval)}`, 'ok') : pill('not armed', 'bad'), + sweep.running && + pill(sweep.runningOn?.length ? `running on ${sweep.runningOn.join(', ')}` : 'running now', 'info'), + spacer(), + muted(last?.nodes > 1 ? `summed over ${last.nodes} nodes’ own slices` : ''), + ], + body, + foot: [ + muted( + 'The sweep catches per-URL drift — an item selling out, one price moving. It walks the whole owned ' + + 'slice at the configured rate, so a large corpus takes hours per pass by design; that is what the ' + + 'canary below exists to cover.' + ), + ], + }); +} + +/** Per-rule canary verdicts, normalized over the node and cluster payload shapes. */ +function perRuleRows(status) { + const last = status.canary?.lastRun; + const hostname = status.node ?? 'this node'; + return (last?.perRule ?? []).map((entry) => ({ + ...entry, + trippedOn: entry.trippedOn ?? (entry.tripped ? [hostname] : []), + actions: entry.actions ?? (entry.action ? [{ hostname, ...entry.action }] : []), + emptyCohortOn: entry.emptyCohortOn ?? (entry.skipped ? [hostname] : []), + })); +} + +/** How an action's refusal reads to someone who has to decide whether it mattered. */ +const ACTION_REASON = { + 'dry-run': ['dry run — nothing invalidated', ''], + 'no-scope': ['no invalidateScope — detection only', ''], + 'holdoff': ['inside canary.holdoff — deliberately not re-stamped', ''], + 'unresolvable-scope': ['invalidateScope names no configured route — NOTHING was invalidated', 'bad'], + 'invalidation-disabled': ['invalidation.enabled is false — NOTHING was invalidated', 'bad'], +}; + +function canaryCard(ctx, status) { + const canary = status.canary ?? {}; + const last = canary.lastRun; + const rows = perRuleRows(status); + const body = []; + + if (canary.unrunNodes?.length) { + body.push(note('warn', [`No canary pass has finished on ${canary.unrunNodes.join(', ')} since startup.`])); + } + if (last?.error) body.push(note('bad', [`Last canary pass failed: ${last.error}`])); + + const empties = rows.filter((row) => row.emptyCohortOn.length); + if (empties.length) { + body.push( + note('warn', [ + `No cohort yet for ${empties.map((row) => row.rule).join(', ')}. A cohort is built by the first sweep ` + + '(or a cheaper key-order sample right after a restart), so an empty one means the mass-change ' + + 'detector is dark for that rule — the sweep still catches the change, hours later.', + ]) + ); + } + + if (rows.length) { + body.push( + table( + [ + 'rule', + { text: 'cohort', right: true }, + { text: 'compared', right: true }, + { text: 'changed', right: true }, + 'verdict', + 'action', + ], + rows.map((row) => { + const tripped = row.trippedOn.length > 0; + const acted = row.actions.filter((action) => action.acted); + const refused = row.actions.filter((action) => !action.acted); + return el('tr', null, [ + el('td', { cls: 'mono', text: row.rule ?? '—' }), + el('td', { cls: 'right mono', text: num(row.cohort) }), + el('td', { cls: 'right mono', text: num(row.compared) }), + el('td', { cls: 'right' }, [ + el('span', { + cls: tripped ? 'pill warn' : 'mono', + text: row.fraction === null || row.fraction === undefined ? '—' : pct(row.changed, row.compared), + title: `${num(row.changed)} changed of ${num(row.compared)} compared`, + }), + ]), + el('td', null, [ + tripped + ? pill(`tripped on ${row.trippedOn.join(', ')}`, 'warn') + : row.compared + ? pill('below threshold', 'ok') + : muted('nothing compared'), + ]), + el('td', null, [ + acted.length ? pill(`invalidated ${acted[0].scope}`, 'ok') : null, + ...refused.map((action) => { + const [text, kind] = ACTION_REASON[action.reason] ?? [action.reason ?? 'not acted', '']; + return pill(text, kind); + }), + !row.actions.length && muted('—'), + ]), + ]); + }) + ) + ); + } else { + body.push(muted('No canary pass has finished since startup.')); + } + + const cohortTotal = Object.values(canary.cohortSizes ?? {}).reduce((acc, size) => acc + (Number(size) || 0), 0); + + return card('Canary', { + head: [ + canary.armedInterval ? pill(`every ${duration(canary.armedInterval)}`, 'ok') : pill('disabled', ''), + canary.running && pill('running now', 'info'), + last?.finishedAt && muted(`last pass ${ago(last.finishedAt)}`), + spacer(), + muted(cohortTotal ? `${num(cohortTotal)} URLs under watch` : ''), + ], + body, + foot: [ + muted( + 'A trip is a threshold crossed against ONE node’s cohort, so the verdict names the nodes that ' + + 'crossed it rather than folding four independent judgements into one. The response is a bulk ' + + 'invalidation, not thousands of re-renders: pre-change snapshots stop serving at once (bots get ' + + 'origin content, correct by definition) while the fleet refills on its own cadence. ' + ), + status.dryRun + ? muted('In dry run a trip is logged and nothing is invalidated — the log line says “WOULD TRIP”.') + : null, + el('span', null, [' ', link('Invalidations →', () => ctx.go('invalidations'))]), + ], + }); +} + +// ---------------------------------------------------------------- per node + +/** Only under cluster scope: at node scope the whole page already IS one node's answer. */ +function nodeTable(status) { + const rows = status.byNode; + if (!rows?.length) return null; + + return card('Per node', { + head: [spacer(), muted('unsummed — a pass is run one node at a time')], + body: [ + table( + [ + 'node', + 'probe', + 'run mode', + 'last sweep', + { text: 'probed', right: true }, + { text: 'changed', right: true }, + { text: 'failed', right: true }, + ], + rows.map((row) => + el('tr', null, [ + el('td', { cls: 'mono', text: row.hostname }), + el('td', null, [ + row.enabled === false ? pill('disabled', 'bad') : pill('enabled', 'ok'), + row.sweepRunning && pill('sweeping', 'info'), + row.canaryRunning && pill('canary', 'info'), + ]), + el('td', null, [row.dryRun === false ? pill('live', 'ok') : pill('dry run', 'warn')]), + el('td', null, [ + row.sweepFinishedAt ? muted(ago(row.sweepFinishedAt)) : muted('never'), + row.error && pill('errored', 'bad'), + ]), + el('td', { cls: 'right mono', text: num(row.probed) }), + el('td', { cls: 'right mono', text: num(row.changed) }), + el('td', { cls: 'right mono', text: num(row.failed) }), + ]) + ) + ), + ], + }); +} + +// ---------------------------------------------------------------- settings + +function settings(ctx) { + return el('div', null, [ + el('div', { cls: 'view-head', style: { marginTop: '20px' } }, [ + el('span', { cls: 'eyebrow', text: 'Settings' }), + spacer(), + muted('staged in this browser until you preview and apply'), + ]), + settingsCard(ctx, { + title: 'Change probe', + prefix: 'changeProbe', + description: + 'What is probed, how fast, and whether a detected change is allowed to act. ratePerSecond is the ' + + 'origin-protection knob — probe endpoints are typically uncached, so every probe is backend work ' + + 'for whoever runs the origin, and it also sizes the sweep (a 200k-URL slice at 10/s is about 5.6 ' + + 'hours per pass). Leave dryRun on until the change rate above has been watched for a while: ' + + 'signatures are written either way, so a dry-run week converges on the true rate rather than ' + + 're-reporting the same delta.', + }), + ]); +} diff --git a/packages/console/src/admin/views/traffic.js b/packages/console/src/admin/views/traffic.js index 90f75436..9f42d82f 100644 --- a/packages/console/src/admin/views/traffic.js +++ b/packages/console/src/admin/views/traffic.js @@ -38,8 +38,8 @@ * * THE BOT FILTER IS CLIENT-SIDE, ALWAYS. Selecting bots re-renders from the payload already in * hand; it never refetches, because the load discipline below is the whole reason this view can - * afford to be this detailed. Three metrics carry a bot (`bot_request`, `bot_serve`, `page_age`) - * and the rest do not — `route_serve`, `origin_fetch`, `duration` and `response_*` have no bot + * afford to be this detailed. Four metrics carry a bot (`bot_request`, `bot_serve`, `page_age`, + * and `prerender_ops`'s `discovery_gated`) and the rest do not — `route_serve`, `origin_fetch`, `duration` and `response_*` have no bot * dimension at all — so a panel that CANNOT honour the filter says "all bots" on its face rather * than quietly showing every crawler's numbers under one crawler's name. * @@ -76,6 +76,7 @@ import { rangePicker, ratioOf, scanFooter, + scopeLabel, segmented, SERIES, stackBy, @@ -160,6 +161,7 @@ export function render(ctx) { el('div', { cls: 'cols' }, [originFetch(data, filter), latency(data, filter)]), el('div', { cls: 'cols' }, [crawlers(data, scope), statusCodes(data, filter)]), routes(ctx, data, cadences, filter), + discoveryGate(ctx, data, filter), breadth(ctx, filter), el('div', { cls: 'scan-foot' }, [scanFooter(data)]), knobs, @@ -248,7 +250,10 @@ function botBar(ctx, data, filter) { ), hidden > 0 && muted(`+${hidden} smaller`), filter && - muted('filters serves, freshness, staleness, page age and the crawler mix — panels marked "all bots" cannot'), + muted( + 'filters serves, freshness, staleness, page age, the crawler mix and the discovery gate — panels ' + + 'marked "all bots" cannot' + ), ]); } @@ -286,16 +291,23 @@ export function cadenceIndex(configPayload, defaultInterval) { const index = new Map(); const options = optionIndex(configPayload); - const add = (path, mode, renderInterval) => { + const add = (path, mode, renderInterval, extra = {}) => { if (typeof path !== 'string' || path === '' || index.has(path)) return; const own = mode === 'prerender' && Number.isFinite(renderInterval) && renderInterval > 0; - index.set(path, { mode, interval: own ? renderInterval : defaultInterval, inherited: !own }); + index.set(path, { mode, interval: own ? renderInterval : defaultInterval, inherited: !own, ...extra }); }; for (const pattern of options.get('ingress.excludePathPatterns')?.effective ?? []) add(pattern, 'passthrough'); for (const entry of options.get('ingress.routes')?.effective ?? []) { if (!entry || typeof entry !== 'object') continue; - add(entry.path, entry.mode === 'passthrough' ? 'passthrough' : 'prerender', Number(entry.renderInterval)); + add(entry.path, entry.mode === 'passthrough' ? 'passthrough' : 'prerender', Number(entry.renderInterval), { + // Both default to "unset", and the difference matters: `discoverTargets` defaults to TRUE + // upstream, so an absent flag is an OPEN route, not an unknown one. `demandFloor` has no + // default — absent means the demand ladder may take this route's pages to any rung. + discoverTargets: entry.discoverTargets !== false, + demandFloor: + Number.isFinite(Number(entry.demandFloor)) && Number(entry.demandFloor) > 0 ? Number(entry.demandFloor) : null, + }); } return index; } @@ -310,6 +322,10 @@ export const cadenceFor = (index, label, defaultInterval) => mode: CLASS_LABELS[label] ?? 'unknown', interval: defaultInterval, inherited: true, + // Null, not `true`: no route matched, so there is no route flag to report. Printing the + // upstream default here would claim a configuration this label does not have. + discoverTargets: null, + demandFloor: null, }; /** Whether a route's staleness is a verdict about US, or just a fact about a path we never cache. */ @@ -1162,9 +1178,16 @@ function routes(ctx, data, cadences, filter) { el('td', { cls: 'right' }, [ integrity ? el('span', { cls: 'pill bad', text: pct(integrity, routeTotal) }) : muted('—'), ]), + el('td', null, [ + cadence.discoverTargets === null ? muted('—') : cadence.discoverTargets ? muted('open') : pill('gated', 'info'), + ]), el('td', { cls: 'right mono' }, [ prerender ? fmtMs(cadence.interval) : '—', prerender && cadence.inherited ? muted(' default') : null, + // The floor bounds how far the demand ladder may accelerate this route's pages, so it + // belongs beside the cadence it modifies rather than in a column of its own that would + // be empty on every deployment that has not set one. + prerender && cadence.demandFloor ? muted(` · floor ${fmtMs(cadence.demandFloor)}`) : null, ]), el('td', { cls: 'right mono', @@ -1199,6 +1222,7 @@ function routes(ctx, data, cadences, filter) { { text: 'miss', right: true }, { text: 'swr+stale', right: true }, anyIntegrity && { text: 'blob', right: true }, + 'discovery', { text: 'cadence', right: true }, { text: 'age median ≈', right: true }, { text: '÷ cadence', right: true }, @@ -1213,7 +1237,9 @@ function routes(ctx, data, cadences, filter) { 'against. A high miss share means its corpus ', 'isn’t covered — flagged only on routes we actually cache, since a passthrough is proxied live by ', 'design. Cadence is read from ingress.routes; “default” means the route sets none and inherits ', - 'render.defaultInterval.', + 'render.defaultInterval, and “floor” is the fastest rung the demand ladder may grant it. ', + '“Gated” means the route no longer mints a target for an unknown URL a bot asks for — those ', + 'requests are still served from the origin, they just never enter the render rotation.', ]), unclassified > 0 && el('div', { cls: 'note warn' }, [ @@ -1226,6 +1252,103 @@ function routes(ctx, data, cadences, filter) { }); } +/** + * What the discovery gate is holding out of the render rotation. + * + * READ IT AS TRAFFIC, NOT AS DENIED MINTS. The plugin counts every cacheable MISS whose target + * creation the gate refused, which includes repeat misses on URLs it has already refused — so this + * is "requests on URLs held out of the rotation", not "URLs prevented". That is the more useful + * number anyway: it is the crawl pressure the gate is absorbing, and its shape over the window is + * what says whether a crawler has moved on or is still walking the same combinatorial space. + * + * NOTHING HERE IS A FAILURE. A gated request is still SERVED — proxied live from the origin, with + * the miss counted on the route table above — it simply never enters the render rotation. The + * panel exists because the alternative to gating is a corpus that grows faster than the fleet can + * keep it fresh, and that shows up nowhere until the whole queue is late. + */ +// The two gate reasons, as named constants rather than quoted literals at the lookup. The +// cross-package route scanner in adminAssets.test.js reads any Map lookup written with a quoted +// name as a call to an admin route of that name — including one inside a comment, which is why +// this one describes the shape instead of showing it. The scanner is deliberately blunt: a +// mistyped route name has to fail in the suite rather than in a browser, and the cost of that is +// spelling lookup keys out as constants here. +const GATE_ROUTE_FLAG = 'route'; +const GATE_BOT_ALLOWLIST = 'bot'; + +function discoveryGate(ctx, data, filter) { + const gated = pick(data, 'prerender_ops', (s) => s.path === 'discovery_gated' && keepBot(filter, s.type)); + const options = optionIndex(configState(ctx).payload); + const bots = options.get('ingress.discoveryBots')?.effective ?? ['*']; + const gatedRoutes = (options.get('ingress.routes')?.effective ?? []).filter( + (entry) => entry && typeof entry === 'object' && entry.discoverTargets === false + ); + const botGateOn = Array.isArray(bots) && !(bots.length === 1 && bots[0] === '*'); + const total = sumCount(gated); + + // Nothing gated AND nothing configured to gate: describe the capability rather than drawing an + // empty chart, which would read as a subsystem that is on and quiet. + if (!total && !botGateOn && !gatedRoutes.length) { + return card('Discovery gate', { + head: [spacer(), pill('not configured', '')], + body: [ + el('p', { cls: 'muted chart-note' }, [ + 'Every route mints a target for any unknown URL a bot asks for, and every bot is trusted to do ', + 'it. On a route whose URL space is combinatorial — facets, filter and sort permutations — that ', + 'is unbounded: crawlers walk novel combinations into permanent render load. ', + el('code', { text: 'discoverTargets' }), + ' on a route, and ', + el('code', { text: 'ingress.discoveryBots' }), + ', are the two gates; both are under Request ingestion on ', + link('Config →', () => ctx.go('config')), + '.', + ]), + ], + }); + } + + const byGate = new Map(); + const byBot = new Map(); + for (const s of gated) { + byGate.set(s.method ?? 'unknown', (byGate.get(s.method ?? 'unknown') ?? 0) + s.count); + byBot.set(s.type ?? 'other', (byBot.get(s.type ?? 'other') ?? 0) + s.count); + } + const ranked = [...byBot.entries()].sort((a, b) => b[1] - a[1]).slice(0, 8); + + return card(`Discovery gate — ${scopeLabel(data)}`, { + head: [ + gatedRoutes.length + ? pill(`${gatedRoutes.length} route${gatedRoutes.length === 1 ? '' : 's'} gated`, 'info') + : null, + botGateOn ? pill(bots.length ? `minting: ${bots.join(', ')}` : 'sitemap-only corpus', 'info') : null, + spacer(), + link('purge what is already in →', () => ctx.go('overview')), + ], + body: [ + el('div', { cls: 'stats' }, [ + stat('Gated misses', fmtCount(total), 'served from the origin, never scheduled'), + stat('By route flag', fmtCount(byGate.get(GATE_ROUTE_FLAG) ?? 0), 'discoverTargets: false'), + stat('By bot allowlist', fmtCount(byGate.get(GATE_BOT_ALLOWLIST) ?? 0), 'ingress.discoveryBots'), + ]), + ranked.length + ? barList( + ranked.map(([bot, count]) => ({ label: bot, value: count })), + { format: fmtCount } + ) + : el('p', { cls: 'muted chart-note' }, [ + 'The gate is configured and refused nothing in this window — no bot asked for an unknown URL ', + 'on a gated route. That is the steady state once crawlers have stopped exploring; it is also ', + 'what a range too narrow to contain a crawl looks like.', + ]), + el('p', { cls: 'muted chart-note' }, [ + 'Counted per gated MISS, so a URL refused a hundred times counts a hundred — this is the crawl ', + 'pressure the gate is absorbing, not a count of URLs prevented. Gating stops NEW targets only: ', + 'everything minted before the flag went on keeps rendering until it is purged, and the ', + 'sitemap pipeline is untouched, so declared URLs on a gated route still schedule normally.', + ]), + ], + }); +} + /** Distinct URLs per bot per day — how much of the corpus crawlers actually walk. */ function breadth(ctx, filter) { const state = ctx.data.breadth; diff --git a/packages/console/src/util/aggregate.js b/packages/console/src/util/aggregate.js index 34d2ac3d..eb45c317 100644 --- a/packages/console/src/util/aggregate.js +++ b/packages/console/src/util/aggregate.js @@ -39,7 +39,20 @@ export const CLUSTER = 'cluster'; * Routes whose numbers are node-local and additive. Everything else in PROXIED_GET reads a * replicated table (or is static), so it is answered by ONE node — see SHARED_NOTE. */ -export const MERGED_GET = Object.freeze(['overview', 'analytics', 'unrouted', 'config']); +export const MERGED_GET = Object.freeze([ + 'overview', + 'analytics', + 'unrouted', + 'config', + // Neither of these ADDS the way analytics does — a probe pass and a purge pass are run one node + // at a time over the keys that node owns, so what the merge produces is every node's own slice + // side by side plus the running tally. They are here rather than in SHARED_NOTE because the + // alternative is worse in the one way this classification exists to prevent: one node's + // "deleted 240,118" answered under a cluster label reads as done when three quarters of the + // keyspace has not been touched. + 'change-probe', + 'discovery-purge', +]); /** * Why a route is answered by a single node under cluster scope, keyed by route. Shown in the @@ -71,6 +84,15 @@ export const NODE_LOCAL_POST = Object.freeze({ // this sweep deletes. 'sweep-orphans': 'The orphan sweep deletes among the keys ONE node owns. Pick a node to sweep it.', 'backlog': 'A backlog snapshot covers the keys ONE node owns. Pick a node to recompute it.', + // A probe pass walks the registry rows THIS node owns and writes their baselines, and the + // canary probes the cohort built from that same slice. Fanned out it would be N passes at N + // times the agreed origin rate, which is the one thing changeProbe.ratePerSecond exists to + // stop — the rate is a promise made to whoever runs the origin, per node. + 'change-probe': + 'A probe pass covers the URLs ONE node owns, at that node’s agreed probe rate. Pick a node to run it.', + // Deletes, owner-scoped, and paced per node — the sweep-orphans argument exactly. + 'discovery-purge': + 'A purge deletes among the keys ONE node owns. Pick a node to run it, and run every node to cover the keyspace.', // `schedule` reads RenderSchedule node-locally (replicateFrom: false, because the table is // residency-pinned and a cross-node read has no timeout). Asked of a node that does not own // the key it truthfully answers "no row" — which reads as "this URL will never render". The @@ -951,13 +973,247 @@ export function mergeConfig(results) { }; } +// ---------------------------------------------------------------- change probe + +/** Pass counters that ADD across nodes: each node probes a disjoint slice of the keyspace. */ +const PROBE_COUNTERS = [ + 'examined', + 'owned', + 'matched', + 'probed', + 'seeded', + 'unchanged', + 'changed', + 'triggered', + 'deferred', + 'failed', + 'errors', +]; + +/** Bound on the pooled failure samples — they exist to name a shape change, not to be a log. */ +const MAX_FAILURE_SAMPLES = 6; + +const sumCounters = (runs, keys) => Object.fromEntries(keys.map((key) => [key, sumOf(runs, (r) => r[key])])); + +/** + * Merge the change-probe status of every node into one cluster answer. + * + * THE COUNTERS ADD, THE VERDICTS DO NOT. A probe pass is owner-scoped — each node walks the + * registry rows it owns — so `probed`, `changed` and the rest partition the corpus and summing + * them is the only way to get a cluster change rate at all. The canary's TRIP is the opposite: it + * is a threshold crossed against ONE node's cohort, and a cluster-wide "tripped: true" folded out + * of four independent verdicts would name no node and match no log line. So trips are reported as + * the nodes that tripped, and the pooled fraction rides alongside as the wider sample it is. + * + * TIME IS TAKEN AS THE OLDEST, never the newest — the same rule the orphan sweep merge follows. + * Nodes stagger their first sweep on purpose (`startJitter`), so their last passes finished at + * different times, and a summary stamped with the freshest of them would present a figure that is + * hours stale on three nodes out of four as current. + * + * A NODE THAT HAS NOT SWEPT CONTRIBUTES ZERO TO EVERY SUM, which is indistinguishable from a node + * that swept and found nothing. `unsweptNodes` is what separates them, and the UI says so rather + * than letting the shortfall read as a smaller change rate. + */ +export function mergeChangeProbe(results) { + const usable = okBodies(results); + if (!usable.length) return allFailed(results, 'merged'); + + const bodies = usable.map((r) => ({ hostname: r.hostname, b: r.body })); + + const sweeps = bodies + .map((r) => ({ hostname: r.hostname, ...(r.b.sweep?.lastRun ?? {}) })) + .filter((r) => r.startedAt); + const canaries = bodies + .map((r) => ({ hostname: r.hostname, ...(r.b.canary?.lastRun ?? {}) })) + .filter((r) => r.startedAt); + + // Rules are configuration, so they are identical by intent and one node's copy is the answer. + // A disagreement is a deploy that skipped a node, which is the Config view's finding and not + // this one's — flagged here, diagnosed there. + const ruleJson = bodies.map((r) => JSON.stringify(r.b.rules ?? [])); + const rules = bodies.find((r) => (r.b.rules ?? []).length)?.b.rules ?? []; + + // Per RULE, across nodes: the counters add (disjoint cohorts), the verdict is named per node. + const perRule = new Map(); + for (const run of canaries) { + for (const entry of run.perRule ?? []) { + const merged = perRule.get(entry.rule) ?? { + rule: entry.rule, + cohort: 0, + ...Object.fromEntries(PROBE_COUNTERS.map((key) => [key, 0])), + compared: 0, + trippedOn: [], + actions: [], + emptyCohortOn: [], + }; + merged.cohort += finite(entry.cohort) || 0; + for (const key of PROBE_COUNTERS) merged[key] += finite(entry[key]) || 0; + merged.compared += finite(entry.compared) || 0; + if (entry.tripped) merged.trippedOn.push(run.hostname); + if (entry.skipped) merged.emptyCohortOn.push(run.hostname); + if (entry.action) merged.actions.push({ hostname: run.hostname, ...entry.action }); + perRule.set(entry.rule, merged); + } + } + // Pooled over the union of the nodes' cohorts: a wider sample of the same population, and the + // number to read when one node's cohort is too small to decide anything on its own. + for (const entry of perRule.values()) { + entry.fraction = entry.compared > 0 ? entry.changed / entry.compared : null; + } + + const failureSamples = sweeps + .flatMap((run) => (run.failureSamples ?? []).map((sample) => ({ hostname: run.hostname, ...sample }))) + .slice(0, MAX_FAILURE_SAMPLES); + + return { + status: 200, + body: { + node: null, + workerIndex: null, + scope: 'cluster', + // ONE node with the probe off is the finding, not "the cluster has it on": that node's + // owned slice is never probed and every panel here keeps looking healthy. + enabled: bodies.some((r) => r.b.enabled), + disabledOn: bodies.filter((r) => r.b.enabled === false).map((r) => r.hostname), + // Dry run only if EVERY node is: one node acting makes "nothing is being triggered" false. + dryRun: bodies.every((r) => r.b.dryRun !== false), + liveOn: bodies.filter((r) => r.b.dryRun === false).map((r) => r.hostname), + ownerScopeNote: bodies[0].b.ownerScopeNote ?? null, + rules, + rulesDiverge: new Set(ruleJson).size > 1, + sweep: { + running: bodies.some((r) => r.b.sweep?.running), + runningOn: bodies.filter((r) => r.b.sweep?.running).map((r) => r.hostname), + armedInterval: bodies.find((r) => Number.isFinite(r.b.sweep?.armedInterval))?.b.sweep.armedInterval ?? null, + sweptNodes: sweeps.length, + unsweptNodes: bodies.filter((r) => !r.b.sweep?.lastRun?.startedAt).map((r) => r.hostname), + lastRun: sweeps.length + ? { + ...sumCounters(sweeps, PROBE_COUNTERS), + dryRun: sweeps.every((run) => run.dryRun !== false), + aborted: sweeps.some((run) => run.aborted), + startedAt: minOf(sweeps, (run) => msOf(run.startedAt)), + finishedAt: minOf(sweeps, (run) => msOf(run.finishedAt)), + error: sweeps.find((run) => run.error)?.error ?? null, + nodes: sweeps.length, + failureSamples, + } + : null, + }, + canary: { + running: bodies.some((r) => r.b.canary?.running), + armedInterval: bodies.find((r) => Number.isFinite(r.b.canary?.armedInterval))?.b.canary.armedInterval ?? null, + cohortSizes: bodies.reduce((acc, r) => { + for (const [label, size] of Object.entries(r.b.canary?.cohortSizes ?? {})) { + acc[label] = (acc[label] ?? 0) + (finite(size) || 0); + } + return acc; + }, {}), + ranNodes: canaries.length, + unrunNodes: bodies.filter((r) => !r.b.canary?.lastRun?.startedAt).map((r) => r.hostname), + lastRun: canaries.length + ? { + perRule: [...perRule.values()], + dryRun: canaries.every((run) => run.dryRun !== false), + startedAt: minOf(canaries, (run) => msOf(run.startedAt)), + finishedAt: minOf(canaries, (run) => msOf(run.finishedAt)), + error: canaries.find((run) => run.error)?.error ?? null, + nodes: canaries.length, + } + : null, + }, + // Every node's own slice, unsummed — the purge/sweep actions are run per node, so this + // is the table an operator works down. + byNode: bodies.map((r) => ({ + hostname: r.hostname, + enabled: r.b.enabled ?? null, + dryRun: r.b.dryRun ?? null, + sweepRunning: !!r.b.sweep?.running, + canaryRunning: !!r.b.canary?.running, + sweepFinishedAt: msOf(r.b.sweep?.lastRun?.finishedAt) || null, + canaryFinishedAt: msOf(r.b.canary?.lastRun?.finishedAt) || null, + probed: num(r.b.sweep?.lastRun?.probed), + changed: num(r.b.sweep?.lastRun?.changed), + failed: num(r.b.sweep?.lastRun?.failed), + error: r.b.sweep?.lastRun?.error ?? r.b.canary?.lastRun?.error ?? null, + })), + sources: sourcesOf(results, { mode: 'merged' }), + }, + }; +} + +// ---------------------------------------------------------------- discovery purge + +const PURGE_COUNTERS = ['examined', 'owned', 'discovered', 'leaseSkipped', 'deleted']; + +/** + * Merge every node's discovered-target purge state. + * + * A PURGE IS RUN PER NODE, ONE AT A TIME, and the keyspace is only covered once every node has + * run — so `byNode` is the substance here and the totals are the running tally, not a result. The + * merge exists to stop the alternative: a single node's "deleted 240,118" answered under a + * "cluster" label, which reads as done when three quarters of the corpus has not been touched. + * + * DIFFERENT NODES MAY HAVE RUN DIFFERENT PREFIXES, and a total summed across two prefixes answers + * a question nobody asked. `urlPrefix` is therefore only reported when every node that ran used + * the same one; otherwise the per-node rows are the whole answer and the UI says so. + */ +export function mergeDiscoveryPurge(results) { + const usable = okBodies(results); + if (!usable.length) return allFailed(results, 'merged'); + + const bodies = usable.map((r) => ({ hostname: r.hostname, b: r.body })); + const runs = bodies.map((r) => ({ hostname: r.hostname, ...r.b })).filter((run) => run.startedAt); + const prefixes = [...new Set(runs.map((run) => run.urlPrefix).filter(Boolean))]; + + return { + status: 200, + body: { + node: null, + scope: 'cluster', + running: bodies.some((r) => r.b.running), + runningOn: bodies.filter((r) => r.b.running).map((r) => r.hostname), + ranNodes: runs.length, + unrunNodes: bodies.filter((r) => !r.b.startedAt).map((r) => r.hostname), + urlPrefix: prefixes.length === 1 ? prefixes[0] : null, + urlPrefixes: prefixes, + // A cluster figure counts as a dry run only if every node's was — one node that actually + // deleted makes "nothing was deleted" false. + dryRun: runs.length ? runs.every((run) => run.dryRun !== false) : true, + canceled: runs.some((run) => run.canceled), + startedAt: minOf(runs, (run) => msOf(run.startedAt)), + // Oldest, and null while any node is still running: a finish time for a pass three nodes + // are still in the middle of is the wrong shape of answer. + finishedAt: runs.some((run) => run.running) ? null : minOf(runs, (run) => msOf(run.finishedAt)), + error: runs.find((run) => run.error)?.error ?? null, + totals: runs.length ? sumCounters(runs, PURGE_COUNTERS) : null, + ownerScopeNote: bodies[0].b.ownerScopeNote ?? null, + byNode: bodies.map((r) => ({ + hostname: r.hostname, + running: !!r.b.running, + urlPrefix: r.b.urlPrefix ?? null, + dryRun: r.b.dryRun ?? null, + ratePerSecond: num(r.b.ratePerSecond), + startedAt: msOf(r.b.startedAt) || null, + finishedAt: msOf(r.b.finishedAt) || null, + canceled: !!r.b.canceled, + error: r.b.error ?? null, + ...Object.fromEntries(PURGE_COUNTERS.map((key) => [key, num(r.b[key])])), + })), + sources: sourcesOf(results, { mode: 'merged' }), + }, + }; +} + // ---------------------------------------------------------------- dispatch const MERGERS = { - overview: mergeOverview, - analytics: mergeAnalytics, - unrouted: mergeUnrouted, - config: mergeConfig, + 'overview': mergeOverview, + 'analytics': mergeAnalytics, + 'unrouted': mergeUnrouted, + 'config': mergeConfig, + 'change-probe': mergeChangeProbe, + 'discovery-purge': mergeDiscoveryPurge, }; export const mergerFor = (route) => MERGERS[route] ?? null; diff --git a/packages/console/src/util/proxy.js b/packages/console/src/util/proxy.js index 468df8b3..18d13b72 100644 --- a/packages/console/src/util/proxy.js +++ b/packages/console/src/util/proxy.js @@ -25,6 +25,11 @@ export const PROXIED_GET = Object.freeze([ 'analytics', 'crawl-breadth', 'metrics', + // Both are node-local reads with a merger of their own (aggregate.js): a probe pass and a + // purge pass act on the keys ONE node owns, so "the cluster" here is every node's own slice + // side by side, never one node's answer wearing a cluster label. + 'change-probe', + 'discovery-purge', ]); export const PROXIED_POST = Object.freeze([ @@ -39,6 +44,8 @@ export const PROXIED_POST = Object.freeze([ 'sitemap', 'sitemap-refresh', 'config-override', + 'change-probe', + 'discovery-purge', ]); /** diff --git a/packages/console/test/adminAssets.test.js b/packages/console/test/adminAssets.test.js index 4a721044..74a832aa 100644 --- a/packages/console/test/adminAssets.test.js +++ b/packages/console/test/adminAssets.test.js @@ -225,6 +225,8 @@ test('a metric the plugin emits is charted by the console, or waived with a reas 'demandLadder', 'invalidationError', 'invalidationReenqueue', + // `probe_${series}` — one emit per finished pass, per counter. Charted on the Change probe view. + 'changeProbe', ]); const emitted = []; diff --git a/packages/console/test/aggregate.test.js b/packages/console/test/aggregate.test.js index c5387fe5..bd50ff69 100644 --- a/packages/console/test/aggregate.test.js +++ b/packages/console/test/aggregate.test.js @@ -786,3 +786,193 @@ test('mergeOverview surfaces a node whose row every PEER holds an older copy of' assert.equal(yc0.spreadMs, 0, 'a healthy row: every copy agrees'); assert.equal(yc0.behind.length, 0); }); + +// ------------------------------------------------------------------ change probe + +const probeStats = (over = {}) => ({ + examined: 10_000, + owned: 2500, + matched: 1000, + probed: 1000, + seeded: 200, + unchanged: 700, + changed: 100, + triggered: 90, + deferred: 10, + failed: 100, + errors: 0, + failureSamples: [], + ...over, +}); + +const probeBody = (node, over = {}) => ({ + enabled: true, + dryRun: true, + node: `${node}.example.com:9926`, + ownerScopeNote: 'Probes only the URLs this node owns; every node sweeps its own slice.', + rules: [{ label: 'price', pathPattern: '^/product/', source: 'request', invalidateScope: 'route:prefix:/product/' }], + sweep: { + running: false, + armedInterval: 86_400_000, + lastRun: { ...probeStats(), dryRun: true, node, startedAt: 1000, finishedAt: 2000, error: null }, + }, + canary: { running: false, armedInterval: 1_800_000, cohortSizes: { price: 500 }, lastRun: null }, + ...over, +}); + +test('change probe: pass counters SUM, because each node probes a disjoint slice of the keyspace', () => { + const { body } = mergerFor('change-probe')([ok('a', probeBody('a')), ok('b', probeBody('b'))]); + assert.equal(body.sweep.lastRun.probed, 2000); + assert.equal(body.sweep.lastRun.changed, 200); + assert.equal(body.sweep.lastRun.nodes, 2); + assert.equal(body.scope, 'cluster'); +}); + +test('change probe: the summary is stamped with the OLDEST pass, never the freshest', () => { + const { body } = mergerFor('change-probe')([ + ok('a', probeBody('a')), + ok( + 'b', + probeBody('b', { + sweep: { + running: false, + armedInterval: 86_400_000, + lastRun: { ...probeStats(), dryRun: true, startedAt: 500_000, finishedAt: 900_000, error: null }, + }, + }) + ), + ]); + // b's pass is much newer. Stamping the sum with it would present a figure that is stale on + // every other node as current. + assert.equal(body.sweep.lastRun.finishedAt, 2000); +}); + +test('change probe: a node that has not swept is NAMED, not folded into the totals as a zero', () => { + const { body } = mergerFor('change-probe')([ + ok('a', probeBody('a')), + ok('b', probeBody('b', { sweep: { running: false, armedInterval: 86_400_000, lastRun: null } })), + ]); + assert.deepEqual(body.sweep.unsweptNodes, ['b.example.com:9926']); + assert.equal(body.sweep.lastRun.nodes, 1); + assert.equal(body.sweep.lastRun.probed, 1000); +}); + +test('change probe: ONE node running live makes the cluster "not a dry run"', () => { + const { body } = mergerFor('change-probe')([ok('a', probeBody('a')), ok('b', probeBody('b', { dryRun: false }))]); + assert.equal(body.dryRun, false); + assert.deepEqual(body.liveOn, ['b.example.com:9926']); +}); + +test('change probe: ONE node with the probe off is the finding, not "the cluster has it on"', () => { + const { body } = mergerFor('change-probe')([ok('a', probeBody('a')), ok('b', probeBody('b', { enabled: false }))]); + assert.equal(body.enabled, true); + assert.deepEqual(body.disabledOn, ['b.example.com:9926']); +}); + +test('change probe: a canary TRIP names the nodes that tripped — a verdict is per cohort', () => { + const canaryRun = (tripped, changed) => ({ + running: false, + armedInterval: 1_800_000, + cohortSizes: { price: 500 }, + lastRun: { + perRule: [ + { + rule: 'price', + cohort: 500, + ...probeStats({ changed, unchanged: 500 - changed, probed: 500, seeded: 0, failed: 0 }), + compared: 500, + fraction: changed / 500, + tripped, + action: tripped ? { acted: true, scope: 'route:prefix:/product/' } : null, + }, + ], + dryRun: false, + startedAt: 1000, + finishedAt: 2000, + error: null, + }, + }); + const { body } = mergerFor('change-probe')([ + ok('a', probeBody('a', { canary: canaryRun(true, 250) })), + ok('b', probeBody('b', { canary: canaryRun(false, 10) })), + ]); + const [rule] = body.canary.lastRun.perRule; + assert.deepEqual(rule.trippedOn, ['a.example.com:9926']); + // Pooled over the union of the two cohorts: 260 of 1000, not the average of 50% and 2%. + assert.equal(rule.compared, 1000); + assert.equal(rule.fraction, 0.26); + assert.deepEqual( + rule.actions.map((action) => action.hostname), + ['a.example.com:9926'] + ); +}); + +test('change probe: rules disagreeing is flagged — the rest of the page is one node’s list', () => { + const { body } = mergerFor('change-probe')([ok('a', probeBody('a')), ok('b', probeBody('b', { rules: [] }))]); + assert.equal(body.rulesDiverge, true); + assert.equal(body.rules.length, 1); +}); + +test('change probe: nothing answering is a 502, never an idle-looking probe', () => { + const { status, body } = mergerFor('change-probe')([down('a'), down('b')]); + assert.equal(status, 502); + assert.equal(body.sources.answered, 0); +}); + +// ------------------------------------------------------------------ discovery purge + +const purgeBody = (node, over = {}) => ({ + node: `${node}.example.com:9926`, + running: false, + urlPrefix: 'https://www.example.com/catalog/', + dryRun: true, + ratePerSecond: 200, + startedAt: 1000, + finishedAt: 5000, + error: null, + examined: 100_000, + owned: 25_000, + discovered: 9000, + leaseSkipped: 12, + deleted: 9000, + canceled: false, + ...over, +}); + +test('discovery purge: the per-node rows are the answer, and the totals are only a tally', () => { + const { body } = mergerFor('discovery-purge')([ok('a', purgeBody('a')), ok('b', purgeBody('b'))]); + assert.equal(body.totals.discovered, 18_000); + assert.equal(body.byNode.length, 2); + assert.equal(body.ranNodes, 2); + assert.equal(body.urlPrefix, 'https://www.example.com/catalog/'); +}); + +test('discovery purge: a node that has never run is named — its targets are still rendering', () => { + const { body } = mergerFor('discovery-purge')([ok('a', purgeBody('a')), ok('b', { node: 'b', running: false })]); + assert.deepEqual(body.unrunNodes, ['b.example.com:9926']); + assert.equal(body.totals.deleted, 9000); +}); + +test('discovery purge: ONE node that actually deleted makes "nothing was deleted" false', () => { + const { body } = mergerFor('discovery-purge')([ok('a', purgeBody('a')), ok('b', purgeBody('b', { dryRun: false }))]); + assert.equal(body.dryRun, false); +}); + +test('discovery purge: two prefixes are not silently added together', () => { + const { body } = mergerFor('discovery-purge')([ + ok('a', purgeBody('a')), + ok('b', purgeBody('b', { urlPrefix: 'https://www.example.com/deals/' })), + ]); + assert.equal(body.urlPrefix, null); + assert.equal(body.urlPrefixes.length, 2); +}); + +test('discovery purge: a pass still running has no finish time, on any node', () => { + const { body } = mergerFor('discovery-purge')([ + ok('a', purgeBody('a')), + ok('b', purgeBody('b', { running: true, finishedAt: null })), + ]); + assert.equal(body.running, true); + assert.deepEqual(body.runningOn, ['b.example.com:9926']); + assert.equal(body.finishedAt, null); +}); diff --git a/packages/console/test/overviewView.test.js b/packages/console/test/overviewView.test.js new file mode 100644 index 00000000..b82a6a5a --- /dev/null +++ b/packages/console/test/overviewView.test.js @@ -0,0 +1,216 @@ +/** + * The Overview's discovered-target purge card. + * + * This is the console's second corpus-DELETING action, and it has one interlock the first does + * not: gating the route has to happen BEFORE the purge, or crawlers re-mint exactly what was + * removed and the pass becomes a rate-limited way to delete corpus and change nothing. The plugin + * refuses that order with a 400; what this card owes the operator is that the refusal is never the + * first they hear of it. + * + * The other thing under test is the never-run state. A node that has never purged answers + * `{ running: false }` and nothing else — no counters at all — and rendering that as a row of + * zeroes would read as a completed, clean census on a node whose discovered targets are all still + * rendering. + */ + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { installDom, find } from './domShim.js'; + +installDom(); + +const { el } = await import('../src/admin/ui.js'); +const { load, render } = await import('../src/admin/views/overview.js'); + +const OVERVIEW = { + generatedAt: Date.now(), + node: 'a.example.com:9926', + workerIndex: 0, + localQueueStatus: 'queued', + control: { cluster: null, knownScopes: [] }, + nodes: [], + counts: null, + countsAsOf: null, + backlog: { enabled: true, interval: 60_000, running: false, lastRun: null }, + intervals: { statusSyncInterval: 1000, jobLeaseTime: 120_000, defaultRenderInterval: 21_600_000 }, + claimFloor: { floorMinute: 0, lagMs: null, oldestLeaseAgeMs: null }, + reconcile: { enabled: true, interval: 60_000, running: false, lastRun: null }, + orphanSweep: { dryRunDefault: true, maxDeletes: 1000, running: false, lastRun: null }, +}; + +const ANALYTICS = { available: true, scope: 'node', node: 'a.example.com:9926', rangeMs: 3_600_000, series: [] }; + +const CENSUS = { + node: 'a.example.com:9926', + running: false, + urlPrefix: 'https://www.example.com/catalog/', + dryRun: true, + ratePerSecond: 200, + startedAt: 1000, + finishedAt: 5000, + error: null, + examined: 100_000, + owned: 25_000, + discovered: 9000, + leaseSkipped: 12, + deleted: 9000, + canceled: false, +}; + +function makeCtx({ overview = OVERVIEW, purge = { running: false } } = {}) { + const views = {}; + const scratch = (id) => (views[id] ??= {}); + const calls = { posts: [] }; + return { + calls, + scratch, + busy: false, + get data() { + return scratch('overview'); + }, + async get(route) { + if (route === 'overview') return { ok: true, body: overview }; + if (route === 'analytics') return { ok: true, body: ANALYTICS }; + if (route === 'discovery-purge') return { ok: true, body: purge }; + return { ok: true, body: null }; + }, + async post(route, data) { + calls.posts.push({ route, data }); + return { ok: true, body: {} }; + }, + async run(fn) { + return fn(); + }, + render() {}, + reload() {}, + go() {}, + }; +} + +const draw = (ctx) => el('div', null, render(ctx)); +const cardTitled = (ctx, title) => + find(draw(ctx), (n) => n.attributes?.class === 'card' && n.textContent.startsWith(title)); +const button = (node, text) => find(node, (n) => n.tagName === 'BUTTON' && n.textContent === text); + +const ready = async (options) => { + const ctx = makeCtx(options); + await load(ctx); + return ctx; +}; + +test('a node that has never purged says so, instead of showing a clean census of zeroes', async () => { + const ctx = await ready(); + const card = cardTitled(ctx, 'Discovered targets'); + assert.ok(card, 'expected a Discovered targets card'); + assert.match(card.textContent, /No purge has run on this node since startup/); + assert.doesNotMatch(card.textContent, /Rows examined/); +}); + +test('the purge button is inert until a census has counted what it would delete', async () => { + const before = await ready(); + assert.equal(button(cardTitled(before, 'Discovered targets'), 'Purge discovered').attributes.disabled, ''); + + const after = await ready({ purge: CENSUS }); + const enabled = button(cardTitled(after, 'Discovered targets'), 'Purge discovered'); + assert.equal(enabled.attributes.disabled, undefined); + assert.match(cardTitled(after, 'Discovered targets').textContent, /9,000/); +}); + +test('a census that already deleted does not offer to delete again', async () => { + // `dryRun: false` means the last pass WAS the purge. Offering the danger button again would + // invite a second full walk of a prefix that has already been cleared. + const ctx = await ready({ purge: { ...CENSUS, dryRun: false } }); + assert.equal(button(cardTitled(ctx, 'Discovered targets'), 'Purge discovered').attributes.disabled, ''); +}); + +test('the census posts a dry run and the purge does not, both carrying the typed prefix', async () => { + const ctx = await ready(); + const card = cardTitled(ctx, 'Discovered targets'); + const input = find(card, (n) => n.tagName === 'INPUT'); + input.value = ' https://www.example.com/catalog/ '; + button(card, 'Dry-run census').listeners.click[0](); + assert.deepEqual(ctx.calls.posts.at(-1), { + route: 'discovery-purge', + data: { urlPrefix: 'https://www.example.com/catalog/', dryRun: true }, + }); +}); + +test('an empty prefix does nothing at all — a bare origin is refused upstream anyway', async () => { + const ctx = await ready(); + const card = cardTitled(ctx, 'Discovered targets'); + button(card, 'Dry-run census').listeners.click[0](); + assert.equal(ctx.calls.posts.length, 0); +}); + +test('the gate-first interlock is stated on the card, not left to the plugin’s 400', async () => { + const ctx = await ready(); + const text = cardTitled(ctx, 'Discovered targets').textContent; + assert.match(text, /discoverTargets: false/); + assert.match(text, /crawlers re-mint what this removes/); + // And the population's boundary: a sitemap-declared URL is corpus by the operator's own + // statement, whatever its traffic looks like. + assert.match(text, /sitemap-declared URL is never/); +}); + +test('a running pass can be stopped, and says which node is running it', async () => { + const ctx = await ready({ + purge: { + ...CENSUS, + running: true, + finishedAt: null, + runningOn: ['b.example.com:9926'], + sources: { mode: 'merged', answered: 2, configured: 2, complete: true, nodes: [] }, + }, + }); + const card = cardTitled(ctx, 'Discovered targets'); + assert.match(card.textContent, /running on b.example.com:9926/); + assert.ok(button(card, 'Stop'), 'a paced pass that outlives the request needs a way to end it'); +}); + +test('under cluster scope the pass refuses and names the keyspace problem', async () => { + const ctx = await ready({ + overview: { ...OVERVIEW, sources: { mode: 'merged', answered: 4, configured: 4, complete: true, nodes: [] } }, + }); + const card = cardTitled(ctx, 'Discovered targets'); + const census = button(card, 'Census (pick a node)'); + assert.ok(census); + assert.equal(census.attributes.disabled, ''); + assert.match(census.attributes.title, /run every node to cover the keyspace/); +}); + +test('a node nobody has purged is named — its discovered targets are still rendering', async () => { + const ctx = await ready({ + purge: { + ...CENSUS, + ranNodes: 1, + unrunNodes: ['b.example.com:9926'], + totals: { examined: 100_000, owned: 25_000, discovered: 9000, leaseSkipped: 12, deleted: 9000 }, + sources: { mode: 'merged', answered: 2, configured: 2, complete: true, nodes: [] }, + }, + }); + assert.match(cardTitled(ctx, 'Discovered targets').textContent, /Never run on b.example.com:9926/); +}); + +test('two nodes on different prefixes are not presented as one total', async () => { + const ctx = await ready({ + purge: { + ...CENSUS, + urlPrefix: null, + urlPrefixes: ['https://www.example.com/catalog/', 'https://www.example.com/deals/'], + ranNodes: 2, + totals: { examined: 200_000, owned: 50_000, discovered: 18_000, leaseSkipped: 24, deleted: 18_000 }, + sources: { mode: 'merged', answered: 2, configured: 2, complete: true, nodes: [] }, + }, + }); + assert.match(cardTitled(ctx, 'Discovered targets').textContent, /add up two populations/); +}); + +test('a pass stopped early reports what it never reached, rather than reading as complete', async () => { + const ctx = await ready({ + purge: { ...CENSUS, canceled: true, dryRun: false, discovered: 9000, deleted: 3000, leaseSkipped: 0 }, + }); + const text = cardTitled(ctx, 'Discovered targets').textContent; + assert.match(text, /stopped early/); + assert.match(text, /~6,000 left under this prefix/); +}); diff --git a/packages/console/test/probeView.test.js b/packages/console/test/probeView.test.js new file mode 100644 index 00000000..4f9a4151 --- /dev/null +++ b/packages/console/test/probeView.test.js @@ -0,0 +1,379 @@ +/** + * The Change probe view, executed. + * + * Four things have to stay true here, and each of them is a way the panel could read as healthy + * while the probe is doing nothing useful: + * + * - The change rate's denominator is the probes that HAD a baseline. A pass that is mostly + * seeding compares almost nothing, and dividing by `probed` would report that as a low + * change rate — "the catalogue is stable" — when the truthful answer is "we have not + * compared anything yet". + * - The pass counters are per FINISHED PASS, so `count` is passes and the recorded value is + * what a pass counted. Summing counts answers "how many passes ran", with a plausible + * number. + * - A dominant failure share is the endpoint-changed-shape alarm, and it is invisible in every + * other number: a failed probe leaves the signature untouched, triggers nothing, and looks + * exactly like a page that did not change. + * - A canary trip is one node's verdict about its own cohort. Folding four of them into a + * single boolean would name no node and match no log line. + */ + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { installDom, find } from './domShim.js'; + +installDom(); + +const { el } = await import('../src/admin/ui.js'); +const { load, render } = await import('../src/admin/views/probe.js'); + +const BUCKETS = 4; +const HOUR = 3_600_000; + +/** A prerender_ops pass counter: `count` is PASSES, `mean` is what each pass counted. */ +const passes = (series, passCount, perPass) => ({ + metric: 'prerender_ops', + path: `probe_${series}`, + method: null, + type: null, + count: passCount, + total: 0, + counts: new Array(BUCKETS).fill(passCount / BUCKETS), + mean: perPass, + median: perPass, + p95: perPass, + means: new Array(BUCKETS).fill(perPass), + p95s: new Array(BUCKETS).fill(perPass), +}); + +// 4 passes, each probing 1000: 400 seeded and 100 failed per pass, so 4,000 probed, 2,000 of them +// compared, and 240 changed. The honest change rate is 240/2,000 = 12%; against `probed` it would +// read as 6% — the same catalogue, a different verdict. +const ANALYTICS = { + available: true, + scope: 'node', + node: 'node-a', + rangeMs: 24 * HOUR, + startMs: 0, + endMs: 24 * HOUR, + bucketMs: 6 * HOUR, + bucketCount: BUCKETS, + truncated: false, + scan: { ms: 4, scanned: 10, kept: 10, cap: 20_000 }, + series: [ + passes('probed', 4, 1000), + passes('seeded', 4, 400), + passes('failed', 4, 100), + passes('changed', 4, 60), + passes('triggered', 4, 55), + passes('deferred', 4, 5), + passes('canary_trip', 2, 1), + passes('invalidated', 1, 1), + ], +}; + +const STATUS = { + enabled: true, + dryRun: true, + node: 'node-a', + ownerScopeNote: 'Probes only the URLs this node owns; every node sweeps its own slice.', + rules: [ + { label: 'price', pathPattern: '^/product/([^/]+)', source: 'request', invalidateScope: 'route:prefix:/product/' }, + ], + sweep: { + running: false, + armedInterval: 24 * HOUR, + lastRun: { + examined: 40_000, + owned: 10_000, + matched: 4000, + probed: 4000, + seeded: 1600, + unchanged: 1760, + changed: 240, + triggered: 220, + deferred: 20, + failed: 400, + errors: 0, + failureSamples: [{ url: 'https://www.example.com/product/a', rule: 'price', error: 'HTTP 500' }], + dryRun: true, + startedAt: Date.now() - HOUR, + finishedAt: Date.now() - 1000, + error: null, + }, + }, + canary: { + running: false, + armedInterval: 30 * 60_000, + cohortSizes: { price: 500 }, + lastRun: { + perRule: [ + { + rule: 'price', + cohort: 500, + changed: 90, + unchanged: 410, + compared: 500, + fraction: 0.18, + tripped: true, + action: { acted: false, reason: 'dry-run' }, + }, + ], + dryRun: true, + startedAt: Date.now() - 60_000, + finishedAt: Date.now() - 30_000, + error: null, + }, + }, +}; + +function makeCtx({ status = STATUS, analytics = ANALYTICS } = {}) { + const views = {}; + const scratch = (id) => (views[id] ??= {}); + const calls = { posts: [], renders: 0, reloads: 0 }; + return { + calls, + scratch, + busy: false, + get data() { + return scratch('probe'); + }, + async get(route) { + if (route === 'change-probe') + return status + ? { ok: true, body: status } + : { ok: false, status: 409, body: { error: 'changeProbe.enabled is false' } }; + if (route === 'analytics') return { ok: true, body: analytics }; + return { ok: true, body: null }; + }, + async post(route, data) { + calls.posts.push({ route, data }); + return { ok: true, body: {} }; + }, + async run(fn) { + return fn(); + }, + render() { + calls.renders++; + }, + reload() { + calls.reloads++; + }, + go() {}, + }; +} + +const draw = (ctx) => el('div', null, render(ctx)); +const tile = (ctx, label) => + find(draw(ctx), (n) => n.attributes?.class === 'stat' && n.children[0]?.textContent === label); +const button = (ctx, text) => find(draw(ctx), (n) => n.tagName === 'BUTTON' && n.textContent === text); + +const ready = async (options) => { + const ctx = makeCtx(options); + await load(ctx); + return ctx; +}; + +test('the change rate is measured against what was COMPARED, not against every probe', async () => { + const ctx = await ready(); + const changed = tile(ctx, 'Changed'); + assert.ok(changed, 'expected a Changed tile'); + assert.match(changed.textContent, /12%/); + assert.match(changed.textContent, /of 2\.0k compared/); +}); + +test('a pass counter is summed by VALUE — counting emits would report the number of passes', async () => { + const ctx = await ready(); + // 4 passes × 1000 = 4,000 probes. Summing `count` would say 4. + assert.match(tile(ctx, 'Probes').textContent, /4\.0k/); +}); + +test('a dominant failure share is called out as the endpoint alarm, not left as a number', async () => { + const ctx = await ready({ + analytics: { + ...ANALYTICS, + series: ANALYTICS.series.map((s) => + s.path === 'probe_failed' ? { ...s, mean: 900, means: s.means.map(() => 900) } : s + ), + }, + }); + const text = draw(ctx).textContent; + assert.match(text, /90% of probes failed/); + assert.match(text, /back on interval-only freshness/); + assert.ok( + find(draw(ctx), (n) => n.attributes?.class === 'note bad'), + 'and it should read as a fault' + ); +}); + +test('a trip that recorded no invalidation says so instead of implying pages were flushed', async () => { + const ctx = await ready(); + // 2 trips, 1 invalidation — the other was the dry run. + assert.match(draw(ctx).textContent, /1 canary trip recorded no invalidation/); +}); + +test('an empty window is explained by the pass cadence, not read as "nothing is probing"', async () => { + const ctx = await ready({ analytics: { ...ANALYTICS, series: [] } }); + const text = draw(ctx).textContent; + assert.match(text, /once per FINISHED pass/); + assert.match(text, /widen the range/); +}); + +// ---- the canary verdict ------------------------------------------------------ + +test('a trip names the node that tripped, because the cohort it judged was that node’s', async () => { + const ctx = await ready(); + assert.match(draw(ctx).textContent, /tripped on node-a/); +}); + +test('a refusal to invalidate is reported with its reason, not as a successful trip', async () => { + const ctx = await ready(); + assert.match(draw(ctx).textContent, /dry run — nothing invalidated/); +}); + +test('an unresolvable invalidateScope is a fault, not a footnote', async () => { + const ctx = await ready({ + status: { + ...STATUS, + dryRun: false, + canary: { + ...STATUS.canary, + lastRun: { + ...STATUS.canary.lastRun, + perRule: [{ ...STATUS.canary.lastRun.perRule[0], action: { acted: false, reason: 'unresolvable-scope' } }], + }, + }, + }, + }); + assert.match(draw(ctx).textContent, /names no configured route — NOTHING was invalidated/); +}); + +test('an empty cohort says the mass-change detector is dark rather than showing a clean zero', async () => { + const ctx = await ready({ + status: { + ...STATUS, + canary: { + ...STATUS.canary, + cohortSizes: { price: 0 }, + lastRun: { + ...STATUS.canary.lastRun, + perRule: [{ rule: 'price', cohort: 0, skipped: 'empty cohort' }], + }, + }, + }, + }); + assert.match(draw(ctx).textContent, /mass-change\s+detector is dark/); +}); + +// ---- running a pass ---------------------------------------------------------- + +test('a pass inherits the configured dry run by default, and can be forced to a dry one', async () => { + const ctx = await ready(); + button(ctx, 'Run sweep').listeners.click[0](); + assert.deepEqual(ctx.calls.posts.at(-1), { route: 'change-probe', data: { action: 'sweep' } }); + + ctx.data.runMode = 'dry'; + button(ctx, 'Run sweep').listeners.click[0](); + assert.deepEqual(ctx.calls.posts.at(-1), { route: 'change-probe', data: { action: 'sweep', dryRun: true } }); +}); + +test('under cluster scope the run buttons refuse and say why — a pass is one node’s rate budget', async () => { + const ctx = await ready({ + status: { + ...STATUS, + node: null, + sources: { mode: 'merged', answered: 4, configured: 4, complete: true, nodes: [] }, + }, + }); + const sweep = button(ctx, 'Sweep (pick a node)'); + assert.ok(sweep, 'expected the button to name the scope problem rather than being silently inert'); + assert.equal(sweep.attributes.disabled, ''); + assert.match(sweep.attributes.title, /that node’s probe rate/); +}); + +test('a disabled probe names the nodes it is off on — their slice is simply absent from every total', async () => { + const ctx = await ready({ + status: { + ...STATUS, + enabled: true, + disabledOn: ['node-c'], + sources: { mode: 'merged', answered: 4, configured: 4, complete: true, nodes: [] }, + }, + }); + // `enabled` true with a disabledOn list is exactly the cluster case: some nodes have it on. + const ctxOff = await ready({ status: { ...STATUS, enabled: false, disabledOn: ['node-c', 'node-d'] } }); + assert.match(draw(ctxOff).textContent, /is false on node-c, node-d/); + assert.match(draw(ctxOff).textContent, /never probed/); + assert.ok(draw(ctx)); +}); + +test('an enabled probe with no rules says no timer is armed rather than showing an idle sweep', async () => { + const ctx = await ready({ + status: { ...STATUS, rules: [], sweep: { running: false, armedInterval: null, lastRun: null } }, + }); + assert.match(draw(ctx).textContent, /no timer is armed/); +}); + +test('an unswept node is named, because it contributes zero to every figure above', async () => { + const ctx = await ready({ + status: { + ...STATUS, + sweep: { ...STATUS.sweep, unsweptNodes: ['node-b'], lastRun: { ...STATUS.sweep.lastRun, nodes: 3 } }, + }, + }); + const text = draw(ctx).textContent; + assert.match(text, /No sweep has finished on node-b/); + assert.match(text, /Oldest of 3 sweeps/); +}); + +test('the failure samples are shown, because a failed probe changes nothing and logs nowhere else', async () => { + const ctx = await ready(); + assert.match(draw(ctx).textContent, /HTTP 500/); +}); + +test('the status read failing still renders the card that turns the probe on', async () => { + const ctx = await ready({ status: null }); + assert.match(draw(ctx).textContent, /changeProbe.enabled is false/); +}); + +test('one node throwing does not delete the three that swept — both are reported', async () => { + // The merge takes the first error it finds across nodes, so an error and a full set of counters + // arrive together whenever the cluster is partly healthy. Showing only the error was dropping + // three good slices; showing only the counters hid that a quarter of the keyspace was missed. + const ctx = await ready({ + status: { + ...STATUS, + sweep: { + ...STATUS.sweep, + lastRun: { ...STATUS.sweep.lastRun, nodes: 4, error: 'read transaction expired' }, + }, + }, + }); + const text = draw(ctx).textContent; + assert.match(text, /read transaction expired/); + assert.match(text, /only the passes that did finish/); + assert.match(text, /Rows examined/); +}); + +test('an errored pass that counted nothing shows the error alone, not a row of dashes', async () => { + const ctx = await ready({ + status: { + ...STATUS, + sweep: { + ...STATUS.sweep, + lastRun: { node: 'node-a', startedAt: 1000, finishedAt: 2000, error: 'boom' }, + }, + }, + }); + const text = draw(ctx).textContent; + assert.match(text, /Last sweep failed: boom/); + assert.doesNotMatch(text, /Rows examined/); + assert.doesNotMatch(text, /No sweep has finished since startup/); +}); + +test('the failure threshold is compared explicitly, so an empty window never reads as failing', async () => { + // probed 0 makes the ratio null. Nothing may turn that into a "probe failures dominate" verdict. + const ctx = await ready({ analytics: { ...ANALYTICS, series: [passes('failed', 1, 0)] } }); + assert.doesNotMatch(draw(ctx).textContent, /probe failures dominate/); +}); diff --git a/packages/console/test/trafficView.test.js b/packages/console/test/trafficView.test.js index bb79c43d..2c0df05d 100644 --- a/packages/console/test/trafficView.test.js +++ b/packages/console/test/trafficView.test.js @@ -138,10 +138,11 @@ const CONFIG = { { path: 'ingress.routes', effective: [ - { match: 'prefix', path: '/product/', renderInterval: HOUR }, + { match: 'prefix', path: '/product/', renderInterval: HOUR, demandFloor: 30 * 60_000 }, { match: 'exact', path: '/' }, { match: 'prefix', path: '/search/', renderInterval: 60_000 }, { match: 'prefix', path: '/legal/', mode: 'passthrough' }, + { match: 'prefix', path: '/catalog/', renderInterval: 2 * HOUR, discoverTargets: false }, ], }, { path: 'ingress.excludePathPatterns', effective: ['/search/'] }, @@ -197,13 +198,44 @@ const ready = async () => { test('an excluded path is passthrough even when ingress.routes also declares it', () => { const index = cadenceIndex(CONFIG, 6 * HOUR); - assert.deepEqual(index.get('/search/'), { mode: 'passthrough', interval: 6 * HOUR, inherited: true }); + assert.deepEqual(index.get('/search/'), { + mode: 'passthrough', + interval: 6 * HOUR, + inherited: true, + }); }); test('a route with its own renderInterval is measured against it, one without inherits the default', () => { const index = cadenceIndex(CONFIG, 6 * HOUR); - assert.deepEqual(index.get('/product/'), { mode: 'prerender', interval: HOUR, inherited: false }); - assert.deepEqual(index.get('/'), { mode: 'prerender', interval: 6 * HOUR, inherited: true }); + assert.deepEqual(index.get('/product/'), { + mode: 'prerender', + interval: HOUR, + inherited: false, + discoverTargets: true, + demandFloor: 30 * 60_000, + }); + assert.deepEqual(index.get('/'), { + mode: 'prerender', + interval: 6 * HOUR, + inherited: true, + discoverTargets: true, + demandFloor: null, + }); +}); + +test('the discovery gate and the demand floor ride on the same join as the cadence', () => { + const index = cadenceIndex(CONFIG, 6 * HOUR); + // `discoverTargets` defaults to TRUE upstream, so an ABSENT flag is an open route and only an + // explicit false is a gate. Reporting "unknown" for the common case would put a dash in the + // column on every route of every deployment that has never used the feature. + assert.equal(index.get('/catalog/').discoverTargets, false); + assert.equal(index.get('/product/').discoverTargets, true); + // A CLASS label matched no route at all, so it carries no route flag — null, never the default. + assert.equal(cadenceFor(index, 'unclassified', HOUR).discoverTargets, null); + assert.equal(cadenceFor(index, 'unclassified', HOUR).demandFloor, null); + // A floor is only reported when it is a usable interval; anything else is "no floor set". + assert.equal(index.get('/product/').demandFloor, 30 * 60_000); + assert.equal(index.get('/catalog/').demandFloor, null); }); test('a route label that is a CLASS, not a configured path, resolves to that class', () => { @@ -583,3 +615,79 @@ test('an unavailable or failed analytics read still renders the knob that turns await load(ctx); assert.match(textOf(ctx), /management\.analytics\.enabled is false/); }); + +// ---- the discovery gate ------------------------------------------------------ +// +// A gated request is still SERVED — it just never enters the render rotation — so nothing about +// this panel is a failure count. What it has to keep straight is that the number is gated MISSES +// (repeat misses on an already-refused URL included), not URLs prevented, and that a deployment +// which has never configured a gate gets the capability described rather than an empty chart. + +const gated = (reason, bot, count) => combo('prerender_ops', 'discovery_gated', reason, bot, count); + +test('the gate panel splits the two gates and ranks the bots behind them', async () => { + const ctx = makeCtx({ + analytics: { + ...ANALYTICS, + series: [...ANALYTICS.series, gated('route', 'Googlebot', 900), gated('bot', 'SomeCrawler', 300)], + }, + }); + await load(ctx); + const text = textOf(ctx); + assert.match(text, /Gated misses/); + assert.match(text, /1\.2k/); + assert.match(text, /SomeCrawler/); + // The two gates answer different questions — a route whose URL space is combinatorial, versus a + // crawler that should not be minting at all — and they are fixed in different places. + assert.match(text, /discoverTargets: false/); + assert.match(text, /ingress.discoveryBots/); +}); + +test('a gated miss is counted as traffic held out, never as a URL prevented', async () => { + const ctx = makeCtx({ + analytics: { ...ANALYTICS, series: [...ANALYTICS.series, gated('route', 'Googlebot', 900)] }, + }); + await load(ctx); + assert.match(textOf(ctx), /not a count of URLs prevented/); +}); + +test('the bot filter narrows the gate panel, because discovery_gated carries the bot', async () => { + const ctx = makeCtx({ + analytics: { + ...ANALYTICS, + series: [...ANALYTICS.series, gated('route', 'Googlebot', 900), gated('route', 'SomeCrawler', 300)], + }, + }); + await load(ctx); + ctx.data.bots = ['SomeCrawler']; + const text = textOf(ctx); + assert.match(text, /300/); + assert.doesNotMatch(text, /1\.2k/, 'the filtered total must not include the bot that was filtered out'); +}); + +test('a deployment with no gate configured is told what the gate is for, not shown an empty chart', async () => { + const ctx = await ready(); + // CONFIG gates /catalog/, so start from a config that gates nothing at all. + const open = { + ...CONFIG, + layers: CONFIG.layers.map((layer) => + layer.path === 'ingress.routes' + ? { ...layer, effective: layer.effective.map(({ discoverTargets, ...rest }) => rest) } + : layer + ), + }; + const plain = makeCtx({ config: open }); + await load(plain); + const text = textOf(plain); + assert.match(text, /not configured/); + assert.match(text, /crawlers walk novel combinations into permanent render load/); + assert.ok(ctx); +}); + +test('a configured gate that refused nothing says so, rather than reading as unconfigured', async () => { + // CONFIG gates /catalog/ and the window carries no discovery_gated rows. + const ctx = await ready(); + const text = textOf(ctx); + assert.match(text, /1 route gated/); + assert.match(text, /refused nothing in this window/); +});