feat(plugin): probe only sitemap-listed targets (opt-in), and re-render a product that rejoins its sitemap; v0.89.0

packages/plugin/METRICS.md

@@ -329,6 +329,16 @@ Grep-able, `[prerender]`-prefixed, and each is the richer record of something th
 - **`change-probe <rule>: pageCheck field <slot>:<fact> … is DISARMED`** — the mapping-defect guard
   (`changeProbe.mappingGuard`) found a mapped field disagreeing with most witnessed pages and stopped it
   triggering. Nothing else announces it; fix the mapping (an edit re-arms it) or remove it.
+- **`Arrival check: N rejoined …`** — the outcome tally of the post-walk sitemap-arrival action
+  (`ingress.routes[].arrivalAction`, `sitemap.arrival`): URLs a walk re-attributed after an EARLIER
+  walk unlinked them, with `render`/`would-render` and the named skips. Deliberately NOT a metric
+  series (a new `sitemap_*` family needs a console panel first); the same tally is `arrivals` on the
+  refresh result and on `GET /sitemap_refresh/<root>`.
+- **`change-probe sweep: changeProbe.scope left N rule-matched targets unprobed and probed only M`** —
+  under `changeProbe.scope: listed`, a pass that skipped more than it kept. Either the corpus really is
+  mostly unlisted (the setting is wrong for this site) or sitemap walks stopped listing what they used to;
+  check `sitemap_removed` and the departure tally. The per-pass count itself is `outOfScope` on the
+  `GET /prerender_admin/change-probe` pass record — the origin requests the scope saved.
 - **Render verdicts per URL** — `Suppressing prerendered url …`, `Prerender failed for … (reason)`,
   `Retrying … (failure strike N)`, `redirected to … which is <class>`. The aggregate is
   the `render` metric's `outcome` series; these lines carry the URL. Most are `info`/`debug` now (they are normal

packages/plugin/README.md

@@ -1291,6 +1291,16 @@ up as a `probe_failed` share and a loud log line, not as schedule churn. An extr
 path yields null is a failure too, so a shape change cannot flip every signature at once and
 mass-trigger. Suppressed targets are skipped — suppression owns its own recheck cadence.
 
+**Scope** (`scope`, default `all`). `listed` probes only targets a sitemap lists now, plus those a walk
+unlinked within `unlistedGrace` (48h by default, measured from `Target.unlistedAt`), and skips targets
+discovered from traffic and products long gone from the sitemap — counted per pass as `outOfScope`, the
+origin requests saved. It is only safe where the sitemap **is** the availability feed and the
+sitemap-departure check is armed (`ingress.routes[].departureAction: render`,
+`sitemap.departure.dryRun: false`), which already re-renders a product's page the day it leaves the
+sitemap; a config warning says so otherwise. Its pair is `arrivalAction: render` on the same route
+(`sitemap.arrival`, dry-run by default), which re-renders a product the day it **rejoins** its
+sitemap instead of leaving its out-of-stock snapshot serving until the next cadence render.
+
 Once a probe rule covers a route's volatile fields, that route's `renderInterval` can usually be
 raised substantially — the interval then only bounds what the probe cannot see (client-side
 content: reviews, image sets), and the render budget freed is what pays for the burst of

packages/plugin/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender",
-	"version": "0.88.0",
+	"version": "0.89.0",
 	"type": "module",
 	"description": "Configurable Harper plugin for prerendering pages for bots and crawlers",
 	"license": "Apache-2.0",

packages/plugin/src/config.js

@@ -40,7 +40,7 @@ import { describeSecret } from './util/redact.js';
 // evaluation time. The count has to come from the compiler rather than from raw config,
 // because the finding's whole job is to catch entries the compiler REJECTED (a typo'd
 // `match`), which the raw array still contains.
-import { prerenderRouteCount } from './util/routeClass.js';
+import { DepartureAction, prerenderRouteCount } from './util/routeClass.js';
 // NOT cyclic, on purpose: changeProbeSpec.js is the probe's pure half and imports nothing of the
 // runtime, precisely so this module can compile prospective rules inside collectConfigWarnings.
 import { inspectProbeRules } from './util/changeProbeSpec.js';
@@ -783,6 +783,34 @@ export const collectConfigWarnings = (target = config, { prerenderRoutes } = {})
 					'nothing is re-rendered and nothing is invalidated until dryRun is turned off'
 			);
 		}
+		// `scope: listed` stops probing a product once it has been out of its sitemap for `unlistedGrace`,
+		// on the premise that the departure check already re-rendered it on the way out. Without an armed
+		// departure nothing does, and the page keeps whatever it said when it left. Read off the RAW route
+		// list, not the compiled one: this may be checking a prospective config (see `prerenderRoutes`).
+		if (target.changeProbe.scope === 'listed') {
+			const { departure } = target.sitemap;
+			const routes = Array.isArray(target.ingress.routes) ? target.ingress.routes : [];
+			const routeDeparts = routes.some(
+				(route) =>
+					route &&
+					route.mode !== 'passthrough' &&
+					(route.departureAction === DepartureAction.RENDER || route.departureAction === DepartureAction.EXPIRE)
+			);
+			if (!departure.enabled || departure.dryRun || !routeDeparts) {
+				add(
+					'warn',
+					'changeProbe.scope',
+					"changeProbe.scope is 'listed' but sitemap departure is not armed (" +
+						(!routeDeparts
+							? 'no prerender route sets departureAction'
+							: !departure.enabled
+								? 'sitemap.departure.enabled is false'
+								: 'sitemap.departure.dryRun is true') +
+						') — a product that leaves its sitemap stops being probed after changeProbe.unlistedGrace and ' +
+						'nothing re-renders it on the way out, so its page keeps its last snapshot until its cadence render'
+				);
+			}
+		}
 		// The canary's bulk action needs the invalidation read path on; the mismatch would otherwise
 		// surface only at trip time, which is 4am during the exact event the operator configured for.
 		if (

packages/plugin/src/configSchema.js

@@ -105,6 +105,7 @@ export const configSchema = group('Prerender plugin configuration.', {
 				'Ordered route list (forwarded mode). Each entry is ' +
 					"{ match: 'exact' | 'prefix' | 'contains', path: string, mode?: 'prerender' | 'passthrough', " +
 					'queryParams?: string[], renderInterval?: number, discoverTargets?: boolean, demandFloor?: number, ' +
+					"departureAction?: 'none' | 'expire' | 'render', arrivalAction?: 'none' | 'render', " +
 					'rawCache?: boolean }.\n\n' +
 					'FIRST MATCH WINS, so order most-specific first. That ordering is what lets a passthrough ' +
 					'carve-out sit inside a prerendered prefix (`/products/clearance/` above `/products/`) ' +
@@ -159,6 +160,19 @@ export const configSchema = group('Prerender plugin configuration.', {
 					'sitemap says something about that product, while a listing URL leaving a paginated sitemap ' +
 					'usually means the catalog was re-bucketed, and acting on it would expire current pages. ' +
 					'Bounded and observable by `sitemap.departure`.\n\n' +
+					"`arrivalAction` (default 'none', prerender routes only) — the other half: what happens when a " +
+					'URL on this route REJOINS a sitemap after an earlier walk unlinked it. On the same retail ' +
+					'catalog a product comes back to the sitemap the day it is restocked, and its cached page — ' +
+					'last rendered while it was unavailable — keeps serving out-of-stock markup until its next ' +
+					'cadence render.\n' +
+					'  none    — re-attribute and nothing else, as before.\n' +
+					'  render  — hard-expire the cached pages and file the URL to render at the current minute: ' +
+					'the departure `render` action, through the same code.\n' +
+					'A rejoin is told from a paginated sitemap\u2019s boundary shear (a URL moving to a later child ' +
+					'is unlinked and re-attached inside ONE walk) by `Target.unlistedAt`, which the unlinking walk ' +
+					'stamps with its own start — only a stamp from an earlier walk counts. Targets unlinked before ' +
+					'v0.89.0 carry no stamp, so their rejoin is not seen. Bounded and observable by ' +
+					'`sitemap.arrival`.\n\n' +
 					'`discoverTargets` (default true, prerender routes only) — whether a bot visiting an UNKNOWN ' +
 					'URL on this route creates a target for it. Set false on routes whose URL space is ' +
 					'combinatorial (faceted navigation, filter/sort permutations): crawlers walking those links ' +
@@ -1133,6 +1147,40 @@ export const configSchema = group('Prerender plugin configuration.', {
 					'holds the sustained rate to ratePerSecond whatever origin latency does.',
 				{ min: 1 }
 			),
+			scope: option(
+				'all',
+				'Which targets the sweep and the canary probe.\n\n' +
+					'"all" (default) probes every owned, rule-matched target, however it entered the registry.\n\n' +
+					'"listed" probes a target only while a sitemap lists it (`sitemapUrl` set) or for ' +
+					'`unlistedGrace` after a walk unlinked it (`Target.unlistedAt`). Targets no sitemap has listed ' +
+					'(discovered from traffic), targets unlinked longer ago than the grace, and targets unlinked ' +
+					'before v0.89.0 (which carry no stamp) are skipped and counted as `outOfScope` in the pass ' +
+					'stats — the origin requests saved per pass. The canary cohort is built from, and re-checked ' +
+					'against, the same selection. Suppressed targets are skipped under either scope, as always.\n\n' +
+					'ONLY SAFE WHERE THE SITEMAP IS THE AVAILABILITY FEED AND DEPARTURE IS ARMED. On a catalog ' +
+					'whose product sitemap drops an item the day it sells out, the departure check ' +
+					"(`ingress.routes[].departureAction: 'render'` with `sitemap.departure.dryRun: false`) already " +
+					'expires and re-renders that page, so probing the unlisted long tail buys little. Anywhere else ' +
+					'this simply stops watching those pages: an unlisted page whose price or availability changes ' +
+					'keeps its snapshot until its cadence render. A config warning names the case where no route ' +
+					'has departure armed, and a pass that skips more rule-matched targets than it probes logs a ' +
+					'warning (most of the watched corpus not being in any sitemap is either a site this setting is ' +
+					'wrong for, or sitemaps that stopped listing what they used to).\n\n' +
+					'A FAILED sitemap walk cannot shrink the scope — it unlinks nothing. A child sitemap that parses ' +
+					'but is truncated can, once it has stayed truncated for longer than the grace. Switching to ' +
+					'"listed" in continuous or anchored mode paces its first cycle against the old, larger slice, ' +
+					'so that cycle finishes early; the next one paces to the real slice.',
+				{ enum: ['all', 'listed'] }
+			),
+			unlistedGrace: option(
+				2 * DAY,
+				'SCOPE "listed" ONLY: how long after a sitemap walk unlinks a target it is still probed. A URL ' +
+					'briefly missing from one walk (a truncated child sitemap, an origin rebuilding its sitemaps) ' +
+					'keeps being watched while later walks decide whether it really left, and a real departure is ' +
+					'still probed while its re-render lands. Measured from the start of the walk that unlinked it. ' +
+					'0 probes only currently-listed targets.',
+				{ unit: 'ms', min: 0 }
+			),
 			reprobeAfter: option(
 				12 * HOUR,
 				'Skip a URL whose stored baseline is younger than this. What makes a sweep RESUMABLE: the ' +
@@ -2210,6 +2258,67 @@ export const configSchema = group('Prerender plugin configuration.', {
 				),
 			}
 		),
+		arrival: group(
+			'What a refresh does about a URL that REJOINS a sitemap — re-attributed by a walk after an ' +
+				'EARLIER walk unlinked it. The action is declared PER ROUTE (`ingress.routes[].arrivalAction`); ' +
+				'this group bounds and observes it, and nothing here does anything until at least one route ' +
+				'opts in. The mirror of `sitemap.departure`, with its own switches so arming one never arms ' +
+				'the other.\n\n' +
+				'SHEAR IS KEPT OUT BY A STAMP, NOT BY WAITING. A walk that unlinks a target stamps ' +
+				'`Target.unlistedAt` with the walk\u2019s own START, and the re-attach compares against the ' +
+				'current walk\u2019s start: a URL that moved to a later child of a paginated index is unlinked ' +
+				'and re-attached inside one walk, so its stamp is this walk\u2019s start and it is not a rejoin. ' +
+				'Rejoins are therefore known the moment they are re-attached; they are still ACTED on after the ' +
+				'walk, by the same executor as departures.\n\n' +
+				'WHAT IT DOES NOT SEE: targets unlinked before v0.89.0 (no stamp — they re-render on their ' +
+				'cadence, as before); rejoins on a `revalidate: true` walk (which files every listed URL due ' +
+				'now anyway). A URL that moves between two ROOT sitemaps looks like a departure in the first ' +
+				'root\u2019s walk and a rejoin in the second\u2019s — the departure check has the same limitation, ' +
+				'and the cost is one render per such URL, inside the ceilings.\n\n' +
+				'Reported as `arrivals` on the refresh result and the progress row (`considered`, `capped`, and ' +
+				'an outcome tally: `render`, or `would-render` under `dryRun`, for actions; `route-opted-out`, ' +
+				'`suppressed`, `unlinked`, `target-gone` and `capped` for the candidates nothing happened to) and ' +
+				'in the walk\u2019s log — not as a metric series.',
+			{
+				enabled: option(
+					true,
+					'Master switch for the arrival action. Routes still have to opt in, so leaving this on costs ' +
+						'nothing until one does; it exists so an operator can stop the behaviour during an incident ' +
+						'without editing the route list.'
+				),
+				dryRun: option(
+					true,
+					'Decide and report, write nothing. SEPARATE from `sitemap.departure.dryRun` on purpose: a ' +
+						'deployment that has already armed departures and then opts a route into arrivals must still ' +
+						'get a measured walk first, because the number that matters — how many rejoins one walk ' +
+						'produces, and in particular the burst when a truncated child sitemap recovers and every URL ' +
+						'it had dropped rejoins at once — is not one anybody has until they have looked. Inheriting ' +
+						'the departure switch would act on the very first walk.'
+				),
+				// -1 = no ceiling on both, for the reasons `sitemap.departure` spells out; read through
+				// `departureLimit`.
+				maxActions: option(
+					5000,
+					'Ceiling on rejoined URLs ACTED ON in one walk, or -1 for no ceiling. Skipped candidates do not ' +
+						'count against it.\n\n' +
+						'WHAT IT GUARDS AGAINST: the rejoin burst. A child sitemap that fetched truncated in one walk ' +
+						'unlinked everything it dropped; when it recovers, every one of those URLs rejoins in the same ' +
+						'walk, and without a ceiling that would expire and re-render all of them at once.\n\n' +
+						'OVERFLOW IS DROPPED FOR GOOD, like a capped departure: the re-attach has already cleared the ' +
+						'stamp, so no later walk sees the URL as a rejoin. It keeps its target, attribution and ' +
+						'cadence; it just never gets its route\u2019s arrival action. -1 removes the ceiling; 0 acts on ' +
+						'nothing (every candidate is still decided and reported).',
+					{ min: -1 }
+				),
+				maxCandidates: option(
+					50000,
+					'Ceiling on rejoined URLs HELD for the post-walk action, or -1 for no ceiling. Bounds memory, as ' +
+						'`sitemap.departure.maxCandidates` does. Overflow is DROPPED FOR GOOD and reported as ' +
+						'`arrivals.capped`. -1 removes the ceiling; 0 collects nothing.',
+					{ min: -1 }
+				),
+			}
+		),
 		userAgent: option(
 			'HarperSitemapCrawler/1.0',
 			'User-Agent for Harper’s sitemap crawler fetch. Unlike the proxy fetch UAs, a sitemap fetch ' +