[harper-joseph]

Plugin 0.97.0, console 0.22.0. Fixes from a full review of the plugin at b312c1d, built in five areas, merged here, then reviewed again as one branch in three more rounds; those findings are fixed too. Every fix has a regression test that fails without it (plugin 1,814 tests, console 484, lint and format clean).

Deploying this needs a worker restart (schema changes, a renamed lease buffer) — see Deploy notes.

What it fixes

Sitemap walk

• A walk with a failed child no longer turns shifted URLs into departures. A URL that moved into a child that failed was hard-expired and re-rendered, then re-rendered again as a rejoin on the next walk. Now only the held URLs on the routes the failed child last held are held back and re-linked; every other departure proceeds.
• A truncated sitemap is a failed child, not a short one: a document whose root element never closes is refused (a truncated gzip decodes without error).
• Shrink guard (sitemap.shrinkGuard): a urlset whose entry count falls below (1 − maxRatio) × its stored count by at least minUrls is refused and counts as a failed child. A refused shrink is accepted only after acceptAfter refusals and either a new origin version (changed Last-Modified or content digest) arriving at least newVersionAfter after the first refusal, or acceptAge. A child the index stops listing is judged after the walk by the URLs it still holds (listed nowhere else), and an index is judged the same way, so renamed children are not a partial index. Every check has the minUrls floor.
• The origin-bypass token goes only to the root sitemap's host or domains, only over https, and redirects are followed manually with the decision re-made per hop (it used to follow redirects to any host with the token attached).
• One departure budget per walk (sitemap.departure.maxPerWalk, 5000): across every child, from normal prunes and accepted shrinks alike, spent only on departures that get an action, and shared round-robin across the children so one child's mass event can't starve the rest. Departures past it are re-linked to the child that unlinked them (validator cleared) and depart on later walks — deferred, not dropped (departure_deferred). A well-formed partial build that loses less than maxRatio can no longer depart everything it dropped in one walk.
• Every hold-back is visible: the refresh result and progress row carry holdBack — each child departures are held back for, why (refused shrink, failed child, budget), since when, and when a refusal can be accepted. A failed child's stored route keys are trusted only under the route table they were computed with (a fingerprint on the row), so a route edit — including a route added above another or an ingress-mode switch — widens its hold-back to its siblings instead of producing false departures.
• A child the index stops listing is pruned (behind the guards) instead of keeping its URLs "listed" forever; a refused omission holds back its routes like a failed child.
• A discovered URL's first sitemap listing gets the new-URL render window; a non-gone suppressed row it lists gets its recheck brought forward.
• A URL listed by two children no longer departs when the other child answers 304 (a narrowed, retried re-read of just the children that could list it).
• The discovery purge spares suppressed rows unless they are http-gone (they were re-minted and re-rendered in a loop).
• Every patch that can race a cross-node delete carries its primary key (a patch to a missing record stores only the patched fields); the orphan sweeps remove key-less stubs after a re-read, deleting only the raw row.
• Urlset rows store the first 500 entries (entryCount keeps the true count); the console pages over what is stored.
• Bulk REST writers (DELETE /sitemaps/<url>, collection revalidate, POST /sitemaps with background: false) run detached from the request transaction.

Change probe

• A pass skips only rows already probed since it (or the pass it resumes) began. It used to skip any URL whose last change was under reprobeAfter old — so a change caught by a daytime or manual sweep hid that URL from the next night's pass. reprobeAfter is retired (accepted, unused).
• The anchored pass can't be lost: it interrupts a dry-run or reseed pass (across workers, re-asserted until honoured), chains after any other pass, and is caught up at boot if no armed pass (or, in dry-run config, dry pass) served the latest anchor. A pass that throws is resumed from its cursor; a row-level ProbeState fault skips only that row. probe_anchor counts each outcome.
• The reseed after a canary trip acts on changes — except pages that all predate the invalidation epoch (so clearing a false trip restores them) and pages already healed after the trip (their claim postdates the epoch and agrees with the new observation). Coverage applies in every sweep while an invalidation is active; a covered change is counted as probe_covered, not changed, and a pre-epoch page whose claim agrees gets a verification so it serves. "Healed" requires every changed slot to be one the page check compares — a change to an unchecked slot (e.g. a strikethrough price) is acted on.
• Render check (changeProbe.renderCheck, default on): a new render whose page claim disagrees with the probe's stored observation gets one detached confirming re-probe, paced from the headroom the sweep leaves under ratePerSecond (the total origin rate never exceeds it); the page is hard-expired and re-filed only if the origin still disagrees, and a re-probe never clears a newer claim.
• Probe counters are emitted per batch (a thrown or restarted pass used to lose every call it made from the offload math); new probe_errors, probe_caught_up, probe_ignored. The first 1,000 failed actions of a pass are retried once after the walk, unless the row moved since.
• "Verified" requires a comparison that happened and agreed. The heartbeat is timer-driven. Rows skipped as fresh don't consume the pacing budget. Optional time-of-day canary schedule (changeProbe.canary.schedule, default off).
• probe_detection_lag: an upper bound on detection lag at every detected change.

Render queue

• The slow-lane retry waits on the effective cadence, not the route ceiling (a failing page with a 48h cadence waited 96h); a changed row retries at lease scale.
• "Render this now" asks get a head start (RenderSchedule.urgentAt, queue.ready.urgentHeadStart 0.5): they used to score 0 and sit behind every overdue row. A detected change still outranks an equally late ask; bulk re-files (collection revalidate, sitemap revalidate: true) are not marked. Both head starts scale with sitemapBoost.
• fileDueNow off the row's owner forwards to the owner (POST /prerender_peer/due-now, peer token; queue.dueNowForward) instead of a whole-row write that replaced the owner's row and its change mark. Falls back to the local write on refusal or connect failure; a plain ask is left to a slow owner.
• A result from a lease granted before a change mark doesn't store the old page (grant times are kept to the millisecond); a provably late result doesn't overwrite a newer page or release the newer lease (generation-checked release, superseded outcome). An ask filed mid-render survives the render.
• A key re-granted at the same due minute counts as a wedge miss, except for outcomes that leave the row due on purpose. Reconcile heartbeats its run; the verification walk stops re-repairing rows with no usable due time (keeper_unschedulable).
• New render change_lag_ms (change mark → render landed) and render_size (decoded size bands).

Serve path

• Proxied responses relay Location (resolved to absolute) and Content-Language — a proxied 3xx used to reach the crawler with no target.
• A page row we found but won't serve (stale, probe-expired, blob fault) is proxied without conditional validators, and no local 304 is built for it. Cached serves carry a weak ETag W/"<lastCached ms>-<device>" and Last-Modified from the render time instead of the origin document's validators.
• Peer rescue checks the owner copy's serve status before serving it.
• HEAD misses go upstream as HEAD, are never captured, and can't reopen a gone target or drop a stored 404. Bodies that won't be sent are drained (≤ 1 MiB, 5 s deadline) or released.
• Proxied origin bodies go through originBodyStream instead of Readable.toWeb: cancelling a toWeb stream in the window after creation throws ERR_INVALID_STATE as an uncaught exception (reproduced 20/20 in that window). Measured equal or cheaper per body; backpressure, truncation errors and the capture tee are tested.
• Invalidation state is held per worker from a subscription (with a 60s backstop re-read and per-request fallback until primed) instead of two point reads per cache-servable request; an apply is visible to the writer's own worker before recordInvalidation returns, and a clear lands with its commit (an aborted clear can't unguard a worker).
• \n-joined stored headers are appended as separate values (repeated vary / x-robots-tag / cache-control were dropped on cache serves). page.serveLinkHeader makes the stored Link header servable — default off, as before, until stored rows are sampled.
• Negative cache: the Target guard is read before a 404 body is teed.
• The peer token header is never forwarded to the origin.

Admin, config, metrics

• POST /prerender_admin/revalidate no longer throws (nextRenderTime was undefined since 0.94.0, and the throw rolled back the render it filed); no-undef is enabled for the plugin with Harper's declared globals.
• Override rows record the file value they replaced (ConfigOverride.fileHash): layers report masking (the file changed since) and redundant (it now matches). An override that would newly drop route or probe-rule entries is refused (409).
• The config-override doorbell and every onConfigApplied listener run detached from the committing request's context.
• The analytics scan folds rows as it streams (identical output, checked against a golden fixture); its window cache is bounded.
• Maxima: four timer intervals ≤ 2^31−1 ms (a larger value fired every 1 ms), render.negative.statuses 4xx integers except 429, admin read limits.
• host.pluginVersion is read at module load.

Console (0.22.0)

Every series this release adds is charted — the metric guard waives none of them:

• Change probe → Change safety: action errors (whether every detected change was acted on), caught up, ignored, covered; anchored-pass outcomes, with a skipped anchor flagged; detection lag from the pass start and from the previous pass (median, ≈p95, per rule); render-check outcomes.
• Queue → Change to cache: change_lag_ms per route — median, ≈p95, mean, trend. Page size: render_size bands and the share over 1 MB by route and device. Filing: due_now_forward outcomes (fallbacks flagged) and keeper_unschedulable.
• Sitemaps → Departures: the departure family grouped by what it means for a page (acted, shear, held back, deferred, skipped, lost), shrink refusals and acceptances, and the live hold-back list from each root's run row.
• Also: the probe view's per-batch wording; sitemap detail pages over the stored entries.

New config options

Option	Default
sitemap.shrinkGuard.maxRatio / .minUrls / .acceptAfter / .acceptAge / .newVersionAfter	0.5 / 1000 / 2 / 48h / 20h
sitemap.departure.maxPerWalk	5000 (lossless; -1 none)
changeProbe.renderCheck	true
changeProbe.canary.schedule	[] (fixed interval)
changeProbe.reprobeAfter	retired — accepted, unused
queue.ready.urgentHeadStart	0.5
queue.dueNowForward.enabled / .timeoutMs	true / 1000 (active only when peerRescue.token is set)
page.serveLinkHeader	false
invalidation.syncInterval	60000

New metrics

probe_errors, probe_caught_up, probe_ignored, probe_covered, probe_anchor, probe_detection_lag, probe_render_mismatch; render change_lag_ms and outcome superseded; render_size; queue_health keeper_unschedulable; prerender_ops due_now_forward (incl. timed-out), sitemap_shrink_refused, sitemap_shrink_accepted; sitemap departure outcomes relinked, deferred, listed-unchanged, unknown-child; invalidation_error view-read-error / view-subscribe-error; serve_error bad-header. probe_* rows are now per batch: sum values, don't count rows. Documented in METRICS.md.

Deploy notes

• Worker restart required. New attributes: RenderSchedule.urgentAt, ProbeState.renderRefiledAt, ConfigOverride.fileHash, and on Sitemap routes, routesFingerprint, shrinkRefusals, shrinkRefusedVersion, shrinkRefusedAt, shrinkAcceptedAt. The lease buffer is renamed (render_queue_v3): expect the usual one-time duplicate-render burst on restart.
• Don't write config overrides during the rolling restart (ConfigOverride is sealed; a row carrying fileHash replicating to a not-yet-restarted node is untested).
• Check stored child sitemaps' hosts and schemes before deploying: a child on a host outside domains, or on http://, is now fetched without the bypass token (and fails if the edge requires it).
• Existing override rows show masking: null until re-saved; rows now identical to the file show redundant: true — safe to clear.
• An anchored node restarted before any armed pass served the latest anchor runs a catch-up pass at boot (once more on the first restart after upgrading, if a dry run was the last pass recorded).
• Departures are now metered at 5000 per walk by default, including on deployments that set sitemap.departure.maxActions: -1: a large legitimate removal (a clearance event) departs over several walks instead of one. Set sitemap.departure.maxPerWalk: -1 to restore the old behaviour.
• A refused product child holds back departures on its route until the shrink is accepted (next rebuild that is still short, or acceptAge); the change probe still catches availability changes meanwhile.

Not in this release

A truncated origin body leaves the crawler's connection open (Harper pipes without an error listener; predates this branch). Own-code failures that answer 500 instead of falling back to the origin; a cap on UA-derived analytics bot labels; expiring non-default device pages on a probe trip; malformed URLs in prefix mode (500 → 400); the probe's swrTtl-relative hard expiry (don't raise page.swrTtl until it is fixed).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request updates the version of the @harperfast/prerender package from 0.96.0 to 0.97.0 in both package.json and package-lock.json. There are no review comments, and I have no additional feedback to provide.