[harper-joseph]

What

1. The health line. Each node writes one [prerender] health {json} line every health.interval (default 5 minutes). The line carries:
   • The bot requests answered since the previous line, as node totals and per prerender route:
     • what answered (source) and the status class;
     • each 5xx filed by cause: own5xx (the plugin's), originUnreachable (the origin never answered), or the origin's own relayed 5xx (the rest);
     • responses over 1 s and 3 s;
     • snapshots older than 1× and 2× their route's interval;
     • truncated deliveries.
   • The queue keeper's state: ok, heartbeat age, ready-set depth.
   • The change probe's state: age of the last completed pass, whether the last pass failed, disarmed fields.
   • The scheduled sitemap walk: the oldest root success, roots that never completed, failing roots, and failed child sitemaps.
2. Sitemap refresh rows keep their last success. SitemapRefresh.lastRefreshed is now restated on every progress write, re-read before each one, and the later value kept. Before, put replaced the row, so a running or failed walk erased the only record of when the root last completed.

Why

Alerting from logs on what crawlers experience. Analytics stay in each node's hdb_analytics until something reads them; this line rides whatever already ships the node's log. It is written whether or not anything is wrong, so a node with no line is itself the signal that the plugin, or every worker on the node, has stopped.

How

• Counting: one call at the end of handleBotRequest, once the status is known. It makes about ten Atomics.add into one shared buffer held through getSab. It doesn't depend on analytics.enabled, never reads hdb_analytics, and makes no closures or strings per request.
  • The count runs in a finally and is guarded, so it can't replace the response.
  • An origin fetch that throws is marked in util/upstream.js, so the handler's 500 is filed as originUnreachable.
  • render-now's timeout 504 is marked madeHere, so it is filed as ours.
• Writing: every worker arms a timer for the next interval boundary. The first to compare-and-swap the boundary into a shared clock drains every counter with Atomics.exchange(…, 0) and writes the line. A busy or wedged worker can't hold it back.
• Live route changes: each worker folds the old layout's buffer into the new one as it switches, route blocks by name, else other. Nothing counted before the change is lost, and a change back finds nothing stale.
• State sections: the writer reads them from state each module already keeps: keeperHealth (renderSchedule.js), probeHealth (changeProbe.js) and sitemapHealth (Sitemap.js). None touches a residency-pinned table. A section that throws or takes over 5 s is written as null and named in err.
• Config: a health group with enabled, interval and level (default notify). On hosts that forward only warn and above from components to their log shipper, set warn.
• Docs: METRICS.md section 4f (fields, LogQL, what to alert on, the limits below), a row in section 5's absence table, and the README config sample.
• Cost: a handful of atomic adds per bot request. Once per interval, one worker drains a few hundred cells, makes a few small node-local reads, and logs about 0.6–1 KB.

Visible elsewhere

• The console's sitemap list now shows a running root's previous completion as "checked … / Last walk", where it showed "—". The admin API's sitemap list and detail rows carry lastRefreshed on running and failed rows too.

How it was checked

• Unit tests (test/health.test.js):
  • per-field and per-route counts, and 5xx filed by cause (own, unreachable, relayed, made here);
  • drain-to-zero and span, the disabled path, one write per boundary, the level;
  • state sections: value, null, throw, and a hang cut off at 5 s;
  • a live route change keeps earlier counts and leaves nothing stale; the 32-route cap.
• Real threads: 4 worker_threads count 80,000 requests into the same shared memory and race for one boundary. Exactly one writes, and the written line plus the remainder equal the total.
• The serve path (botServePath.test.js), with analytics off: an origin miss, a relayed 502, an origin that hangs up (originUnreachable), an old cached snapshot and a handler 500, each filed correctly.
• The sitemap fix and readers (sitemapWalkGuards.test.js):
  • a failed walk keeps the last completed time on every write;
  • a walk overtaken by another restates the newer completion;
  • failedChildren counts; removed roots, children and the all marker are skipped.
• probeHealth (changeProbe.test.js): the completed-pass age, a failed last pass, disarmed fields, and a dry run not counted on an armed probe.
• Revert checks: the sitemap carry-forward, the overtaken walk, the route-change fold and the unreachable-origin split each have a test that fails with that fix reverted.
• Local end to end on Harper Pro 5.2.15 in Docker with 16 workers. Logging matched a managed host: root log at warn on std streams, components' log at info, file only.
  • 2,080 bot requests: 1,650 × 200, 200 relayed 502, 200 × 404, 50 slow, 30 to an unreachable origin. The lines summed exactly: originUnreachable 30, own5xx 0, every route right.
  • One line per 10 s interval, written by whichever worker claimed it.
  • A rolling restart of all 16 workers lost nothing.
  • At notify the line reached only the log file; at warn it also reached container output.
• LogQL: regexp → line_format → json → unwrap was checked on a local Loki 3.5.3 against lines in this format. That covered sums, ratios, bracketed route paths, state fields and the missing-line query. Malformed lines raised no query error.
• Review: an independent review found the route-change loss, the invisible child failures, the overtaken walk, render-now's 504, the dry-run fallback and several doc gaps. All are fixed above.
• npm run lint, npm run format:check and the plugin suite (2,111 tests) pass. This repo has no PR CI.

Known limits (documented in METRICS.md 4f)

• A process restart loses up to one interval of counts. So does a worker restart that also changes the route table, because the new workers never knew the old layout.
• A request counted while a line drains can split its fields across two lines. Sums over a window stay exact.
• Snapshot age is judged against the route's interval, not a target's own stored interval.
• A root whose very first walk failed has no sitemap row yet, so it isn't counted.

Not verified

• Production: one writer per node at production worker counts, and the line on a hosted Loki. The first deploy confirms both.
• Thresholds: none are set here. They should come from a week of real lines.

Release

prerender-v0.108.0: 0.106.1 is #256 and 0.107.0 is #254. After merge, I'll cut the release; consumers bump the tarball and, where needed, set health.level: warn.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a new 'health line' feature that periodically logs a JSON line containing metrics about bot requests and background tasks (queue keeper, change probe, sitemap walks) for log-based alerting. The metrics are tracked in shared memory across workers. The review feedback suggests executing the state providers in parallel using Promise.all instead of sequentially to prevent accumulated timeouts if multiple providers hang.

[gemini-code-assist]

The state providers are currently executed sequentially within a for...of loop. If one or more providers hang or take a long time to respond, the sequential execution will accumulate their delays (up to N * STATE_TIMEOUT_MS, which could be 15 seconds or more). Running them in parallel using Promise.all ensures that the total execution time is capped at STATE_TIMEOUT_MS (5 seconds), improving the responsiveness and reliability of the health line generation.

	const entries = Object.entries(stateProviders);
	const results = await Promise.all(
		entries.map(async ([name, read]) => {
			try {
				const section = await withTimeout(
					Promise.resolve().then(() => read(nowMs)),
					STATE_TIMEOUT_MS
				);
				return { name, value: section ?? null };
			} catch (e) {
				getLogger().debug?.('[prerender] health: the ' + name + ' section failed: ' + (e?.message ?? String(e)));
				return { name, value: null, error: true };
			}
		})
	);
	for (const { name, value, error } of results) {
		line[name] = value;
		if (error) {
			errors.push(name);
		}
	}

[harper-joseph]

Agreed. Fixed in 354de6b: the sections are now read with Promise.all, so sections that hang share a single 5 s deadline. Results are still written in provider order, with failures named in err. The test now hangs two sections and cuts both off with one 5 s tick, which a sequential read cannot pass.

[harper-joseph]

Review round (independent agent review, plus my own pass), fixed in 1617b05, e202eb9 and 9ea5c29:

• An unreachable origin was counted as our own 5xx. It's now its own field, originUnreachable.
• A live route change lost every count since the last line, and a change back brought old counts back. Counts are now folded into the new layout.
• A sitemap index whose children all fail still showed as succeeding. Added failedChildren.
• A walk overtaken by another could restate an older completion. The last success is now re-read before each write.
• render-now's timeout 504 was counted as the origin's. It's now ours.
• probeHealth's fallback could count a dry run on an armed probe. It now applies the same check as anchorServedSince.
• Doc gaps. METRICS.md 4f now covers the other bucket, where lastPassAgeH is measured from, the snapshot age baseline, requests split across two lines, and the restart that changes the route table.

Each behavioural fix has a test that fails with the fix reverted. The plugin suite (2,111 tests) passes, and I re-ran the local Docker check on the new head. The PR description is rewritten to match.