Repository navigation
console v0.9.0: read the probe's origin-pressure signals and the purge's spare-visited pass - #137
Merged
Merged
Conversation
…'s spare-visited pass; v0.9.0
The console last caught up at plugin v0.55.0. Since then v0.56.0 and v0.57.0 added
three prerender_ops series and reshaped both node-local passes, and none of it was
being read — the metric-coverage guard exempts `metrics.changeProbe` because it names
its series with a template, so three new signals shipped invisible with a green suite
on both sides. One of them is the one the catalog says to ALERT on.
Change probe
- probe_throttled: origin pushback (429/502/503/504, connect/read timeouts). This is
the only signal that the probe is loading an origin that cannot take it, and the
sweep answers it by halving its own rate — so it covers less of the corpus per pass
while the change rate, the failure share and the trigger count all keep the shape
they had. Its own alarm, a head pill, and a per-node column, since the probe rate is
agreed and held PER NODE and a cluster total names nobody to go and look at.
- probe_fresh: rows skipped because a baseline was younger than reprobeAfter. Counted
against the rows a pass CONSIDERED, not against its probes — a skipped URL was never
attempted — and flagged past half, which means reprobeAfter sits too close to
sweepInterval and real cadence is quietly two sweep intervals.
- probe_unreadable: registry rows whose key would not decode. A storage-layer
escalation, said as one: no setting on that page reaches them.
- The three ways a pass can stop early are no longer one "Interrupted" label. Standing
down for a reseed is routine; giving up on a refusing origin means the slice was
never covered, and the counters are a partial count.
Discovered-target purge
- skipVisited, defaulted ON in the console though the plugin defaults it off — a
plugin default that changed behaviour for existing callers would be the wrong kind
of change, a console default is a suggestion to a human. Sent on the census as well
as the purge, so the number an operator approves is the number that happens.
- visitedSkipped/errors are now terms in the "not reached" subtraction. Without them a
pass that spared 40% of a prefix on purpose reported that 40% as keyspace it never
got to — a completed pass rendered as an interrupted one. (unreadable is not a term:
those rows never entered `discovered`.)
- abortedOnErrors and the delete-failure samples, because a failed delete leaves a row
in the corpus that appears in no render metric, no serve metric and nowhere else.
The merge learns the same distinctions: counters that sum, a pacing window taken as the
worst node's rather than a mean that was true nowhere, and the nodes that gave up or ran
a different delete predicate named instead of folded into a total.
Guard: a second metric contract test covers the probe family by the catalog's own
values list, which is what the regex-based one structurally cannot see. Verified to fail
when a series goes unread. The other dynamic families stay unguarded and the test says
so — they need a per-series decision that does not belong in a catch-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request bumps the version of @harperfast/prerender-console to 0.9.0 and introduces several UI and aggregation improvements to the management console. Key additions include support for sparing bot-visited targets (skipVisited) during discovery purges, handling and displaying origin pushback/throttling, tracking skipped fresh rows from resumable sweeps, and surfacing unreadable registry rows. It also updates the aggregation logic to correctly merge these new metrics across cluster nodes and adds comprehensive test coverage for these features. I have no feedback to provide as there are no review comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The console last caught up at plugin v0.55.0. Plugin v0.56.0 and v0.57.0 added three
prerender_opsseries and reshaped both node-local passes, and the console read none of it.It shipped invisible because the metric-coverage guard exempts
metrics.changeProbe— it names its series with aprobe_${series}template, which the guard's regex structurally cannot read. Green suite on both sides. One of the three missed series is the one the catalog says to alert on.Change probe
probe_throttledprobe_freshreprobeAfterprobeddrops after a restart. Sustained, it meansreprobeAftersits too close tosweepIntervaland real cadence is quietly two sweep intervals.probe_unreadablethrottledgets its own alarm, a head pill, and a per-node column — the probe rate is agreed and held per node, so pushback is a fact about one node's conversation with the origin, and a cluster total names nobody to go and look at.freshis counted against the rows a pass considered, not against its probes: a skipped URL was never attempted, so it sits outsideprobedentirely.unreadableis stated as a storage-layer escalation, because no setting on that page reaches those rows.Interruptedlabel. Standing down for a reseed is routine; giving up on a refusing origin means the slice was never covered and the counters are a partial count.Discovered-target purge
skipVisited, defaulted ON in the console although the plugin defaults it off. A plugin default that changed behaviour for existing callers would be the wrong kind of change; a console default is a suggestion to a human. Sent on the census as well as the purge, so the number an operator approves is the number that happens.visitedSkippedanderrorsare now terms in the "not reached" subtraction. Without them a pass that spared 40% of a prefix on purpose reported that 40% as keyspace it never got to — a completed pass rendered as an interrupted one. (unreadableis deliberately not a term: the walk skips those rows before anything reads theirsitemapUrl, so they never entereddiscovered.)abortedOnErrorsand the delete-failure samples, because a failed delete leaves a row in the corpus that appears in no render metric, no serve metric, and nowhere else.Cluster merge
Counters that sum, sum. The pacing window is taken as the worst node's, never a mean — averaging one throttled node against three healthy ones answers "was the probe being held back" with a number that was true nowhere. Nodes that gave up on a refusing origin, or ran a different delete predicate over the same prefix, are named rather than folded into a total.
Guard
A second metric contract test covers the probe family via the catalog's own
valueslist — the source the regex-based guard cannot reach. Verified by hand to fail when a series goes unread, and to pass again when restored.The other dynamic families (
sitemap_*,demand_*,queue_healthgauges,invalidation_*) stay unguarded, and the test says so explicitly. They are legitimately unread from analytics — the queue gauges come from the overview endpoint, the demand series have no panel — so guarding them means a waiver list stating a decision per series. That belongs with whoever makes those decisions, not in a catch-up.Not in this PR
changeProbeStatus()does not expose a rule'sstatusSignals, so the probe view's rules table cannot show which rules treat a 4xx as an observation rather than a failure. That needs a plugin field and therefore a plugin release train; the console's failure-share narrative stays correct without it. Worth a follow-up.Verification
packages/console: 210 pass, 0 fail (was 185 — 25 new tests)packages/plugin: 850 pass, 0 fail, untouchednpm run lintandnpm run format:checkclean at the root🤖 Generated with Claude Code