[harper-joseph]

Problem

The Bloom ring behind render.demand recorded every prerender-class request from every bot, with no
filter on either axis. Two consequences, and both fail silently.

It counts the wrong bots. recordVisit was called with a URL and nothing else — the bot name was never
passed in, so no filtering was possible. A third-party scraper's visit promoted a page exactly as hard as a
search engine's. On the deployment this was measured against, ~36% of bot requests come from crawlers the
site is not being optimised for.

It counts URLs the ladder can never act on. The call sat before resolveResource and before
maybeSchedule, so it ran ahead of the discovery gate. Every URL on a prerender route was recorded,
including two large classes that own no Target and never will: misses on discovery-gated routes
(combinatorial facet URLs — discoverTargets: false stops the mint, not the recording) and URLs the origin
404s. On the same deployment those were ~1.2M requests/day, ~25% of all bot traffic.

Why that is a correctness problem, not a cost one. The ring's false-positive rate is fill^k, so it
degrades off a cliff rather than gradually. At the default k=7, bitsPerSlice = 1<<20:

distinct URLs per slice	fill	false-positive rate
100,000 (the documented sizing)	0.49	0.7%
320,000	0.88	41%
640,000 (measured in production)	0.986	91%

At 91% the ring answers "visited" for essentially everything, so the ladder promotes the whole corpus to its
floor. Nothing else moves: the decision counters look healthy, fast_fraction is unaffected, targets sit at
plausible rungs. The cadence has simply stopped being demand-driven, and the only metric that says so is
demand_fill — which, as it happens, was documented in a way that hides it (below).

Changes

render.demand.bots — same shape and matching rules as ingress.discoveryBots: ['*'] (default)
counts every bot, [] counts none, a list counts exactly those, case-insensitively. Default is unchanged
behaviour, so this is opt-in. The two allowlists share a compile helper but not their cached state, and
there is a test pinning that they stay independent.

They are deliberately separate knobs rather than one: minting corpus is permanent, while counting demand
only reallocates cadence inside a budget that already exists, so a deployment can reasonably answer the two
questions differently.

A rotation gate. recordVisit moves behind resolveResource into a new recordDemand, so it can read
cacheStatus — a sharper test than resource.miss:

• Any status that found a page row counts: hit, swr, stale, invalidated, blob-missing,
  blob-timeout, peer-rescue. A page row exists only where a Target does, so treating those as unvisited
  would demote pages that are merely late — the false-negative direction this filter must not have.
• A miss counts only when this request is what puts the URL into the rotation: a cacheable 200 on a
  route that mints targets. Not on a discovery-gated route, not on a non-200.
• bypass (non-GET) and skip (a deliberate cache bypass) never looked for a page row, so they
  cannot claim one — and they are not miss either, since a miss looked and found nothing. They take
  neither branch. That is the conservative reading of an ambiguous request, not a claim that on-demand
  traffic is illegitimate: resolveResource stamps cacheStatus before the render-now branch, so an
  on-demand render of an unknown URL arrives as a plain miss and does record.

Reading guidance for demand_fill, in both the metric catalog and METRICS.md. It is a sawtooth that
resets to ~0 at each sliceMs rollover and climbs until the next one, so the documented "average it,
never sum" reports the middle of the ramp. Because the exponent is 7, that distinction is the whole
signal: a mean of 0.75 implies 13% false positives, and the 0.986 peak of the same slice implies 91%.
Take max/p95. bitsPerSlice's own doc now carries the sizing curve and says to cut what goes in before
raising it, since that row replicates on every flush.

For the human reviewer

• The one behaviour change I am least sure about is the move across the await. recordVisit used to
  run before resolveResource; it now runs after. If resolveResource throws (an origin connect timeout,
  ~0.2% of proxy fetches) no visit is recorded where one previously was. I think that is correct — we
  cannot classify a URL whose origin fetch failed — but it is a real difference and it is not free.
• A residual false negative I chose to accept: a page on a discovery-gated route that is stale for an
  entire slice, and served from origin every time within it, is not recorded. Measured on the reference
  deployment that is ~712 requests/day against ~59k catalog cache serves, and the same URLs are recorded
  from their cache hits in the same slice. Using cacheStatus rather than resource.miss is what keeps
  this down to the all-slice case instead of every stale serve.
• No new metric. The gates' effect is observable as demand_fill falling, and adding a series would
  couple this to the console's metric-coverage guard. Happy to add demand_gated (reason, bot) mirroring
  discovery_gated if reviewers would rather see it counted directly.
• These gates alone may not be sufficient on a corpus this size — dropping the junk should roughly halve
  distinct inserts, which takes fill from 0.986 to ~0.88 and the false-positive rate from 91% to ~41%.
  Better, still not good. A deployment at this scale will also want bots set and/or bitsPerSlice raised;
  that is why the sizing curve is now in the schema doc rather than left as folklore.
• recordDemand is exported for tests only, following recordServeOutcome in the same file.

What operators should expect on deploy

A saturated ring promotes everything, so today's rung distribution is not evidence of demand — on the
reference deployment 100% of sampled PDPs sit at their demandFloor and 90% of catalog at theirs. Once
fill comes down the ladder starts discriminating for the first time, and pages nothing in
render.demand.bots visits will demote toward their route base. That is the feature working, but it is
a real cadence change on real pages and it will look like a regression on a freshness dashboard.

Total render demand should fall with it, since fewer targets hold a fast rung. Watch demand_fill's peak,
demand_promoted / demand_demoted, and the render rate.

One deploy will probably not be enough. Dropping the junk should roughly halve distinct inserts, which
takes fill from 0.986 to ~0.88 and the false-positive rate from 91% to ~41% — better, still not a working
filter. Setting bots cuts further. Size bitsPerSlice from the measured new peak rather than from an
estimate; that is why the sizing curve is now in the schema doc.

Verification

• node --test in packages/plugin: 934 pass, 0 fail. New test/demandSignal.test.js (16 assertions)
  drives recordDemand through the real visit filter — record → flush → refresh → visitedWithin —
  rather than a stub, so a wiring change that records the cache key instead of the device-free URL fails
  there. Four new userAgent.test.js cases cover the allowlist and its independence from discovery.
• npm run lint and npm run format:check clean at the root.
• Two defects were caught by the tests during development and are fixed here: the mint branch initially
  omitted its miss precondition (so a bypass with a 200 was recorded), and skip was not in the
  never-looked set.
• Not verified end-to-end. This repo has no PR CI and the change is on the bot serve path; the
  behaviour it targets is only observable at production distinct-URL rates. The intended live check is
  demand_fill's peak before and after, per node.

Review coverage

Written and self-reviewed by Claude (Opus 5). No outside-model review ran — no independent
second-family coverage on this diff.

[gemini-code-assist]

Code Review

This pull request introduces a new render.demand.bots configuration option to restrict which bots' visits count as demand for the render ladder, helping to prevent Bloom filter ring saturation. It refactors the demand recording logic into a more robust recordDemand function that filters out non-eligible requests (such as bypasses, skips, and certain misses) and checks bot eligibility. Additionally, it updates metrics documentation to advise reading the peak of demand_fill rather than its mean, and adds comprehensive unit tests to verify the new behavior. There are no review comments to address, so no further feedback is provided.