Metrics: no counter pairs route with bot; add route_bot_serve (route × bot × outcome) #263

Summary

No metric the plugin emits has both the route and the bot, so "Googlebot on /browse/*: 91% from cache, 1.1% errors" can't be computed from counters. We're building a crawl-analytics dashboard for SEO teams on top of the plugin's metrics. Its main table is route × bot showing views, % from cache and errors, and today the only source for that table is the request sampler, which is an estimate at production sample rates.

Asking for one new counter, route_bot_serve (route × bot × outcome). We're happy to send the PR.

Why the current metrics can't answer it

Each metric has three positional slots (METRICS.md fact 4), and none pairs route with bot:

bot_miss has route × bot, but only for origin|miss serves. Cache serves, and every status code, never get both dimensions on one row.

The marginals can't be recombined: two traffic mixes that are mirror images of each other produce identical rows.

Reproduction

Any single node on 0.108.1 with two routes configured, e.g.:

'@harperfast/prerender':
  ingress:
    mode: forwarded
    deviceTypeSource: header
    routes:
      - { match: prefix, path: '/ip/', queryParams: [] }
      - { match: prefix, path: '/cp/', queryParams: [] }

Send 8 bot requests (mix A), wait for the analytics flush, then sum get_analytics for bot_serve by type and route_serve by path. Wait more than a minute, so the next window starts clean, then do the same for the mirror mix B. Script used:

// Two mirror-image traffic mixes against one node; print what bot_serve and route_serve record for each.
import { request, Agent } from 'node:https';
const APP = 19927, OPS = 9926, ORIGIN_HOST = 'host.docker.internal:8080'; // adjust
const AUTH = 'Basic ' + Buffer.from('admin:admin').toString('base64');
process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0'; // self-signed sandbox certs
const agent = new Agent({ rejectUnauthorized: false });
const UA = {
	Googlebot: 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
	GPTBot: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot',
};
const send = (bot, path) => new Promise((ok, no) => request({ host: '127.0.0.1', port: APP, path, agent,
	headers: { 'user-agent': UA[bot], 'x-forwarded-host': ORIGIN_HOST, 'x-forwarded-proto': 'http', 'x-device-type': 'desktop' } },
	(r) => { r.resume(); r.on('end', () => ok(r.statusCode)); }).on('error', no).end());
const analytics = async (metric, start) => (await (await fetch(`https://127.0.0.1:${OPS}`, { method: 'POST',
	headers: { 'content-type': 'application/json', authorization: AUTH },
	body: JSON.stringify({ operation: 'get_analytics', metric, start_time: start }) })).json());
const sum = (rows, key) => { const o = {}; for (const r of rows) { const k = key(r); if (k) o[k] = (o[k] ?? 0) + r.count; } return o; };

async function run(name, mix) {
	const start = Date.now();
	let i = 0;
	for (const [bot, route, n] of mix) for (let k = 0; k < n; k++) await send(bot, `${route}${100 + i++}`);
	const want = mix.reduce((a, m) => a + m[2], 0);
	for (;;) { // wait for the analytics flush
		await new Promise((r) => setTimeout(r, 5000));
		const bs = (await analytics('bot_serve', start)).filter((r) => r.type === 'Googlebot' || r.type === 'GPTBot');
		if (bs.reduce((a, r) => a + r.count, 0) < want) continue;
		const rs = (await analytics('route_serve', start)).filter((r) => r.path === '/ip/' || r.path === '/cp/');
		console.log(`\n${name}: sent ${mix.map(([b, r, n]) => `${n}× ${b} → ${r}*`).join(', ')}`);
		console.log('  bot_serve   by bot  :', JSON.stringify(sum(bs, (r) => r.type)));
		console.log('  route_serve by route:', JSON.stringify(sum(rs, (r) => r.path)));
		return;
	}
}
await run('Mix A', [['Googlebot', '/ip/', 3], ['Googlebot', '/cp/', 1], ['GPTBot', '/ip/', 1], ['GPTBot', '/cp/', 3]]);
await new Promise((r) => setTimeout(r, 70000)); // next analytics bucket, so mix A can't leak into mix B
await run('Mix B', [['Googlebot', '/ip/', 1], ['Googlebot', '/cp/', 3], ['GPTBot', '/ip/', 3], ['GPTBot', '/cp/', 1]]);

Actual output (harper-pro 5.3.5, plugin 0.108.1, run 2026-11-07):

Mix A: sent 3× Googlebot → /ip/*, 1× Googlebot → /cp/*, 1× GPTBot → /ip/*, 3× GPTBot → /cp/*
  bot_serve   by bot  : {"Googlebot":4,"GPTBot":4}
  route_serve by route: {"/ip/":4,"/cp/":4}

Mix B: sent 1× Googlebot → /ip/*, 3× Googlebot → /cp/*, 3× GPTBot → /ip/*, 1× GPTBot → /cp/*
  bot_serve   by bot  : {"Googlebot":4,"GPTBot":4}
  route_serve by route: {"/ip/":4,"/cp/":4}

Expected: some metric that tells A (Googlebot mostly on /ip/) from B (Googlebot mostly on /cp/). Nothing does.

What we verified

All of this ran in a local sandbox with a fabricated origin, not on any customer cluster.

• Setup: 4 replicating harper-pro 5.3.5 nodes, plugin 0.108.1, plus a puppeteer render worker.

• The existing counters are exact. We replayed a seeded mix of 5,000 bot requests across 10 bots, 4 routes and 2 devices, with 91.8% served from cache. Summed across nodes, hdb_analytics matched the known counts exactly on every check:

  • bot_request by bot
  • bot_serve by bot, and by source
  • route_serve by route
  • origin_fetch by status, against the origin-served samples

  So the counters are the right primitive; they just don't have this combination.

• The sampler matches too, but only at rate 1. With sample: { by: request, rate: 1 }, the sampled log gives bot × route × status exactly, and that check passed as well. At production rates it's an estimate, it's spread across per-node files rather than in hdb_analytics, and every reader has to page through every node's /prerender_admin/samples.

Proposal

One counter, recorded next to botServe and routeServe in recordServeOutcome (src/http_handlers/bot_request.js:244-245 in 0.108.1):

route_bot_serve   counter   path: route   method: botName   type: outcome
outcome = <source>:<status class>    e.g. cache:2xx, origin:4xx, rendered:2xx

// metrics.js
routeBotServe: (route, botName, outcome) => server.recordAnalytics(true, 'route_bot_serve', route, botName, outcome),

// bot_request.js, recordServeOutcome
metrics.routeBotServe(route, request.botName, `${info.source}:${statusClass(resource.statusCode)}`);

• Closed sets only (METRICS.md rule 7):
  • routes come from config, using the same label route_serve already uses
  • bot names come from the registry plus deriveUnknownBots
  • outcome is about 6 sources × 4 status classes
• Cost. It's one more counter bump, so the per-request cost is negligible (fact 2). The durable cost is active combos per node per period, roughly routes × active bots × outcomes in use. That's a few hundred on a typical site, the same order as bot_serve.
• Existing metrics stay as they are. We considered reusing route_serve's deviceType slot for the bot, but that breaks the positional contract existing dashboards read.
• Open question: which status to count. recordServeOutcome runs before deliverResource, so resource.statusCode is what's available there. If you'd rather count conditional 304s the way the sampler does, the call could move after delivery and take response.status.

Smaller asks (optional)

1. Crawl breadth per bot × route. The crawlStats HLL is per bot per day. Per bot × route would answer "distinct pages Googlebot crawled under /ip/". Happy to drop this if the sketch memory is a concern.

2. Four crawlers for the registry (src/configSchema.js, around line 3259):

   • Amzn-SearchBot
   • Amzn-User
   • Meta-WebIndexer
   • facebookexternalhit

   None of the four is in 0.108.1. deriveUnknownBots already gives them usable names, so nothing breaks, but registering them would pin their display names and bot classes.

Questions

1. Is the counter acceptable, and does route_bot_serve fit your naming?
2. Would you take a PR from us? It would add unit tests in the style of the existing recordServeOutcome emission tests. Our 4-node replay would also check the new metric exactly against known counts.
3. Roughly which release could it land in? Until then we'll show sample-based estimates, labelled as estimates.