Skip to content

the dashboard's agent numbers counted our own probes and missed every crawler - #2750

Merged
Flotapponnier merged 1 commit into
devfrom
feat/crm-agent-traffic
Sep 30, 2026
Merged

Flotapponnier merged 1 commit into
devfrom
feat/crm-agent-traffic

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

The CRM's audience page answered "who reads the site" from PostHog, and PostHog cannot answer it.

Its SDK is client-side JavaScript, so nothing that skips JS is ever in it, and the server capture in src/lib/analytics-server.ts is wired to three routes, two of which sit behind a 60 s and a one hour edge cache and therefore emit one event per cache fill rather than per read. Every HTML page is ISR-cached, so a crawler fetching a bench page runs no code of ours at all.

What the page showed, and what was true

PostHog, whole history: 317 server reads, 19 of them from an identified AI agent. The top three "agent families" were curl, browser and other; 205 of the 317 were our own curl probes and the Blackbox exporter. Read as adoption, that says agents ignore the site.

Vercel, over the same three days:

Category Requests
ai_crawler 75,429
search_engine_crawler 8,001
ai_assistant 2,005

The two sources are not close because they are not measuring the same thing.

What this does

It does not duplicate the collection. Vercel already logs every edge request with a classified bot, cache hits included, so crm/lib/vercel-obs.ts queries it as a new snapshot section beside benches, harness and dune. The PostHog panels stay, now saying what they do and do not count.

The AI total is published twice, with and without the single largest AI bot. One crawler, meta-externalagent, was 73,192 of the 75,429 and was looping over fifteen static pages (3,570 requests to /contribute, 3,501 to /about) rather than indexing, so a single number would read as adoption when it is one misbehaving bot. 14,533 of its requests missed the cache and reached the origin, which is worth a separate look.

The API

/v2/observability/query, the one vercel metrics uses, is not in the public REST docs. The header of vercel-obs.ts keeps what was learned the hard way, so the next person does not repeat it:

  • the filter dialect is OData with single quotes (environment eq 'production'), not the CLI's environment:production, which the API rejects
  • orderBy must name a rollup column, vercel_request_count_sum
  • grouped dimensions come back snake_cased (bot_name) though groupBy takes them camelCased
  • granularity accepts only 5m/15m/1h/1d, so whole-window totals are summed in code
  • the per-path query 408s over 30 days and serves in seconds over 7, hence its own shorter window

Notes

  • Retention starts when Observability was enabled on the team, 2026-09-28, so earlier days are real zeros. firstDayWithData is carried so the page says so instead of implying the site was quiet.
  • Without VERCEL_API_TOKEN, VERCEL_TEAM_ID and VERCEL_PROJECT_ID the section reports itself unconfigured and every other section is unaffected. They need adding to the Railway service before this shows anything.
  • scripts/agents-probe.ts runs the loader once and prints what it returns, for checking the queries against the live API without a full refresh.

Verified: tsc --noEmit clean, 45 tests pass (5 new, covering the twice-published AI total, per-bot summing, unknown categories carried through, the empty-days rule, and path summing), and the loader was run against live Vercel.

… crawler

The audience page answered "who reads the site" from PostHog, and PostHog
cannot answer it. Its SDK is client-side JavaScript, so nothing that skips JS
is ever in it, and the server capture in src/lib/analytics-server.ts is wired
to three routes, two of which sit behind a 60 s and a one hour edge cache and
therefore emit one event per cache fill rather than per read. Every HTML page
is ISR-cached, so a crawler fetching a bench page runs no code of ours at all.

What the page showed on 2026-09-30: 317 server reads over its whole history,
of which 19 from an identified AI agent, and the top three "agent families"
were curl, browser and other, 205 of the 317 being our own curl probes and the
Blackbox exporter. Read as adoption, that says agents ignore the site.

Vercel, over the same three days, counted 75,429 requests from AI crawlers,
8,001 from search engine crawlers and 2,005 from AI assistants. The two
sources are not close because they are not measuring the same thing.

So this does not duplicate the collection. Vercel already logs every edge
request with a classified bot, cache hits included, and lib/vercel-obs.ts
queries it as a new snapshot section beside benches, harness and dune. The
PostHog panels stay, now saying what they do and do not count.

The API, /v2/observability/query, is the one `vercel metrics` uses and is not
in the public REST docs, so the header of vercel-obs.ts keeps what was learned
the hard way: the filter dialect is OData with single quotes, orderBy must
name a rollup column, grouped dimensions come back snake_cased, granularity
accepts only 5m/15m/1h/1d so whole-window totals are summed in code, and the
per-path query 408s over 30 days and runs over 7.

The AI total is published twice, with and without the single largest AI bot.
One crawler, meta-externalagent, was 73,192 of the 75,429 and was looping over
fifteen static pages rather than indexing, so a single number would read as
adoption when it is one misbehaving bot.

Retention starts when Observability was enabled on the team, 2026-09-28, so
earlier days are real zeros; firstDayWithData is carried so the page can say
so instead of implying the site was quiet.

Without the three VERCEL_* variables the section says it is not configured and
every other section is unaffected.
@Flotapponnier
Flotapponnier merged commit 092b81d into dev Sep 30, 2026
2 checks passed
@Flotapponnier
Flotapponnier deleted the feat/crm-agent-traffic branch September 30, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant