Repository navigation
the dashboard's agent numbers counted our own probes and missed every crawler - #2750
Merged
Merged
Conversation
… crawler The audience page answered "who reads the site" from PostHog, and PostHog cannot answer it. Its SDK is client-side JavaScript, so nothing that skips JS is ever in it, and the server capture in src/lib/analytics-server.ts is wired to three routes, two of which sit behind a 60 s and a one hour edge cache and therefore emit one event per cache fill rather than per read. Every HTML page is ISR-cached, so a crawler fetching a bench page runs no code of ours at all. What the page showed on 2026-09-30: 317 server reads over its whole history, of which 19 from an identified AI agent, and the top three "agent families" were curl, browser and other, 205 of the 317 being our own curl probes and the Blackbox exporter. Read as adoption, that says agents ignore the site. Vercel, over the same three days, counted 75,429 requests from AI crawlers, 8,001 from search engine crawlers and 2,005 from AI assistants. The two sources are not close because they are not measuring the same thing. So this does not duplicate the collection. Vercel already logs every edge request with a classified bot, cache hits included, and lib/vercel-obs.ts queries it as a new snapshot section beside benches, harness and dune. The PostHog panels stay, now saying what they do and do not count. The API, /v2/observability/query, is the one `vercel metrics` uses and is not in the public REST docs, so the header of vercel-obs.ts keeps what was learned the hard way: the filter dialect is OData with single quotes, orderBy must name a rollup column, grouped dimensions come back snake_cased, granularity accepts only 5m/15m/1h/1d so whole-window totals are summed in code, and the per-path query 408s over 30 days and runs over 7. The AI total is published twice, with and without the single largest AI bot. One crawler, meta-externalagent, was 73,192 of the 75,429 and was looping over fifteen static pages rather than indexing, so a single number would read as adoption when it is one misbehaving bot. Retention starts when Observability was enabled on the team, 2026-09-28, so earlier days are real zeros; firstDayWithData is carried so the page can say so instead of implying the site was quiet. Without the three VERCEL_* variables the section says it is not configured and every other section is unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CRM's audience page answered "who reads the site" from PostHog, and PostHog cannot answer it.
Its SDK is client-side JavaScript, so nothing that skips JS is ever in it, and the server capture in
src/lib/analytics-server.tsis wired to three routes, two of which sit behind a 60 s and a one hour edge cache and therefore emit one event per cache fill rather than per read. Every HTML page is ISR-cached, so a crawler fetching a bench page runs no code of ours at all.What the page showed, and what was true
PostHog, whole history: 317 server reads, 19 of them from an identified AI agent. The top three "agent families" were
curl,browserandother; 205 of the 317 were our own curl probes and the Blackbox exporter. Read as adoption, that says agents ignore the site.Vercel, over the same three days:
The two sources are not close because they are not measuring the same thing.
What this does
It does not duplicate the collection. Vercel already logs every edge request with a classified bot, cache hits included, so
crm/lib/vercel-obs.tsqueries it as a new snapshot section besidebenches,harnessanddune. The PostHog panels stay, now saying what they do and do not count.The AI total is published twice, with and without the single largest AI bot. One crawler,
meta-externalagent, was 73,192 of the 75,429 and was looping over fifteen static pages (3,570 requests to/contribute, 3,501 to/about) rather than indexing, so a single number would read as adoption when it is one misbehaving bot. 14,533 of its requests missed the cache and reached the origin, which is worth a separate look.The API
/v2/observability/query, the onevercel metricsuses, is not in the public REST docs. The header ofvercel-obs.tskeeps what was learned the hard way, so the next person does not repeat it:environment eq 'production'), not the CLI'senvironment:production, which the API rejectsorderBymust name a rollup column,vercel_request_count_sumbot_name) thoughgroupBytakes them camelCasedNotes
firstDayWithDatais carried so the page says so instead of implying the site was quiet.VERCEL_API_TOKEN,VERCEL_TEAM_IDandVERCEL_PROJECT_IDthe section reports itself unconfigured and every other section is unaffected. They need adding to the Railway service before this shows anything.scripts/agents-probe.tsruns the loader once and prints what it returns, for checking the queries against the live API without a full refresh.Verified:
tsc --noEmitclean, 45 tests pass (5 new, covering the twice-published AI total, per-bot summing, unknown categories carried through, the empty-days rule, and path summing), and the loader was run against live Vercel.