Repository navigation
a panel's 7d chart ignored the chain tab, so First to report read the same everywhere - #3362
Merged
Merged
Conversation
… same everywhere
Lucas spotted it on production: bench 001's "First to report" view showed
Codex 68.8% and Serialized 29.5% identically on Base, BNB Chain, Solana and
Robinhood Chain, while the head-lag view beside it moved per chain as it
should. The shares are a per-chain race, so one set of numbers on four chains
is not a coincidence, it is an unfiltered query.
It was. Two paths build a panel's PromQL and only one of them scoped it:
values + 24h series specToBenchmark -> applyDimensionsToSpec -> injectLabels
7d + 30d series /api/series/[slug]?panel=... -> hand-rolled
The second exists because slimBenchmarkForCache and the snapshot both strip
seriesByProvider7d/30d on purpose, so the chart refetches them on tab click.
That route loads the raw spec off disk, never passes it through
applyDimensionsToSpec, and injected only the provider selector:
avg by (aggregator) (head_lag_first_share_pct{aggregator="codex"})
averaged over every chain and every region. The blob branch above it is
correctly keyed by filterSig, but it can never hit for a panel because the
series it looks for are the ones that were stripped, so every request fell
through to the unscoped build. Bench 001 declares the selector as `{}`, which
is what made it invisible in review: it looks filtered and matches everything.
Both paths now call one exported panelMetricQuery, so they cannot drift again,
and ignore_dimensions is honoured in both rather than in one (bench 282's
blank-on-71-of-75-views trap). The route's own cache key moves to v10 because
entries written before this hold a cross-chain average under a per-chain key.
Ten tests on the shared function. Verified they fail: putting the dropped
dimensions back turns 5 of them red, so this is a guard and not a guess.
Not a regression from today's Prometheus work. head_lag_first_share_pct comes
from the monitor job, which was kept, and no head_lag metric was touched.
This was referenced Oct 29, 2026
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
…ther tab (#3364) This is the bug Lucas actually reported, and #3362 was not it. Bench 001's First to report showed the same shares on Base, BNB, Solana and Robinhood while head lag beside it moved per chain. Florent narrowed it to "only 7D and 30D", which is the whole diagnosis: 1H, 6H and 24H render from the embedded per-filter series, and 7D, 30D, 90D and 1Y are lazy-fetched. The lazy caches were keyed by panel id alone: if (lazyPanel7d[activePanelId] && lazyPanel30d[activePanelId]) return; The effects DO list chainProp in their deps and DO re-run on a tab change, and the fetch URL has carried the chain since the main series was fixed for this same reason. It never got there: the first chain opened filled the entry, and every later chain hit that guard and returned before fetching. What it published, measured against the live series with EU-WEST selected: base Serialized 74.7 % the chart named Codex bnb Mobula 64.8 % the chart named Codex robinhood Serialized 93.3 % the chart named Codex solana Codex 73.4 % right, by luck So three chains out of four named the wrong leader, which is worse than the "same numbers everywhere" it looked like. Keyed on panel + chain + region + tier now, through one panelSeriesCacheKey helper, reads and writes together. The 90d/1y writes are included: they had been left on the old key by the first pass, which would have made that cache never hit and refetch on every render. Seven tests on the key, including the two orderings a naive concatenation would collide. Note #3362 is still a real fix and stays: /api/series rebuilds a panel's series from the raw spec whenever the blob misses, and that path dropped the dimensions. It is latent rather than what Lucas saw, because the blob does carry the per-filter panel series and was answering correctly throughout. The PR body claimed otherwise and has been corrected.
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
…ther tab (#3364) This is the bug Lucas actually reported, and #3362 was not it. Bench 001's First to report showed the same shares on Base, BNB, Solana and Robinhood while head lag beside it moved per chain. Florent narrowed it to "only 7D and 30D", which is the whole diagnosis: 1H, 6H and 24H render from the embedded per-filter series, and 7D, 30D, 90D and 1Y are lazy-fetched. The lazy caches were keyed by panel id alone: if (lazyPanel7d[activePanelId] && lazyPanel30d[activePanelId]) return; The effects DO list chainProp in their deps and DO re-run on a tab change, and the fetch URL has carried the chain since the main series was fixed for this same reason. It never got there: the first chain opened filled the entry, and every later chain hit that guard and returned before fetching. What it published, measured against the live series with EU-WEST selected: base Serialized 74.7 % the chart named Codex bnb Mobula 64.8 % the chart named Codex robinhood Serialized 93.3 % the chart named Codex solana Codex 73.4 % right, by luck So three chains out of four named the wrong leader, which is worse than the "same numbers everywhere" it looked like. Keyed on panel + chain + region + tier now, through one panelSeriesCacheKey helper, reads and writes together. The 90d/1y writes are included: they had been left on the old key by the first pass, which would have made that cache never hit and refetch on every render. Seven tests on the key, including the two orderings a naive concatenation would collide. Note #3362 is still a real fix and stays: /api/series rebuilds a panel's series from the raw spec whenever the blob misses, and that path dropped the dimensions. It is latent rather than what Lucas saw, because the blob does carry the per-filter panel series and was answering correctly throughout. The PR body claimed otherwise and has been corrected.
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
… tab (#3363) * a panel's 7d chart ignored the chain tab, so First to report read the same everywhere (#3362) Lucas spotted it on production: bench 001's "First to report" view showed Codex 68.8% and Serialized 29.5% identically on Base, BNB Chain, Solana and Robinhood Chain, while the head-lag view beside it moved per chain as it should. The shares are a per-chain race, so one set of numbers on four chains is not a coincidence, it is an unfiltered query. It was. Two paths build a panel's PromQL and only one of them scoped it: values + 24h series specToBenchmark -> applyDimensionsToSpec -> injectLabels 7d + 30d series /api/series/[slug]?panel=... -> hand-rolled The second exists because slimBenchmarkForCache and the snapshot both strip seriesByProvider7d/30d on purpose, so the chart refetches them on tab click. That route loads the raw spec off disk, never passes it through applyDimensionsToSpec, and injected only the provider selector: avg by (aggregator) (head_lag_first_share_pct{aggregator="codex"}) averaged over every chain and every region. The blob branch above it is correctly keyed by filterSig, but it can never hit for a panel because the series it looks for are the ones that were stripped, so every request fell through to the unscoped build. Bench 001 declares the selector as `{}`, which is what made it invisible in review: it looks filtered and matches everything. Both paths now call one exported panelMetricQuery, so they cannot drift again, and ignore_dimensions is honoured in both rather than in one (bench 282's blank-on-71-of-75-views trap). The route's own cache key moves to v10 because entries written before this hold a cross-chain average under a per-chain key. Ten tests on the shared function. Verified they fail: putting the dropped dimensions back turns 5 of them red, so this is a guard and not a guess. Not a regression from today's Prometheus work. head_lag_first_share_pct comes from the monitor job, which was kept, and no head_lag metric was touched. * the panel chart cached one chain's 7d series and served it on every other tab (#3364) This is the bug Lucas actually reported, and #3362 was not it. Bench 001's First to report showed the same shares on Base, BNB, Solana and Robinhood while head lag beside it moved per chain. Florent narrowed it to "only 7D and 30D", which is the whole diagnosis: 1H, 6H and 24H render from the embedded per-filter series, and 7D, 30D, 90D and 1Y are lazy-fetched. The lazy caches were keyed by panel id alone: if (lazyPanel7d[activePanelId] && lazyPanel30d[activePanelId]) return; The effects DO list chainProp in their deps and DO re-run on a tab change, and the fetch URL has carried the chain since the main series was fixed for this same reason. It never got there: the first chain opened filled the entry, and every later chain hit that guard and returned before fetching. What it published, measured against the live series with EU-WEST selected: base Serialized 74.7 % the chart named Codex bnb Mobula 64.8 % the chart named Codex robinhood Serialized 93.3 % the chart named Codex solana Codex 73.4 % right, by luck So three chains out of four named the wrong leader, which is worse than the "same numbers everywhere" it looked like. Keyed on panel + chain + region + tier now, through one panelSeriesCacheKey helper, reads and writes together. The 90d/1y writes are included: they had been left on the old key by the first pass, which would have made that cache never hit and refetch on every render. Seven tests on the key, including the two orderings a naive concatenation would collide. Note #3362 is still a real fix and stays: /api/series rebuilds a panel's series from the raw spec whenever the blob misses, and that path dropped the dimensions. It is latent rather than what Lucas saw, because the blob does carry the per-filter panel series and was answering correctly throughout. The PR body claimed otherwise and has been corrected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reported by Lucas on production. Bench 001's First to report view shows Codex 68.8% and Serialized 29.5% identically on Base, BNB Chain, Solana and Robinhood Chain, while the Head lag view right beside it moves per chain as it should (Base 0.08/0.08 s, Robinhood 0.76/0.67 s).
The shares are a per-chain race. One set of numbers across four chains is not a coincidence, it is an unfiltered query.
Why
Two paths build a metric panel's PromQL, and only one of them scoped it:
specToBenchmark→applyDimensionsToSpec→injectLabels/api/series/[slug]?panel=...The second path exists because
slimBenchmarkForCacheand the snapshot builder both stripseriesByProvider7d/seriesByProvider30don purpose, so the chart refetches them on tab click. That route loads the raw spec off disk, never passes it throughapplyDimensionsToSpec, and injected only the provider selector:averaged over every chain and every region.
The blob branch above it is correctly keyed by
filterSig, but it can never hit for a panel, because the series it looks for are exactly the ones that were stripped. So every panel-series request fell through to the unscoped build.Bench 001 declares the selector as
{}, which is what made this survive review: it looks filtered and matches everything.The fix
One exported
panelMetricQuery(panel, labels)that both paths call, so they cannot drift again, plusdimensionLabels(filters)to build the label set the loader uses (tierexcluded, it selects the cohort;allis not a filter).ignore_dimensionsis now honoured in both paths instead of one. That matters: appending a dimension a metric has no label for matches nothing and empties the panel in silence, which is how bench 282 shipped a "Billing units per request" chart blank on 71 of 75 views.The route's own cache key goes to v10: entries written before this hold a cross-chain average under a per-chain key.
Verification
Ten tests on the shared function, covering the bare name, the
{}form bench 001 uses, an existing selector, a label the panel already pins,ignore_dimensions, and the provider selector the route prepends afterwards.They were made to fail first. Putting the dropped dimensions back turns 5 of the 10 red, so this is a guard rather than a guess.
Gates:
pnpm test763 pass 0 fail,tsc --noEmitclean,pnpm validate258 specs valid, eslint clean on the three files.Not a regression from today
head_lag_first_share_pctcomes from themonitorjob, which the Prometheus cleanup kept, and nohead_lagmetric was touched. The route's panel path has had this shape since #2724.Needs a cherry-pick to
mainafter this lands: the bench is live on production, which is where Lucas is looking.