Repository navigation
a badge printed a chain's name over a figure measured across every chain - #3403
Merged
Merged
Conversation
Measured on staging after the board had already stopped doing it:
/api/badge/rpc-api-coverage/alchemy?chain=robinhood
"#1/6 (Robinhood Chain), 100.0%"
/api/badge/rpc-api-coverage/blockmachine?chain=robinhood
"#2/6 (Robinhood Chain), 95.9%"
/api/badge/rpc-api-coverage/quicknode?chain=robinhood
"#4/6 (Robinhood Chain), 91.1%"
Two things are wrong at once. Alchemy leads that chain only because the one
method it was not asked, trace_block, is the one every other provider there
refuses, which is exactly why the board publishes it outside the ranking with
"asked on only 86% of cells". And the percentages are the ALL-CHAINS figures
under a chain label: Robinhood's real numbers are 85.7 for BlockMachine and
85.7 for Quicknode, not 95.9 and 91.1.
The second is a documented approximation. rankOfChain says so in its own
comment: the per-chain leader stash is forced to rank 1 and everyone else is
ordered by their unfiltered p50, "soft approximation, full per-chain
leaderboards live on the bench page chain tabs". The exact path exists
(rankOfCell, fed by a spec's rank_matrix_query) and bench 311 declares none, so
it falls through. Declaring one would fix the figures and NOT the crown,
because the crown is in the value rather than in the approximation.
This commit fixes the first only, which is the part that states something
false rather than something imprecise.
A row the bench publishes outside its ranking is not ranked here either. Three
rank paths exist and all three would have published it, so the check sits
before them rather than inside each. The answer is placeholderSvg, which
already exists for "no defensible position this cycle", names the bench and
claims no rank. The guard can only withhold a rank, never invent one.
Badges are embedded in other people's README files, so this is the one surface
where a false position outlives the page it came from.
No test: the route is a Next handler whose rank paths need the whole bench
object, and a mock thorough enough to exercise this is a bigger change than
the guard. Verified against the live case on staging instead, which is the
evidence that matters: the same URL that read "#1/6" must stop claiming a rank
while the board still shows Alchemy outside the ranking.
Measured on production, 2026-10-29, one provider on one bench:
/api/badge/perp-fees/hyperliquid #6/11 (all chains) 0.048%
/api/badge/perp-fees/hyperliquid?chain=ETH #6/11 (ETH) 0.048%
/api/badge/perp-fees/hyperliquid?chain=BTC #6/11 (BTC) 0.048%
/api/badge/perp-fees/hyperliquid?chain=SOL #6/11 (SOL) 0.048%
One figure, four labels, three of which it does not have.
The exact per-chain path needs a spec's rank_matrix_query. Of the 30 live
chain-dimensioned benches, 22 declare none, so the request fell through to
rankOfChain, which forced the chain's own leader to rank 1 and read everyone
else's UNFILTERED median: value: sortedOthers[idx].ms.p50, sorted by the same
unfiltered median. Only slot 1 was chain-aware. The result is a hybrid that
neither "ETH" nor "all chains" can label honestly, and the label was taken from
the request rather than from the figure.
So the hybrid is gone rather than relabelled. A scope label is now only ever
printed over a figure measured in that scope, and when the scope cannot be
ranked the honest answer depends on what was asked: a chain alone gets the
bench's real cross-chain ranking, labelled "all chains", which is also how the
embedder sees their ?chain= was not honoured; a region, with or without a
chain, gets no rank at all, because the aggregate is not a stand-in for a
region (it is a blend of the regions the reader wanted to compare).
The figure and its label now leave one function together. That pairing is the
fault, so it is the thing under test: badge-scope.test.ts pins the perp-fees
case directly (no chain may reach the label), both withholding cases, and the
cohort and default-view parts of each label. Mutation-checked three ways, each
caught: reintroducing the chain label on the aggregate, letting a region fall
back, and dropping the cohort from the cell label.
/badges was the other half. Its embed modal offered a Chain picker for every
bench that declared the dimension, which on those 22 meant a control that
silently did nothing, so the picker now follows what the badge can answer. It
reads cellRanks, the same signal the badge route looks up, so the two cannot
disagree and a cycle whose matrix query failed goes quiet instead of promising
a scope it cannot serve. Checked against the live aggregate blob in both
directions: 8 benches keep the picker (aggregator-head-lag, bridge-quote-latency,
bridge-quote-latency-solana, evm-quote-latency, metadata-coverage,
rpc-capabilities, rpc-reliability, token-quote-coverage), 22 lose it, and the
230 live benches with no chain dimension are untouched.
Badges live in other people's README files, which is why withholding is a
placeholder rather than a 404 and why this is the surface where a false scope
outlives the page it came from.
tsc clean, 816 tests pass, 260 specs valid.
# Conflicts: # src/app/api/badge/[slug]/[provider]/route.ts
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
…ng calls (#3404) src/lib/snapshot.ts:293 carries the four per-chain and per-cell stashes when a render lacks one, and its doc comment says why, unprompted: per-chain / per-cell stashes (bestPerChain, cellRanks, ...) are computed by separate Prom queries that can fail on an otherwise-healthy cycle. When the new render lacks one and the existing snapshot has it, carry the old value forward so scoped badges / placeholders don't 404 after a cold start. That reasoning is right and it is on the wrong path. writeSnapshot has no caller: nothing in src/ or worker/ imports @/lib/snapshot at all, so the carry only ever existed for the KV fallback, and the site reads the worker's blob. mergeWithPrevious, which builds that blob, carries provider rows and panel values under a ceiling and never mentions cellRanks, bestPerChain, worstPerChain or providersPerChain. So one failed matrix query blanked them for a cycle, and after today that is visible rather than theoretical: a chain-scoped badge is served from cellRanks and withholds the rank without it (#3403), and /badges offers its Chain picker only where cellRanks says the badge can answer. The figure and the control that asks for it went together, on the 8 benches that have cells. Two of those, metadata-coverage and token-quote-coverage, publish 2 and 3 cells, so for them "the query degraded" and "the query failed" are the same reading. The ceiling is the interesting part and it is why this is not three lines. A stash has no observedAt, and prev.builtAt is restamped every sweep, so a carry dated from builtAt reads as newly observed on every pass and never expires: the unbounded carry this file's own header is about, in the function it warns about. The observation time therefore goes in WorkerState beside the per-row one, optional so blobs written before it still parse, falling back to the previous blob's builtAt exactly once. The window is the one the rows already get, which is the bench's own expected_freshness_seconds where it declares one, a day otherwise. A ranking carried past the window the bench claims its data is good for is its own lie. An empty stash counts as a failure rather than an answer, so a {} cannot displace a populated carry and then be stamped fresh, which is how a blank outlives the gap that produced it while claiming it was just measured. A filtered view carries nothing, same rule as the rows: load.ts computes cells only on the unfiltered load. Eight tests. The one that matters walks 180 one-minute sweeps, feeding each sweep's own state back in, and asserts the observation time never moves before the stash drops at the ceiling: a carry that merely looks right passes every other test in the file. Mutation-checked three ways, each caught: dating the carry from builtAt (6 failures, including that one), removing the ceiling (4), and treating an empty stash as an answer (1). tsc clean, 834 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3402 (same file, one commit below). Review the top commit; the diff
collapses to mine once #3402 merges.
Measured on production, 2026-10-29, one provider on one bench:
One figure, four labels, three of which it does not have.
The exact per-chain path needs a spec's
rank_matrix_query. Of the 30 livechain-dimensioned benches, 22 declare none, so the request fell through to
rankOfChain, which forced the chain's own leader to rank 1 and read everyoneelse's UNFILTERED median (
value: sortedOthers[idx].ms.p50, sorted by the sameunfiltered median). Only slot 1 was chain-aware. The result is a hybrid that
neither "ETH" nor "all chains" can label honestly, and the label came from the
request rather than from the figure. This is the half #3402 named and left:
"the percentages are the ALL-CHAINS figures under a chain label".
So the hybrid is gone rather than relabelled. A scope label is now only printed
over a figure measured in that scope, and when the scope cannot be ranked the
honest answer depends on what was asked:
which is also how the embedder sees their
?chain=was not honoured.stand-in for a region; it is a blend of the regions the reader wanted to
compare.
The figure and its label now leave one function together, because choosing them
in two places is the fault. That makes the pairing testable, so it is tested:
badge-scope.test.tspins the perp-fees case directly (no chain may reach thelabel), both withholding cases, and the cohort and default-view parts of each
label. Mutation-checked three ways, each caught: reintroducing the chain label
on the aggregate, letting a region fall back, dropping the cohort from the cell
label.
/badgeswas the other half. Its embed modal offered a Chain picker forevery bench that declared the dimension, which on those 22 meant a control that
silently did nothing but hand out the URL above. The picker now follows what the
badge can answer. It reads
cellRanks, the same signal the badge route looksup, so the two cannot disagree and a cycle whose matrix query failed goes quiet
instead of promising a scope it cannot serve. Checked against the live aggregate
blob in both directions:
bridge-quote-latency-solana, evm-quote-latency, metadata-coverage,
rpc-capabilities, rpc-reliability, token-quote-coverage.
evm-quote-output, gas-estimation, launchpad-wars, memecoin-platforms,
mev-protect-rpc, perp-cost-slope, perp-execution-quality, perp-fees,
perp-liq-rate, perp-mark-price-lag, rpc-api-coverage,
terminal-avg-trade-size, terminal-execution-quality, terminal-fill-quality,
terminal-swap-transactions, trading-app-daily-volume,
trading-platform-wallets, trading-platform-wars, wallet-labels-coverage.
Nothing is lost by hiding it: the per-chain boards are on the bench page's chain
tabs, which are real per-chain measurements, and declaring a
rank_matrix_queryon any of those 22 brings its picker back with an exact figure behind it.
Badges live in other people's README files, which is why a withheld rank is a
placeholder rather than a 404, and why this is the one surface where a false
scope outlives the page it came from.
One seam left deliberately:
/snippet?format=jsonstill echoes the requestedscope.chainnext tobadge_url. It is accurate about thebenchUrlit alsoreturns (the bench page does honour chain tabs), so relabelling it would make
the link description wrong to make the image description right.
tsc clean, 816 tests pass, 260 specs valid.