Repository navigation
central prometheus, strip per-harness ops - #4
Merged
Merged
Conversation
5 tasks done
bakabhai993
pushed a commit
to bakabhai993/OpenChainBench
that referenced
this pull request
Sep 12, 2026
central prometheus, strip per-harness ops
Flotapponnier
added a commit
that referenced
this pull request
Oct 23, 2026
#3241) It ranks #4 at 290 bps on the bench and renders "FM" instead of a logo, because the slug reached the spec and none of the site's registries, while pump-bridge is in four of them. The precedent was already set and its own comment says why: pump-bridge carries pump.fun's logo and brand colour because it IS pump, reached by a different rail. FOMO over Mobula is the same case, so it gets FOMO's violet, FOMO's logo and FOMO's links. brand.ts the colour, so the bar is FOMO's rather than a default providers.ts the display name logo-manifest.ts FOMO's own logo, which is on disk provider-registry the product page's url, description and handle The description says what the ROW is rather than what Mobula is: the row measures FOMO's trades over that rail, and a reader clicking it is asking about FOMO's route rather than about the bridge company. Checked both routes field by field against axiom, a row that has been on this bench for months, rather than against a list of what I just added, so anything I had not thought of would still show: spec provider, brand colour, display name, logo, product registry, and the logo file actually on disk. Both complete on all of them.
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
…ng (#3402) Measured on staging after the board had already stopped doing it: /api/badge/rpc-api-coverage/alchemy?chain=robinhood "#1/6 (Robinhood Chain), 100.0%" /api/badge/rpc-api-coverage/blockmachine?chain=robinhood "#2/6 (Robinhood Chain), 95.9%" /api/badge/rpc-api-coverage/quicknode?chain=robinhood "#4/6 (Robinhood Chain), 91.1%" Two things are wrong at once. Alchemy leads that chain only because the one method it was not asked, trace_block, is the one every other provider there refuses, which is exactly why the board publishes it outside the ranking with "asked on only 86% of cells". And the percentages are the ALL-CHAINS figures under a chain label: Robinhood's real numbers are 85.7 for BlockMachine and 85.7 for Quicknode, not 95.9 and 91.1. The second is a documented approximation. rankOfChain says so in its own comment: the per-chain leader stash is forced to rank 1 and everyone else is ordered by their unfiltered p50, "soft approximation, full per-chain leaderboards live on the bench page chain tabs". The exact path exists (rankOfCell, fed by a spec's rank_matrix_query) and bench 311 declares none, so it falls through. Declaring one would fix the figures and NOT the crown, because the crown is in the value rather than in the approximation. This commit fixes the first only, which is the part that states something false rather than something imprecise. A row the bench publishes outside its ranking is not ranked here either. Three rank paths exist and all three would have published it, so the check sits before them rather than inside each. The answer is placeholderSvg, which already exists for "no defensible position this cycle", names the bench and claims no rank. The guard can only withhold a rank, never invent one. Badges are embedded in other people's README files, so this is the one surface where a false position outlives the page it came from. No test: the route is a Next handler whose rank paths need the whole bench object, and a mock thorough enough to exercise this is a bigger change than the guard. Verified against the live case on staging instead, which is the evidence that matters: the same URL that read "#1/6" must stop claiming a rank while the board still shows Alchemy outside the ranking.
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
…ain (#3403) * a rank badge crowned a provider the bench publishes outside its ranking Measured on staging after the board had already stopped doing it: /api/badge/rpc-api-coverage/alchemy?chain=robinhood "#1/6 (Robinhood Chain), 100.0%" /api/badge/rpc-api-coverage/blockmachine?chain=robinhood "#2/6 (Robinhood Chain), 95.9%" /api/badge/rpc-api-coverage/quicknode?chain=robinhood "#4/6 (Robinhood Chain), 91.1%" Two things are wrong at once. Alchemy leads that chain only because the one method it was not asked, trace_block, is the one every other provider there refuses, which is exactly why the board publishes it outside the ranking with "asked on only 86% of cells". And the percentages are the ALL-CHAINS figures under a chain label: Robinhood's real numbers are 85.7 for BlockMachine and 85.7 for Quicknode, not 95.9 and 91.1. The second is a documented approximation. rankOfChain says so in its own comment: the per-chain leader stash is forced to rank 1 and everyone else is ordered by their unfiltered p50, "soft approximation, full per-chain leaderboards live on the bench page chain tabs". The exact path exists (rankOfCell, fed by a spec's rank_matrix_query) and bench 311 declares none, so it falls through. Declaring one would fix the figures and NOT the crown, because the crown is in the value rather than in the approximation. This commit fixes the first only, which is the part that states something false rather than something imprecise. A row the bench publishes outside its ranking is not ranked here either. Three rank paths exist and all three would have published it, so the check sits before them rather than inside each. The answer is placeholderSvg, which already exists for "no defensible position this cycle", names the bench and claims no rank. The guard can only withhold a rank, never invent one. Badges are embedded in other people's README files, so this is the one surface where a false position outlives the page it came from. No test: the route is a Next handler whose rank paths need the whole bench object, and a mock thorough enough to exercise this is a bigger change than the guard. Verified against the live case on staging instead, which is the evidence that matters: the same URL that read "#1/6" must stop claiming a rank while the board still shows Alchemy outside the ranking. * a badge printed a chain's name over a figure measured across every chain Measured on production, 2026-10-29, one provider on one bench: /api/badge/perp-fees/hyperliquid #6/11 (all chains) 0.048% /api/badge/perp-fees/hyperliquid?chain=ETH #6/11 (ETH) 0.048% /api/badge/perp-fees/hyperliquid?chain=BTC #6/11 (BTC) 0.048% /api/badge/perp-fees/hyperliquid?chain=SOL #6/11 (SOL) 0.048% One figure, four labels, three of which it does not have. The exact per-chain path needs a spec's rank_matrix_query. Of the 30 live chain-dimensioned benches, 22 declare none, so the request fell through to rankOfChain, which forced the chain's own leader to rank 1 and read everyone else's UNFILTERED median: value: sortedOthers[idx].ms.p50, sorted by the same unfiltered median. Only slot 1 was chain-aware. The result is a hybrid that neither "ETH" nor "all chains" can label honestly, and the label was taken from the request rather than from the figure. So the hybrid is gone rather than relabelled. A scope label is now only ever printed over a figure measured in that scope, and when the scope cannot be ranked the honest answer depends on what was asked: a chain alone gets the bench's real cross-chain ranking, labelled "all chains", which is also how the embedder sees their ?chain= was not honoured; a region, with or without a chain, gets no rank at all, because the aggregate is not a stand-in for a region (it is a blend of the regions the reader wanted to compare). The figure and its label now leave one function together. That pairing is the fault, so it is the thing under test: badge-scope.test.ts pins the perp-fees case directly (no chain may reach the label), both withholding cases, and the cohort and default-view parts of each label. Mutation-checked three ways, each caught: reintroducing the chain label on the aggregate, letting a region fall back, and dropping the cohort from the cell label. /badges was the other half. Its embed modal offered a Chain picker for every bench that declared the dimension, which on those 22 meant a control that silently did nothing, so the picker now follows what the badge can answer. It reads cellRanks, the same signal the badge route looks up, so the two cannot disagree and a cycle whose matrix query failed goes quiet instead of promising a scope it cannot serve. Checked against the live aggregate blob in both directions: 8 benches keep the picker (aggregator-head-lag, bridge-quote-latency, bridge-quote-latency-solana, evm-quote-latency, metadata-coverage, rpc-capabilities, rpc-reliability, token-quote-coverage), 22 lose it, and the 230 live benches with no chain dimension are untouched. Badges live in other people's README files, which is why withholding is a placeholder rather than a 404 and why this is the surface where a false scope outlives the page it came from. tsc clean, 816 tests pass, 260 specs valid.
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
… we have (#3405) Verified on staging after #3402 landed, which is how this surfaced: alchemy ?chain=robinhood "measurement pending" alchemy unscoped "measurement pending" quicknode ?chain=robinhood "#4/6 (Robinhood Chain), 91.1%" (control) The guard works: Alchemy no longer claims a rank and a ranked provider still answers. The wording does not. Alchemy's figure exists, 100.0 percent over the 12 cells we could ask, and the board publishes it with its cause. Only the RANK is withheld. placeholderSvg hardcodes "Measurement pending" in the aria-label, the <title> and the visible line, so my guard replaced a false rank with a false status, which is the same fault facing the other way. placeholderSvg now takes an optional status line. The default is unchanged, so the three callers that mean it keep saying "Measurement pending": a transient scoped miss, an unknown cell, a bench with no cell data. The unranked path passes "Not ranked". NO FIGURE IN THE BADGE, deliberately, and this is the part I changed my mind on My first version read `Not ranked · ${fmtUnit(providerRow.ms.p50, b.unit)}`. That is wrong for the same reason rankOfChain was: `b` is the unfiltered bench, so under a ?chain= request it would print a cross-chain value behind a scoped label. It happens to be invisible on this bench, because Alchemy reads 100.0 both all-chains and on Robinhood, and it would be a live defect on any bench where those differ. "Not ranked" is true on every scope and asserts no number; the board carries the figure. So: a badge that cannot defend a position says so, and says nothing else.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refactors the harness model to a clean producer/scraper split: every harness exposes /metrics, a single shared Prometheus scrapes them all.
What's added
What's removed
What's updated
Net effect
Follow-up (Railway)
After merge: create OpenChainBench Railway project with 3 services - prometheus (from infrastructure/prometheus/), aggregator-head-lag, bridge-monitor (each from harnesses//). Update YAML prom_url to the new Prom URL. Old Mobula Railway services can stay up as fallback during migration.