Repository navigation
decentralized hosting model - #6
Merged
Merged
Conversation
…om scrapes public urls
bakabhai993
pushed a commit
to bakabhai993/OpenChainBench
that referenced
this pull request
Sep 12, 2026
decentralized hosting model
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
Measured on production, 2026-10-29, one provider on one bench:
/api/badge/perp-fees/hyperliquid #6/11 (all chains) 0.048%
/api/badge/perp-fees/hyperliquid?chain=ETH #6/11 (ETH) 0.048%
/api/badge/perp-fees/hyperliquid?chain=BTC #6/11 (BTC) 0.048%
/api/badge/perp-fees/hyperliquid?chain=SOL #6/11 (SOL) 0.048%
One figure, four labels, three of which it does not have.
The exact per-chain path needs a spec's rank_matrix_query. Of the 30 live
chain-dimensioned benches, 22 declare none, so the request fell through to
rankOfChain, which forced the chain's own leader to rank 1 and read everyone
else's UNFILTERED median: value: sortedOthers[idx].ms.p50, sorted by the same
unfiltered median. Only slot 1 was chain-aware. The result is a hybrid that
neither "ETH" nor "all chains" can label honestly, and the label was taken from
the request rather than from the figure.
So the hybrid is gone rather than relabelled. A scope label is now only ever
printed over a figure measured in that scope, and when the scope cannot be
ranked the honest answer depends on what was asked: a chain alone gets the
bench's real cross-chain ranking, labelled "all chains", which is also how the
embedder sees their ?chain= was not honoured; a region, with or without a
chain, gets no rank at all, because the aggregate is not a stand-in for a
region (it is a blend of the regions the reader wanted to compare).
The figure and its label now leave one function together. That pairing is the
fault, so it is the thing under test: badge-scope.test.ts pins the perp-fees
case directly (no chain may reach the label), both withholding cases, and the
cohort and default-view parts of each label. Mutation-checked three ways, each
caught: reintroducing the chain label on the aggregate, letting a region fall
back, and dropping the cohort from the cell label.
/badges was the other half. Its embed modal offered a Chain picker for every
bench that declared the dimension, which on those 22 meant a control that
silently did nothing, so the picker now follows what the badge can answer. It
reads cellRanks, the same signal the badge route looks up, so the two cannot
disagree and a cycle whose matrix query failed goes quiet instead of promising
a scope it cannot serve. Checked against the live aggregate blob in both
directions: 8 benches keep the picker (aggregator-head-lag, bridge-quote-latency,
bridge-quote-latency-solana, evm-quote-latency, metadata-coverage,
rpc-capabilities, rpc-reliability, token-quote-coverage), 22 lose it, and the
230 live benches with no chain dimension are untouched.
Badges live in other people's README files, which is why withholding is a
placeholder rather than a 404 and why this is the surface where a false scope
outlives the page it came from.
tsc clean, 816 tests pass, 260 specs valid.
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
…ain (#3403) * a rank badge crowned a provider the bench publishes outside its ranking Measured on staging after the board had already stopped doing it: /api/badge/rpc-api-coverage/alchemy?chain=robinhood "#1/6 (Robinhood Chain), 100.0%" /api/badge/rpc-api-coverage/blockmachine?chain=robinhood "#2/6 (Robinhood Chain), 95.9%" /api/badge/rpc-api-coverage/quicknode?chain=robinhood "#4/6 (Robinhood Chain), 91.1%" Two things are wrong at once. Alchemy leads that chain only because the one method it was not asked, trace_block, is the one every other provider there refuses, which is exactly why the board publishes it outside the ranking with "asked on only 86% of cells". And the percentages are the ALL-CHAINS figures under a chain label: Robinhood's real numbers are 85.7 for BlockMachine and 85.7 for Quicknode, not 95.9 and 91.1. The second is a documented approximation. rankOfChain says so in its own comment: the per-chain leader stash is forced to rank 1 and everyone else is ordered by their unfiltered p50, "soft approximation, full per-chain leaderboards live on the bench page chain tabs". The exact path exists (rankOfCell, fed by a spec's rank_matrix_query) and bench 311 declares none, so it falls through. Declaring one would fix the figures and NOT the crown, because the crown is in the value rather than in the approximation. This commit fixes the first only, which is the part that states something false rather than something imprecise. A row the bench publishes outside its ranking is not ranked here either. Three rank paths exist and all three would have published it, so the check sits before them rather than inside each. The answer is placeholderSvg, which already exists for "no defensible position this cycle", names the bench and claims no rank. The guard can only withhold a rank, never invent one. Badges are embedded in other people's README files, so this is the one surface where a false position outlives the page it came from. No test: the route is a Next handler whose rank paths need the whole bench object, and a mock thorough enough to exercise this is a bigger change than the guard. Verified against the live case on staging instead, which is the evidence that matters: the same URL that read "#1/6" must stop claiming a rank while the board still shows Alchemy outside the ranking. * a badge printed a chain's name over a figure measured across every chain Measured on production, 2026-10-29, one provider on one bench: /api/badge/perp-fees/hyperliquid #6/11 (all chains) 0.048% /api/badge/perp-fees/hyperliquid?chain=ETH #6/11 (ETH) 0.048% /api/badge/perp-fees/hyperliquid?chain=BTC #6/11 (BTC) 0.048% /api/badge/perp-fees/hyperliquid?chain=SOL #6/11 (SOL) 0.048% One figure, four labels, three of which it does not have. The exact per-chain path needs a spec's rank_matrix_query. Of the 30 live chain-dimensioned benches, 22 declare none, so the request fell through to rankOfChain, which forced the chain's own leader to rank 1 and read everyone else's UNFILTERED median: value: sortedOthers[idx].ms.p50, sorted by the same unfiltered median. Only slot 1 was chain-aware. The result is a hybrid that neither "ETH" nor "all chains" can label honestly, and the label was taken from the request rather than from the figure. So the hybrid is gone rather than relabelled. A scope label is now only ever printed over a figure measured in that scope, and when the scope cannot be ranked the honest answer depends on what was asked: a chain alone gets the bench's real cross-chain ranking, labelled "all chains", which is also how the embedder sees their ?chain= was not honoured; a region, with or without a chain, gets no rank at all, because the aggregate is not a stand-in for a region (it is a blend of the regions the reader wanted to compare). The figure and its label now leave one function together. That pairing is the fault, so it is the thing under test: badge-scope.test.ts pins the perp-fees case directly (no chain may reach the label), both withholding cases, and the cohort and default-view parts of each label. Mutation-checked three ways, each caught: reintroducing the chain label on the aggregate, letting a region fall back, and dropping the cohort from the cell label. /badges was the other half. Its embed modal offered a Chain picker for every bench that declared the dimension, which on those 22 meant a control that silently did nothing, so the picker now follows what the badge can answer. It reads cellRanks, the same signal the badge route looks up, so the two cannot disagree and a cycle whose matrix query failed goes quiet instead of promising a scope it cannot serve. Checked against the live aggregate blob in both directions: 8 benches keep the picker (aggregator-head-lag, bridge-quote-latency, bridge-quote-latency-solana, evm-quote-latency, metadata-coverage, rpc-capabilities, rpc-reliability, token-quote-coverage), 22 lose it, and the 230 live benches with no chain dimension are untouched. Badges live in other people's README files, which is why withholding is a placeholder rather than a 404 and why this is the surface where a false scope outlives the page it came from. tsc clean, 816 tests pass, 260 specs valid.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…lculators off every bench page Fixes from the 2026-10-30 re-audits of benches 309 and 310 (data and SEO passes), applied to bench 312 as well since it shares the code. Ranks. Product pages ranked only the prices that hold while the ledger and the calculator rank every priced row except an "at least" price at or under the leader, so /products/portal read "#8 of 8" in its bench list and "#6 of 6" in the table under it, and Privy "#7 of 8" beside "not ranked". src/lib/cost-rank.ts now holds the one rule; the product tables and the product metadata use it, with the same denominator. Machine surfaces. /api/stat ignored ?bucket=, so ?kind=trading&bucket=100k answered with the 10k cell while echoing only the kind; it now applies the size. /api/stat and the MCP rankings carry priceReading ("at most", "at least"), so Privy's floor is no longer a plain $6,500 in JSON. The variant API copies the headline seoTitle and seoDescription instead of resolving them on a filtered view under the headline's label. A new spec field, headline_slice, replaces "measured on AI agents, 50 transactions each, Medium size" in the citable headline with what the cell is. The Dataset JSON-LD of a bench with ledger_reliability: false no longer lists sample_size or percentile axes. Weight. Every bench page downloaded the cost calculators and their curves (68 KB gzip) because the shared route imported them statically; they now load through next/dynamic from a client wrapper, SSR kept. Display. The calculators print cents when a bill has cents (Dfns $2,916.67, not $2,917); product appearance rows print a published price whole ($51,299, not $51.3K); a $0 floor reads "$0 on its published rates" instead of "at least $0". Vendors priced only by the pricing benches get an H1 that says pricing, and one on several of them (Circle, Turnkey, Openfort, ZeroDev) a title naming every kind and a description from the first headline bill it has, instead of bench 309's alone. Copy. 309: Portal's meter in the Units bullet and the transactions FAQ, the changelog sentences out of the Dynamic finding and the cohort, the false Openfort cross-reference dropped, Privy's 100k bills read from the curve, the freshness gap stated. 310: the bot FAQ and finding name the curve's winners (Magic ties Circle at 10,000 transactions), Dfns and Crossmint in Units, Circle's chain scope and USDC rebate, bot and payout cells in the intro, a title that does not promise one winner. Validator. A question asking for the cheapest must be answered by a rank cell in its first sentence, and only a rank cell exempts a winner sentence (any vendor cell did). Proved by putting the old bot answer back: it fails. Harnesses (309, 310, 312). The site curves are re-read every 5 minutes and the mismatch alert fires after 15 (was hourly and 2h, blind to the 30 minutes production priced from two catalogues on 2026-10-30); the release order is in the wallet-cost README. Cache keys bench-unfiltered-v131, all-benchmarks-v124, bench-filters-v69.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…lculators off every bench page Fixes from the 2026-10-30 re-audits of benches 309 and 310 (data and SEO passes), applied to bench 312 as well since it shares the code. Ranks. Product pages ranked only the prices that hold while the ledger and the calculator rank every priced row except an "at least" price at or under the leader, so /products/portal read "#8 of 8" in its bench list and "#6 of 6" in the table under it, and Privy "#7 of 8" beside "not ranked". src/lib/cost-rank.ts now holds the one rule; the product tables and the product metadata use it, with the same denominator. Machine surfaces. /api/stat ignored ?bucket=, so ?kind=trading&bucket=100k answered with the 10k cell while echoing only the kind; it now applies the size. /api/stat and the MCP rankings carry priceReading ("at most", "at least"), so Privy's floor is no longer a plain $6,500 in JSON. The variant API copies the headline seoTitle and seoDescription instead of resolving them on a filtered view under the headline's label. A new spec field, headline_slice, replaces "measured on AI agents, 50 transactions each, Medium size" in the citable headline with what the cell is. The Dataset JSON-LD of a bench with ledger_reliability: false no longer lists sample_size or percentile axes. Weight. Every bench page downloaded the cost calculators and their curves (68 KB gzip) because the shared route imported them statically; they now load through next/dynamic from a client wrapper, SSR kept. Display. The calculators print cents when a bill has cents (Dfns $2,916.67, not $2,917); product appearance rows print a published price whole ($51,299, not $51.3K); a $0 floor reads "$0 on its published rates" instead of "at least $0". Vendors priced only by the pricing benches get an H1 that says pricing, and one on several of them (Circle, Turnkey, Openfort, ZeroDev) a title naming every kind and a description from the first headline bill it has, instead of bench 309's alone. Copy. 309: Portal's meter in the Units bullet and the transactions FAQ, the changelog sentences out of the Dynamic finding and the cohort, the false Openfort cross-reference dropped, Privy's 100k bills read from the curve, the freshness gap stated. 310: the bot FAQ and finding name the curve's winners (Magic ties Circle at 10,000 transactions), Dfns and Crossmint in Units, Circle's chain scope and USDC rebate, bot and payout cells in the intro, a title that does not promise one winner. Validator. A question asking for the cheapest must be answered by a rank cell in its first sentence, and only a rank cell exempts a winner sentence (any vendor cell did). Proved by putting the old bot answer back: it fails. Harnesses (309, 310, 312). The site curves are re-read every 5 minutes and the mismatch alert fires after 15 (was hourly and 2h, blind to the 30 minutes production priced from two catalogues on 2026-10-30); the release order is in the wallet-cost README. Cache keys bench-unfiltered-v132, all-benchmarks-v125, bench-filters-v70.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…lculators off every bench page (#3433) Fixes from the 2026-10-30 re-audits of benches 309 and 310 (data and SEO passes), applied to bench 312 as well since it shares the code. Ranks. Product pages ranked only the prices that hold while the ledger and the calculator rank every priced row except an "at least" price at or under the leader, so /products/portal read "#8 of 8" in its bench list and "#6 of 6" in the table under it, and Privy "#7 of 8" beside "not ranked". src/lib/cost-rank.ts now holds the one rule; the product tables and the product metadata use it, with the same denominator. Machine surfaces. /api/stat ignored ?bucket=, so ?kind=trading&bucket=100k answered with the 10k cell while echoing only the kind; it now applies the size. /api/stat and the MCP rankings carry priceReading ("at most", "at least"), so Privy's floor is no longer a plain $6,500 in JSON. The variant API copies the headline seoTitle and seoDescription instead of resolving them on a filtered view under the headline's label. A new spec field, headline_slice, replaces "measured on AI agents, 50 transactions each, Medium size" in the citable headline with what the cell is. The Dataset JSON-LD of a bench with ledger_reliability: false no longer lists sample_size or percentile axes. Weight. Every bench page downloaded the cost calculators and their curves (68 KB gzip) because the shared route imported them statically; they now load through next/dynamic from a client wrapper, SSR kept. Display. The calculators print cents when a bill has cents (Dfns $2,916.67, not $2,917); product appearance rows print a published price whole ($51,299, not $51.3K); a $0 floor reads "$0 on its published rates" instead of "at least $0". Vendors priced only by the pricing benches get an H1 that says pricing, and one on several of them (Circle, Turnkey, Openfort, ZeroDev) a title naming every kind and a description from the first headline bill it has, instead of bench 309's alone. Copy. 309: Portal's meter in the Units bullet and the transactions FAQ, the changelog sentences out of the Dynamic finding and the cohort, the false Openfort cross-reference dropped, Privy's 100k bills read from the curve, the freshness gap stated. 310: the bot FAQ and finding name the curve's winners (Magic ties Circle at 10,000 transactions), Dfns and Crossmint in Units, Circle's chain scope and USDC rebate, bot and payout cells in the intro, a title that does not promise one winner. Validator. A question asking for the cheapest must be answered by a rank cell in its first sentence, and only a rank cell exempts a winner sentence (any vendor cell did). Proved by putting the old bot answer back: it fails. Harnesses (309, 310, 312). The site curves are re-read every 5 minutes and the mismatch alert fires after 15 (was hourly and 2h, blind to the 30 minutes production priced from two catalogues on 2026-10-30); the release order is in the wallet-cost README. Cache keys bench-unfiltered-v132, all-benchmarks-v125, bench-filters-v70.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repositions the project around the federation model: each contributor hosts their own harness on whatever infra they prefer; the project shares only one Prometheus that scrapes public /metrics URLs. Maintainers never see contributor secrets.
What's updated
infrastructure/prometheus/prometheus.yml
*.railway.internal:port(single-Railway-project model) to public HTTPS URLs (federation model)host:label to each job to credit who runs each harnessinfrastructure/README.md
harnesses/README.md
CONTRIBUTING.md
/contribute page
root README.md
What this does NOT change
Operational reality
Mobula currently hosts the existing two harnesses (aggregator-head-lag, bridge-monitor) on its Railway infrastructure, sponsoring those benchmarks. The OpenChainBench Prometheus described in this PR is a target architecture: when it's up, the YAMLs'
prom_urlwill switch to it in a one-line change.