Repository navigation
bench 033: we published Route as the fastest EVM quote API on four chains it never enters - #3052
Merged
Merged
Conversation
…ains it never enters The audit's two blockers and two of its majors, all live on production. 1. Route quotes Robinhood Chain alone. Its all-chains figure was the mean of 3 Robinhood cells, 215 ms, which ranked above KyberSwap's mean of 15 cells across 5 chains, 225 ms. /products/route read "ranks #1 of 6". Route does lead Robinhood; KyberSwap leads the four chains Route has no row on. The headline said the opposite of the evidence under it. The loader already knew: the cell-rank marginals admit a provider only when it covers every cell of the collapsed dimension, and the comment there calls it Simpson's bias. That rule governed badges and never the ledger or the product-page rank, which is where the claim is read. score_scope gains `full_coverage` next to the existing `contested_chains`. The existing one does not catch this: it asks whether a chain was contested, and Robinhood is, with three providers. The new one asks whether the providers compared cover the same ground. A provider missing from any chain with data becomes unranked carrying what it covers ("Robinhood only"), stays visible, and keeps its own chain tab where the comparison is like for like. Opt-in per bench; no other spec changes behaviour. 2. Sentences promising to describe the active tab always rendered the aggregate, so the Ethereum tab said "Route currently leads at 215 ms" with no Route row on it and KyberSwap leading at 199. The copy now says "across all chains and regions", which is what it actually shows. The deeper fix, re-rendering unscoped placeholders against the variant in the variant route, is not in here. 3. LI.FI's Success column read 62%. 1,602 of its 4,320 daily probes are throttled by the free-tier budget we probe it on, so that number is our access, published under their name. The FAQ now says so. 4. Enso's formula still promised "~20% of probes hit 429, remaining samples keep p50 and p90 stable" while the key's credits are exhausted and nothing is recorded at all. Tests: six on applyFullCoverageScope, including that a fully covering provider is untouched, that one chain is not a cross-chain comparison, and that a label over the twenty-character ceiling falls back to a count.
Flotapponnier
added a commit
that referenced
this pull request
Oct 19, 2026
#3052 stopped the page calling Route the fastest EVM quote API overall, which was the point. But demanding coverage of every chain also unranked Mobula, Bebop and CoW, each covering four of five and each exactly the comparison a reader wants. /api/stat went from "6 ranked providers" to 2. Robinhood joined this bench on 2026-10-17 and three of the six live providers reach it. A chain most of the field does not reach is not the ground the comparison stands on; it is a tab of its own. The rule now scores over the common basket: the chains more than half the live providers cover. Everyone is rescored over that basket, so the published number is the one the ranking follows, the same property contested_chains states. Only a provider with nothing in the basket leaves the order, which here is Route and only Route. The threshold is strictly more than half, not at least half. Robinhood carries exactly three of six, and "at least" let it back into the basket, which made the whole rule a no-op. The test fixture now mirrors the real cohort, six live providers across five chains, and would have caught it. Four tests added or rewritten: the thin-chain provider leaves the order, the four-of-five providers stay in it, the published value is the mean over the basket rather than over everything, and the leader after the fix is the one that leads the shared chains.
Flotapponnier
added a commit
that referenced
this pull request
Oct 19, 2026
The re-run audit's pass A opened on it: Route published at exactly 0 ms p50 on 717 samples. That is my own fix from #3052 and #3053. Taking a row out of the order, I zeroed its value because the spec-declared unranked path does, and the ledger rendered "0 ms" in p50, p90 and p99. A wall-clock round trip of zero is impossible. A reader sees either a broken row or Route being instant, and the second is worse than the claim the fix was written to remove. Nothing needed the zero. citation.ts picks the leader from rows filtered on the label, the ledger pins labelled rows below the ranked field and excludes them from ranks and the field mean, and capital-hub filters on the label too. The label alone does the work. Route now keeps its measured 215 ms and sits in its own block under "Robinhood only", which is both true and useful: it is the number a reader wants once they know what it covers. Pass A also claimed the Robinhood explainer and badge named a 0 ms leader. They did not: bestPerChain carries Route at 216.209 and the page renders "at 217 ms". The finding was right about the ledger and wrong about the explainer.
Flotapponnier
added a commit
that referenced
this pull request
Oct 19, 2026
…untouched (#3057) * the unranked row read 0 ms, which is not a latency The re-run audit's pass A opened on it: Route published at exactly 0 ms p50 on 717 samples. That is my own fix from #3052 and #3053. Taking a row out of the order, I zeroed its value because the spec-declared unranked path does, and the ledger rendered "0 ms" in p50, p90 and p99. A wall-clock round trip of zero is impossible. A reader sees either a broken row or Route being instant, and the second is worse than the claim the fix was written to remove. Nothing needed the zero. citation.ts picks the leader from rows filtered on the label, the ledger pins labelled rows below the ranked field and excludes them from ranks and the field mean, and capital-hub filters on the label too. The label alone does the work. Route now keeps its measured 215 ms and sits in its own block under "Robinhood only", which is both true and useful: it is the number a reader wants once they know what it covers. Pass A also claimed the Robinhood explainer and badge named a 0 ms leader. They did not: bestPerChain carries Route at 216.209 and the page renders "at 217 ms". The finding was right about the ledger and wrong about the explainer. * the rescored tail was a copy of the median, and the region tabs were untouched Both blockers of the re-run audit, both introduced by #3052 and #3053. 1. Rescoring over the basket wrote the p50 mean into p90, p99 and mean alike, so the tail columns became copies of the median. KyberSwap read 220 ms in all four while its measured p99 is 931, and CoW's 4,961 ms Arbitrum tail hid behind 1,658. A desk reads the p99 column precisely because it is not the median. The per-chain fan-out now carries every percentile, not just the p50, and each one is averaged over the basket on its own. 2. The rule only ran on the unfiltered view, so Route still ranked first on the US East and EU West tabs: the exact claim #3052 was written to remove, surviving one click away from the page it was removed from. The fan-out now also runs on a view filtered by something other than chain when the bench declares a score_scope. A chain tab needs nothing, being one chain already, and the extra loads are paid only by a bench that asked for the scope. The test fixture carries tails far from its medians, so a regression that copies the p50 into p90 and p99 fails loudly rather than passing on numbers that happen to be close.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The audit's two blockers and two of its majors, all live on production right now.
1. Route ranked #1 of 6 by comparing one chain against five
Route quotes Robinhood Chain alone.
/products/routeread "ranks #1 of 6, 215 ms p50". Route genuinely leads Robinhood; KyberSwap leads the four chains Route has no row on. The headline said the opposite of the evidence beneath it.The loader already knew.
load.tsadmits a provider into a cell-rank marginal only when it covers every cell of the collapsed dimension, and the comment there calls it Simpson's bias. That rule governed badges, never the ledger or the product-page rank, which is where the claim is actually read.score_scopegainsfull_coveragebeside the existingcontested_chains. The existing one does not catch this: it asks whether a chain was contested, and Robinhood is, with three providers. The new one asks whether the providers being compared cover the same ground. A provider missing from any chain with data becomes unranked carrying what it covers ("Robinhood only"), stays visible, and keeps its own chain tab where the comparison is like for like.Opt-in per bench. No other spec changes behaviour.
2. Tab sentences always rendered the aggregate
The Ethereum tab said "Route currently leads at 215 ms" with no Route row on it and KyberSwap leading at 199 ms. Same on Singapore. The copy now says "across all chains and regions", which is what it actually shows.
The deeper fix — re-rendering unscoped placeholders against the variant in
variant/route.ts— is not in here. The copy change stops the false claim; the mechanism still deserves its own PR.3. LI.FI's 62% Success was our rate-limit budget
1,602 of its 4,320 daily probes are throttled on the free tier we probe it on. That number is our access, published under their name. The FAQ now says so explicitly, with the arithmetic.
4. Enso's formula promised stability while recording nothing
It still read "~20% of probes hit 429, remaining samples keep p50 and p90 stable" while the key's credits are exhausted and no latency is recorded at all.
Tests
Six on
applyFullCoverageScope: the single-chain provider leaves the ranking and says what it covers, a fully covering provider is untouched, partial coverage is counted rather than named, one chain is not a cross-chain comparison, a label over the 20-char ceiling falls back to a count, and an unavailable provider is left alone.Gates: 251 specs valid, tsc clean, 603 tests pass.