Repository navigation
a product page claimed first place on a chain where the board publishes no rank - #3425
Merged
Merged
Conversation
The adversarial audit of bench 311 came back FIX THEN SHIP, and the two
blockers it found below the worker lag share one cause: every selector on this
bench aggregates `rpc_api_supported` and `rpc_api_probe_total` with no `api`
filter. So each probe added in the last few hours silently joined the
headline, the success rate, the sample size, the rank gate and the freshness
column. I added four probes and two ladder counters and did not look at what
already summed them.
WHAT IT PUBLISHED, measured by the audit against the live blob.
The two eth_call legs write the capability gauge, so each chain contributed
nine cells where the copy describes seven:
provider published under the stated 7-cell method
Quicknode 93.06 91.07
Chainstack 95.83 94.64
GetBlock 81.94 76.79
Dwellir 91.67 89.29
BlockMachine 96.83 95.92
Every ranked headline sat 0.9 to 5.2 points high, the GetBlock to BlockMachine
gap read 19.1 where it is 14.9, and the sample said n=72 where the copy says
56. The order and the gate did not move, which is why nothing looked wrong.
The range ladder writes a probe counter and no capability gauge, so the
success denominator grew by two APIs while the numerator did not. Every ranked
row read 81.8182 percent, which is 72/88, for cells that all reached a
verdict: the figure this spec defines is 100. The tooltip then invented
failures from it, "13 failed of 72 measurements", none of which happened. On
the Robinhood tab Alchemy's unranked badge read 73 percent for what is 6 of 7.
THE FIX is an allowlist of the seven capability questions on all 67
selectors, written out rather than globbed: `eth_getlogs_(nofilter|address|
topics)` and the four others by name. A wildcard would let the next probe in
the same way, and the ladder's own counters are called `getlogs_range_*`,
which a careless `eth_getlogs_.*` would not catch today and might tomorrow.
Measurements keep their own views; they have no business in a share of cells
served.
THREE MORE FROM THE SAME AUDIT, each bounded:
The freshness column took `max by (provider)` over every api, and the eth_call
stamps are rewritten hourly while capability verdicts move every six hours. So
the column would have sat near one hour forever while a capability pass
stalled, which is the exact failure its own methodology bullet promises to
expose. It now reads the OLDEST capability stamp.
The Solana view pinned `chain="solana"` and that pin survives dimension
injection, so it rendered Solana values on every EVM tab: on the Ethereum tab
it showed chainstack 0, quicknode 0, getblock 0. The bench declares Solana out
of scope in its own methodology. The view is gone; the probe stays in the
harness for the Solana bench that owns the question.
The FAQ answered "Not yet ... planned and is deliberately absent" about the
window measurement, directly above the two columns that publish it. Rewritten
to describe the ladder.
AND TWO CORRECTIONS TO MY OWN COPY. The cadence bullet said every method is
asked every six hours; the two eth_call legs have run hourly since #3412. And
both the spec and my commit message for #3420 said the legs share one cold
connection. They do not: the cold transport sets DisableKeepAlives, so each
leg dials its own and pays its own handshake. That is not a detail, because
the handshake noise is what the difference carries, and `> 0` only removes the
half that falls below zero. The copy now says so, and the view is described as
a number rather than a ranking until the hourly lane gives it enough samples.
Cache keys move in lockstep, v126 / v119 / v64, rebased onto the v125 another
session landed while this was being written.
260 specs valid, tsc clean, 828 tests pass.
…es no rank Third blocker from the bench 311 audit, and the same shape as the badge faults removed this morning: a rank handed to a row the board itself declines to rank, and a tie resolved by whoever happens to be first in the YAML. bestPerChain took `sorted[0]` of every live row on a chain. Two live consequences. A GATED ROW WAS CROWNED. Alchemy reads 100 on the Robinhood tab and the board publishes it UNRANKED there, with its cause printed beside it: the one method it was not asked is the one the rest of the cohort refuses. The stash does not read `unrankedLabel`, so /products/alchemy rendered "#1 of 5 on Robinhood Chain" while the bench declined to rank the row. The 5 is the ranked pool, which does not contain Alchemy, so the denominator described a different set from the numerator. A TIE WENT TO SPEC ORDER. On Ethereum, Quicknode, Chainstack, Dwellir and BlockMachine all read 100. The sort is stable, so first-declared won: /products/quicknode showed "#1 of 5" on Ethereum, Base, Arbitrum, BNB Chain and Arc while three equal providers got nothing. A tie is not a ranking, and this repo spent the year removing that shape from boards. A chip is the same claim in a smaller box. So a gated row cannot lead, and a shared best value elects nobody. The conservative answer is no chip rather than a chip three others have an equal right to. The trailer end applies only the first rule: a tie for worst harms nobody, but naming a gated row the worst reads as a verdict on a row whose whole point is that it was not compared. The rule is its own module because it is two sentences that are easy to get right and easy to lose in a sixty-line loop, and because it needed the exact production cases replayed rather than a neighbouring shape. Ten tests carrying the Robinhood and Ethereum readings as they were measured. Mutation-checked by restoring `sorted[0]`, which fails five of them. Four cache keys move: bench-unfiltered-v127, all-benchmarks-v120, bench-filters-v65 and providers-v34. The last one is the point: the index feeds the chip, so a cached one keeps claiming the rank for its whole 900 s TTL. Rebased onto the v33 another session landed while this was being written. Stacked on #3424, which fixes the same audit's other two findings. 260 specs valid, tsc clean, 838 tests pass.
Flotapponnier
force-pushed
the
fix/per-chain-leader-gate
branch
from
October 30, 2026 15:57
358d5e6 to
35b697d
Compare
# Conflicts: # src/lib/spec.ts
# Conflicts: # src/lib/providers.ts # src/lib/spec.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
a product page claimed first place on a chain where the board publishes no rank
Third blocker from the bench 311 audit, and the same shape as the badge faults
removed this morning: a rank handed to a row the board itself declines to
rank, and a tie resolved by whoever happens to be first in the YAML.
bestPerChain took
sorted[0]of every live row on a chain. Two liveconsequences.
A GATED ROW WAS CROWNED. Alchemy reads 100 on the Robinhood tab and the board
publishes it UNRANKED there, with its cause printed beside it: the one method
it was not asked is the one the rest of the cohort refuses. The stash does not
read
unrankedLabel, so /products/alchemy rendered "#1 of 5 on RobinhoodChain" while the bench declined to rank the row. The 5 is the ranked pool,
which does not contain Alchemy, so the denominator described a different set
from the numerator.
A TIE WENT TO SPEC ORDER. On Ethereum, Quicknode, Chainstack, Dwellir and
BlockMachine all read 100. The sort is stable, so first-declared won:
/products/quicknode showed "#1 of 5" on Ethereum, Base, Arbitrum, BNB Chain
and Arc while three equal providers got nothing. A tie is not a ranking, and
this repo spent the year removing that shape from boards. A chip is the same
claim in a smaller box.
So a gated row cannot lead, and a shared best value elects nobody. The
conservative answer is no chip rather than a chip three others have an equal
right to. The trailer end applies only the first rule: a tie for worst harms
nobody, but naming a gated row the worst reads as a verdict on a row whose
whole point is that it was not compared.
The rule is its own module because it is two sentences that are easy to get
right and easy to lose in a sixty-line loop, and because it needed the exact
production cases replayed rather than a neighbouring shape. Ten tests carrying
the Robinhood and Ethereum readings as they were measured. Mutation-checked by
restoring
sorted[0], which fails five of them.Four cache keys move: bench-unfiltered-v127, all-benchmarks-v120,
bench-filters-v65 and providers-v34. The last one is the point: the index
feeds the chip, so a cached one keeps claiming the rank for its whole 900 s
TTL. Rebased onto the v33 another session landed while this was being written.
Stacked on #3424, which fixes the same audit's other two findings.
260 specs valid, tsc clean, 838 tests pass.