Repository navigation
every probe added today joined the headline without being asked - #3424
Merged
Merged
Conversation
The adversarial audit of bench 311 came back FIX THEN SHIP, and the two
blockers it found below the worker lag share one cause: every selector on this
bench aggregates `rpc_api_supported` and `rpc_api_probe_total` with no `api`
filter. So each probe added in the last few hours silently joined the
headline, the success rate, the sample size, the rank gate and the freshness
column. I added four probes and two ladder counters and did not look at what
already summed them.
WHAT IT PUBLISHED, measured by the audit against the live blob.
The two eth_call legs write the capability gauge, so each chain contributed
nine cells where the copy describes seven:
provider published under the stated 7-cell method
Quicknode 93.06 91.07
Chainstack 95.83 94.64
GetBlock 81.94 76.79
Dwellir 91.67 89.29
BlockMachine 96.83 95.92
Every ranked headline sat 0.9 to 5.2 points high, the GetBlock to BlockMachine
gap read 19.1 where it is 14.9, and the sample said n=72 where the copy says
56. The order and the gate did not move, which is why nothing looked wrong.
The range ladder writes a probe counter and no capability gauge, so the
success denominator grew by two APIs while the numerator did not. Every ranked
row read 81.8182 percent, which is 72/88, for cells that all reached a
verdict: the figure this spec defines is 100. The tooltip then invented
failures from it, "13 failed of 72 measurements", none of which happened. On
the Robinhood tab Alchemy's unranked badge read 73 percent for what is 6 of 7.
THE FIX is an allowlist of the seven capability questions on all 67
selectors, written out rather than globbed: `eth_getlogs_(nofilter|address|
topics)` and the four others by name. A wildcard would let the next probe in
the same way, and the ladder's own counters are called `getlogs_range_*`,
which a careless `eth_getlogs_.*` would not catch today and might tomorrow.
Measurements keep their own views; they have no business in a share of cells
served.
THREE MORE FROM THE SAME AUDIT, each bounded:
The freshness column took `max by (provider)` over every api, and the eth_call
stamps are rewritten hourly while capability verdicts move every six hours. So
the column would have sat near one hour forever while a capability pass
stalled, which is the exact failure its own methodology bullet promises to
expose. It now reads the OLDEST capability stamp.
The Solana view pinned `chain="solana"` and that pin survives dimension
injection, so it rendered Solana values on every EVM tab: on the Ethereum tab
it showed chainstack 0, quicknode 0, getblock 0. The bench declares Solana out
of scope in its own methodology. The view is gone; the probe stays in the
harness for the Solana bench that owns the question.
The FAQ answered "Not yet ... planned and is deliberately absent" about the
window measurement, directly above the two columns that publish it. Rewritten
to describe the ladder.
AND TWO CORRECTIONS TO MY OWN COPY. The cadence bullet said every method is
asked every six hours; the two eth_call legs have run hourly since #3412. And
both the spec and my commit message for #3420 said the legs share one cold
connection. They do not: the cold transport sets DisableKeepAlives, so each
leg dials its own and pays its own handshake. That is not a detail, because
the handshake noise is what the difference carries, and `> 0` only removes the
half that falls below zero. The copy now says so, and the view is described as
a number rather than a ranking until the hourly lane gives it enough samples.
Cache keys move in lockstep, v126 / v119 / v64, rebased onto the v125 another
session landed while this was being written.
260 specs valid, tsc clean, 828 tests pass.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…es no rank Third blocker from the bench 311 audit, and the same shape as the badge faults removed this morning: a rank handed to a row the board itself declines to rank, and a tie resolved by whoever happens to be first in the YAML. bestPerChain took `sorted[0]` of every live row on a chain. Two live consequences. A GATED ROW WAS CROWNED. Alchemy reads 100 on the Robinhood tab and the board publishes it UNRANKED there, with its cause printed beside it: the one method it was not asked is the one the rest of the cohort refuses. The stash does not read `unrankedLabel`, so /products/alchemy rendered "#1 of 5 on Robinhood Chain" while the bench declined to rank the row. The 5 is the ranked pool, which does not contain Alchemy, so the denominator described a different set from the numerator. A TIE WENT TO SPEC ORDER. On Ethereum, Quicknode, Chainstack, Dwellir and BlockMachine all read 100. The sort is stable, so first-declared won: /products/quicknode showed "#1 of 5" on Ethereum, Base, Arbitrum, BNB Chain and Arc while three equal providers got nothing. A tie is not a ranking, and this repo spent the year removing that shape from boards. A chip is the same claim in a smaller box. So a gated row cannot lead, and a shared best value elects nobody. The conservative answer is no chip rather than a chip three others have an equal right to. The trailer end applies only the first rule: a tie for worst harms nobody, but naming a gated row the worst reads as a verdict on a row whose whole point is that it was not compared. The rule is its own module because it is two sentences that are easy to get right and easy to lose in a sixty-line loop, and because it needed the exact production cases replayed rather than a neighbouring shape. Ten tests carrying the Robinhood and Ethereum readings as they were measured. Mutation-checked by restoring `sorted[0]`, which fails five of them. Four cache keys move: bench-unfiltered-v127, all-benchmarks-v120, bench-filters-v65 and providers-v34. The last one is the point: the index feeds the chip, so a cached one keeps claiming the rank for its whole 900 s TTL. Rebased onto the v33 another session landed while this was being written. Stacked on #3424, which fixes the same audit's other two findings. 260 specs valid, tsc clean, 838 tests pass.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…es no rank (#3425) * every probe added today joined the headline without being asked The adversarial audit of bench 311 came back FIX THEN SHIP, and the two blockers it found below the worker lag share one cause: every selector on this bench aggregates `rpc_api_supported` and `rpc_api_probe_total` with no `api` filter. So each probe added in the last few hours silently joined the headline, the success rate, the sample size, the rank gate and the freshness column. I added four probes and two ladder counters and did not look at what already summed them. WHAT IT PUBLISHED, measured by the audit against the live blob. The two eth_call legs write the capability gauge, so each chain contributed nine cells where the copy describes seven: provider published under the stated 7-cell method Quicknode 93.06 91.07 Chainstack 95.83 94.64 GetBlock 81.94 76.79 Dwellir 91.67 89.29 BlockMachine 96.83 95.92 Every ranked headline sat 0.9 to 5.2 points high, the GetBlock to BlockMachine gap read 19.1 where it is 14.9, and the sample said n=72 where the copy says 56. The order and the gate did not move, which is why nothing looked wrong. The range ladder writes a probe counter and no capability gauge, so the success denominator grew by two APIs while the numerator did not. Every ranked row read 81.8182 percent, which is 72/88, for cells that all reached a verdict: the figure this spec defines is 100. The tooltip then invented failures from it, "13 failed of 72 measurements", none of which happened. On the Robinhood tab Alchemy's unranked badge read 73 percent for what is 6 of 7. THE FIX is an allowlist of the seven capability questions on all 67 selectors, written out rather than globbed: `eth_getlogs_(nofilter|address| topics)` and the four others by name. A wildcard would let the next probe in the same way, and the ladder's own counters are called `getlogs_range_*`, which a careless `eth_getlogs_.*` would not catch today and might tomorrow. Measurements keep their own views; they have no business in a share of cells served. THREE MORE FROM THE SAME AUDIT, each bounded: The freshness column took `max by (provider)` over every api, and the eth_call stamps are rewritten hourly while capability verdicts move every six hours. So the column would have sat near one hour forever while a capability pass stalled, which is the exact failure its own methodology bullet promises to expose. It now reads the OLDEST capability stamp. The Solana view pinned `chain="solana"` and that pin survives dimension injection, so it rendered Solana values on every EVM tab: on the Ethereum tab it showed chainstack 0, quicknode 0, getblock 0. The bench declares Solana out of scope in its own methodology. The view is gone; the probe stays in the harness for the Solana bench that owns the question. The FAQ answered "Not yet ... planned and is deliberately absent" about the window measurement, directly above the two columns that publish it. Rewritten to describe the ladder. AND TWO CORRECTIONS TO MY OWN COPY. The cadence bullet said every method is asked every six hours; the two eth_call legs have run hourly since #3412. And both the spec and my commit message for #3420 said the legs share one cold connection. They do not: the cold transport sets DisableKeepAlives, so each leg dials its own and pays its own handshake. That is not a detail, because the handshake noise is what the difference carries, and `> 0` only removes the half that falls below zero. The copy now says so, and the view is described as a number rather than a ranking until the hourly lane gives it enough samples. Cache keys move in lockstep, v126 / v119 / v64, rebased onto the v125 another session landed while this was being written. 260 specs valid, tsc clean, 828 tests pass. * a product page claimed first place on a chain where the board publishes no rank Third blocker from the bench 311 audit, and the same shape as the badge faults removed this morning: a rank handed to a row the board itself declines to rank, and a tie resolved by whoever happens to be first in the YAML. bestPerChain took `sorted[0]` of every live row on a chain. Two live consequences. A GATED ROW WAS CROWNED. Alchemy reads 100 on the Robinhood tab and the board publishes it UNRANKED there, with its cause printed beside it: the one method it was not asked is the one the rest of the cohort refuses. The stash does not read `unrankedLabel`, so /products/alchemy rendered "#1 of 5 on Robinhood Chain" while the bench declined to rank the row. The 5 is the ranked pool, which does not contain Alchemy, so the denominator described a different set from the numerator. A TIE WENT TO SPEC ORDER. On Ethereum, Quicknode, Chainstack, Dwellir and BlockMachine all read 100. The sort is stable, so first-declared won: /products/quicknode showed "#1 of 5" on Ethereum, Base, Arbitrum, BNB Chain and Arc while three equal providers got nothing. A tie is not a ranking, and this repo spent the year removing that shape from boards. A chip is the same claim in a smaller box. So a gated row cannot lead, and a shared best value elects nobody. The conservative answer is no chip rather than a chip three others have an equal right to. The trailer end applies only the first rule: a tie for worst harms nobody, but naming a gated row the worst reads as a verdict on a row whose whole point is that it was not compared. The rule is its own module because it is two sentences that are easy to get right and easy to lose in a sixty-line loop, and because it needed the exact production cases replayed rather than a neighbouring shape. Ten tests carrying the Robinhood and Ethereum readings as they were measured. Mutation-checked by restoring `sorted[0]`, which fails five of them. Four cache keys move: bench-unfiltered-v127, all-benchmarks-v120, bench-filters-v65 and providers-v34. The last one is the point: the index feeds the chip, so a cached one keeps claiming the rank for its whole 900 s TTL. Rebased onto the v33 another session landed while this was being written. Stacked on #3424, which fixes the same audit's other two findings. 260 specs valid, tsc clean, 838 tests pass.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…as inside the measurement (#3427) * every probe added today joined the headline without being asked The adversarial audit of bench 311 came back FIX THEN SHIP, and the two blockers it found below the worker lag share one cause: every selector on this bench aggregates `rpc_api_supported` and `rpc_api_probe_total` with no `api` filter. So each probe added in the last few hours silently joined the headline, the success rate, the sample size, the rank gate and the freshness column. I added four probes and two ladder counters and did not look at what already summed them. WHAT IT PUBLISHED, measured by the audit against the live blob. The two eth_call legs write the capability gauge, so each chain contributed nine cells where the copy describes seven: provider published under the stated 7-cell method Quicknode 93.06 91.07 Chainstack 95.83 94.64 GetBlock 81.94 76.79 Dwellir 91.67 89.29 BlockMachine 96.83 95.92 Every ranked headline sat 0.9 to 5.2 points high, the GetBlock to BlockMachine gap read 19.1 where it is 14.9, and the sample said n=72 where the copy says 56. The order and the gate did not move, which is why nothing looked wrong. The range ladder writes a probe counter and no capability gauge, so the success denominator grew by two APIs while the numerator did not. Every ranked row read 81.8182 percent, which is 72/88, for cells that all reached a verdict: the figure this spec defines is 100. The tooltip then invented failures from it, "13 failed of 72 measurements", none of which happened. On the Robinhood tab Alchemy's unranked badge read 73 percent for what is 6 of 7. THE FIX is an allowlist of the seven capability questions on all 67 selectors, written out rather than globbed: `eth_getlogs_(nofilter|address| topics)` and the four others by name. A wildcard would let the next probe in the same way, and the ladder's own counters are called `getlogs_range_*`, which a careless `eth_getlogs_.*` would not catch today and might tomorrow. Measurements keep their own views; they have no business in a share of cells served. THREE MORE FROM THE SAME AUDIT, each bounded: The freshness column took `max by (provider)` over every api, and the eth_call stamps are rewritten hourly while capability verdicts move every six hours. So the column would have sat near one hour forever while a capability pass stalled, which is the exact failure its own methodology bullet promises to expose. It now reads the OLDEST capability stamp. The Solana view pinned `chain="solana"` and that pin survives dimension injection, so it rendered Solana values on every EVM tab: on the Ethereum tab it showed chainstack 0, quicknode 0, getblock 0. The bench declares Solana out of scope in its own methodology. The view is gone; the probe stays in the harness for the Solana bench that owns the question. The FAQ answered "Not yet ... planned and is deliberately absent" about the window measurement, directly above the two columns that publish it. Rewritten to describe the ladder. AND TWO CORRECTIONS TO MY OWN COPY. The cadence bullet said every method is asked every six hours; the two eth_call legs have run hourly since #3412. And both the spec and my commit message for #3420 said the legs share one cold connection. They do not: the cold transport sets DisableKeepAlives, so each leg dials its own and pays its own handshake. That is not a detail, because the handshake noise is what the difference carries, and `> 0` only removes the half that falls below zero. The copy now says so, and the view is described as a number rather than a ranking until the hourly lane gives it enough samples. Cache keys move in lockstep, v126 / v119 / v64, rebased onto the v125 another session landed while this was being written. 260 specs valid, tsc clean, 828 tests pass. * a product page claimed first place on a chain where the board publishes no rank Third blocker from the bench 311 audit, and the same shape as the badge faults removed this morning: a rank handed to a row the board itself declines to rank, and a tie resolved by whoever happens to be first in the YAML. bestPerChain took `sorted[0]` of every live row on a chain. Two live consequences. A GATED ROW WAS CROWNED. Alchemy reads 100 on the Robinhood tab and the board publishes it UNRANKED there, with its cause printed beside it: the one method it was not asked is the one the rest of the cohort refuses. The stash does not read `unrankedLabel`, so /products/alchemy rendered "#1 of 5 on Robinhood Chain" while the bench declined to rank the row. The 5 is the ranked pool, which does not contain Alchemy, so the denominator described a different set from the numerator. A TIE WENT TO SPEC ORDER. On Ethereum, Quicknode, Chainstack, Dwellir and BlockMachine all read 100. The sort is stable, so first-declared won: /products/quicknode showed "#1 of 5" on Ethereum, Base, Arbitrum, BNB Chain and Arc while three equal providers got nothing. A tie is not a ranking, and this repo spent the year removing that shape from boards. A chip is the same claim in a smaller box. So a gated row cannot lead, and a shared best value elects nobody. The conservative answer is no chip rather than a chip three others have an equal right to. The trailer end applies only the first rule: a tie for worst harms nobody, but naming a gated row the worst reads as a verdict on a row whose whole point is that it was not compared. The rule is its own module because it is two sentences that are easy to get right and easy to lose in a sixty-line loop, and because it needed the exact production cases replayed rather than a neighbouring shape. Ten tests carrying the Robinhood and Ethereum readings as they were measured. Mutation-checked by restoring `sorted[0]`, which fails five of them. Four cache keys move: bench-unfiltered-v127, all-benchmarks-v120, bench-filters-v65 and providers-v34. The last one is the point: the index feeds the chip, so a cached one keeps claiming the rank for its whole 900 s TTL. Rebased onto the v33 another session landed while this was being written. Stacked on #3424, which fixes the same audit's other two findings. 260 specs valid, tsc clean, 838 tests pass. * a floor we reached was printed as a limit they set, and a handshake was inside the measurement The audit's last two findings. Both are measurements that needed rebuilding rather than numbers that needed clamping, which is why they were held back from the blocker pass. THE LADDER DID NOT SAY WHAT ITS NUMBER MEANT. `climb` keeps the last accepted rung when one of OUR calls fails above it, and publishes it the same way it publishes a provider's refusal. Live, from the audit: Quicknode reads 10,000 on seven chains after an HTTP error at the 100,000 rung, and GetBlock 1,000 on Arc after one at 2,000. Both render as the provider's guard. Either may be. Nothing in the figure says we never asked higher. The gauge now carries a `bound` label with three values, and they are not interchangeable: refused the provider declined the next rung. The figure is its guard. failed our own call failed above it. The figure is a floor we reached. top every rung passed. The guard is above the ladder. The two window columns select `bound=~"refused|top"`, so a floor from our own failure leaves the cell blank instead of claiming a limit, and the reading stays in Prometheus for whoever is debugging. The pass line says which case it was, including the floor, so the next person reads it from the log rather than rediscovering it from a board. A rate-limit reply also stops counting as a width refusal. It says we asked too often, not that the window is too wide, and reading it as a limit publishes our own cadence as their policy. Same rule that already refuses to classify a timeout. The match list is deliberately narrow: anything unrecognised stays a width refusal, because that is the common case and widening it would erase true measurements. THE WORK TERM HAD A HANDSHAKE INSIDE IT. I wrote that the two eth_call legs share one cold connection. They do not: coldTransport sets DisableKeepAlives, so each leg dialled its own and paid its own DNS, TCP and TLS. A handshake is about 25 ms from this container and the difference being measured is a few milliseconds, so the noise was larger than the signal and it crossed zero in both directions. The sign guard from #3420 removes only the half below zero; the audit found GetBlock at plus 0.568 ms on Ethereum, which would sort first on a lower-is-better view for exactly the reason a negative would. So both legs are primed now. `coverageProbe` gains a `warm` field, off for every capability question, where the socket cannot change the answer and a prime would double the spend for nothing, and on for the pair, where the handshake is the thing in the way. The measured call then rides a pooled connection and the handshake is paid outside it. Cost: one unmeasured call per leg per pass. On the hourly lane that is 1,440 requests a month per endpoint, against the smallest budget in the cohort of 300,000. Nine tests, carrying the production messages as they were read. Mutation checked three ways: removing the throttle rule fails the throttle test, dropping the failed bound fails the bound test, and the earlier sign-guard tests still hold. Cache keys move in lockstep, v128 / v121 / v66. 260 specs valid, tsc clean, 838 site tests, go build, go vet and go test clean. * the throttle needle "429" also matches a block number Caught re-reading my own fix from an hour ago, before the audit could. acceptedSpan never sees the HTTP status: probe.go discards a non-200 body, so only the JSON-RPC message reaches it. Matching "429" as a substring there does not find a throttle, it finds three digits. Both of these are width refusals, which is the measurement this ladder exists to take: eth_getLogs range of 4291 blocks exceeds the 2000-block limit query returned more than 10000 results from block 14290000 Either would have been read as our problem rather than their policy, made the rung non-definitive, and dropped a true reading. The failure is silent: a provider simply stops publishing a window and nothing says why. The spelled-out forms cover a real throttle without that risk, so the bare digits are gone. Two tests now carry messages whose width or block number contains 429, and a mutation that puts the needle back fails them. This is the same shape as a case-sensitive needle reading a page as empty earlier today: a matcher that catches more than it names. go build, go vet, go test clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
every probe added today joined the headline without being asked
The adversarial audit of bench 311 came back FIX THEN SHIP, and the two
blockers it found below the worker lag share one cause: every selector on this
bench aggregates
rpc_api_supportedandrpc_api_probe_totalwith noapifilter. So each probe added in the last few hours silently joined the
headline, the success rate, the sample size, the rank gate and the freshness
column. I added four probes and two ladder counters and did not look at what
already summed them.
WHAT IT PUBLISHED, measured by the audit against the live blob.
The two eth_call legs write the capability gauge, so each chain contributed
nine cells where the copy describes seven:
Every ranked headline sat 0.9 to 5.2 points high, the GetBlock to BlockMachine
gap read 19.1 where it is 14.9, and the sample said n=72 where the copy says
56. The order and the gate did not move, which is why nothing looked wrong.
The range ladder writes a probe counter and no capability gauge, so the
success denominator grew by two APIs while the numerator did not. Every ranked
row read 81.8182 percent, which is 72/88, for cells that all reached a
verdict: the figure this spec defines is 100. The tooltip then invented
failures from it, "13 failed of 72 measurements", none of which happened. On
the Robinhood tab Alchemy's unranked badge read 73 percent for what is 6 of 7.
THE FIX is an allowlist of the seven capability questions on all 67
selectors, written out rather than globbed:
eth_getlogs_(nofilter|address| topics)and the four others by name. A wildcard would let the next probe inthe same way, and the ladder's own counters are called
getlogs_range_*,which a careless
eth_getlogs_.*would not catch today and might tomorrow.Measurements keep their own views; they have no business in a share of cells
served.
THREE MORE FROM THE SAME AUDIT, each bounded:
The freshness column took
max by (provider)over every api, and the eth_callstamps are rewritten hourly while capability verdicts move every six hours. So
the column would have sat near one hour forever while a capability pass
stalled, which is the exact failure its own methodology bullet promises to
expose. It now reads the OLDEST capability stamp.
The Solana view pinned
chain="solana"and that pin survives dimensioninjection, so it rendered Solana values on every EVM tab: on the Ethereum tab
it showed chainstack 0, quicknode 0, getblock 0. The bench declares Solana out
of scope in its own methodology. The view is gone; the probe stays in the
harness for the Solana bench that owns the question.
The FAQ answered "Not yet ... planned and is deliberately absent" about the
window measurement, directly above the two columns that publish it. Rewritten
to describe the ladder.
AND TWO CORRECTIONS TO MY OWN COPY. The cadence bullet said every method is
asked every six hours; the two eth_call legs have run hourly since #3412. And
both the spec and my commit message for #3420 said the legs share one cold
connection. They do not: the cold transport sets DisableKeepAlives, so each
leg dials its own and pays its own handshake. That is not a detail, because
the handshake noise is what the difference carries, and
> 0only removes thehalf that falls below zero. The copy now says so, and the view is described as
a number rather than a ranking until the hourly lane gives it enough samples.
Cache keys move in lockstep, v126 / v119 / v64, rebased onto the v125 another
session landed while this was being written.
260 specs valid, tsc clean, 828 tests pass.