Repository navigation
the race scored trades from pools the bench does not measure - #2809
Merged
Merged
Conversation
Florent spotted it on the page: OKX first to report on 95.93% of Solana trades, while its own head lag p50 there is 209 ms against Mobula's 11 ms. Two series on the same panel saying opposite things. The race scored any trade two feeds both reported. That was the whole eligibility rule, and it was correct for as long as every feed was subscribed to the bench pool. OKX and Birdeye are subscribed per TOKEN, so since they shipped the book has been dominated by trades from other pools that only those two can see, each one scored as a race between them alone. Measured on Solana over one hour: races closed 77,776 trades on the bench pool ~1,200 firsts awarded to OKX 75,151 on 1,210 pool trades of its own Mobula: first on 85 of the 109 pool trades it reported, about 78%, published as 0.08% Mobula's share is forced toward zero by arithmetic, not by latency: the numerator is its wins, the denominator is every race in the window including the tens of thousands it could not enter. Base 84.90% and BNB 85.90% were the same thing, less visibly. The fix needed membership and timing to stop being the same question. Voiding races the reference never saw does not work on Base, where the reference IS the chain-wide flashblock stream and matches every transaction on the chain. So poolTrades is a second set, written only by the pool subscriptions in reference_monitor.go and never by the flashblock stream. Timing still comes from the reference, so Base keeps its flashblock zero point; the new set only decides what is eligible. Chains with no pool subscription are left unfiltered rather than voided. Robinhood has no endpoint we trust, and voiding everything there would read on the page as four dead feeds instead of an unmeasured chain. Safe to check at close time: the pool subscription lands within ~130 ms of the fastest feed on Solana (measured, ref lag p50 -0.106 s for Mobula, -0.130 s for Serialized), comfortably inside the 3 s raceWindow. race.go had no test file at all, which is how this shipped. The new one asserts both directions, and the two tests that matter fail with the guard removed while the two guarding against over-voiding still pass. The Solana head_lag column moves with this, since it is produced in closeLocked. Base's head_lag does not: it comes from resolveOne against the chain-wide reference, which is the separate scope question still open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Oct 4, 2026
Flotapponnier
added a commit
that referenced
this pull request
Oct 4, 2026
…ce (#2811) #2809 shipped and the Solana race population is still about 25,000 an hour against a pool that every pool-scoped feed reports at roughly 600. I cannot tell from the existing metrics which of three things is true, and I have already drawn one wrong conclusion from them tonight. The three candidates: - the pool subscription genuinely observes far more than the feeds report, so the population is legitimate and the feeds cover a few percent of it - poolTrades is populated beyond the pool, so the filter passes off-pool trades - the filter is not reached for most races Without a voided counter, "the guard is working" and "the guard never runs" produce the same observable from outside. So: head_lag_pool_observed_total{chain} what the pool subscription sees head_lag_pool_trades_entries live size of the membership set head_lag_races_voided_total{chain,reason} off_pool vs single_participant The wrong conclusion worth recording: I reported ~1,200 trades on the Solana bench pool, taken from head_lag_ref_matches_total{outcome="matched"}. That counter has a third outcome I had not looked at, ahead_of_reference, which is where every feed faster than the pool subscription lands on Solana. Counting both columns puts each pool-scoped feed at ~580 to 600 trades an hour, which also matches the historical race rate of 100 to 900 before the token-scoped feeds arrived. The direction of #2809 holds; that number did not. No behaviour change. Counters only. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 4, 2026
…ool figure (#2812) Two debts from #2809, now that #2811's counters can measure instead of infer. headLagFirst was incremented inside the winners loop, which runs BEFORE the providers < 2 return. A trade one feed alone reported therefore handed that feed a first while never entering the race count or the share history: a numerator moving without its denominator, the same defect that published OKX at 95.93% on Solana, and it survived the first fix. The lag recording stays inside the loop on purpose, since single-feed lags are deliberately kept and only the race is unscored. race_test.go gains the case. single_participant now also reaches the voided counter, which it did not before. And the number I published in #2809 is corrected. The bullet said roughly 1,200 trades on the Solana bench pool, read from head_lag_ref_matches_total{outcome="matched"} alone. That counter has a third outcome, ahead_of_reference, where every feed faster than the pool subscription lands on Solana. What the new counters say directly: pool observed solana 465/h bnb 2,903/h base 269/h races scored solana 254/h bnb 2,806/h base 316/h voided off_pool solana 576,530/h bnb 14,893/h base 11,707/h So the filter does bite, hard, and Solana is back inside the 100 to 900 races an hour it ran before the token-scoped feeds arrived. The 25,070/h I reported after deploying #2809 was a rate() window straddling the container restart, which is the artefact I had warned about in that same PR and then read anyway. The shares now agree with the latency column, which is the check that found the bug: base mobula 71.37% serialized 23.50% okx 5.13% bnb mobula 60.54% serialized 35.70% codex 10.36% okx 0.29% solana serialized 81.03% mobula 54.61% okx 0.00% Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 5, 2026
It said Solana scores 254 races an hour. Five consecutive 15-minute windows read 203, 25, 46, 37 and 186, so 254 was one sample of something that swings by a factor of eight. A race needs two feeds to report the same pool trade, which is a sparse coincidence, so the jitter is the quantity's nature rather than a fault. Replaced with the order of magnitude and the comparison that actually carries the claim: it is back to the rate it ran at before the token-scoped feeds arrived. Two numbers from the #2809 commit message and PR body are also wrong, and recorded here since they are not in published copy and cannot be edited there: "Mobula was first on 85 of the 109 pool trades it reported, about 78%" used the matched-only denominator, which omits ahead_of_reference. On the actual races the figure is 114 of 415, about 27.5%, which is what the page publishes and what the fix was always about: it was 0.08%. "~1,200 trades on the bench pool" came from the same mistake and was already corrected in #2812. The defect and the fix are unaffected: OKX took 5 firsts in 415 Solana races after the filter, against 75,151 an hour before it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spotted on the live page: OKX first to report on 95.93% of Solana trades, while its own head lag p50 there is 209 ms against Mobula's 11 ms. Two series on the same panel saying opposite things.
What the race was scoring
Any trade two feeds both reported. That was the whole eligibility rule, and it was correct for as long as every feed was subscribed to the bench pool. OKX and Birdeye are subscribed per token, so they see every pool on it while the other four see only ours. Since they shipped, the book has been dominated by trades from pools this bench does not measure, each scored as a race between those two alone.
Measured on Solana over one hour:
And the number that makes it concrete: Mobula was first on 85 of the 109 pool trades it reported, about 78%. The page published 0.08%.
Its share is forced toward zero by arithmetic, not latency. The numerator is its wins; the denominator is every race in the window, including the tens of thousands it could not enter:
Base (84.90%) and BNB (85.90%) were the same defect, less visibly.
Why the obvious fix does not work
Voiding races the reference never saw fails on Base, where the reference is the chain-wide flashblock stream and therefore matches every transaction on the chain. Membership and timing were being answered by one
refClock, and they are different questions.So
poolTradesis a second set, written only by the pool subscriptions inreference_monitor.goand never by the flashblock stream. Timing still comes fromreference, so Base keeps its flashblock zero point; the new set only decides what is eligible.Chains with no pool subscription are left unfiltered rather than voided. Robinhood has no endpoint we trust, and voiding everything there would read on the page as four dead feeds rather than an unmeasured chain.
Checking at close time is safe
The pool subscription lands within ~130 ms of the fastest feed on Solana (measured: ref lag p50 -0.106 s for Mobula, -0.130 s for Serialized, -0.040 s for Codex), comfortably inside the 3 s
raceWindow. A genuine pool trade is present by the time its race closes.race.go had no test file
Which is how this shipped. The new one asserts both directions, and I verified the discrimination rather than assuming it: with the guard removed,
TestRaceVoidsTradesOutsideTheBenchPoolandTestRaceMembershipDoesNotUseTheChainWideReferencefail, while the two tests guarding against over-voiding still pass.Expected effect on published numbers
Solana drops from 77,776 races an hour to roughly 1,200, OKX's first share falls to whatever it earns on the bench pool, and Mobula's rises toward the ~78% it was already achieving. The Solana
head_lagcolumn moves with it, since that figure is produced incloseLocked.Base's
head_lagdoes not move: it comes fromresolveOneagainst the chain-wide reference, which is the separate scope question still open.Gates
go build ./..., the fullgo test ./cmd/script/suite, andpnpm validate(235 specs valid) all pass. Methodology now states the eligibility rule and says plainly that figures published for OKX and Birdeye on the Solana column before today were measured on a wider set of trades than the other four feeds.🤖 Generated with Claude Code