Skip to content

the race scored trades from pools the bench does not measure - #2809

Merged
Flotapponnier merged 1 commit into
mainfrom
fix/race-bench-pool-scope
Oct 4, 2026
Merged

Flotapponnier merged 1 commit into
mainfrom
fix/race-bench-pool-scope

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Spotted on the live page: OKX first to report on 95.93% of Solana trades, while its own head lag p50 there is 209 ms against Mobula's 11 ms. Two series on the same panel saying opposite things.

What the race was scoring

Any trade two feeds both reported. That was the whole eligibility rule, and it was correct for as long as every feed was subscribed to the bench pool. OKX and Birdeye are subscribed per token, so they see every pool on it while the other four see only ours. Since they shipped, the book has been dominated by trades from pools this bench does not measure, each scored as a race between those two alone.

Measured on Solana over one hour:

races closed 77,776
trades on the bench pool ~1,200
firsts awarded to OKX 75,151 (on 1,210 pool trades of its own)

And the number that makes it concrete: Mobula was first on 85 of the 109 pool trades it reported, about 78%. The page published 0.08%.

Its share is forced toward zero by arithmetic, not latency. The numerator is its wins; the denominator is every race in the window, including the tens of thousands it could not enter:

headLagFirstShare.Set(float64(wins[a]) / float64(len(hist)) * 100)

Base (84.90%) and BNB (85.90%) were the same defect, less visibly.

Why the obvious fix does not work

Voiding races the reference never saw fails on Base, where the reference is the chain-wide flashblock stream and therefore matches every transaction on the chain. Membership and timing were being answered by one refClock, and they are different questions.

So poolTrades is a second set, written only by the pool subscriptions in reference_monitor.go and never by the flashblock stream. Timing still comes from reference, so Base keeps its flashblock zero point; the new set only decides what is eligible.

Chains with no pool subscription are left unfiltered rather than voided. Robinhood has no endpoint we trust, and voiding everything there would read on the page as four dead feeds rather than an unmeasured chain.

Checking at close time is safe

The pool subscription lands within ~130 ms of the fastest feed on Solana (measured: ref lag p50 -0.106 s for Mobula, -0.130 s for Serialized, -0.040 s for Codex), comfortably inside the 3 s raceWindow. A genuine pool trade is present by the time its race closes.

race.go had no test file

Which is how this shipped. The new one asserts both directions, and I verified the discrimination rather than assuming it: with the guard removed, TestRaceVoidsTradesOutsideTheBenchPool and TestRaceMembershipDoesNotUseTheChainWideReference fail, while the two tests guarding against over-voiding still pass.

Expected effect on published numbers

Solana drops from 77,776 races an hour to roughly 1,200, OKX's first share falls to whatever it earns on the bench pool, and Mobula's rises toward the ~78% it was already achieving. The Solana head_lag column moves with it, since that figure is produced in closeLocked.

Base's head_lag does not move: it comes from resolveOne against the chain-wide reference, which is the separate scope question still open.

Gates

go build ./..., the full go test ./cmd/script/ suite, and pnpm validate (235 specs valid) all pass. Methodology now states the eligibility rule and says plainly that figures published for OKX and Birdeye on the Solana column before today were measured on a wider set of trades than the other four feeds.

🤖 Generated with Claude Code

Florent spotted it on the page: OKX first to report on 95.93% of Solana
trades, while its own head lag p50 there is 209 ms against Mobula's 11 ms.
Two series on the same panel saying opposite things.

The race scored any trade two feeds both reported. That was the whole
eligibility rule, and it was correct for as long as every feed was
subscribed to the bench pool. OKX and Birdeye are subscribed per TOKEN, so
since they shipped the book has been dominated by trades from other pools
that only those two can see, each one scored as a race between them alone.

Measured on Solana over one hour:

  races closed                    77,776
  trades on the bench pool        ~1,200
  firsts awarded to OKX           75,151   on 1,210 pool trades of its own
  Mobula: first on 85 of the 109 pool trades it reported, about 78%,
  published as 0.08%

Mobula's share is forced toward zero by arithmetic, not by latency: the
numerator is its wins, the denominator is every race in the window
including the tens of thousands it could not enter.

Base 84.90% and BNB 85.90% were the same thing, less visibly.

The fix needed membership and timing to stop being the same question.
Voiding races the reference never saw does not work on Base, where the
reference IS the chain-wide flashblock stream and matches every transaction
on the chain. So poolTrades is a second set, written only by the pool
subscriptions in reference_monitor.go and never by the flashblock stream.
Timing still comes from the reference, so Base keeps its flashblock zero
point; the new set only decides what is eligible.

Chains with no pool subscription are left unfiltered rather than voided.
Robinhood has no endpoint we trust, and voiding everything there would read
on the page as four dead feeds instead of an unmeasured chain.

Safe to check at close time: the pool subscription lands within ~130 ms of
the fastest feed on Solana (measured, ref lag p50 -0.106 s for Mobula,
-0.130 s for Serialized), comfortably inside the 3 s raceWindow.

race.go had no test file at all, which is how this shipped. The new one
asserts both directions, and the two tests that matter fail with the guard
removed while the two guarding against over-voiding still pass.

The Solana head_lag column moves with this, since it is produced in
closeLocked. Base's head_lag does not: it comes from resolveOne against the
chain-wide reference, which is the separate scope question still open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Flotapponnier
Flotapponnier merged commit 8f23353 into main Oct 4, 2026
2 checks passed
Flotapponnier added a commit that referenced this pull request Oct 4, 2026
…ce (#2811)

#2809 shipped and the Solana race population is still about 25,000 an hour
against a pool that every pool-scoped feed reports at roughly 600. I cannot
tell from the existing metrics which of three things is true, and I have
already drawn one wrong conclusion from them tonight.

The three candidates:

  - the pool subscription genuinely observes far more than the feeds report,
    so the population is legitimate and the feeds cover a few percent of it
  - poolTrades is populated beyond the pool, so the filter passes off-pool
    trades
  - the filter is not reached for most races

Without a voided counter, "the guard is working" and "the guard never runs"
produce the same observable from outside. So:

  head_lag_pool_observed_total{chain}        what the pool subscription sees
  head_lag_pool_trades_entries               live size of the membership set
  head_lag_races_voided_total{chain,reason}  off_pool vs single_participant

The wrong conclusion worth recording: I reported ~1,200 trades on the Solana
bench pool, taken from head_lag_ref_matches_total{outcome="matched"}. That
counter has a third outcome I had not looked at, ahead_of_reference, which
is where every feed faster than the pool subscription lands on Solana.
Counting both columns puts each pool-scoped feed at ~580 to 600 trades an
hour, which also matches the historical race rate of 100 to 900 before the
token-scoped feeds arrived. The direction of #2809 holds; that number did
not.

No behaviour change. Counters only.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier added a commit that referenced this pull request Oct 4, 2026
…ool figure (#2812)

Two debts from #2809, now that #2811's counters can measure instead of infer.

headLagFirst was incremented inside the winners loop, which runs BEFORE the
providers < 2 return. A trade one feed alone reported therefore handed that
feed a first while never entering the race count or the share history: a
numerator moving without its denominator, the same defect that published OKX
at 95.93% on Solana, and it survived the first fix. The lag recording stays
inside the loop on purpose, since single-feed lags are deliberately kept and
only the race is unscored.

race_test.go gains the case. single_participant now also reaches the voided
counter, which it did not before.

And the number I published in #2809 is corrected. The bullet said roughly
1,200 trades on the Solana bench pool, read from
head_lag_ref_matches_total{outcome="matched"} alone. That counter has a third
outcome, ahead_of_reference, where every feed faster than the pool
subscription lands on Solana. What the new counters say directly:

  pool observed       solana 465/h    bnb 2,903/h   base 269/h
  races scored        solana 254/h    bnb 2,806/h   base 316/h
  voided off_pool     solana 576,530/h  bnb 14,893/h  base 11,707/h

So the filter does bite, hard, and Solana is back inside the 100 to 900
races an hour it ran before the token-scoped feeds arrived. The 25,070/h I
reported after deploying #2809 was a rate() window straddling the container
restart, which is the artefact I had warned about in that same PR and then
read anyway.

The shares now agree with the latency column, which is the check that found
the bug:

  base    mobula 71.37%  serialized 23.50%  okx 5.13%
  bnb     mobula 60.54%  serialized 35.70%  codex 10.36%  okx 0.29%
  solana  serialized 81.03%  mobula 54.61%  okx 0.00%

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier added a commit that referenced this pull request Oct 5, 2026
It said Solana scores 254 races an hour. Five consecutive 15-minute windows
read 203, 25, 46, 37 and 186, so 254 was one sample of something that swings
by a factor of eight. A race needs two feeds to report the same pool trade,
which is a sparse coincidence, so the jitter is the quantity's nature rather
than a fault.

Replaced with the order of magnitude and the comparison that actually
carries the claim: it is back to the rate it ran at before the token-scoped
feeds arrived.

Two numbers from the #2809 commit message and PR body are also wrong, and
recorded here since they are not in published copy and cannot be edited
there:

  "Mobula was first on 85 of the 109 pool trades it reported, about 78%"
  used the matched-only denominator, which omits ahead_of_reference. On the
  actual races the figure is 114 of 415, about 27.5%, which is what the page
  publishes and what the fix was always about: it was 0.08%.

  "~1,200 trades on the bench pool" came from the same mistake and was
  already corrected in #2812.

The defect and the fix are unaffected: OKX took 5 firsts in 415 Solana races
after the filter, against 75,151 an hour before it.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant