Repository navigation
instrument the pool filter, because every read of it has been inference - #2811
Merged
Merged
Conversation
#2809 shipped and the Solana race population is still about 25,000 an hour against a pool that every pool-scoped feed reports at roughly 600. I cannot tell from the existing metrics which of three things is true, and I have already drawn one wrong conclusion from them tonight. The three candidates: - the pool subscription genuinely observes far more than the feeds report, so the population is legitimate and the feeds cover a few percent of it - poolTrades is populated beyond the pool, so the filter passes off-pool trades - the filter is not reached for most races Without a voided counter, "the guard is working" and "the guard never runs" produce the same observable from outside. So: head_lag_pool_observed_total{chain} what the pool subscription sees head_lag_pool_trades_entries live size of the membership set head_lag_races_voided_total{chain,reason} off_pool vs single_participant The wrong conclusion worth recording: I reported ~1,200 trades on the Solana bench pool, taken from head_lag_ref_matches_total{outcome="matched"}. That counter has a third outcome I had not looked at, ahead_of_reference, which is where every feed faster than the pool subscription lands on Solana. Counting both columns puts each pool-scoped feed at ~580 to 600 trades an hour, which also matches the historical race rate of 100 to 900 before the token-scoped feeds arrived. The direction of #2809 holds; that number did not. No behaviour change. Counters only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 4, 2026
…ool figure (#2812) Two debts from #2809, now that #2811's counters can measure instead of infer. headLagFirst was incremented inside the winners loop, which runs BEFORE the providers < 2 return. A trade one feed alone reported therefore handed that feed a first while never entering the race count or the share history: a numerator moving without its denominator, the same defect that published OKX at 95.93% on Solana, and it survived the first fix. The lag recording stays inside the loop on purpose, since single-feed lags are deliberately kept and only the race is unscored. race_test.go gains the case. single_participant now also reaches the voided counter, which it did not before. And the number I published in #2809 is corrected. The bullet said roughly 1,200 trades on the Solana bench pool, read from head_lag_ref_matches_total{outcome="matched"} alone. That counter has a third outcome, ahead_of_reference, where every feed faster than the pool subscription lands on Solana. What the new counters say directly: pool observed solana 465/h bnb 2,903/h base 269/h races scored solana 254/h bnb 2,806/h base 316/h voided off_pool solana 576,530/h bnb 14,893/h base 11,707/h So the filter does bite, hard, and Solana is back inside the 100 to 900 races an hour it ran before the token-scoped feeds arrived. The 25,070/h I reported after deploying #2809 was a rate() window straddling the container restart, which is the artefact I had warned about in that same PR and then read anyway. The shares now agree with the latency column, which is the check that found the bug: base mobula 71.37% serialized 23.50% okx 5.13% bnb mobula 60.54% serialized 35.70% codex 10.36% okx 0.29% solana serialized 81.03% mobula 54.61% okx 0.00% Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 4, 2026
Carries #2811 and #2812 across: the three diagnostic counters, the first awarded past the single-participant return, and the corrected pool figure in the methodology. The worker clones dev at BUILD time, so the bullet only reaches the page from here. Verified on dev's base: go build ./..., the full go test ./cmd/script/ suite, and pnpm validate (235 specs valid). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 18, 2026
I wrote the Help as "Trades observed by the bench pool subscription" in #2811 and then read the counter back as a trade rate while checking the new alert rules. On EVM it is neither. eth_subscribe on a pool address delivers every event the pool emits (Swap, Mint, Burn, Collect), so one swap can arrive as several. Measured on the BNB pool: 49,672 events an hour against roughly 18,380 swaps, a factor of 2.7. Solana logsSubscribe is per transaction, so there the two coincide, which is why the mislabel survived: every number I had checked it against was Solana. Help string only, no behaviour change. The counter is still exactly right for what the race filter uses it for, which is telling a live pool subscription from a dead one, and for the two alert rules added today that test it against zero. It is wrong only as a trade count, which is how I was about to read it. Rides the next harness deploy; not worth one of its own. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#2809 is deployed (confirmed: Railway EU is running
8f233531) and the Solana race population is still ~25,000/h against a pool every pool-scoped feed reports at ~600/h. Three things could be true and the existing metrics cannot separate them:poolTradesis populated beyond the pool, so the filter passes off-pool tradesWithout a voided counter, "the guard is working" and "the guard never runs" are the same observable from outside. This adds:
head_lag_pool_observed_total{chain}head_lag_pool_trades_entrieshead_lag_races_voided_total{chain,reason}off_poolvssingle_participantA number I got wrong in #2809
I reported ~1,200 trades on the Solana bench pool, from
head_lag_ref_matches_total{outcome="matched"}. That counter has a third outcome I had not looked at,ahead_of_reference, which is where every feed faster than the pool subscription lands on Solana. Counting both columns puts each pool-scoped feed at ~580-600 trades/h:That also matches the historical race rate (100-900/h) before the token-scoped feeds arrived, measured with
offset. The direction of #2809 holds, and the 0.08% vs 78% defect in the share denominator is unaffected. The pool-trade figure quoted in its commit message and in the methodology bullet does not, and I will correct the published bullet once these counters say what the real population is.No behaviour change. Counters only.
🤖 Generated with Claude Code