Repository navigation
hotfix(head-lag): a full race book no longer eats pool trades; Codex edge errors end the subscription - #3257
Merged
Merged
Conversation
…ge error ends the subscription
Off-pool voids were kept for raceRetention (10 min). On Solana OKX and
Birdeye produce ~4,800 of them a minute per region, ~52,000 held against
raceMaxOpen = 50,000: the book sat full, observe() dropped every new hash
including bench-pool trades, and races were scored only in bursts as old
voids expired. On 2026-10-23 ~70% of pool trades never raced, and a feed
whose arrival was dropped could lose a race to a slower one admitted after
a slot freed. Off-pool voids now leave after 30 s, and a drop on a full
book is counted (head_lag_races_voided_total{reason="book_full"}).
Codex edge nodes also end a subscription with a single "next" carrying
only errors (EDGE_ORIGIN_UNAVAILABLE, seen on iad.edge) and keep the
socket open. That now forces a reconnect instead of waiting 10 minutes
for the flow watchdog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #3256 to main: the agg services on Railway build from main.
Off-pool voids were kept for raceRetention (10 min). On Solana OKX and
Birdeye produce ~4,800 of them a minute per region, ~52,000 held against
raceMaxOpen = 50,000: the book sat full, observe() dropped every new hash
including bench-pool trades, and races were scored only in bursts as old
voids expired. On 2026-10-23 ~70% of pool trades never raced, and a feed
whose arrival was dropped could lose a race to a slower one admitted after
a slot freed. Off-pool voids now leave after 30 s, and a drop on a full
book is counted (head_lag_races_voided_total{reason="book_full"}).
Codex edge nodes also end a subscription with a single "next" carrying
only errors (EDGE_ORIGIN_UNAVAILABLE, seen on iad.edge) and keep the
socket open. That now forces a reconnect instead of waiting 10 minutes
for the flow watchdog.