Repository navigation
head-lag: a stalled Serialized socket was scored as 100 s of provider lag - #2986
Merged
Merged
Conversation
… lag (#2985) On 2026-10-14 the us-east probe read Serialized at 60-118 s while eu-west and sgp read under 1 s and the other feeds in the same process read 1-3 s. Most spikes fall in windows where every socket on that box was dropping (reference node, Mobula, OKX, GeckoTerminal). The others redial within seconds; the Serialized client waited out a 90 s read deadline, so a frozen path that resumed delivered its backlog at once. - ping every 10 s and read with a 20 s deadline (two missed pongs); verified against ws.serialized.xyz, pongs return in ~100 ms - reset the reconnect backoff after a session that streamed for over 2 min; every drop after the fifth was waiting the 60 s ceiling - log each trade over 3 s with hash, Serialized's `at` and our receive time, so the next spike can be checked tx by tx against their logs - drop and log a trade delivered twice, and do not score a re-delivery after its race closed as a late arrival (cherry picked from commit 19294fb)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On 2026-10-14 the us-east probe read Serialized at 60-118 s, while eu-west and sgp read under 1 s and the other feeds in the same process read 1-3 s. Most of those spikes fall in windows where every socket on that box was dropping: the reference node, Mobula, OKX and GeckoTerminal. The other feeds redial within seconds. The Serialized client waited out a 90 s read deadline, so when a frozen path resumed, its whole backlog arrived at once.
Changes
ws.serialized.xyzon an idle socket for 70 s: pongs return in about 100 ms.atand our receive time. The next spike can then be checked tx by tx against their own send logs.Not in this PR
The
head_lag_secondsgauge still holds the last value until a newer trade replaces it. That is shared by every feed, so it gets its own change.