Skip to content

the WebSocket arm goes from 138 million messages a month to about six - #3018

Merged
Flotapponnier merged 1 commit into
devfrom
fix/ws-message-budget
Oct 17, 2026
Merged

Flotapponnier merged 1 commit into
devfrom
fix/ws-message-budget

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

the WebSocket arm goes from 138 million messages a month to about six

QuickNode wrote to say the monthly limit was reached. Measured before
touching anything, over 24h: QuickNode took 4,598,911 pushes, 138M a month,
against 583,140 HTTP probe calls. The arm is 237 times the HTTP probe by
event count.

The guard we had was looking the other way. rpc_keyed_quota_used_ratio
reads 1.17% for QuickNode because it counts HTTP calls and knows nothing
about sockets, which is why nothing warned before the vendor did.

And this is not one vendor's problem. Chainstack 124M, Validation Cloud
124M, Dwellir 112M, GetBlock 99M a month. QuickNode is simply the first to
have written.

Two cuts, measured rather than guessed.

The three fastest chains leave the arm. Robinhood alone is 40% of all
pushes, Solana 19%, Arbitrum 14%, against Ethereum's 0.4%: a chain producing
a block every 100 ms pushes 120 times what a 12 s chain does, to answer the
same question. That is 73% of the volume for a third of the pages. Their
three specs go with them; none was ever on main, so no redirect is owed.

What remains listens ten minutes in every hour. The window is derived from
the wall clock, and that detail is the whole correctness of it: if two
providers computed different boundaries they would listen to different
blocks, each would be its own leader, every lead would read zero and the
board would report that every provider is fastest. A silent failure worse
than an outage, so wsWindowAt is pinned by a test that asks several cells
at skewed instants and requires them to open together.

A close on the window deadline is not a disconnect. rpc_ws_disconnect_total
is a published result about a provider, and our own scheduler must not write
to it; the read goroutine racing the deadline is covered too.

The skip list is compiled in rather than read from the environment, because
it exists only because a bill was reached, and a lost variable must not be
able to restore one in silence. RPC_KEYED_WS_SKIP_CHAINS overrides it,
"none" empties it, and a duty window as long as its period restores the
continuous behaviour without a code change.

Expected effect per provider: 138M -> about 6M a month, a 95% cut, with six
of the nine benches intact.

Gates: gofmt clean on the file touched, go build, go vet, 4 new Go tests,
253 specs valid, 561 site tests.

QuickNode wrote to say the monthly limit was reached. Measured before
touching anything, over 24h: QuickNode took 4,598,911 pushes, 138M a month,
against 583,140 HTTP probe calls. The arm is 237 times the HTTP probe by
event count.

The guard we had was looking the other way. `rpc_keyed_quota_used_ratio`
reads 1.17% for QuickNode because it counts HTTP calls and knows nothing
about sockets, which is why nothing warned before the vendor did.

And this is not one vendor's problem. Chainstack 124M, Validation Cloud
124M, Dwellir 112M, GetBlock 99M a month. QuickNode is simply the first to
have written.

Two cuts, measured rather than guessed.

The three fastest chains leave the arm. Robinhood alone is 40% of all
pushes, Solana 19%, Arbitrum 14%, against Ethereum's 0.4%: a chain producing
a block every 100 ms pushes 120 times what a 12 s chain does, to answer the
same question. That is 73% of the volume for a third of the pages. Their
three specs go with them; none was ever on main, so no redirect is owed.

What remains listens ten minutes in every hour. The window is derived from
the wall clock, and that detail is the whole correctness of it: if two
providers computed different boundaries they would listen to different
blocks, each would be its own leader, every lead would read zero and the
board would report that every provider is fastest. A silent failure worse
than an outage, so `wsWindowAt` is pinned by a test that asks several cells
at skewed instants and requires them to open together.

A close on the window deadline is not a disconnect. `rpc_ws_disconnect_total`
is a published result about a provider, and our own scheduler must not write
to it; the read goroutine racing the deadline is covered too.

The skip list is compiled in rather than read from the environment, because
it exists only because a bill was reached, and a lost variable must not be
able to restore one in silence. RPC_KEYED_WS_SKIP_CHAINS overrides it,
"none" empties it, and a duty window as long as its period restores the
continuous behaviour without a code change.

Expected effect per provider: 138M -> about 6M a month, a 95% cut, with six
of the nine benches intact.

Gates: gofmt clean on the file touched, go build, go vet, 4 new Go tests,
253 specs valid, 561 site tests.
@Flotapponnier
Flotapponnier merged commit 461bb13 into dev Oct 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant