Skip to content

the OKX monitor acked its subscribe and then decoded nothing - #2795

Merged
Flotapponnier merged 1 commit into
mainfrom
fix/okx-frame-decoding
Oct 4, 2026
Merged

Flotapponnier merged 1 commit into
mainfrom
fix/okx-frame-decoding

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Fixes #2791 in production. Found by deploying to one region and watching it:
the socket connected, both chains acked, and 0 series appeared. The log line
written for exactly this case, subscribe acked, waiting for frames, never
progressed to pushing trades.

Two decoding bugs, both silent

The read loop skips any frame it cannot parse, so a decoder that rejects
everything looks identical to a feed that sends nothing.

1. data is a single object, not the array every other OKX channel uses.
[]okxTrade rejected every frame.

2. timestamp is a JSON number on the WebSocket, while the REST endpoint of
the same product sends it quoted. A string field failed the entire object on
that one field, with the hash parsed correctly immediately above it:

json: cannot unmarshal number into Go struct field trade.timestamp of type string
hash="0xf5469898b251f5b313f84f9b9a123426414c31339ede1f34a1768c3d1fd3ed91"

raw=137  dataFrames=136  arrayOK=0  objectOK=0  rejected=136

Why the original verification missed both

The Node probe that validated the feed had written
Array.isArray(m.data) ? m.data : [m.data], and read the timestamp through
JavaScript's loose typing. It absorbed both differences without a word.

A probe more tolerant than the real client proves nothing about the real
client.
That is the lesson worth keeping from this PR.

The fix

okxFrameTrades takes either shape. okxMillis takes the timestamp quoted or
bare and never fails the decode: losing a whole trade over a field that neither
measured chain publishes would cost far more than losing the field.

Verification

Against the live gateway, with the monitor's own decoder rather than a copy:

decoded 180 trades, 180 with a hash

That check ships as TestOKXLiveFeed, opt-in so CI never depends on a third
party, and it fails with "an ack is not evidence, only decoded trades are".

Three unit tests pin the wire shapes: object and array, timestamp bare and
quoted, and the unusable cases that must leave the trade intact rather than
drop it.

Deploy note

Only EU West was deployed, deliberately, before this was caught. The other two
agg services still run the 29 September build and were never exposed. After this
merges, all three need railway redeploy --service <name> --from-source --yes.

Checks

  • go build ./... and go test ./cmd/script/ clean, 9 OKX tests
  • live decoder check passes against the gateway

🤖 Generated with Claude Code

Deployed to one region and watched it: the socket connected, both chains
acked, and 0 series appeared. The log line I had written for exactly this
case, "subscribe acked, waiting for frames", never progressed to
"pushing trades".

Two decoding bugs, both silent because the read loop skips a frame it
cannot parse.

First, `data` is a single OBJECT on this channel, not the array every
other OKX channel uses, so `[]okxTrade` rejected every frame.

Second, and the one that survived the first fix: `timestamp` arrives as a
JSON NUMBER on the WebSocket while the REST endpoint of the same product
sends it quoted. A `string` field failed the whole object on that one
field, with the hash already parsed correctly immediately above it.
Measured against the live gateway: 136 of 136 frames rejected.

Neither surfaced earlier because the Node probe that validated the feed
wrote `Array.isArray(m.data) ? m.data : [m.data]` and read the timestamp
through JavaScript's loose typing, so it absorbed both differences
without a word. A probe tolerant where the real client is strict proves
nothing about the real client.

okxFrameTrades accepts either shape; okxMillis accepts the timestamp
quoted or bare and never fails the decode, because losing a whole trade
over a field neither measured chain publishes would cost more than
losing the field.

Verified with the monitor's own decoder against the gateway rather than
a copy of it: 180 trades decoded, 180 carrying a hash. That check ships
as TestOKXLiveFeed, opt-in so CI never depends on a third party, and it
fails with "an ack is not evidence, only decoded trades are".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Flotapponnier
Flotapponnier merged commit e478cf1 into main Oct 4, 2026
2 checks passed
@Flotapponnier
Flotapponnier deleted the fix/okx-frame-decoding branch October 4, 2026 10:07
Flotapponnier added a commit that referenced this pull request Oct 4, 2026
Eight PRs of bench 001 work went straight to main this week on "mets live
now", and the fast-forward back to dev that AGENTS.md prescribes after a
hotfix was never done. The two branches have since diverged, so a
fast-forward is no longer possible: dev carries its own merges of the same
work under different SHAs plus commits main does not have.

This ports the one bench, by file, rather than reconciling twenty commits.
Every line it removes from dev is a line main deliberately replaced:

  Timestamp string           dev still types the OKX millis field as a
                             string, which is the bug #2795 fixed: the
                             WebSocket sends a JSON number and 136 of 136
                             frames were rejected
  plain websocket.Dialer     replaced by the pinned Chrome ClientHello in
                             #2802, without which Cloudflare 403s every
                             dial from Railway
  the 403 retry loop         #2799 gave it a circuit breaker after it
                             hammered a third party from three regions
  TestOKXDoesNotCoverBNB     re-derived as a coupling in #2805

The spec comes with it, which is the operational reason this cannot wait:
the materialize worker clones dev at BUILD time, so until this lands a
worker rebuild would pick up a spec that still says OKX and Birdeye are
measured on Base and Solana only.

Verified on dev's base, not main's: go build ./... and
go test ./cmd/script/ -run 'Birdeye|OKX|Reference|Chain' pass, pnpm validate
reports 235 specs valid.

The wider divergence is untouched and still needs a decision.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant