Repository navigation
the Birdeye monitor retried a refusal forever, from three regions - #2799
Merged
Merged
Conversation
Deployed to one region and watched it. Every dial is answered 403 from Railway, and the monitor did what I wrote it to do: back off, retry, back off, retry, for as long as it runs. Three regions and two chains each, against an endpoint refusing all six, indefinitely. That is hammering a third party for no measurement, and it ran in production before I caught it. The 403 is not transient and not a header problem, which is what my own error message claimed. The same dial, same headers, same binary: laptop, residential 101 OVH VPS, datacenter 101 Railway EU West 403 So the edge is scoring the egress IP. Retrying cannot change that, and a monitor that cannot tell "come back later" from "not from here" will keep asking forever. A 403 now returns a sentinel, and three consecutive ones disable that chain with a message saying what is actually wrong. Ordinary disconnects still reconnect as before; a test pins that distinction, because a breaker that also swallowed a dropped socket would silently stop measuring a provider that was merely flaky. This does not make Birdeye work from Railway. It makes its failure quiet and honest instead of loud and useless. Whether to route Railway through a proxy, run this chain from the VPS, or drop Birdeye is a separate decision with its own costs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HTTP_PROXY and HTTPS_PROXY are set on all three Railway services, and getProxyDialer() reads them. geckoterminal_monitor.go and both mobula paths already dial that way; I hardcoded a bare websocket.Dialer, so Birdeye went direct from an egress IP Cloudflare refuses. That is the whole reason it returned 403 from Railway while the same binary got 101 from a laptop and from the OVH VPS. I had reported that no proxy was configured, which was wrong: my grep was anchored to the start of the line and missed both variables. The breaker in the previous commit stays. If the proxy is also refused, three 403s still disable the chain rather than retrying forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
force-pushed
the
fix/birdeye-forbidden-breaker
branch
from
October 4, 2026 12:59
551bf75 to
037a07b
Compare
Flotapponnier
added a commit
that referenced
this pull request
Oct 4, 2026
Eight PRs of bench 001 work went straight to main this week on "mets live
now", and the fast-forward back to dev that AGENTS.md prescribes after a
hotfix was never done. The two branches have since diverged, so a
fast-forward is no longer possible: dev carries its own merges of the same
work under different SHAs plus commits main does not have.
This ports the one bench, by file, rather than reconciling twenty commits.
Every line it removes from dev is a line main deliberately replaced:
Timestamp string dev still types the OKX millis field as a
string, which is the bug #2795 fixed: the
WebSocket sends a JSON number and 136 of 136
frames were rejected
plain websocket.Dialer replaced by the pinned Chrome ClientHello in
#2802, without which Cloudflare 403s every
dial from Railway
the 403 retry loop #2799 gave it a circuit breaker after it
hammered a third party from three regions
TestOKXDoesNotCoverBNB re-derived as a coupling in #2805
The spec comes with it, which is the operational reason this cannot wait:
the materialize worker clones dev at BUILD time, so until this lands a
worker rebuild would pick up a spec that still says OKX and Birdeye are
measured on Base and Solana only.
Verified on dev's base, not main's: go build ./... and
go test ./cmd/script/ -run 'Birdeye|OKX|Reference|Chain' pass, pnpm validate
reports 235 specs valid.
The wider divergence is untouched and still needs a decision.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caught by deploying to one region and watching it, not by a test.
Every dial from Railway is answered 403, and the monitor did exactly what I wrote it to do: back off, retry, back off, retry, indefinitely. Three regions and two chains each, against an endpoint refusing all six. That is hammering a third party for no measurement, and it ran in production before I caught it.
The 403 is not what my own error message said
It claimed a missing or stale User-Agent. Same dial, same headers, same binary:
So the edge is scoring the egress IP. Retrying cannot change that.
The fix
A 403 returns a sentinel; three consecutive ones disable that chain with a message saying what is actually wrong. Ordinary disconnects still reconnect as before, and a test pins that distinction, because a breaker that also swallowed a dropped socket would silently stop measuring a provider that was merely flaky.
What this does not do
It does not make Birdeye work from Railway. It makes the failure quiet and honest instead of loud and useless. The column will read "Feed down" rather than filling the logs.
Three options, each with its own cost, and none of them mine to pick:
getProxyDialeralready readsHTTP_PROXY; none is currently set)Checks
go build ./...andgo test ./cmd/script/clean, 7 Birdeye tests🤖 Generated with Claude Code