Skip to content

five RPC pages were measuring endpoints that had stopped answering (cherry-pick of #2935) - #2937

Closed
Flotapponnier wants to merge 1 commit into
mainfrom
fix/rpc-dead-endpoints-main
Closed

Flotapponnier wants to merge 1 commit into
mainfrom
fix/rpc-dead-endpoints-main

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Cherry-pick of #2935, already merged to dev.

This has to reach main to have any effect: the Railway rpc-capabilities
services deploy from main, so a dev-only merge changes which providers the
page declares without changing which endpoints are probed. Verified by
measurement, not assumption: after merging #2935 to dev and redeploying all
three regions with --from-source, Prometheus still showed only the old provider
set, and main carries the ETC repair from this morning (which did take
effect) while it does not carry this one.

Without this merge the five pages declare providers nobody probes and publish
empty rows.

Gates on this branch: go build, go vet, pnpm validate (245 specs), and an
endpoint-match check confirming every URL the specs publish is the URL the
harness probes.

A sweep of the 64 thin and unread RPC pages probed all 159 configured
endpoints with the call each chain actually speaks, and 14 pages had at
least one dead endpoint. The dominant cause is not random rot: PublicNode
stopped serving five of our chains. The identical request still succeeds
on ethereum-rpc.publicnode.com and 404s on moonbeam from the same edge
with DNS resolving, so this is a withdrawal, not an outage.

Repaired here, each replacement probed with eth_getBlockByNumber or the
chain's own method, kept only after returning the same head as another
source on 10 of 10 tries:

  moonbeam    publicnode and tenderly 404, and the Foundation endpoint
              refuses the TLS handshake from two independent stacks
              (LibreSSL 3.3.6 and OpenSSL 3.5.5), so it cannot be
              measured from our probes whatever the cause. Left drpc
              alone, added onfinality and unitedbloc.
  moonriver   same PublicNode withdrawal, same Foundation refusal. Added
              drpc and unitedbloc, both of which the spec had excluded in
              August for a dead route and dead DNS and which answer again.
  astar       1RPC answers about 15% of the time, the same ok_rate that
              already got it dropped from Oasis Sapphire. Added the Astar
              Foundation endpoint and drpc.
  cosmos-hub  Lava returns 410 Gone, consistent with its removal in #2926.
              Replaced with ecostake.
  gravity     Ankr requires a key on this chain now and answers 403, so it
              can never produce a sample. Replaced with thirdweb.

Two things found in passing and fixed:

  Moonbeam's intro claimed four endpoints and carried a clause with no
  subject. Moonriver's intro still listed its August cohort and named two
  providers as excluded that work today. Both rewritten against what is
  actually measured.

  gravity-official was published with no endpoint field at all, on a page
  whose own description promises URLs for every no-key endpoint. Added
  rpc.gravity.xyz, the host the harness has been probing all along.

Gravity keeps its "publicnode" provider label even though that row is the
Foundation's own host. Renaming it would start a new series and blank the
column for a full day, and the page already displays it correctly.

Not recovered, recorded here so the next sweep does not re-probe them:
dymension, nibiru, conflux, merlin, swellchain, regen and nolus have no
third keyless endpoint that answers, so they stay under the three-provider
gate.
@Flotapponnier

Copy link
Copy Markdown
Collaborator Author

Closing: this already shipped. It was a cherry-pick to main, and the dev
twin (#2935) reached main through the release merge #3021 on 2026-10-18.

Checked rather than assumed: all 151 substantial lines this PR adds are
already present in main's copy of the files it touches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant