Skip to content

start the Validation Cloud probe, and keep it off production - #2978

Closed
Flotapponnier wants to merge 2 commits into
mainfrom
fix/keyed-probe-validationcloud
Closed

Flotapponnier wants to merge 2 commits into
mainfrom
fix/keyed-probe-validationcloud

Conversation

@Flotapponnier

@Flotapponnier Flotapponnier commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Two commits, both needed on main, and together they do exactly one thing: start measuring, without anything appearing on production.

1. The probes ignore the endpoints already set on them

One file, seven lines: validationcloud joins the provider list in the keyed probe harness and gets a budget.

The three probe services build from main. Until this lands they run a binary with no validationcloud in its provider list, and a cell is probed only when both its env var and its provider exist. The eighteen endpoint variables have been sitting on rpc-keyed-us, rpc-keyed-eu and rpc-keyed-sgp doing nothing. Prometheus confirms it: five keyed providers reporting series, Validation Cloud absent.

2. A provider only dev declares was about to get a production product page

Found while checking the first commit was safe to merge.

The providers blob is built by the worker from the dev checkout and read by both deployments. confineToThisDeploy already dropped appearances on benches that do not exist here, which covers a dev-only bench. It did not cover the other half: a bench that exists on both branches while only dev's copy declares the provider. That is precisely the state we are in.

So the moment commit 1 starts the probe and the worker republishes, the blob would carry a validationcloud profile whose appearances all point at benches main does serve. The filter would pass it, and /products/validationcloud would have rendered on production crediting six benchmark tables that do not list it.

Bench pages were never at risk: spec.ts already prunes blob results to the providers the local YAML declares, which is why the six RPC pages on production stay clean. This commit applies the same rule to the product index, the one surface that reads the blob's provider list directly.

Four tests, run against the repo's real specs. They also fail the day someone adds the provider to main's YAML, which is the moment the page becomes correct.

What production looks like after this merges

Unchanged. No page, no spec, no copy is touched by either commit, and the new guard only removes something that should not have been there. The six specs that declare Validation Cloud are on dev, so the rows appear on staging only.

What starts happening

The probes rebuild and sample every 120 s from each region. The staging rows fill within minutes; the 24 hour median needs a day to be a real median.

Gates

pnpm typecheck, pnpm test 479 pass (4 new), pnpm lint clean on the touched files, go build ./... && go vet ./... in the harness module.

… them

One file, seven lines: validationcloud joins the provider list and gets a
budget. Nothing else from dev comes with it.

The probe services build from main, so until this lands they run a binary whose
provider list has no validationcloud in it, and a cell is only probed when both
its env var AND its provider exist. The eighteen endpoint variables have been
on rpc-keyed-us, rpc-keyed-eu and rpc-keyed-sgp since yesterday, doing nothing.

This changes nothing a reader sees on production: the six RPC specs that
declare the provider are on dev and render on staging only. What it changes is
that samples start flowing, so those staging rows stop being empty.

The budget is a runaway guard at 600k requests per region per month, not a
quota. Six chains at the 120 s cadence cost 130k. It wants replacing with the
real figure once the account's plan is disclosed, which is also what the row's
"plan not yet disclosed" tag is waiting on.
The providers blob is built by the worker from the dev checkout and read by
both deployments, so production can be handed a profile that only makes sense
on staging. confineToThisDeploy already dropped appearances on benches that do
not exist here, which covers a dev-only bench. It did not cover the other half:
a bench that exists on BOTH branches while only dev's copy declares a given
provider.

That is the normal state while a provider is trialled on staging, and it is the
state right now. The six RPC specs that declare Validation Cloud are on dev;
main's copies of those same benches do not declare it. As soon as the probe
starts and the worker republishes, the blob carries a validationcloud profile
whose appearances all point at benches main does serve, so the filter passed it
and /products/validationcloud would have rendered on production, crediting six
benchmark tables that do not list it.

The bench pages were never at risk: spec.ts prunes blob results to the
providers the local YAML declares. This is the same rule, applied to the
product index, which is the only surface that reads the blob's provider list
directly.

Four tests against the repo's real specs. They also fail the day someone adds
the provider to main's YAML, which is the moment the page becomes correct.
@Flotapponnier Flotapponnier changed the title the keyed probes ignore the Validation Cloud endpoints already set on them start the Validation Cloud probe, and keep it off production Oct 14, 2026
@Flotapponnier

Copy link
Copy Markdown
Collaborator Author

Closing: the premise no longer holds.

Validation Cloud was removed from the keyed cohort entirely on 2026-10-18
(#3034), because it had never produced a single measurement:

rpc_health        0 on all 27 (chain, region) series
rpc_call_total    result="http_err"  22018
                  result="timeout"       5
                  successes              0
latency series    none exist at all

The harness log gave the cause, identically on every chain and region:
http_err err=status 401, plus websocket: bad handshake on the WS arm. The
probe itself was healthy and running at full cadence, 19,437 calls in 24h, the
same rate as QuickNode. The key does not authenticate.

Starting the probe is not the open question any more; getting a key that works
is. The 54 Railway URL and WS variables are backed up outside the repo and can
be restored one command each when Validation Cloud supplies one.

Validation Cloud remains live and correct on rpc-cost (167) and
rpc-free-tier (1,219,510), which are not probes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant