Repository navigation
recording rules for the cold-connection arm - #2899
Merged
Merged
Conversation
#2896 made the keyed probe measure both arms deliberately: rpc_latency_milliseconds is now explicitly the WARM one (pooled connection) and rpc_latency_cold_milliseconds is the same call on a connection that is never reused. Both are flowing from all three regions. Without these rules the cold series exists but no page can read it, because every spec queries the precomputed ocb: shapes rather than raw gauges. Four rules mirroring the warm ones, same group, same 60 s interval, and no tier filter — the keyed harness emits the same metric names with tier="keyed", so one rule set covers both cohorts and all three regions, exactly as the comment in prometheus.yml says of the existing ones. ## Context for whoever reads this later Every probe used to share one implicit http.DefaultTransport. Whether a call rode a pooled connection or paid a fresh TLS handshake depended on how often its HOSTNAME happened to be touched, which is not evenly distributed: a provider serving nine chains from one hostname stayed warm, a provider using a hostname per chain went cold between 60 s ticks and paid about 25 ms. That gap, not node speed, was most of the published keyed ranking. Three of four providers were reported 4 to 6x slower than they answer. Series before 2026-10-08 are a mixture of both arms and are not comparable with either one after it. ## Still to come The nine keyed specs do not read the cold series yet. The plan is a ranked WARM column plus a non-ranked cold-start figure, rather than two competing rankings: the cold number mixes provider quality with the distance from our probe regions to their TLS termination, so it is partly a fact about where Railway put our containers. Warm isolates what the provider controls and is the number a reader elsewhere can transpose. Validated with the repo's own js-yaml: group ocb_rpc_capabilities, 11 rules.
Flotapponnier
added a commit
that referenced
this pull request
Oct 8, 2026
The nine keyed benches now carry a third ACCESS value, "Private, cold start", alongside "Public, no key" and "Private, API key". Same endpoints, same call, measured on a connection that is never reused: DNS, TCP and the TLS handshake are paid every time. That is what a serverless function or a one-shot client experiences; the API key cohort is the pooled equivalent a long-running backend gets. 36 provider clones, four per spec, pointed at the ocb:rpc_latency_cold_* rules added in #2899. Verified against Prometheus before shipping: the generated p50 query for getblock on Ethereum returns 23.8 ms, so the column renders rather than blanking. ## Why a third button and not a nested selector The ask was a selector that appears inside the Private tab. The tier dimension partitions PROVIDERS, not PromQL labels, and the dimension machinery can inject a label but cannot swap a metric NAME. Warm and cold are the same providers against two different metrics, so a nested control needs new code in benchmark-body.tsx plus a new dimension kind. A third tier value needs none. It is also arguably the clearer surface: one row, three mutually exclusive cohorts, no hidden state. The nested version stays possible later if the three buttons read badly. Provider slugs are suffixed `-cold` because the panel label IS the provider slug and a duplicate would collide — the bug that blanked five ledger columns on bench 284. The `provider=` label inside each query still points at the real provider, so the series resolve. ## Caveats a reader should get, and the specs do not say yet The cold figure mixes provider quality with the distance from our probe regions to their TLS termination, so it is partly a fact about where Railway put our containers. Warm isolates what the provider controls. If only one number is quoted, it should be the warm one. Both arms have about an hour of history: the keyed warm series was reset today after a measurement bug, and the cold series did not exist before this morning. Percentiles will be noisy until tomorrow. pnpm validate: 245 specs.
Flotapponnier
added a commit
that referenced
this pull request
Oct 8, 2026
…#2903) Everything from the connection-reuse incident landed on main: the two-arm harness (#2896), the cold recording rules (#2899) and the cold-start cohort on the nine keyed specs (#2902). The materialise worker clones DEV at build time, so none of it can reach a page until the specs exist here. That is why ?tier=keyed-cold currently serves the public provider list: the worker's snapshot predates the new tier and falls through to the default cohort. Same content as main: 9 specs the keyed-cold cohort, 36 provider clones on the cold metric probe.go warm and cold transports, alternating, plus the gated conn trace metrics.go rpc_latency_cold_milliseconds and its histogram rules ocb:rpc_latency_cold_milliseconds:{p50,p90,p99,mean}_24h pnpm validate 246 specs, go build clean. After this merges the worker still needs a rebuild, and there is roughly a 72-minute display floor before the pages show it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
recording rules for the cold-connection arm
#2896 made the keyed probe measure both arms deliberately: rpc_latency_milliseconds
is now explicitly the WARM one (pooled connection) and rpc_latency_cold_milliseconds
is the same call on a connection that is never reused. Both are flowing from all
three regions. Without these rules the cold series exists but no page can read it,
because every spec queries the precomputed ocb: shapes rather than raw gauges.
Four rules mirroring the warm ones, same group, same 60 s interval, and no tier
filter — the keyed harness emits the same metric names with tier="keyed", so one
rule set covers both cohorts and all three regions, exactly as the comment in
prometheus.yml says of the existing ones.
Context for whoever reads this later
Every probe used to share one implicit http.DefaultTransport. Whether a call rode
a pooled connection or paid a fresh TLS handshake depended on how often its
HOSTNAME happened to be touched, which is not evenly distributed: a provider
serving nine chains from one hostname stayed warm, a provider using a hostname per
chain went cold between 60 s ticks and paid about 25 ms. That gap, not node speed,
was most of the published keyed ranking. Three of four providers were reported 4
to 6x slower than they answer.
Series before 2026-10-08 are a mixture of both arms and are not comparable with
either one after it.
Still to come
The nine keyed specs do not read the cold series yet. The plan is a ranked WARM
column plus a non-ranked cold-start figure, rather than two competing rankings:
the cold number mixes provider quality with the distance from our probe regions to
their TLS termination, so it is partly a fact about where Railway put our
containers. Warm isolates what the provider controls and is the number a reader
elsewhere can transpose.
Validated with the repo's own js-yaml: group ocb_rpc_capabilities, 11 rules.