Repository navigation
keyed latency: measure warm and cold connections deliberately, and publish both - #2896
Merged
Merged
Conversation
…blish both The implicit transport was deciding the ranking. Until now every probe used http.DefaultTransport, shared with the whole process and sized by Go's defaults. Whether a call rode a pooled connection or paid a fresh TLS handshake was an accident of how often its HOSTNAME happened to be touched, and that is not evenly distributed. GetBlock serves all nine chains from ONE hostname with the key in the path, so that host is hit nine times a cycle and stays warm. Every other provider uses a hostname per chain, touched once per 60 s. A handshake costs about 25 ms from these containers, and 16 + 25 is 41, which is exactly what this harness published for QuickNode against a measured 8. Measured 2026-10-08 from inside the eu probe container, 25 samples per provider, pooled, Ethereum: provider measured directly published by this harness, same minutes getblock 15.6 ms 15-19 ms quicknode 8.3 ms 25-41 ms chainstack 9.0 ms 32-105 ms alchemy 13.3 ms 37-71 ms Three of four providers were reported 4 to 6x slower than they answer. ## What this changes Two explicit transports: warmTransport pools with an IdleConnTimeout longer than the probe interval, coldTransport never reuses. Each endpoint alternates between them, so the call volume is exactly what it was and each arm lands 720 samples per day per region. The keyed budgets have headroom (600k-2M per region per month against ~195k used) but doubling a metered key to answer a methodology question is the wrong trade. Warm is what a server-side application with a connection pool gets. Cold is what a one-shot client or a serverless invocation gets. Neither is "the" latency, and the gap between them is worth reading on its own. rpc_latency_milliseconds becomes the WARM arm. The cold arm is a new metric name rather than a `mode` label on the existing one, deliberately: the nine keyed specs and the recording rules select on the current label set, and adding a dimension to a metric a spec already queries silently splits a published column into two series and blanks it. Anyone reading a chart across today should know the series before this commit are a mixture of both arms and are not comparable with either one after it. Both failure paths now delete both series. Leaving the cold one behind would freeze a dead endpoint at its last good number in the half of the data nobody is watching. go build and go vet clean. The nine keyed pages still need a second column and the disclosure that goes with it; that is the next change, not this one.
# Conflicts: # harnesses/rpc-keyed-latency/cmd/script/probe.go
Flotapponnier
added a commit
that referenced
this pull request
Oct 8, 2026
…mages (#2897) Two merges to main (#2895, #2896) each triggered a build that reported SUCCESS, and the container really was replaced: /proc/1 showed a boot at 18:21:35 and pid 1 was /app/monitor. But grepping the running binary for symbols from both commits returned zero on all four of them. The builder was serving a cached image whose `COPY . .` layer predated the changes, so six deployments shipped the same binary. A BUILD_REF arg between the source copy and the go build gives the layers below something that changes. The echo is there so the arg is actually consumed; a bare ARG is not a cache key. Also recording the diagnostic that cost an hour, because it will recur: the binary's mtime is NOT a way to tell whether a deploy landed. It comes from the image layer, so it reads old even in a container that booted a minute ago. I read 16:35 six times and concluded nothing was deploying, which was wrong — the containers were fine, the image was stale. Grep the running binary for a string you just added. That is the test that works.
Flotapponnier
added a commit
that referenced
this pull request
Oct 8, 2026
#2896 made the keyed probe measure both arms deliberately: rpc_latency_milliseconds is now explicitly the WARM one (pooled connection) and rpc_latency_cold_milliseconds is the same call on a connection that is never reused. Both are flowing from all three regions. Without these rules the cold series exists but no page can read it, because every spec queries the precomputed ocb: shapes rather than raw gauges. Four rules mirroring the warm ones, same group, same 60 s interval, and no tier filter — the keyed harness emits the same metric names with tier="keyed", so one rule set covers both cohorts and all three regions, exactly as the comment in prometheus.yml says of the existing ones. ## Context for whoever reads this later Every probe used to share one implicit http.DefaultTransport. Whether a call rode a pooled connection or paid a fresh TLS handshake depended on how often its HOSTNAME happened to be touched, which is not evenly distributed: a provider serving nine chains from one hostname stayed warm, a provider using a hostname per chain went cold between 60 s ticks and paid about 25 ms. That gap, not node speed, was most of the published keyed ranking. Three of four providers were reported 4 to 6x slower than they answer. Series before 2026-10-08 are a mixture of both arms and are not comparable with either one after it. ## Still to come The nine keyed specs do not read the cold series yet. The plan is a ranked WARM column plus a non-ranked cold-start figure, rather than two competing rankings: the cold number mixes provider quality with the distance from our probe regions to their TLS termination, so it is partly a fact about where Railway put our containers. Warm isolates what the provider controls and is the number a reader elsewhere can transpose. Validated with the repo's own js-yaml: group ocb_rpc_capabilities, 11 rules.
Flotapponnier
added a commit
that referenced
this pull request
Oct 8, 2026
…#2903) Everything from the connection-reuse incident landed on main: the two-arm harness (#2896), the cold recording rules (#2899) and the cold-start cohort on the nine keyed specs (#2902). The materialise worker clones DEV at build time, so none of it can reach a page until the specs exist here. That is why ?tier=keyed-cold currently serves the public provider list: the worker's snapshot predates the new tier and falls through to the default cohort. Same content as main: 9 specs the keyed-cold cohort, 36 provider clones on the cold metric probe.go warm and cold transports, alternating, plus the gated conn trace metrics.go rpc_latency_cold_milliseconds and its histogram rules ocb:rpc_latency_cold_milliseconds:{p50,p90,p99,mean}_24h pnpm validate 246 specs, go build clean. After this merges the worker still needs a rebuild, and there is roughly a 72-minute display floor before the pages show it.
Flotapponnier
added a commit
that referenced
this pull request
Oct 8, 2026
…#2905) #2896 added warm and cold arms to stop the ranking depending on accidental connection reuse. It made that dependency worse. Alternating the two arms doubled each one's effective interval, from 60 s to 120 s, and most endpoints drop an idle connection inside that window. The connection trace on eu says it plainly: shared.eu-central-1.getblock.io reused=TRUE tls= 0ms wispy-white-bush.ethereum-mainnet.quiknode.pro reused=false tls=10ms neat-dawn-shape.bsc.quiknode.pro reused=false tls=10ms ethereum-mainnet.core.chainstack.com reused=false tls= 9ms polygon-mainnet.g.alchemy.com reused=false tls=10ms GetBlock serves all nine chains from one hostname, so the other chains' traffic keeps its socket hot every ~13 s. Everyone else uses a hostname per chain, touched once per 120 s, and pays a handshake every time. The "warm" column was therefore warm for one provider and cold for the other three, which is the original bug amplified rather than removed. Every keyed number published since #2896 is wrong, including the p50 table I read off it. ## The fix primeWarm sends one unmeasured request through warmTransport before the measured one, so the measurement is guaranteed to ride a pooled socket instead of hoping one survived. The response is drained before closing: a body left unread never returns to the pool, which would defeat the whole call. The quota guard now reserves two requests on a warm tick instead of one, so it counts what we actually send. That is +50% volume against budgets running 600k-2M per region per month on ~195k used. Determinism is the point. A warm figure that depends on whether a socket happened to survive is not a measurement, and it systematically favours whoever reuses one hostname across many chains. ## Verification PROBE_TRACE_CONN=1 is already set on rpc-keyed-eu. The check is that every hostname logs reused=true on warm ticks, not just GetBlock's. go build and go vet clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
keyed latency: measure warm and cold connections deliberately, and publish both
The implicit transport was deciding the ranking.
Until now every probe used http.DefaultTransport, shared with the whole process
and sized by Go's defaults. Whether a call rode a pooled connection or paid a
fresh TLS handshake was an accident of how often its HOSTNAME happened to be
touched, and that is not evenly distributed.
GetBlock serves all nine chains from ONE hostname with the key in the path, so
that host is hit nine times a cycle and stays warm. Every other provider uses a
hostname per chain, touched once per 60 s. A handshake costs about 25 ms from
these containers, and 16 + 25 is 41, which is exactly what this harness
published for QuickNode against a measured 8.
Measured 2026-10-08 from inside the eu probe container, 25 samples per
provider, pooled, Ethereum:
provider measured directly published by this harness, same minutes
getblock 15.6 ms 15-19 ms
quicknode 8.3 ms 25-41 ms
chainstack 9.0 ms 32-105 ms
alchemy 13.3 ms 37-71 ms
Three of four providers were reported 4 to 6x slower than they answer.
What this changes
Two explicit transports: warmTransport pools with an IdleConnTimeout longer
than the probe interval, coldTransport never reuses. Each endpoint alternates
between them, so the call volume is exactly what it was and each arm lands 720
samples per day per region. The keyed budgets have headroom (600k-2M per region
per month against ~195k used) but doubling a metered key to answer a
methodology question is the wrong trade.
Warm is what a server-side application with a connection pool gets. Cold is
what a one-shot client or a serverless invocation gets. Neither is "the"
latency, and the gap between them is worth reading on its own.
rpc_latency_milliseconds becomes the WARM arm. The cold arm is a new metric
name rather than a
modelabel on the existing one, deliberately: the ninekeyed specs and the recording rules select on the current label set, and adding
a dimension to a metric a spec already queries silently splits a published
column into two series and blanks it.
Anyone reading a chart across today should know the series before this commit
are a mixture of both arms and are not comparable with either one after it.
Both failure paths now delete both series. Leaving the cold one behind would
freeze a dead endpoint at its last good number in the half of the data nobody
is watching.
go build and go vet clean. The nine keyed pages still need a second column and
the disclosure that goes with it; that is the next change, not this one.