From 17393e510cafa1bfb2ece737a24aa9161afdeef5 Mon Sep 17 00:00:00 2001 From: Flotapponnier <160007691+Flotapponnier@users.noreply.github.com> Date: Thu, 8 Oct 2026 20:57:29 +0200 Subject: [PATCH] recording rules for the cold-connection arm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #2896 made the keyed probe measure both arms deliberately: rpc_latency_milliseconds is now explicitly the WARM one (pooled connection) and rpc_latency_cold_milliseconds is the same call on a connection that is never reused. Both are flowing from all three regions. Without these rules the cold series exists but no page can read it, because every spec queries the precomputed ocb: shapes rather than raw gauges. Four rules mirroring the warm ones, same group, same 60 s interval, and no tier filter — the keyed harness emits the same metric names with tier="keyed", so one rule set covers both cohorts and all three regions, exactly as the comment in prometheus.yml says of the existing ones. ## Context for whoever reads this later Every probe used to share one implicit http.DefaultTransport. Whether a call rode a pooled connection or paid a fresh TLS handshake depended on how often its HOSTNAME happened to be touched, which is not evenly distributed: a provider serving nine chains from one hostname stayed warm, a provider using a hostname per chain went cold between 60 s ticks and paid about 25 ms. That gap, not node speed, was most of the published keyed ranking. Three of four providers were reported 4 to 6x slower than they answer. Series before 2026-10-08 are a mixture of both arms and are not comparable with either one after it. ## Still to come The nine keyed specs do not read the cold series yet. The plan is a ranked WARM column plus a non-ranked cold-start figure, rather than two competing rankings: the cold number mixes provider quality with the distance from our probe regions to their TLS termination, so it is partly a fact about where Railway put our containers. Warm isolates what the provider controls and is the number a reader elsewhere can transpose. Validated with the repo's own js-yaml: group ocb_rpc_capabilities, 11 rules. --- .../recording_rules/ocb_rpc_capabilities.yml | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/infrastructure/monitoring/prometheus/recording_rules/ocb_rpc_capabilities.yml b/infrastructure/monitoring/prometheus/recording_rules/ocb_rpc_capabilities.yml index 625b6e53e..a47e1d97a 100644 --- a/infrastructure/monitoring/prometheus/recording_rules/ocb_rpc_capabilities.yml +++ b/infrastructure/monitoring/prometheus/recording_rules/ocb_rpc_capabilities.yml @@ -30,3 +30,30 @@ groups: expr: sum by (provider, chain, region) (rate(rpc_call_total[24h])) - record: ocb:rpc_call:increase_24h expr: sum by (provider, chain, region) (increase(rpc_call_total[24h])) + + # Cold-connection arm, added 2026-10-08 with the keyed probe fix. + # + # rpc_latency_milliseconds is now explicitly the WARM arm: a probe on a + # pooled connection. rpc_latency_cold_milliseconds is the same call on a + # connection that is never reused, so it carries DNS + TCP + TLS every + # time. The keyed harness alternates between the two, which is why each + # arm lands half the samples of the old single series. + # + # Why the arms exist at all: every probe used to share one implicit + # transport, so whether a call was warm or cold depended on how often its + # HOSTNAME happened to be touched. A provider serving nine chains from one + # hostname stayed warm; a provider using a hostname per chain went cold + # between 60 s ticks and paid a ~25 ms handshake. That gap, not node + # speed, was most of the published ranking. + # + # No tier filter, same as the rules above: the keyed harness emits the + # same metric names with tier="keyed", so one rule set covers both + # cohorts and all three regions. + - record: ocb:rpc_latency_cold_milliseconds:p50_24h + expr: quantile_over_time(0.50, rpc_latency_cold_milliseconds[24h]) + - record: ocb:rpc_latency_cold_milliseconds:p90_24h + expr: quantile_over_time(0.90, rpc_latency_cold_milliseconds[24h]) + - record: ocb:rpc_latency_cold_milliseconds:p99_24h + expr: quantile_over_time(0.99, rpc_latency_cold_milliseconds[24h]) + - record: ocb:rpc_latency_cold_milliseconds:mean_24h + expr: avg_over_time(rpc_latency_cold_milliseconds[24h])