Skip to content

recording rules for the cold-connection arm #2899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 1 commit into from
Oct 8, 2026
Merged

recording rules for the cold-connection arm #2899

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions infrastructure/monitoring/prometheus/recording_rules/ocb_rpc_capabilities.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,30 @@ groups:
expr: sum by (provider, chain, region) (rate(rpc_call_total[24h]))
- record: ocb:rpc_call:increase_24h
expr: sum by (provider, chain, region) (increase(rpc_call_total[24h]))

# Cold-connection arm, added 2026-10-08 with the keyed probe fix.
#
# rpc_latency_milliseconds is now explicitly the WARM arm: a probe on a
# pooled connection. rpc_latency_cold_milliseconds is the same call on a
# connection that is never reused, so it carries DNS + TCP + TLS every
# time. The keyed harness alternates between the two, which is why each
# arm lands half the samples of the old single series.
#
# Why the arms exist at all: every probe used to share one implicit
# transport, so whether a call was warm or cold depended on how often its
# HOSTNAME happened to be touched. A provider serving nine chains from one
# hostname stayed warm; a provider using a hostname per chain went cold
# between 60 s ticks and paid a ~25 ms handshake. That gap, not node
# speed, was most of the published ranking.
#
# No tier filter, same as the rules above: the keyed harness emits the
# same metric names with tier="keyed", so one rule set covers both
# cohorts and all three regions.
- record: ocb:rpc_latency_cold_milliseconds:p50_24h
expr: quantile_over_time(0.50, rpc_latency_cold_milliseconds[24h])
- record: ocb:rpc_latency_cold_milliseconds:p90_24h
expr: quantile_over_time(0.90, rpc_latency_cold_milliseconds[24h])
- record: ocb:rpc_latency_cold_milliseconds:p99_24h
expr: quantile_over_time(0.99, rpc_latency_cold_milliseconds[24h])
- record: ocb:rpc_latency_cold_milliseconds:mean_24h
expr: avg_over_time(rpc_latency_cold_milliseconds[24h])
Loading