Repository navigation
feat(web_rtc): log a periodic webrtc listener summary with GET service time - #278
Conversation
…e time The browser listener kept every counter in `WebRtcServerSnapshot` and never logged it, so during V2-1795 nobody could tell from production logs whether fleet nodes were queueing browser GETs on the four global bulk-response slots. `ListenerResources` gains seven relaxed counters: `get_service_count`, `get_service_total_ms`, `get_service_max_ms`, `get_service_failures`, `bulk_wait_count`, `bulk_wait_total_ms` and `bulk_wait_max_ms`, recorded by `record_bulk_wait` and `record_get_service`. The request worker times itself from the moment it starts, so a GET's service time covers the wait for its source and global bulk permits as well as the work; that wait is also reported on its own. A GET whose processing errors is counted in `get_service_failures` and left out of the service-time totals. The snapshot carries those counters plus `bulk_slots_in_use` and `bulk_slots_max`, and a third branch of the listener task's `tokio::join!` emits them every `WEBRTC_SUMMARY_INTERVAL_SECS` (300 s, the replication and chunk-RPC cadence) as two `webrtc listener summary (cumulative)` INFO lines on target `ant_node::web_rtc::traffic`, told apart by `group`. Sampling only reads atomics and semaphore permit counts, so the loop holds no lock across an await and never blocks the accept loop or a request worker. Both that loop and the pre-existing address-refresh loop move out of `spawn` into `log_listener_summaries` and `refresh_browser_endpoint_loop`, so the `tokio::join!` is three symmetric calls and `spawn` stays under `clippy::too_many_lines`. No cap, policy or admission behaviour changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
f9501df to
78202bb
Compare
MaidSafe-Hermes
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES — the emitted metric contract is inaccurate.
At docs/adr/ADR-0015-direct-browser-clients-over-webrtc-direct.md:344-346 and src/web_rtc.rs:707-709, the text says every field except capacity/slot fields is a cumulative counter that can be differenced for rates. That is false for running, active_connections, active_channels, active_requests, and transport active/connecting connection counts (gauges), and for get_service_max_ms / bulk_wait_max_ms (lifetime high-water marks). Differencing these can report meaningless rates and conceal sustained high latency. Please classify additive counters, instantaneous gauges, and lifetime maxima separately in the ADR and rustdoc.
Also align the latency comments in src/web_rtc.rs: get_service_total_ms and record_get_service say timing starts at request admission, but start_request starts the clock when the spawned worker begins. Either document that boundary consistently or move the start time if admission-to-response latency is intended.
Coverage: 2/2 seats reviewed (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). The seats disagreed on verdict; sol-1 required changes and glm-1 approved while flagging the same metric-contract error. I confirmed the error against the source, so it blocks approval.
Checks at this head: Quick checks, Rust 1.95 MSRV, release build, filesystem jobs, ADR validation and PR checks pass; macOS/Ubuntu/Windows tests are still pending. Security Audit fails on crypto-bigint 0.5.5 (RUSTSEC-2026-0387); this PR does not change Cargo.lock, so that failure is inherited, not introduced here.
… boundary Review of WithAutonomi#278 found the emitted metric contract inaccurate. The ADR and the rustdoc on `log_webrtc_listener_summary` said every field but the capacity and slot gauges was a counter accumulated since process start, so a query could diff two lines for a rate. That is false for `running`, `active_connections`, `active_channels`, `active_requests` and the transport active/connecting gauges, which are instantaneous levels, and for `get_service_max_ms` and `bulk_wait_max_ms`, which are lifetime high-water marks: a diff of either is 0 for any window that does not beat the all-time worst case, so differencing them conceals sustained high latency instead of revealing it. Both now classify every field as an additive counter, an instantaneous gauge, a lifetime maximum or a configured capacity, name the additive counters as the only ones a query may diff, and give mean GET latency as the `get_service_total_ms` diff over the `get_service_count` diff. `get_service_total_ms` and `record_get_service` also claimed the clock starts at request admission, while `start_request` starts it when the spawned worker begins. The measurement point is unchanged — admission completes before frame assembly and PQ decryption, which do not belong in bulk-slot service time — so the docs now state that boundary and `admitted_at` becomes `worker_started_at`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both points are correct — thanks. Fixed in 22502f9. 1. Metric classificationThe old sentence ("every other field is a counter accumulated since process
One thing I did not change: the message string is still 2. Timing boundaryAlso correct — I fixed the documentation rather than moving the clock, and renamed the
The docs on the field, the recorder and the ADR now all state the same ChecksAgreed on |
MaidSafe-Hermes
left a comment
There was a problem hiding this comment.
Focused re-review of the follow-up delta from 78202bb to 22502f9; this is not a fresh full-PR panel.
Both seats reviewed and approve the delta (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). The ADR and logger rustdoc now distinguish additive counters, instantaneous gauges, lifetime maxima and configured capacities. The timing documentation and variable name now consistently identify the worker-start boundary. I found no remaining blocker in the reviewed changes.
The branch-protected checks (linear-link and pr-template) pass. Ubuntu tests, Quick checks, MSRV, release build, filesystem checks and ADR validation also pass. macOS and Windows tests are still pending; this follow-up changes documentation and renames a local variable, with no runtime behavior change. Security Audit remains failed on the known inherited crypto-bigint 0.5.5 advisory (RUSTSEC-2026-0387); this PR does not change Cargo.lock.
Brings the branch up to main f37429a, which adds pressure-gated retention (WithAutonomi#271), get-or-closer chunk GETs (WithAutonomi#249), the WebRTC UPnP mapping (WithAutonomi#264) and listener summary (WithAutonomi#278), and the consolidated PR CI (WithAutonomi#269). The only conflict was the ADR index: ADR-0024 now sits before main's ADR-0027 and ADR-0028. None of main's changes touch the fetch scheduler.
Takes the per-source scheduler branch at f568f91, which merged upstream main f37429a, so this branch carries main's pressure-gated retention (WithAutonomi#271), get-or-closer chunk GETs (WithAutonomi#249), WebRTC changes (WithAutonomi#264, WithAutonomi#278) and the consolidated PR CI (WithAutonomi#269) through it. The only conflict was the ADR index: ADR-0025 now sits before main's ADR-0027 and ADR-0028. The resulting tree equals a direct merge of this branch with main apart from that index.
Linear issue
Closes V2-1871
Risk tier
Observability only: no cap, policy or admission change. The request worker gains
an
Instant::now()and, for GETs, two relaxedfetch_add/fetch_maxpairs; thenew summary loop only reads atomics and semaphore permit counts, so it holds no
lock across an await and never blocks the accept loop or a request worker.
Compatibility
WebRtcServerSnapshotgains ninepubfields and
WebRtcServerDiagnosticsalog_webrtc_listener_summary()method.Both live in the private
web_rtcmodule, so no external name is affected;the only readers are
src/devnet.rsandtests/webrtc_direct_devnet.rs.Semver impact
Test evidence
Run locally at rustc 1.99.0, both clean:
cargo fmt --all -- --checkandcargo clippy --all-targets --all-features -- -D warnings. The crate deniespanic,unwrap_usedandexpect_used, so the ms conversion goes throughu64::try_from(..).unwrap_or(u64::MAX). Everything else is left to this PR'schecks, including
cargo check --lib --no-default-featuresand the test matrix.Security Auditfails onRUSTSEC-2026-0387in the transitivecrypto-bigint.That is unrelated to this PR, which adds no dependency and does not touch
Cargo.lock.Tests added or extended in
src/web_rtc.rs, exercised by this PR's checks:diagnostics_report_capacity_rate_and_byte_pressure_without_leaking_slotsextended to assert the seven new counters start at 0,
bulk_slots_in_useis 0, and
bulk_slots_maxis 1 at that test's 8-byte frame budget.latency_counters_accumulate_and_the_summary_logs_the_snapshot— twodurations through each recorder, asserting count, sum and max, that
bulk_slots_maxis 4 at the default config, and thatlog_webrtc_listener_summaryemits both lines without panicking.one_get_records_a_bulk_wait_and_a_service_time— drives one GET throughthe real
start_requestworker against a storage-disabledServerState,asserting
bulk_wait_count == 1,get_service_count == 1,get_service_failures == 0, and that the bulk slot is released after theresponse is dropped.
Not yet observed on a testnet; the line is INFO on a 300 s cadence, same as the
replication and chunk-RPC summaries.
New dependency
none
ADR
docs/adr/ADR-0015-direct-browser-clients-over-webrtc-direct.md— the"Internal listener diagnostics" section now documents the line, its cadence and
every field. (n/a as a gate at T1; updated because the ADR owns this surface.)
Mitigation / rollback
Revert the commit, or raise
WEBRTC_SUMMARY_INTERVAL_SECSif the cadence provesnoisy. The instrumentation is read-only accounting, so dropping it cannot affect
request admission or responses.
🤖 Generated with Claude Code