Node gRPC server rejects frequent client keepalive pings and needs more than 2 hours to drop a connection whose TCP peer still ACKs but sends no gRPC frames #125

Found by code review and reproduced in a local test; never seen in production.

Checked against node dev 9ccd263 and main 739eab9 (v0.5.7); grpc-go v1.85.0 from go.mod; Go 1.27.3. PR #130 contains the fix.

Summary

StartGRPCListener (controller/rpc/service.go:36-42 at 9ccd263) creates the gRPC server without grpc.KeepaliveParams and without grpc.KeepaliveEnforcementPolicy, so grpc-go's server defaults apply (internal/transport/defaults.go:38-40 in grpc-go v1.85.0):

• the server pings a connection only after it has received nothing for 2 hours, and closes it 20 s after an unacknowledged ping;
• a client ping counts as a strike when it comes less than 5 minutes after the previous ping while a stream is active, or less than 2 hours after it on a connection with no active stream; after more than two strikes, with no data or headers sent by the server in between, the server sends GOAWAY too_many_pings (grpc-go v1.85.0 http2_server.go:880-925, read from the source).

Which cases take more than 2 hours

The 2-hour delay applies only when the TCP peer of the node keeps acknowledging but no gRPC frames arrive:

• a TCP-terminating proxy or load balancer between the panel and the node whose other side is dead or stalled, or
• a panel process that hangs while its kernel still acknowledges TCP.

In both cases, without the fix the node keeps the connection until grpc-go's default ping goes unanswered, 2 hours and 20 seconds after the last frame it received. In a local test (macOS, loopback, a TCP proxy that keeps both sockets open but stops forwarding) the node had not closed its side after 70 s.

When the panel host disappears or the path drops packets, the kernel detects it without the fix. The following comes from reading grpc-go and the Go net package; it was not measured:

• grpc-go sets TCP_USER_TIMEOUT on Linux to the keepalive Timeout whenever the keepalive Time is not infinite (internal/transport/http2_server.go:244-248). With the defaults that is 20 s.
• Go enables TCP keepalive on accepted connections, with 15 s idle, a 15 s interval and 9 probes (net/dial.go:20-27 in Go 1.27.3).

So on Linux such a connection is closed by the kernel, not by grpc-go's 2-hour ping. I did not measure how long that takes. On other systems grpc-go does not set TCP_USER_TIMEOUT (internal/syscall/syscall_nonlinux.go:67-70).

Impact

• Availability only; no data loss.
• The node's own keep-alive watchdog (controller/controller.go:339-372 at 9ccd263) already stops the core, when keep_alive is set, once no panel request has arrived for keep_alive + 5 s (checked every 5 s). What stays open in the cases above is the stale gRPC connection itself.
• The current panel client sends no pings, so it does not hit too_many_pings. PasarGuard/node_bridge_py main c5632b65a10a (v0.10.0, PasarGuardNodeBridge/grpclib.py:88-89) passes Configuration(_keepalive_timeout=10, ...) and does not set _keepalive_time. This is from reading the source; I did not run the panel against a node.
• A client that pings every 10 s on an idle connection is closed with GOAWAY too_many_pings; in a local test this happened after about 30 s.

Reproduction

The test controller/rpc/keepalive/keepalive_test.go from PR #130 starts a real node gRPC listener over TLS on loopback. On 9ccd263, with only the test files added (go test -race -count=1 ./controller/rpc/keepalive/, Go 1.27.3, macOS arm64):

    keepalive_test.go:145: the node dropped an idle panel connection that pings every 10 s after 30s; state now IDLE
--- FAIL: TestNodeKeepsAnIdlePanelConnectionThatPingsEveryTenSeconds (30.03s)
    keepalive_test.go:230: the node kept a connection to a panel that stopped answering for 70 s; it must ping an idle panel and close the connection when the ping is not acknowledged
--- FAIL: TestNodeClosesAConnectionWhosePanelStoppedAnswering (75.02s)

Proposed fix

Pass both options to grpc.NewServer:

• keepalive.ServerParameters{Time: 30 * time.Second, Timeout: 20 * time.Second}. The node pings a connection after 30 s with nothing received and closes it when the ping is not acknowledged within 20 s. Timeout stays at grpc-go's default 20 s, so TCP_USER_TIMEOUT on Linux keeps its current value.
• keepalive.EnforcementPolicy{MinTime: 5 * time.Second, PermitWithoutStream: true}. The node accepts client pings every 5 s or more, also on an idle connection.

Duplicate search

Searched again on 2026-11-07, leaving out this issue and PR #130:

• repo:PasarGuard/node keepalive: Harden node lifecycle and API handling #78 (open PR, "Harden node lifecycle and API handling"), xray: the version probe has no timeout, so a hung xray binary blocks Start and every other API call #127 (open issue, xray version probe), Node stops xray on keepalive timeout and does not recover automatically #50 (closed issue, node stops xray on keepalive timeout), feat: add OFFLINE_GRACE_PERIOD env to extend keepAlive timeout #33 (closed PR, OFFLINE_GRACE_PERIOD), feat(wireguard): manage WireGuard peers via Xray's inbound UserManager #64 (closed PR, WireGuard peers).
• repo:PasarGuard/node too_many_pings: no other results.

#127 is about the xray version probe. #50 and #33 are about the application-level keep-alive timer, not gRPC transport keepalive. The diff of #78 does not add KeepaliveParams or KeepaliveEnforcementPolicy.