fix(supervisor-middleware): configure HTTP/2 keepalive on the middleware gRPC channel #2474

Agent Diagnostic

Found while reviewing the design plan for #2428 (WebSocket message introspection middleware), during an investigation of how gRPC deadlines and connection liveness work for supervisor middleware.

• Read crates/openshell-supervisor-middleware/src/remote.rs, crates/openshell-supervisor-middleware/src/lib.rs, crates/openshell-sandbox/src/lib.rs, and the tonic Endpoint API for the resolved version.
• Baseline tested: main at deced871.
• Searched existing issues for middleware keepalive and keepalive: no matching issue found. The two sandbox exec hang reports ([Bug]: sandbox exec hangs after receiving complete gRPC response — never outputs data, never exits #828, [Bug]: sandbox exec hangs after receiving complete gRPC response — never outputs data, never exits #1046) are a different code path.
• tonic resolves to 0.14.6 (Cargo.lock:6928). All of http2_keep_alive_interval, keep_alive_timeout, keep_alive_while_idle, and http2_adaptive_window are available on Endpoint in that version.

Findings:

1. The remote middleware channel is built at crates/openshell-supervisor-middleware/src/remote.rs:26-57 and configures only connect_timeout(5s) plus encoding and decoding message size limits. No HTTP/2 keepalive, no TCP keepalive, no idle timeout.

2. The channel is long-lived and idle between evaluations. connect_middleware_registry at crates/openshell-sandbox/src/lib.rs:2533-2542 builds it once, and reconcile_middleware_registry at crates/openshell-sandbox/src/lib.rs:2552-2606 early-returns at :2559-2562 unless the desired registration set changes, so the same channel is reused indefinitely. A sandbox with a middleware policy attached to a rarely-contacted host holds an idle HTTP/2 connection for the lifetime of the sandbox.

3. Every other long-lived gRPC client in the repo already configures keepalive, and the middleware channel is the only outlier. The canonical helper is standard_endpoint at crates/openshell-sdk/src/transport.rs:28-39, whose doc comment states it "Centralizes timeouts and HTTP/2 keepalive so behavior is consistent across transport branches". The most complete variant is crates/openshell-core/src/grpc_client.rs:134-146. The same pattern appears in crates/openshell-cli/src/tls.rs:343-397 and crates/openshell-tui/src/lib.rs:563-567. The server side sets the counterpart at crates/openshell-server/src/multiplex.rs:194-203, with a comment noting that supervisors hold long-lived sessions.

Description

The supervisor middleware gRPC channel does not configure HTTP/2 keepalive, so an idle channel is not kept alive and a dead peer is not detected until the next evaluation attempts to use the connection.

Expected: the middleware channel follows the same keepalive configuration as every other long-lived gRPC client in the repo, so idle connections survive intermediary idle timeouts and dead peers are detected proactively.

Actual: the channel is created with connect_timeout and message size limits only. An intermediary idle timeout, a load balancer idle reap, a middleware service redeploy, or a GOAWAY silently invalidates the connection. The failure surfaces on the next middleware evaluation, which then fails and is handled by that attachment's on_error. Under fail_closed this denies a request that should have succeeded. Under fail_open it bypasses inspection for that request and emits a detection finding for what is actually a transport configuration gap rather than a middleware fault.

tonic's Channel does reconnect on its own after a failure, so the impact today is bounded to failed evaluations rather than a permanently broken registry. That bound is why this has not been noticed. It stops holding for long-lived streaming RPCs, which is why this surfaced during the #2428 review: a bidirectional stream that is idle for the duration of a quiet WebSocket session has no equivalent recovery, and losing it terminates the session under fail_closed.

Fixing this before #2428 lands removes a whole class of confusing failure from the streaming work, and it is a correctness improvement for the existing HTTP middleware path on its own merits.

Reproduction Steps

1. Register an operator middleware service against a gateway, per [[openshell.supervisor.middleware]] in the gateway config (crates/openshell-server/src/config_file.rs:178-197).
2. Attach it to a host in a sandbox policy via network_middlewares with on_error: fail_closed.
3. Place any idle-reaping hop between the supervisor and the middleware service, or simply restart the middleware service.
4. Leave the sandbox idle past that idle timeout without making a matching request.
5. Make a request matching the attachment. The evaluation fails on a stale connection and the request is denied under fail_closed, despite the middleware service being healthy.

The same behavior is observable without an intermediary by inspecting the constructed Endpoint: no keepalive is set, so no HTTP/2 PING frames are ever emitted on an idle middleware channel.

Environment

• OS: macOS 15.2 (darwin 25.5.0)
• OpenShell: main at deced871
• tonic: 0.14.6 (Cargo.lock:6928)
• Latest release checked: tested against current main, which is ahead of the latest release
• Possible duplicates checked: yes, searched middleware keepalive and keepalive across open and closed issues, no match

Proposed Fix

Apply the existing repo-standard keepalive configuration to the middleware channel in crates/openshell-supervisor-middleware/src/remote.rs, matching crates/openshell-core/src/grpc_client.rs:134-146:

• http2_keep_alive_interval(10s)
• keep_alive_while_idle(true)
• keep_alive_timeout(20s)
• http2_adaptive_window(true)

Keep the existing connect_timeout and message size limits unchanged. No configuration schema change is required, and MiddlewareServiceFileConfig is deny_unknown_fields, so keeping this out of the operator schema is deliberate: these are platform constants matching every other client in the repo, not something an operator should tune.

Consider whether crates/openshell-gateway-interceptors/src/plan.rs:862-874 should be fixed in the same change. It builds a bare Endpoint::from_shared(...).connect() with no keepalive and no connect timeout at all, so it has the same gap plus a missing connect bound.

Testing

• Assert the constructed endpoint carries the expected keepalive settings, following whatever pattern openshell-core or openshell-sdk already uses for their equivalents.
• Verify an evaluation succeeds against a middleware service that has been idle past a configured intermediary idle timeout.