bug(proxy): 60s reqwest total timeout kills streaming inference responses mid-generation #641

Summary

The openshell-router creates its reqwest::Client with a 60-second total request timeout (lib.rs: .timeout(Duration::from_secs(60))). For streaming inference responses (/v1/chat/completions with stream: true), this timeout covers the entire request lifecycle — from initial send through the last streamed chunk. When model generation takes longer than 60 seconds (common with large models at ~8-15 tok/s), reqwest aborts the connection mid-stream.

This was introduced alongside the streaming fix in #260 / PR #261, which correctly switched from buffered to chunked streaming but inherited the existing 60s total timeout.

Reproduction

1. Configure inference.local to route to a self-hosted vLLM instance serving a large model (e.g., Nemotron-3-Super-120B)
2. Send a chat completion request that produces a long response (500+ tokens at ~8-15 tok/s = 30-60+ seconds of generation)
3. After exactly 60 seconds, the streaming response is truncated
4. Sandbox proxy logs: WARN openshell_sandbox::proxy: error reading upstream response chunk error=upstream protocol error: failed to read response chunk: error decoding response body

Evidence

Error timestamps from the sandbox proxy show exact 60-second intervals between consecutive failures (client retries that each hit the same timeout):

02:01:54.632105 -> 02:02:54.656482: 60.0s
02:41:50.410839 -> 02:42:50.434817: 60.0s
02:42:50.434817 -> 02:43:50.455857: 60.0s
03:27:49.791945 -> 03:28:49.817736: 60.0s
03:28:49.817736 -> 03:29:49.841861: 60.0s

Root Cause

In crates/openshell-router/src/lib.rs, the reqwest client is configured with:

.timeout(Duration::from_secs(60))

reqwest's .timeout() is a total request timeout — it does not reset when data is received. For streaming SSE responses, this means the timeout fires even though the upstream is actively sending data.

The error path in proxy.rs (line ~1032-1034):

Err(e) => {
    warn!(error = %e, "error reading upstream response chunk");
    break;
}

...breaks out of the chunk-reading loop and then writes the chunked terminator (0\r\n\r\n), causing the downstream client to see a truncated-but-technically-valid HTTP response.

Suggested Fix

Replace the total .timeout() with a read timeout that resets on each chunk:

// Before (total timeout — kills long-running streams):
reqwest::Client::builder()
    .timeout(Duration::from_secs(60))
    .build()

// After (read timeout — only fires if upstream goes silent):
reqwest::Client::builder()
    .read_timeout(Duration::from_secs(120))
    .connect_timeout(Duration::from_secs(30))
    .build()

This preserves the protection against hung upstreams (fires if no data arrives for 120s) while allowing arbitrarily long streaming generations.

Environment

• OpenShell sandbox version: bundled in NemoClaw container (openshell-sandbox binary, vLLM 0.17.1 backend)
• Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 (~8-15 tok/s generation)
• Platform: NVIDIA DGX Spark GB10, Linux 6.17.0