feat(sandbox): make L7 inference proxy CHUNK_IDLE_TIMEOUT configurable per route

Problem Statement

The L7 inference proxy in openshell-sandbox aborts streaming inference responses when no chunk arrives within CHUNK_IDLE_TIMEOUT, a hardcoded constant (crates/openshell-sandbox/src/proxy.rs:39). PR #834 raised it from 30 s to 120 s in v0.0.30 to accommodate reasoning models, but the constant remains baked into the binary with no operator override.

120 s is enough for typical generation, but several legitimate workloads exceed it on the prefill phase alone — at which point the proxy injects an SSE error and truncates a perfectly healthy stream:

Large initial tool-call returns / pasted context. A single user turn carrying a 20K+ token tool result has to be prefilled before the model emits any token.
Full-context reprocess after prefix-cache invalidation. Long-running OpenClaw conversations can carry hundreds of thousands of tokens. When something near the start of the context changes (a system-prompt edit, memory injection, tool-result rewrite), every subsequent token is no longer prefix-cacheable and the entire context must be re-prefilled. Empirically measured single-request prefill on this setup (NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 served by local vLLM on an ASUS Ascent GX10 / NVIDIA GB10, DGX Spark–compatible, otherwise idle GPU):

tokens elapsed tok/s vs 120 s cap

16,000 6 s 2647 5 %

64,000 26 s 2441 22 %

128,000 61 s 2094 51 %

256,000 158 s 1616 132 %

400,000 293 s 1356 245 %

779,685 866 s 901 722 %

Rate degrades from 2647 → 901 tok/s as size grows (consistent with attention's O(N²) work). 256K already crosses the cap on a fresh, otherwise-idle GPU. 780K — close to Nemotron-3-Super's 1M context window — takes ~14.4 minutes of pure prefill, ~7× the cap. Extrapolating to a true 1M reprocess, ~20–25 minutes is realistic. Full data and methodology in Agent Investigation below.
Backend restart / KV-cache eviction. vLLM losing its KV cache (process restart, OOM-driven eviction, cache pressure from concurrent users) forces the next request to re-prefill from scratch with no prefix-cache hits. Same math as above.
GPU contention from concurrent OpenClaw sessions. OpenClaw can spawn sub-agents and run multiple sessions against the same backend. Per-request prefill rate divides roughly with concurrency, so the safe single-request ceiling above (~250K tokens at 120 s) drops to ~125K with two concurrent prefills, ~80K with three. Workloads that fit comfortably under the cap in isolation can fail under realistic load.

In every case the upstream is healthy and would happily stream tokens given enough time; only the proxy's hardcoded ceiling is at fault. The 120 s value is a reasonable default but not a universal one — the right ceiling depends on the operator's hardware, model, and expected prompt sizes.

Proposed Design

Make the per-chunk idle timeout configurable per inference route, mirroring the existing timeout_secs plumbing.

Config plumbing. Add chunk_idle_timeout_secs: Option<u64> to RouteConfig / ResolvedRoute in openshell-router, alongside the existing timeout_secs. None = use the compiled-in default (currently 120 s); 0 = disable the per-chunk timeout entirely (consistent with how --timeout 0 already means "default" for total request timeout).
CLI surface. Extend openshell inference set and openshell inference update with --chunk-idle-timeout <secs> (default 0 = unchanged/use default). Mirror the existing --timeout flag behavior. Show the resolved value in openshell inference get.
Proxy wiring. In route_inference_request (crates/openshell-sandbox/src/proxy.rs:1261), replace the constant with the route-resolved value:
```
let chunk_idle = route.chunk_idle_timeout.unwrap_or(DEFAULT_CHUNK_IDLE_TIMEOUT);
```
Keep DEFAULT_CHUNK_IDLE_TIMEOUT = 120s as the floor when unset, so existing deployments behave identically.
Truncation event metadata. Include the resolved timeout value in the OCSF streaming response chunk idle timeout event so operators can see in logs which timeout fired.
Validation. Sanity-check the value at config load (e.g., reject < 5 s to avoid foot-guns; cap at some upper bound like 3600 s to avoid lock-up scenarios).

User-facing change is additive and backward-compatible.

Alternatives Considered

Just bump the constant again. No value satisfies all workloads — 600 s helps a 200K-token reprocess but penalizes operators on weak hardware who'd rather fail-fast on stuck streams. A constant always picks one operating point.
Derive from existing --timeout. Reusing route.timeout_secs as both total-request and per-chunk idle timeout would conflate two distinct safety properties: total lifetime (cap on runaway requests) vs. per-chunk liveness (catch a genuinely dead stream). Operators want to set them independently — e.g., 3600 s total, 300 s per-chunk idle.
Auto-tune from a backend probe. Inferring the right value from /v1/models or a warmup probe is too magical and brittle; it can't know what the operator's worst-case prompt looks like.
Disable the per-chunk timeout entirely for self-hosted backends. Removes a useful safety net that catches genuinely stuck streams (network partitions, hung backend processes). Should remain available as an opt-in (--chunk-idle-timeout 0) but not the default.

Agent Investigation

Reproduced and isolated with NemoClaw v0.0.10 + OpenShell v0.0.26 + local vLLM serving NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 on an ASUS Ascent GX10 (NVIDIA GB10 GPU, 20-core Cortex-X925, aarch64; DGX Spark–compatible reference platform):

vLLM logs show every aborted request was killed exactly 30 s after acceptance; matches CHUNK_IDLE_TIMEOUT = 30s in proxy.rs:34 of v0.0.26.
Streaming curl from inside the sandbox through the L7 proxy with a 25K–30K random-token prompt: aborted at 30 s, vLLM logs the abort, client receives the format_sse_error("response truncated: chunk idle timeout exceeded") frame.
Same prompt sent direct host → vLLM (bypassing the sandbox): completed normally with no abort. Confirms the cap lives in openshell-sandbox, not in vLLM, the OpenAI SDK, undici, or OpenClaw's agents.defaults.llm.idleTimeoutSeconds (which is a client-side iterator timer that only fires after the proxy has already aborted).
Closed PR fix(inference): prevent silent truncation of large streaming responses #834 / commit 355d845d already shows maintainers see the constant as a tuning knob ("Reasoning models … can pause for 60+ seconds … 120 s provides headroom"). This proposal is the next step: lift it from a compile-time constant to a per-route config.

Prefill benchmark methodology (numbers in Problem Statement table): non-streaming /v1/chat/completions requests sent direct to vLLM, max_tokens: 5 to make generation a negligible fraction of total time, each request prefixed with a fresh UUID to defeat prefix-cache hits. Verified usage.prompt_tokens_details.cached_tokens == 0 on every response. Cross-validated synthetic random-number prompts against real book text (Romeo & Juliet through Moby Dick and Chambers Dictionary): rates match within ~3 % at the same token sizes.

Full benchmark data (16 sweep points, random + book text)

tokens	elapsed	tok/s	source
15,974	6.04 s	2647	random
31,850	12.19 s	2614	random
42,216	16.82 s	2510	alice-in-wonderland
44,881	17.95 s	2501	romeo-and-juliet
63,696	26.09 s	2441	random
71,105	29.97 s	2373	the-great-gatsby
104,166	47.76 s	2181	frankenstein
127,372	60.82 s	2094	random
170,463	90.60 s	1881	wuthering-heights
178,023	96.82 s	1839	pride-and-prejudice
254,541	157.49 s	1616	random
263,972	168.64 s	1565	jane-eyre
317,863	218.20 s	1457	moby-dick
326,075	226.87 s	1437	city-of-god-volume-i
397,856	293.47 s	1356	random
779,685	865.70 s	901	chambers-dictionary

Checklist

I've reviewed existing issues and the architecture docs
This is a design proposal, not a "please build this" request

feat(sandbox,router,cli): honor route timeout on streaming chunk-idle

Authorship note: this comment was drafted by Claude (Anthropic's AI assistant) from a reproduction + design review we did together on my hardware, then reviewed and edited by me. The linked branch came out of that same session.

Quick update with reproduction evidence and a proposed implementation that's simpler than what I sketched in the issue body. A working branch is pushed to my public fork — details at the end.

Reproduction

Reproduced on an NVIDIA DGX Spark (GB10, aarch64) running vLLM with nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4, --tool-call-parser=qwen3_coder, --enable-auto-tool-choice. OpenShell v0.0.32, NemoClaw v0.0.23, route timeout 3600s.

OpenClaw prompt used (deliberately verbose — without the "no placeholder / single write call / don't check first" constraints, the model tends to sidestep the long tool-call by generating a Python script or using short placeholder paragraphs, which doesn't exercise the failure):

Please save a very long story (at least 100 paragraphs, each with detailed descriptions and multiple sentences, total length at least 10000 words) to /sandbox/.openclaw/workspace/story.md. You do not need to worry about token limits, they have been configured to accommodate large outputs. Do not use placeholder text, truncated paragraphs, or repeated content — write the full original story inline in a single write call. Do not check if the file exists first, just write it.

Without patch (baseline, hardcoded CHUNK_IDLE_TIMEOUT = 120s):

event	timestamp	Δ since last upstream chunk
vLLM last SSE delta (`[tool_calls: "file": "/sandbox/.openclaw/workspace/story.md"]`)	13:57:27	0
Sandbox OCSF `NET:FAIL [MED] inference.local:443`	13:59:27	+120.000s
vLLM `Aborted request(s) chatcmpl-97432e7552bd26e3-a6e94d66`	13:59:27	+120.000s

vLLM throughput metrics show continuous generation at ~20 tok/s throughout the 120s window — the backend was healthy, it simply wasn't flushing SSE chunks because the qwen3_coder tool-call parser buffers the arguments field until the call is well-formed. That's the same 120s boundary whose inline code comment at proxy.rs:37-38 cites nemotron-3-super as the motivating model.

This is a distinct failure mode from the prefill case in the issue body: active generation, not a cold start. Same symptom, different trigger — reinforces that no fixed constant is correct across workloads.

Simpler design than the one I originally proposed

A closer pass through the code by Claude surfaced send_backend_request_streaming at crates/openshell-router/src/backend.rs:241 — it explicitly sets no total timeout for streaming, and the comment reads "the total duration is unbounded — liveness is enforced by the caller's per-chunk idle timeout instead." So route.timeout is currently unused on the streaming path. Reusing this config field is a cleaner implementation than the separate chunk_idle_timeout_secs field I pitched:

Reuse route.timeout as the effective per-chunk idle timeout for streaming, floored at the existing 120s default so operators on defaults see no regression. No new CLI flag, no new config field, no new proto field — the existing openshell inference set --timeout <secs> already does the right thing on both streaming and non-streaming paths.

The whole semantic core is one helper:

// crates/openshell-router/src/config.rs
pub const DEFAULT_CHUNK_IDLE_TIMEOUT: Duration = Duration::from_secs(120);

/// Effective per-chunk idle timeout for a streaming request on this route.
pub fn effective_chunk_idle_timeout(route_timeout: Duration) -> Duration {
    route_timeout.max(DEFAULT_CHUNK_IDLE_TIMEOUT)
}

Plumbed through:

openshell-router::backend::StreamingProxyResponse — new chunk_idle_timeout: Duration field, populated at response construction (live + mock paths).
openshell-sandbox::proxy — delete the hardcoded constant, read resp.chunk_idle_timeout at the streaming loop. Include the effective value in the OCSF truncation event metadata so operators can see which timeout fired.
openshell-cli — one-line docstring update on inference set / inference update --timeout noting the streaming-path meaning.

Total: ~50 LoC + ~130 LoC of tests across 4 files (vs. ~200 LoC + proto changes in the original proposal).

What's lost vs. the original: setting total and chunk_idle independently. But there's no total timeout on streaming today, so the "3600s total, 300s chunk" scenario isn't actually representable in the current code — this change gives the existing knob a meaning on a path where it currently has none, rather than conflating two existing knobs. If a future workload ever needs independent tuning, adding an optional --chunk-idle-timeout override on top is backward-compatible.

Runtime validation

Rebuilt openshell-sandbox with the patch, swapped it into the running cluster, rebounced the sandbox pod, kept the existing --timeout 3600s on the route, and reran the same repro:

metric	baseline (120s hardcoded)	with patch + `--timeout 3600`
Story written to disk	no — proxy aborted at t+120s	yes — 12,547 words / 109 paragraphs / 80.7 KB
Sandbox `NET:FAIL [MED] inference.local:443` events in window	fired at exactly t+120.000s	0
vLLM `Aborted request(s)` in window	fired at exactly t+120.000s	0
Big write request duration	killed at 120s	615s (10m 15s)
Max inter-chunk SSE gap within that request	n/a — killed before emitting	586s (~10 min)

That 586-second inter-chunk gap is the key evidence: on the unpatched baseline the proxy would have fired NET:FAIL [MED] at t+120s and truncated the stream. With the patch, the proxy waited, received the chunk, and forwarded it to the client. No change to the user's workflow beyond openshell inference update --timeout 3600.

The 586s number also rules out "just bump the hardcode to a larger fixed value like 10 minutes" as a real fix. Before running this, I'd have guessed 10 minutes was more than enough — after, 586s is already uncomfortably close to 600s, and this was a single-request run on an otherwise-idle GPU. Re-running the same test a few times (the model's sampling is non-deterministic), or adding vLLM concurrency from a second sandbox (which divides per-request throughput), would push gaps past 10 minutes without difficulty. A configurable ceiling tied to route.timeout is the correct shape; any fixed constant will eventually bite someone.

Branch

Implementation + tests on a public branch in my fork: vnicolici:feat/chunk-idle-timeout-per-route (compare view against current main). Single commit, rebased onto latest main; cargo test -p openshell-router is clean (17/17 including 5 new timeout-specific tests for both the pure helper and the plumbed StreamingProxyResponse.chunk_idle_timeout on live + mock paths).

Not vouched yet so a direct PR would get auto-closed by vouch-check. If the design looks right, happy to open a Vouch Request — otherwise feel free to pick up the branch directly, or let me know what shape you'd want the PR in.

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: Medium
Confidence: High — current-main prototype and focused tests pass

Summary

Make the existing inference route timeout value effective for streamed responses by treating it as the per-chunk idle timeout, with the current 120-second value retained as a floor. This preserves default behavior while making openshell inference update --timeout <seconds> useful for models that legitimately buffer tool-call output or spend several minutes in prefill.

Scope

crates/openshell-router/src/backend.rs: carry the matched route timeout on streaming responses.
crates/openshell-router/src/lib.rs: preserve the route timeout on buffered mock responses.
crates/openshell-supervisor-network/src/proxy.rs: derive the effective chunk-idle timeout from the test/production floor and matched route timeout, use it for streaming reads, and include the effective value in the OCSF failure event.
crates/openshell-cli/src/main.rs: clarify the existing --timeout behavior for streaming routes.
crates/openshell-router/tests/backend_integration.rs: verify live and mock streaming responses carry the matched route timeout.
docs/sandboxes/inference-routing.mdx: document buffered total-timeout behavior versus streaming per-chunk idle behavior, the 120-second floor, and a long-running example.

Implementation Steps

Propagate the raw matched route.timeout through StreamingProxyResponse for live and mock paths.
In the supervisor proxy, compute max(CHUNK_IDLE_TIMEOUT, route_timeout) and use it around each upstream chunk read.
Add the effective idle duration to the existing OCSF timeout message.
Update CLI help and inference-routing documentation so operators know that values above 120 seconds raise the streaming idle ceiling.
Add focused tests, including a delayed upstream chunk that exceeds the test-only floor but remains below the configured route timeout.

Test Plan

Unit tests: supervisor-network coverage for default truncation and configured extended idle; router config/constructor assertions.
Integration tests: router live-response propagation plus mock-response propagation.
E2E tests: N/A — no e2e/ files change; the existing issue includes runtime validation with a 586-second healthy inter-chunk gap.
Local gates: cargo test -p openshell-router, cargo test -p openshell-supervisor-network, cargo fmt --all -- --check, and mise run pre-commit.

Risks & Open Questions

Reusing route.timeout gives the currently unused streaming value a concrete meaning; a future independent total-stream timeout can remain additive.
The current test-only idle floor is 100 ms. Tests must propagate the raw route timeout and use zero where the existing truncation test expects the floor, avoiding accidental 120-second waits.
A broad proxy refactor in PR feat(supervisor-middleware): add network egress middleware #2027 may create mechanical merge churn but does not currently supersede this behavior.

Documentation Impact

Update docs/sandboxes/inference-routing.mdx; no gateway TOML fields or driver defaults change.

Revision 1 — current-main implementation plan

This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

Navigation Menu

Repository navigation

feat(sandbox): make L7 inference proxy CHUNK_IDLE_TIMEOUT configurable per route #866

Problem Statement

Proposed Design

Alternatives Considered

Agent Investigation

Checklist

Reproduction

Simpler design than the one I originally proposed

Runtime validation

Branch

Implementation Plan

Summary

Scope

Implementation Steps

Test Plan

Risks & Open Questions

Documentation Impact

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

tokens	elapsed	tok/s	vs 120 s cap
16,000	6 s	2647	5 %
64,000	26 s	2441	22 %
128,000	61 s	2094	51 %
256,000	158 s	1616	132 %
400,000	293 s	1356	245 %
779,685	866 s	901	722 %

feat(sandbox): make L7 inference proxy CHUNK_IDLE_TIMEOUT configurable per route #866

Description

Problem Statement

Proposed Design

Alternatives Considered

Agent Investigation

Checklist

Activity

vnicolici commented on Apr 24, 2026

Reproduction

Simpler design than the one I originally proposed

Runtime validation

Branch

ericksoa commented on Jul 8, 2026

Implementation Plan

Summary

Scope

Implementation Steps

Test Plan

Risks & Open Questions

Documentation Impact

github-actions commented on Aug 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions