Repository navigation
bug(proxy): inference.local buffers entire streaming response, inflating TTFB #260
Description
Summary
The sandbox proxy's inference.local interception path fully buffers upstream streaming responses (SSE) before sending any bytes back to the client. This converts a streaming response (fast TTFB, incremental tokens) into a buffered response (slow TTFB, instant completion), causing downstream clients with TTFB timeouts to abort before the proxy finishes buffering.
Actual Behavior
- Client sends
POST /v1/chat/completionswith"stream": truetoinference.local - Proxy forwards to upstream (e.g. NVIDIA NIM endpoint), which begins streaming SSE events (TTFB ~200ms)
response.bytes().awaitinproxy_to_backend()blocks for the full generation time (10-60s)- Proxy sends nothing back to the client during this entire period
- Client's TTFB/first-event timeout fires and aborts the request
- Proxy eventually finishes buffering and tries to write to a closed connection
Expected Behavior
The proxy should forward response headers and SSE chunks to the client incrementally as they arrive from the upstream, preserving the streaming semantics and sub-second TTFB.
Root Cause
Three layers conspire to prevent streaming:
navigator-router/src/backend.rs:112-115—response.bytes().awaitcollects the entire upstream response body into memory before returningProxyResponsestruct (backend.rs:9-13) — stores body as a singlebytes::Bytesblob with no streaming abstractionformat_http_response()(l7/inference.rs:244-246) — replaces upstreamTransfer-Encoding: chunkedwithContent-Lengthbased on buffered body size, then writes everything in a singlewrite_allcall
Affected Path
Only the HTTPS/CONNECT inference interception path (inference.local) is affected. The plain HTTP forward proxy path uses copy_bidirectional and streams correctly.
Call chain: handle_inference_interception() → route_inference_request() → proxy_with_candidates() → proxy_to_backend() → response.bytes().await
Fix Direction
- Add a streaming variant of
proxy_to_backend()that returns headers +impl Stream<Item = Bytes>instead of a fully-bufferedProxyResponse - Have
route_inference_request()write response headers immediately, then forward body chunks incrementally to the TLS client - For streaming responses, emit
Transfer-Encoding: chunkedinstead ofContent-Length
Activity
johntmyers commentedon Mar 12, 2026
Implementation Plan
Issue type:
fixComplexity: Medium
Confidence: High — clear path, all APIs exist
Summary
The inference proxy path buffers the entire upstream response via
response.bytes().awaitbefore sending anything to the client. The fix adds a streaming variant of the proxy backend that returns response headers immediately and streams body chunks incrementally using HTTP chunked transfer encoding. Non-streaming (buffered) responses continue to work as before.Scope
crates/navigator-router/src/backend.rs: AddStreamingProxyResponsetype andproxy_to_backend_streaming()that returns headers +reqwest::Response(for incremental.chunk()reads) instead of buffering the entire bodycrates/navigator-router/src/lib.rs: Addproxy_with_candidates_streaming()method onRouter, export new typescrates/navigator-sandbox/src/l7/inference.rs: Addformat_http_response_header()(status line + headers for chunked TE),format_chunk(), andformat_chunk_terminator()helperscrates/navigator-sandbox/src/proxy.rs: Updateroute_inference_request()to use the streaming path — write headers first, then stream body chunks via chunked TEImplementation Steps
StreamingProxyResponseandproxy_to_backend_streaming()inbackend.rs— extract shared request-building into a helper to avoid duplication with the existing buffered pathproxy_with_candidates_streaming()onRouterinlib.rs, export new typesinference.rs(format_http_response_header,format_chunk,format_chunk_terminator)route_inference_request()inproxy.rsto use streaming: write headers immediately, loop onresponse.chunk(), write each chunk in chunked TE framing, terminateTest Plan
inference.rs,build_backend_requestextraction inbackend.rsproxy_to_backend_streaming()against wiremock (already a dev-dependency)examples/local-inference/inference.pyscript serves as the acceptance test (TTFB << total time when streaming)Risks & Open Questions
reqwest::Responseis notDebug—StreamingProxyResponsecannot deriveDebug(minor; use manual impl or skip)proxy_to_backend()(buffered) stays for mock routes which are inherently non-streamingDocumentation Impact
Revision 1 — initial plan
fix(proxy): stream inference responses instead of buffering entire body
johntmyers commentedon Mar 12, 2026
Implementation Complete
PR: #261
What was built
Added a streaming proxy path for
inference.localthat writes response headers immediately and forwards body chunks incrementally using HTTP chunked transfer encoding, replacing the previousresponse.bytes().awaitbuffering that inflated TTFB.Tests
examples/local-inference/inference.pywith TTFB instrumentationDocs updated
examples/local-inference/README.md— updated with NVIDIA provider quick startThe issue will auto-close when the PR is merged.
fix(proxy): stream inference responses instead of buffering entire bo…
fix(proxy): stream inference responses instead of buffering entire body
fix(proxy): stream inference responses instead of buffering entire bo…