bug(supervisor): a request that waits for a middleware evaluation slot loses its upstream connection #4462

User Story

I use OpenShell to run coding agents (Claude Code and Codex) in sandboxes. The agents act on GitHub through its REST API.
I build supervisor middleware that sends a few risky requests to a person for approval (#4406). Most requests pass at once.
I directly encountered this bug in a load test on v0.1.3: requests that my middleware allows at once failed.
This affects every client of a host with middleware, when some evaluations of that sandbox are slow.

Problem Statement

A sandbox runs at most 32 middleware evaluations at the same time. Other requests wait for a free slot, with no deadline.

For HTTPS, the supervisor opens the upstream connection before the request waits for a slot. While the request waits, that upstream connection stays open and idle. An upstream that closes idle connections closes it. When the slot comes, the request fails with an empty reply.

This also happens to a request that the middleware allows at once. The request never had a slow evaluation. It failed only because it waited in the queue.

Impact / Why This Matters

• One slow middleware evaluation path can break all traffic of a sandbox to every host that the middleware covers.
• The agent sees curl: (52) Empty reply from server, not a middleware error. Nothing tells it to retry.
• The failure depends on the idle timeout of the upstream. api.github.com closes an idle connection after about 5 s. In my tests, queued GitHub requests failed after 7 to 27 s.
• Today the only workaround is to keep every evaluation fast, far below the upstream idle timeout. A middleware cannot safely take more than a few seconds for any request.

Measurements on v0.1.3, one VM sandbox. A burst of N requests to a test host, each held 10 s by the middleware. Three seconds after the burst started, one GET to api.github.com from the same sandbox. The middleware allows that GET at once.

A second sandbox stayed normal in all runs.

Acceptance Criteria

• With 32 evaluations held for 10 s in a sandbox, a request from that sandbox that the middleware allows at once gets the upstream response, not an empty reply.
• The time that a request waits for an evaluation slot does not make the request fail.
• The docs state the evaluation limit per sandbox (32 running, 64 waiting) and what happens to a waiting request.

Reproduction Steps

1. Use the example examples/supervisor-middleware-content-guard. In its HTTP request handler, wait 10 s before it answers when the request host is httpbin.org. Answer at once for all other hosts.
2. Register the middleware in the gateway config with timeout = "30s". Attach it in the sandbox policy to httpbin.org and api.github.com. Allow curl to reach both hosts.
3. In the sandbox, start 32 requests to the slow host in the background:
   for i in $(seq 32); do curl -s -o /dev/null https://httpbin.org/get & done
4. After 3 s, in the same sandbox, run:
   curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://api.github.com/zen
5. Result: curl: (52) Empty reply from server, with a time of 7 s or more.
6. Control: remove the middleware from the policy. Repeat steps 3 and 4. The GET to api.github.com returns 200 in less than 1 s.

My runs used a local HTTP test host on the Mac in place of httpbin.org. That host answered after 10 s. The middleware held its requests for 10 s.

Suggested UX (if applicable)

No change to configuration or API. A request that waited for a slot behaves like a request that did not wait.

Environment

• OpenShell: v0.1.3 (latest release), Homebrew, brew services
• OS: macOS 27.0.1, Apple M2 Pro, 32 GB
• Deployment: local gateway, VM compute driver, one sandbox per test (node:22-bookworm)
• Middleware: operator-run, http://127.0.0.1, allow_insecure_transport = true, gateway timeout = "30s"

Logs

# sandbox, GET to api.github.com during a 32-request burst (middleware attached)
curl: (52) Empty reply from server 000 ms=7074

# same GET during a 90-request burst with no middleware (control)
github 200 ms=437

# supervisor.err.log at 64 or more open connections (related, see note)
WARN openshell_sandbox_backend::runtime: Sandbox network relay streams exhausted; new sandbox TCP connections are waiting for a relay to close

Related note: at 96 open connections, new connections from the same sandbox to any host wait about 7.5 s, with or without middleware (supervisor log above). Because of this limit, a sandbox cannot reach the documented 503 for a full middleware queue (32 + 64): I saw no 503 at 110 requests.