Repository navigation
[Bug]: sandbox exec hangs after receiving complete gRPC response — never outputs data, never exits #828
Description
Agent Diagnostic
Investigated using openshell sandbox exec -vvv trace on a DGX Spark host (aarch64). The gRPC exec flow completes server-side, but the CLI never prints stdout and never terminates.
Findings from -vvv trace:
- TLS handshake to local gateway (
127.0.0.1:8080) succeeds, h2 connection established. - The
ExecSandboxgRPC request is sent on streamStreamId(1). - Server responds with HTTP 200 headers (
Headers { stream_id: StreamId(1), flags: END_HEADERS }). - 1,780 bytes of command output are received in a
Dataframe onStreamId(1). - Server sends trailing headers with
END_HEADERS | END_STREAM— the stream transitions toClosed(EndStream). - At this point the gRPC call is fully complete.
- The CLI never writes the received data to stdout and never exits. Instead, the h2 connection stays
Openand enters a keepalive loop, exchangingPINGframes every ~10 seconds indefinitely. - Without an external
timeout, the process hangs until the SSH session times out (exit code 255).
Likely root cause: The client-side exec output streaming code receives the ExecSandboxEvent data frames but does not flush them to stdout or signal process exit after the server closes the stream. The h2 connection teardown / process exit path is not triggered by the stream completing.
Description
openshell sandbox exec hangs indefinitely after the remote command completes. The gRPC response (stdout data + trailing headers with END_STREAM) is received in full, but the CLI never prints output and never exits.
This affects both --tty and --no-tty modes. The --timeout flag on exec does not help (tested with --timeout 15); only an external timeout command or SSH session timeout terminates the process.
The ssh-config + SSH workaround (openshell sandbox ssh-config → ssh -T -F) works correctly for the same commands.
Reproduction Steps
# 1. Verify sandbox is ready
openshell sandbox list
# NAME NAMESPACE CREATED PHASE
# my-assistant openshell 2026-03-22 08:31:11 Ready
# 2. Try exec — hangs forever, no output
openshell sandbox exec -n my-assistant --no-tty -- echo "hello"
# (hangs — no output, never exits)
# 3. Same with --tty
openshell sandbox exec -n my-assistant --tty -- echo "hello"
# (hangs — no output, never exits)
# 4. With --timeout flag (still hangs — timeout flag has no effect)
openshell sandbox exec -n my-assistant --no-tty --timeout 15 -- echo "hello"
# (hangs for 15+ seconds, no output, does not exit)
# 5. Wrapped in external timeout — exits with 124 (killed), still no output
timeout 30 openshell sandbox exec -vvv --no-tty --timeout 15 -n my-assistant -- echo "test"
# (verbose trace shows 1780 bytes received, stream closed, then PING loop until kill)
# 6. SSH workaround works fine
openshell sandbox ssh-config my-assistant > /tmp/sc
ssh -T -F /tmp/sc openshell-my-assistant "echo hello"
# hello ← works instantlyExpected Behavior
openshell sandbox exec -n my-assistant -- echo "hello" should print hello to stdout and exit with code 0, similar to docker exec or kubectl exec.
Actual Behavior
The CLI receives the complete gRPC response (verified via -vvv trace showing 1,780 bytes of data and END_STREAM) but never prints output and never exits. The h2 connection remains open, sending PING keepalive frames every 10 seconds indefinitely.
Environment
- OpenShell version:
openshell m-dev(installed viainstall.sh, which resolved to v0.0.28) - Host OS: Ubuntu (NVIDIA DGX Spark), Linux 6.17.0-1008-nvidia aarch64
- Host architecture: aarch64
- Sandbox image: Default NemoClaw sandbox (created 2026-03-22)
- Gateway: Local (
127.0.0.1:8080) - Install method:
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
Abbreviated -vvv Trace (Key Moments)
# Connection established
2026-04-14T13:55:04 DEBUG connecting to 127.0.0.1:8080
2026-04-14T13:55:04 DEBUG connected to 127.0.0.1:8080
2026-04-14T13:55:04 DEBUG Using ciphersuite TLS13_AES_256_GCM_SHA384
# Request sent, response headers received
2026-04-14T13:55:04 DEBUG send frame=Headers { stream_id: StreamId(1), flags: END_HEADERS }
2026-04-14T13:55:04 DEBUG received frame=Headers { stream_id: StreamId(1), flags: END_HEADERS }
# 1780 bytes of command output received
2026-04-14T13:55:04 DEBUG received frame=Data { stream_id: StreamId(1) }
2026-04-14T13:55:04 TRACE recv_data; size=1780; connection=5242880; stream=2097152
# Stream fully closed by server
2026-04-14T13:55:04 DEBUG received frame=Headers { stream_id: StreamId(1), flags: END_HEADERS | END_STREAM }
2026-04-14T13:55:04 TRACE recv_close: HalfClosedLocal => Closed
2026-04-14T13:55:04 TRACE transition_after; stream=StreamId(1); state=Closed(EndStream)
# CLI does NOT exit — enters PING keepalive loop
2026-04-14T13:55:14 DEBUG send frame=Ping { ack: false, payload: [...] }
2026-04-14T13:55:14 DEBUG received frame=Ping { ack: true, payload: [...] }
2026-04-14T13:55:24 DEBUG send frame=Ping { ack: false, payload: [...] }
2026-04-14T13:55:24 DEBUG received frame=Ping { ack: true, payload: [...] }
# ... repeats every 10s until external kill
Impact
sandbox execis unusable for scripting, CI/CD, and agent tool-use (its primary purpose per feat(cli): addopenshell sandbox execsubcommand #750)- Workaround exists:
openshell sandbox ssh-config+ssh -T -F(used in production by NemoClaw bridges) - The
--timeoutflag onexecalso appears non-functional (does not terminate the command)
Agent-First Checklist
- I pointed my agent at the repo and had it investigate this issue
- I loaded relevant skills (e.g.,
debug-openshell-cluster,openshell-cli) - My agent could not resolve this — the diagnostic above explains why (client-side CLI bug in gRPC response handling)
Activity
mjamiv commentedon Apr 17, 2026
Negative datapoint — not reproduced on x86_64 with host CLI v0.0.29 / v0.0.31
Running 4 sandboxes on a single cluster host (Ubuntu 24.04, Linux 6.8 x86_64, gateway
ghcr.io/nvidia/openshell/cluster:0.0.16, supervisor in-pod v0.0.31 fleet-wide as of 2026-04-17). Host CLI upgraded v0.0.16 → v0.0.29 → v0.0.31 on 2026-04-16.openshell sandbox execis used heavily in production paths:exec)exec)docker cp+ per-pod validation viaexec)Across both v0.0.29 and v0.0.31 host CLI: no hangs observed. Commands stream stdout and exit cleanly with the expected code. No
PINGkeepalive loops.--ttyand--no-ttyboth work. Typical response time 1–3 s for short commands, proportional to output for long ones.Differences from the reporter's setup that may be relevant to scoping:
m-devbuildIf this is useful for bisecting, happy to run targeted exec traces on our fleet.
nv-ddave commentedon May 1, 2026
Re-test on macOS arm64 + OpenShell v0.0.36 — partially fixed, but stdin handling still hangs
Re-ran the original reproducer on macOS arm64 with the latest released OpenShell (
v0.0.36, 2026-04-23) against a local sandbox. Findings: the hang still reproduces when stdin is inherited from a non-closing source, but goes away the moment stdin is closed. That points the bug at the CLI's stdin handling rather than the response/exit path that was originally suspected.Environment
openshell 0.0.36(installed viainstall.sh, latest tagged release)ReadyphaseTest matrix (every call wrapped in a host-side
gtimeoutto detect hangs)openshell sandbox exec -n my-assistant --no-tty -- echo higtimeout 20SIGKILLsopenshell sandbox exec -n my-assistant --tty -- echo hiopenshell sandbox exec -n my-assistant --no-tty -- sh -c 'seq 1 5000'openshell sandbox exec -n my-assistant --no-tty --timeout 5 -- echo hi--timeoutflag does not break the hangopenshell sandbox exec -n my-assistant --no-tty -- echo hi </dev/nullprintf x | openshell sandbox exec -n my-assistant --no-tty -- catecho unused | openshell sandbox exec -n my-assistant --no-tty -- echo ignoring-stdinopenshell sandbox exec -n my-assistant --no-tty -- sh -c 'echo a; echo b 1>&2; exit 42' </dev/nullopenshell sandbox exec -n my-assistant --no-tty -- sh -c 'seq 1 5000' </dev/nullcat /host/file | openshell sandbox exec -n my-assistant --no-tty -- sh -c 'cat > /sandbox/dst'cat | ssh ...upload patternnemoclaw-start ..., ~45 s of output) viaexec ... </dev/nullssh-config+ssh -T -Fbaselineopenshell sandbox exec -n no-such-sandbox --no-tty -- echo hi </dev/nullNotFound(much better than SSH's exit-255 ambiguity)What the data suggests
--timeoutflag still has no effect on the hang (Test 4). External host-side timeouts are still required.Likely fix area
The CLI should signal
END_STREAMon the request side once it detects there's no more stdin to forward — either because the user closed it (</dev/null) or because no stdin was attached interactively. Right now it appears to keep the request half open indefinitely, which keeps the h2 stream alive and prevents the process from exiting after the server closes its half.Operational workaround we're using
Until the stdin handling is fixed, every caller of
openshell sandbox execMUST close stdin explicitly:cmd </dev/nullchild_process.spawnstdio: ['ignore', 'pipe', 'pipe']subprocess.runstdin=subprocess.DEVNULLWith that single discipline, exec is a viable replacement for the
ssh-config+ SSH workaround across our use cases.Happy to run more targeted traces (
-vvv, gRPC-level captures, or a controlled comparison with closed vs open stdin on the same connection) if useful for the fix.github-actions commentedon Aug 30, 2026
This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.