Repository navigation
sandbox exec hangs indefinitely (phantom AND healthy containers); sandbox list reports Ready for containers that don't exist, across gateway restarts #2733
Description
Summary
Two related behaviors observed under load on 2026-08-12 (OpenShell sandboxes backing an agent pipeline, Docker backend, NixOS host):
- The sandbox ledger lists
Readyentries for containers that do not exist, and the ledger survives gateway restarts, so the lie persists. openshell sandbox exechangs indefinitely instead of erroring — both against those phantom entries and, intermittently, against healthy containers.
Evidence
Phantom Ready entries. After several sandboxes' containers were removed (docker rm -f), and after killing and respawning openshell-gateway, openshell sandbox list still showed all four as Ready:
openclaw-agent-dev-issue-42-c1432479 ... Ready
openclaw-agent-dev-issue-42-r1786573300-138434ac ... Ready
openclaw-agent-dev-issue-42-r1786574308-137f6703 ... Ready
openclaw-agent-dev-issue-42-r1786575103-1380f2eb ... Ready <- container never existed in docker ps
For the last entry, docker ps -a had no matching container at any point we observed — Ready with no referent.
Exec hangs, phantom case: openshell sandbox exec -n <phantom> -- true blocked >9 minutes with no output and no error until externally killed.
Exec hangs, healthy case: later the same evening, sandbox exec -n <name> -- true against a container that was demonstrably fine — docker exec <same container> true returned instantly — blocked 27+ minutes. So the hang is in the CLI→gateway→container path, not the container.
Gateway process ages during observations: one 5.5 days old, and a freshly respawned one — behavior identical, so this is not stale-daemon-specific.
Why it hurts
An automation pipeline that provisions a sandbox per task uses exec -- true as its readiness probe. A probe that can hang forever converts every downstream timeout into a mystery: the pipeline's failure surfaces minutes later, attributed to whatever was waiting. A bounded error ("no such container", or any timeout) would have turned three separate multi-minute incidents into immediate, correctly-attributed failures.
Asks
sandbox execshould fail fast when the backing container is absent, and support/observe a client-side timeout rather than blocking indefinitely.sandbox listshould reconcile against the backend (or mark entries whose container is gone) rather than reporting persisted state asReady.
Happy to gather gateway logs or traces on the next occurrence — the setup reproduces this under load and is instrumented.
Activity
krishicks commentedon Aug 13, 2026
Agent triage, including request for info:
Analysis
Verdict: credible bug report, but it currently combines two potentially unrelated failures and lacks enough evidence to validate either root cause. I would classify it as needs information before rewriting.
1. Persisted Ready sandbox without a container
This would be a bug if the CLI and gateway are using the same Docker daemon.
The current contract says Ready means both:
The gateway also runs an immediate reconciliation sweep at startup and then every 60 seconds. For an old store record absent from the Docker driver snapshot, it rechecks the backend and removes the record. Therefore a missing container remaining Ready across a gateway restart contradicts intended behavior.
However, we need to establish:
docker ps -a \ --filter label=openshell.ai/sandbox-id=<sandbox-id>Without this, a daemon/context mismatch remains plausible.
2. sandbox exec hanging
This is likely a separate issue and may duplicate #828/#1046.
The CLI currently reads non-terminal stdin to EOF before making the exec RPC. If an automation process supplies an inherited pipe that remains open, even exec -- true can block indefinitely before contacting the gateway. The known discriminating test is:
Other relevant facts:
We need to know whether:
rocket-turkey commentedon Aug 13, 2026
Requested info, plus one retraction and one confirmation.
Environment: CLI 0.0.85, gateway 0.0.85, same install for the supervisor. Single Docker daemon, default context,
unix:///var/run/docker.sock; the gateway user is in thedockergroup and every query below hit that same socket. Host is NixOS, gateway runs under systemd.1. The "phantom Ready" claim: retracted. Your label-based lookup found the container immediately:
All four
Readyentries from the report have live containers today, same daemon. The mechanism of our error: container names are decorated asopenshell-<sandbox-name>-<sandbox-id>, and our Aug-12 search keyed on the bare sandbox name in a way that missed them. "Ready with no referent" was our search failing, not your ledger lying — the ledger was right. Withdrawn.2. The exec hang: confirmed client-side, pre-RPC, exactly as you describe. Against a healthy
Readysandbox, from the same shell, minutes apart:openshell sandbox exec -n <name> --no-tty -- true </dev/null→ returns immediately, rc 0.openshell sandbox exec -n <name> --timeout 15 -- truewith an inherited-pipe stdin → hung indefinitely; killed by an outertimeout 30.--timeout 15did not bound it.ExecSandboxfor this sandbox (the successful one). The hung attempt never produced an RPC — consistent with the CLI blocking on stdin read-to-EOF before contacting the gateway, which also explains why--timeoutcannot help: the command phase is never reached.So: likely duplicate of #828/#1046 as you suggested; the automation context (an agent pipeline where execs inherit long-lived pipes) makes the default read-to-EOF behavior a reliable hang rather than an intermittent one. Happy to close this issue in favor of those, with the
--timeout-cannot-bound-the-stdin-phase observation carried over — that part may still be worth a docs note or a bounded pre-RPC read.github-actions commentedon Aug 28, 2026
This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
johntmyers commentedon Sep 17, 2026