Sandboxes stuck in Error after Podman machine restart — no recovery path #2179

Agent Diagnostic

• Loaded debug-openshell-cluster skill and openshell-cli skill context
• Ran openshell sandbox list — all 4 sandboxes show Error phase after Podman Desktop stopped and restarted the Podman machine
• Ran podman ps -a — all sandbox containers exist in Exited (143) state (SIGTERM from machine shutdown), meaning the containers are recoverable
• Ran brew services restart openshell to trigger resume_persisted_sandboxes() — sandboxes remain in Error
• Inspected crates/openshell-server/src/compute/mod.rs:750 (resume_persisted_sandboxes) — the function skips sandboxes already in Error phase (line 775: if !sandbox_phase_should_be_running(phase) { continue; })
• Inspected sandbox_phase_should_be_running at line 1978 — only Unspecified, Provisioning, Ready, and Unknown phases are considered resumable; Error is excluded
• Root cause: The initial Podman machine restart marked sandboxes as Error (containers exited). On subsequent gateway restarts, the resume sweep skips Error-phase sandboxes, so they can never be recovered — even though the underlying containers still exist and could be restarted
• The agent could not resolve this — there is no CLI command (sandbox restart, sandbox resume) or API to transition a sandbox out of the Error phase without deleting and recreating it

Description

When a Podman machine is stopped (e.g. via Podman Desktop) and restarted, all running sandbox containers receive SIGTERM (exit code 143). The gateway's reconciliation loop or startup resume marks them as Error with reason BackendResourceMissing or similar.

After restarting the Podman machine and the gateway, the sandboxes remain permanently stuck in Error because resume_persisted_sandboxes() explicitly skips sandboxes in Error phase. The underlying Podman containers still exist (Exited (143)) and could be restarted with podman start, but there is no mechanism to bring the gateway's sandbox state back to Ready.

Expected behavior: After a Podman/Docker machine restart followed by a gateway restart, sandboxes whose containers still exist should be recoverable — either automatically by the resume sweep, or via a CLI command like openshell sandbox restart <name>.

Actual behavior: Sandboxes are permanently stuck in Error. The only recovery is openshell sandbox delete + recreate, which loses sandbox state, uploaded files, and configuration.

Reproduction Steps

1. Create one or more sandboxes with a Podman-backed gateway: openshell sandbox create --name test
2. Wait for sandboxes to reach Ready phase
3. Stop the Podman machine (e.g. stop Podman Desktop, or podman machine stop)
4. Restart the Podman machine (podman machine start or restart Podman Desktop)
5. Restart the gateway (brew services restart openshell on macOS)
6. Run openshell sandbox list — all sandboxes show Error
7. Run podman ps -a — containers still exist as Exited (143)
8. Restart the gateway again — sandboxes remain Error (resume sweep skips them)

Environment

• OS: macOS 26.5.2 (Darwin 25.5.0)
• Podman: 6.0.0 (via Podman Desktop)
• OpenShell CLI: 0.0.79-dev.1+gf7aa3aa3
• OpenShell Gateway: 0.0.77 (Homebrew service)
• Compute driver: Podman

Logs

$ openshell sandbox list
NAME          CREATED              PHASE
ai-beacon     2026-07-02 14:48:08  Error
examinator    2026-07-08 08:15:29  Error
s1            2026-07-08 08:42:40  Error
sonarr-drone  2026-07-08 09:49:08  Error

$ podman ps -a --format '{{.Names}} {{.Status}}'
openshell-sandbox-ai-beacon Exited (143) 2 hours ago
openshell-sandbox-examinator Exited (143) 2 hours ago
openshell-sandbox-s1 Exited (143) 2 hours ago
openshell-sandbox-sonarr-drone Exited (143) 2 hours ago

$ openshell logs ai-beacon --since 5m
[1783510768.375] [gateway] [WARN ] [openshell_server::compute] Container exited with code 143

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why