Sandbox stuck in Phase: Error despite healthy container and working gateway RPC channel #3440

User Story

As an operator running a production NemoClaw/OpenShell sandbox with a live messaging
channel (Telegram) attached, after the gateway's log-push RPC connection to a sandbox
is disrupted and the underlying container later recovers on its own, I need the gateway
to correctly re-recognize the sandbox as Ready, so that the web dashboard and CLI-based
recovery commands remain usable for that sandbox without requiring a destructive
destroy + re-onboard.

Problem Statement

A sandbox (harmony) is permanently stuck in Phase: Error as reported by
openshell sandbox list and nemoclaw <name> status, even though:

• The underlying Docker container is running and healthy (docker ps shows
  Up ... (healthy))
• The sandbox is actively serving real traffic (Telegram bot polling getUpdates
  every ~30s and successfully processing real conversations via GPT-5.4 inference)
• The gateway's gRPC channel to the sandbox (127.0.0.1:8080) shows healthy
  ESTABLISHED connections with no drops or timeouts
• nemoclaw doctor reports every other check as ok (Docker daemon, OpenShell CLI,
  gateway binding, gateway connection, inference route)

Both nemoclaw <name> start and nemoclaw <name> gateway restart fail to clear the
Error phase, and no combination of these commands re-triggers a successful readiness
check against the already-running, already-healthy container.

Suspected root cause / trigger event:

The sandbox was healthy prior to a host-level event: .wslconfig was edited (setting
a memory cap) and wsl --shutdown was run to apply it. This interrupted the
sandbox↔gateway internal log-push RPC connection:

log push RPC failed ... peer closed connection without sending TLS close_notify

This connection never re-established on its own, and the sandbox's underlying Docker
container eventually crashed (exit 255). After a later docker start <container>
brought the container back up (Docker-level health: healthy, service resumed
normally), the OpenShell/NemoClaw state layer never re-synced — it appears the
gateway's own reconciliation/recovery scan may have run at a moment when the
container was down, latched the sandbox into Error, and does not re-check
sandboxes already marked Error.

This looks similar in nature to #1765 ("fix: resume unspecified sandbox phase") —
a gateway-restart-time reconciliation scan misclassifying a sandbox state if it runs
during a window when the container isn't up yet, and not re-scanning sandboxes
already in Error.

Impact / Why This Matters

• The messaging service itself is unaffected — the sandbox works, real customer
  conversations are being handled correctly throughout.
• However, the OpenClaw web dashboard is completely unusable for this sandbox,
  because it depends on the sandbox being recognized as Ready. Any
  dashboard-dependent workflow (monitoring, manual intervention, session
  management via the web UI) is blocked.
• CLI-based recovery paths (start, gateway restart, recover) do not clear the
  stuck state, and there is no documented CLI command to force a phase re-check or
  manually clear Error short of destroying and rebuilding the sandbox (destroy +
  re-onboard), which is high-risk for a production sandbox and was avoided.
• There is currently no accessible log output for the gateway's internal decision
  process (stdout/stderr for the gateway process point to an internal socket, not a
  file; no systemd journal entries; no *.log/*.jsonl files found under
  ~/.local/state/nemoclaw), making this hard to self-diagnose further.
• Because the sandbox is marked Error, downstream doctor checks that depend on
  reaching it through the named gateway are also skipped (Inference route (gateway), Runtime channel registry), reducing the diagnostic visibility into
  the very state that needs debugging.

Acceptance Criteria

• A sandbox whose container has recovered to Up (healthy) after an interruption
  is correctly re-recognized as Ready without requiring destroy + re-onboard.
• A CLI command (or flag on start/recover) exists to force the gateway to
  re-run its readiness/reconciliation check against a sandbox currently in Error,
  independent of whatever cached decision put it there.
• The gateway's internal readiness/reconciliation decision (the one that currently
  causes nemoclaw start / gateway restart to wait ~41–59s and then re-latch to
  Error) is surfaced in --json/verbose output, so the failing check is
  identifiable without needing gateway-internal log access.

Reproduction Steps

1. Have a sandbox running normally under NemoClaw/OpenShell with a live messaging
   channel (Telegram) attached.
2. Cause an abrupt disruption to the gateway↔sandbox log-push RPC connection (in
   our case: wsl --shutdown while the sandbox was active).
3. Underlying container eventually exits.
4. docker start <container> brings the container back to Up (healthy), and the
   in-container service resumes working correctly (verified via real Telegram
   conversations, real GPT-5.4 inference calls in logs).
5. openshell sandbox list continues to report Phase: Error for this sandbox
   indefinitely.

Note: exact trigger not confirmed reproducible on demand — observed once via the
above sequence.

Diagnostics performed (ruling out other causes):

• docker ps -a: Up ... (healthy), confirmed via repeated checks over time
• docker inspect (RestartCount, StartedAt/FinishedAt): manually restarted via
  docker start; no crash loop
• nemoclaw <name> doctor --json: all checks ok except Sandbox / Live sandbox
  = fail, detail: "harmony present (Error)"
• Docker daemon health (docker version, docker info): fully healthy
• Gateway process (ps aux): single long-running background process (not
  systemd-managed), stable, no crash/restart
• lsof -p <gateway pid>: only holds fds on its own SQLite state db; fuser on
  that db shows no other process holding a lock
• nemoclaw <name> gateway restart: reports gateway health passed but fails with
  Warning: OpenShell ForwardTcp 18789 for harmony did not start: OpenShell forward service exited before binding 0.0.0.0:18789 (status 1) — Failure layer: forward recovery failure
• Port 18789: confirmed free via lsof -i :18789 and ss -tlnp — not a port
  conflict
• gRPC channel to sandbox, port 8080: ss -tlnp/ss -tnp show healthy
  ESTABLISHED connections from the gateway process to itself via 127.0.0.1:8080
  (routed into sandbox netns) — no drops, no timeouts
• nemoclaw <name> start: detects container is already running, attempts to
  "restore sandbox startup state," waits ~41–59s, then reports:
  Sandbox 'harmony' entered 'Error' state
• In-sandbox logs (nemoclaw <name> logs --follow): show completely normal
  operation — policy load, network namespace setup, Landlock rules applied, live
  Telegram getUpdates polling every ~30s, successful GPT-5.4 inference calls,
  successful sendMessage

Environment

• nemoclaw: v0.0.124
• openshell CLI: 0.0.116
• OpenClaw (in-sandbox agent): 2026.7.1 (base image), upgraded to 2026.9.4 via
  plugin work in a separate sandbox
• Docker Desktop: 29.8.0 (server), Docker Desktop 4.91.0 (239619)
• Host: WSL2 Ubuntu, NVIDIA GeForce RTX 5060 Ti
• Gateway process: /home/abt/.local/bin/openshell-gateway, run as a plain
  background process (not systemd-managed)

Logs

log push RPC failed ... peer closed connection without sending TLS close_notify

Warning: OpenShell ForwardTcp 18789 for harmony did not start: OpenShell forward
service exited before binding 0.0.0.0:18789 (status 1)
Failure layer: forward recovery failure

{
  "schemaVersion": 1,
  "sandbox": "harmony",
  "status": "fail",
  "failed": 1,
  "warnings": 0,
  "checks": [
    { "group": "Host", "label": "CLI build", "status": "ok", "detail": "dist/nemoclaw.js present" },
    { "group": "Host", "label": "Docker daemon", "status": "ok", "detail": "server 29.8.0" },
    { "group": "Host", "label": "OpenShell CLI", "status": "ok", "detail": "/home/abt/.local/bin/openshell" },
    { "group": "Gateway", "label": "Registered gateway binding", "status": "ok", "detail": "resolved to 'nemoclaw'" },
    { "group": "Gateway", "label": "OpenShell status", "status": "ok", "detail": "connected to nemoclaw" },
    { "group": "Sandbox", "label": "Live sandbox", "status": "fail", "detail": "harmony present (Error)", "hint": "run `nemoclaw harmony status` or `nemoclaw harmony logs --follow`" },
    { "group": "Inference", "label": "Route", "status": "ok", "detail": "openai-api / gpt-5.4" },
    { "group": "Inference", "label": "Inference route (gateway)", "status": "info", "detail": "skipped because the sandbox is not reachable through its named gateway" },
    { "group": "Inference", "label": "Provider health (upstream)", "status": "info", "detail": "OpenAI health requires OPENAI_API_KEY; skipping model-invocation probe instead of reporting endpoint reachability as healthy." },
    { "group": "Inference", "label": "Serving process", "status": "info", "detail": "not checked — serving-process probing is not implemented" },
    { "group": "Sandbox", "label": "Agent version", "status": "ok", "detail": "OpenClaw v2026.7.1" },
    { "group": "Sandbox", "label": "Lifecycle registration", "status": "ok", "detail": "registry entry has lifecycle metadata for snapshot, rebuild, upgrade, recovery, and reboot" },
    { "group": "Sandbox", "label": "Config permissions", "status": "ok", "detail": "runtime config permission contract verified" },
    { "group": "Messaging", "label": "Channels", "status": "info", "detail": "telegram enabled; telegram inbound delivery is not inferred from conflict signatures", "hint": "run `nemoclaw harmony channels status --channel telegram` to probe inbound delivery" },
    { "group": "Messaging", "label": "Runtime channel registry", "status": "info", "detail": "skipped because the sandbox is not reachable through its named gateway", "hint": "fix the gateway and live sandbox checks, then rerun `nemoclaw harmony doctor`" },
    { "group": "Local services", "label": "Ollama", "status": "info", "detail": "not reachable or invalid response at http://127.0.0.1:11434/api/tags" },
    { "group": "Local services", "label": "cloudflared", "status": "info", "detail": "stopped", "hint": "no cloudflared process; run `nemoclaw tunnel start` to start it" }
  ]
}