Repository navigation
feat: gateway resume from existing state and persistent SSH handshake secret #487
Description
Problem Statement
When Docker restarts (or the cluster container is lost for any reason), running openshell gateway start offers only two options:
- Destroy everything and recreate — loses all sandboxes, k3s state, etcd data
- Do nothing — pretends the gateway exists but nothing is actually running
There is no path to resume from the existing Docker volume state. Users lose all their sandboxes on every Docker restart.
Additionally, the SSH handshake HMAC secret is regenerated on every container start (in cluster-entrypoint.sh) and injected via sed into the HelmChart CR. This means every container restart invalidates existing sandbox SSH sessions, even though the sandboxes themselves may still be running.
Proposed Design
Gateway Resume
When openshell gateway start is called and existing state is detected (Docker volume exists, but container is stopped or gone), the gateway should automatically resume from that state instead of prompting for destroy/recreate.
New behavior by state:
| State | Current behavior | New behavior |
|---|---|---|
| Running | Prompt "Destroy and recreate?" | Return immediately ("already running") |
| Stopped container | Prompt "Destroy and recreate?" | Auto-resume |
| Volume only (no container) | Prompt "Destroy and recreate?" | Auto-resume |
--recreate flag |
Destroy + redeploy | No change |
Implementation approach:
- Add a
resume: boolflag toDeployOptionsinopenshell-bootstrap - When resuming, skip the
destroy_gateway_resourcescall and fall through to the existing idempotentensure_*calls (ensure_network,ensure_volume,ensure_container,start_container) - The existing
clean_stale_nodeshandles stale k3s node entries from prior containers - The existing
reconcile_pkireuses valid TLS secrets from the volume's k3s state - On resume failure, only clean up the container/network — preserve the volume so the user can retry
- The auto-bootstrap path (
sandbox create) should try resume first, falling back to recreate if resume fails
Docker restart policy:
- Add
unless-stoppedrestart policy to the container'sHostConfigso Docker automatically restarts the container when the daemon comes back gateway stopexplicitly stops the container, which overrides theunless-stoppedpolicy
SSH Handshake Secret Persistence
Move the SSH handshake secret from ephemeral generation in cluster-entrypoint.sh to a proper Kubernetes Secret that persists in etcd (on the Docker volume):
- Create a K8s secret
openshell-ssh-handshakewith asecretkey containing the hex-encoded HMAC key - The bootstrap Rust code reconciles this secret the same way it reconciles TLS PKI — check if it exists, reuse if valid, generate new if missing
- The Helm chart StatefulSet references the secret via
secretKeyRefinstead of a plain value - Remove the sed-based injection from
cluster-entrypoint.shand the__SSH_HANDSHAKE_SECRET__placeholder from the HelmChart CR
Files involved:
| File | Change |
|---|---|
crates/openshell-bootstrap/src/constants.rs |
New SSH_HANDSHAKE_SECRET_NAME constant |
crates/openshell-bootstrap/src/docker.rs |
Restart policy in ensure_container, cleanup_gateway_container function |
crates/openshell-bootstrap/src/lib.rs |
DeployOptions.resume, resume branch in deploy_gateway_with_logs, reconcile_ssh_handshake_secret |
crates/openshell-cli/src/run.rs |
gateway_admin_deploy auto-resume logic |
crates/openshell-cli/src/bootstrap.rs |
Auto-bootstrap resume-first with recreate fallback |
deploy/docker/cluster-entrypoint.sh |
Remove SSH secret generation/sed injection |
deploy/docker/cluster-healthcheck.sh |
Add SSH handshake secret health check |
deploy/helm/openshell/templates/statefulset.yaml |
secretKeyRef instead of plain value |
deploy/helm/openshell/values.yaml |
sshHandshakeSecretName instead of sshHandshakeSecret |
deploy/kube/manifests/openshell-helmchart.yaml |
Remove sshHandshakeSecret placeholder |
tasks/scripts/cluster-deploy-fast.sh |
Use K8s secret instead of Helm values |
Alternatives Considered
-
Only add restart policy, no resume logic — Handles the common Docker restart case but doesn't cover
gateway stopthengateway start, manualdocker rm, or OOM kills. -
Prompt for resume vs recreate — Adds friction to the common case. Since
--recreateexists for explicit clean starts, auto-resume is better UX. -
Store SSH handshake secret on host filesystem — Would survive container restarts but creates a separate state management problem. K8s secrets are already the pattern used for TLS PKI and live in etcd on the persistent volume.
Agent Investigation
Traced the full code path for openshell gateway start:
check_existing_gateway(docker.rs:1030) detects volume/container statedeploy_gateway_with_logs(lib.rs:293) had a binary "destroy or error" check — no resume pathensure_volume(docker.rs:383) is already idempotent (no-op if volume exists)ensure_container(docker.rs:446) handles "no container" by creating one with existing volumeclean_stale_nodes(runtime.rs:376) removes stale k3s NotReady nodesreconcile_pki(lib.rs:798) reuses existing TLS secrets from the volumecluster-entrypoint.shcopies fresh charts/manifests on every start and k3s handles data directory resume
SSH handshake secret traced through 7 stages: generation in entrypoint, sed into HelmChart CR, Helm values, StatefulSet env var, server startup, sandbox pod injection, sandbox SSH verification. The secret was regenerated on every container start, breaking existing SSH sessions.
Definition of Done
-
openshell gateway startauto-resumes from existing volume state (no prompt) -
openshell gateway start --recreatestill destroys and rebuilds from scratch - Container has
unless-stoppedrestart policy - SSH handshake secret persists as K8s secret
openshell-ssh-handshake - SSH handshake secret survives container restart (sandbox SSH sessions remain valid)
- Auto-bootstrap (
sandbox create) tries resume first, falls back to recreate - Resume failure preserves the Docker volume (no data loss on transient errors)
- Health check verifies SSH handshake secret exists
-
cluster-deploy-fast.shcreates/reuses K8s secret instead of Helm values - All existing tests pass
Activity
feat(bootstrap): resume gateway from existing state and persist SSH h…
feat(bootstrap): resume gateway from existing state and persist SSH h…
mjamiv commentedon Mar 27, 2026
Production impact — multi-agent OpenClaw fleet on Mac Minis
We are running 3 OpenClaw agents (Slack-connected, policy-enforced) inside OpenShell sandboxes on dedicated Mac Minis. The agents handle project work daily — document review, RFP analysis, engineering support.
Every Docker restart destroys all sandbox state. Config, workspace, agent memory, session history, cron jobs — gone. A Mac reboot means 10-15 minutes of manual re-setup per agent: gateway start, sandbox create, CoreDNS patch, policy apply, WS proxy patch upload, OpenClaw bundle upload, config restore, gateway start, Slack re-pair.
Workaround: automated backup/restore
We built a host-side backup/restore layer:
.openclaw/state + workspace viaopenshell sandbox downloadgateway start→sandbox create→ CoreDNS patch → policy apply → state upload → gateway start. ~3-5 min, unattended.It works, but fights the ephemeral design at every step. CoreDNS needs re-patching on every gateway recreate. Proxy patches need re-uploading. Policies need re-applying. The full K3s + etcd + pod lifecycle restarts from scratch.
Why this issue matters
Gateway resume with persistent state would eliminate the rebuild cycle entirely. If
openshell gateway startdetected existing Docker volume state and resumed instead of destroy/recreate, agents would survive reboots natively. SSH handshake persistence is equally critical — without it, every restart invalidates sandbox SSH sessions even when the pod survives.Suggested addition:
openshell sandbox backup/restoreEven with gateway resume, full sandbox rebuilds are sometimes needed (volume corruption, machine migration). First-class
openshell sandbox backupandopenshell sandbox restoreCLI commands would close the gap. We have working scripts that could serve as a reference implementation — happy to contribute upstream.Environment
This is the #1 blocker for production deployment. Everything else works.
cloudant42 commentedon Mar 29, 2026
Similar issue here, but in a smaller local Mac setup intended as a minimal and non-invasive client-like environment.
In our case, gateway/sandbox persistence after restart is the main operational problem. We can work around it with backup/restore scripts, but that is still tedious and disruptive for a day2day setup.
C-Bjorn commentedon Mar 29, 2026
Adding a production data point to support prioritizing this.
We're running multiple NemoClaw sandboxes on a dedicated Linux host (several agents per sandbox, accumulated memories in Neo4j, MCP integrations). After a power failure last week, every sandbox was unreachable due to the SSH handshake secret regeneration. Recovery took ~24 hours because the agent tasked with fixing it had no documentation on this behavior, saw
handshake verification failed, and concluded it needed to reinstall everything from scratch.The actual correct recovery — delete+recreate sandbox pods, restore from host-side config backups — works, but it's fragile: it requires a 30-minute backup window, manual token re-entry for any secrets not in the backup, and a npm reinstall inside each sandbox. As you scale to more sandboxes and more agent state, the blast radius of a reboot grows linearly.
The proposed fix (reconcile the secret into etcd like TLS PKI, reference via
secretKeyRef) is the right architecture. A few things we validated via source review that might be useful:SSH_HANDSHAKE_SECRET:-$(...)default pattern incluster-entrypoint.shalready supports override via env var, which means the persistence path is additive — you can reconcile the secret into a K8s Secret first, then pass it as an env var to the entrypoint, without changing the injection mechanism on day one.openshell-server(confirmed viagrpc.rsandhttp.rs) — pod recreation is the only recovery path today, which makes this fix load-bearing for any reliability story around reboots.identity_driftstate in NemoClaw's CLI currently directs users tonemoclaw onboardas the recovery action. That's destructive — it wipes all sandbox state. Once feat: gateway resume from existing state and persistent SSH handshake secret #487 ships,identity_driftshould become rare enough that the error message can safely say "contact support" rather than suggesting a nuclear option. Worth coordinating with the NemoClaw team (related: feat(cli): add nemoclaw reconnect command for post-reboot recovery NemoClaw#910, feat(cli): add reconnect command NemoClaw#960).Happy to test a build or provide more detail on the failure scenario.
Mirai1129 commentedon Mar 30, 2026
Same issue here, a Linux server reboot wipes out all services and sandboxes, forcing me to re-upload my settings into the sandbox again and again. Hoping for a quick fix or at least a recommended workaround to persist the sandbox state.
feat(bootstrap): resume gateway from existing state and persist SSH h…
vnicolici commentedon Apr 2, 2026
This issue is very serious, as it can cause data loss. Personally I lost many hours already due to this issue. I had to recreate my sandboxes probably around 10 times from scratch until now, as my DGX Spark receives updates that require reboots almost daily.
Because of this issue, I lost the OpenClaw memory and other files stored inside the sandboxes a few times already, and some work I was doing in other sandboxes. Not the end of the world, but not optimal either.
As a workaround, today I implemented cron scripts that automatically back up the sandbox filesystems on the host machine every minute. Then I attempted another reboot, to see what happens.
On reboot, the container didn't start automatically. Upon starting it manually, the sandboxes were inaccessible. After looking at the logs and discovering the
handshake verificationerrors, I then searched this repo and finally discovered this is a known issue.In my opinion, due to the severity and the data loss potential, and especially since this is not a documented behavior, this is not actually a missing feature, but a bug.
I certainly understand this is currently an alpha, and such things are to be expected, and from what I see there is work in progress to fix this issue, which is great. However, the issue not being mentioned in the documentation, despite being known for 2 weeks at this point, is not that great.
Until this issue is fixed, I think a warning about this issue should be included in the documentation for both OpenShell and NemoClaw. This way, new users will know what to expect, and may be able implement workarounds to prevent data loss, not to mention time loss.