Repository navigation
bug(driver-podman): per-sandbox secrets and volumes are orphaned when a container disappears without DeleteSandbox #2352
Description
Agent Diagnostic
Reproduced live against main at 8cf2673c (2026-07-17, the commit behind release v0.0.86), with the gateway, CLI, Podman driver, and openshell/supervisor:dev image all rebuilt from that commit so no branch code was involved.
- Traced the create path (
PodmanComputeDriver::create_sandbox,crates/openshell-driver-podman/src/driver.rs) and the store-side removal paths (apply_deleted_locked,prune_missing_sandbox,crates/openshell-server/src/compute/mod.rs). - Ran a standalone Podman-backed gateway (
mise run gateway), created a sandbox, removed its container out of band, and observed what survived. - Confirmed no reconciliation exists for either resource family:
git grep list_secrets/reconcile_orphanedincrates/openshell-driver-podman/returns nothing onmain. - Searched open and closed issues for
orphaned secret,secret reconciliation,podman secret cleanup,openshell-token,secret leak: no matching issue. The search was full-text, so an issue using different vocabulary could have been missed. - Corroborating evidence that this happens in normal use, not just under a synthetic trigger: before starting, this machine had four
openshell-token-*secrets, ~10 days old, with zero containers present. They were removed to get a clean baseline for the reproduction below. - Surfaced during review of feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy #2245, which adds a third per-sandbox secret with the same lifecycle. That PR's family is deliberately out of scope here — see the Scope note.
Description
The Podman driver creates two per-sandbox resources before it creates the container, and removes them in exactly two places: the create path's own error cleanup, and delete_sandbox. Neither runs if the container disappears without a DeleteSandbox call — whether because the gateway died mid-create, or the container was removed out of band.
Affected resources on main:
- Workspace volume — created at
driver.rs:509, removed only in create-error cleanup (driver.rs:515,528,543,557,564,580) anddelete_sandbox(driver.rs:672). openshell-token-*secret — created atdriver.rs:101, removed only in those same paths anddelete_sandbox(driver.rs:681).
When the container goes missing, the server removes its store record through apply_deleted_locked (compute/mod.rs:1381) without ever asking the driver to clean up. Two paths reach it:
- The watch path — the Podman watcher emits a deleted event,
apply_deletedfires immediately, no grace period. - The prune sweep —
prune_missing_sandbox(compute/mod.rs:1476-1517) afterORPHAN_GRACE_PERIOD(300s).
apply_deleted_locked calls cleanup_sandbox_owned_records, which deletes only store objects (SSH sessions, sandbox settings). Nothing in either path touches Podman. The record naming the sandbox disappears while its volume and secret persist, leaving a bearer token and a workspace volume with nothing left in the system to attribute them to.
For the token secret this is a leaked credential (CWE-459, missing release of resource after effective lifetime). For the volume it is unbounded disk growth.
Expected: per-sandbox resources whose container no longer exists are reclaimed — either by the driver reconciling against live container IDs, or by the removal path asking the driver to clean up.
Scope note
PR #2245 adds a third family, openshell-proxy-auth-*, with the same lifecycle and a higher-value credential. It is not in scope here because it is not on main yet, but a fix should cover it too rather than be redone later. See #2245 (comment) for the scope discussion.
Reproduction Steps
Verified end to end on main at 8cf2673c.
- Start a standalone Podman-backed gateway:
mise run gateway - Create a sandbox:
openshell sandbox create --name orphan-repro-main - Note the per-sandbox resources:
podman secret ls,podman volume ls - Remove the container out of band:
podman rm -f openshell-sandbox-orphan-repro-main - Wait ~10 seconds, then:
openshell sandbox list(empty),podman secret lsandpodman volume ls(both still present)
Observed timeline, sandbox ID 1eadfcbe-ad00-4ef9-9c95-1f92be936441:
| Time (UTC) | Event |
|---|---|
| 16:32:37 | CreateSandbox — secret, volume, container created |
| 16:35:38 | podman rm -f (out-of-band removal) |
| 16:35:48 | Watcher: "Container already removed when inspecting after event, emitting deleted event" |
| 16:35:48 | Store record deleted via apply_deleted_locked |
| 16:37:41 | Secret and volume still present |
| 16:39:44 | Still present, past the 300s grace and ~4 reconcile sweeps; zero cleanup log lines |
The same end state is reachable by killing the gateway between secret creation and container creation (driver.rs:509-580), which additionally strands the volume created moments earlier.
Environment
- OS: macOS (Darwin 25.5.0), Apple Silicon; the affected code is platform-independent
- Podman: 5.8.1, rootless,
podman machine(libkrun) - OpenShell:
mainat8cf2673c, matching release v0.0.86 — gateway, CLI, driver, and supervisor image all rebuilt from this commit - Latest release checked: yes — v0.0.86 is latest and contains the affected code
- Possible duplicates checked: yes — none found (see diagnostic for terms and caveat)
Logs
2026-07-19T16:35:48.490839Z INFO openshell_driver_podman::watcher: Container already removed when inspecting after event, emitting deleted event container_id=a9f3aafedbcd… action=stop
2026-07-19T16:35:48.491922Z INFO openshell_driver_podman::watcher: Container already removed when inspecting after event, emitting deleted event container_id=a9f3aafedbcd… action=die
$ openshell sandbox list
No sandboxes found.
$ podman secret ls --format '{{.Name}}'
openshell-token-1eadfcbe-ad00-4ef9-9c95-1f92be936441
$ podman volume ls --format '{{.Name}}'
openshell-sandbox-1eadfcbe-ad00-4ef9-9c95-1f92be936441-workspace
Activity
elezar commentedon Jul 20, 2026
One question I would have here is whether this indicates a general driver-agnostic gap in the existing gateway <-> driver API?
feloy commentedon Jul 20, 2026
Yes — this is a driver-agnostic gap in the gateway↔driver API, not just a Podman-specific bug.
The
ComputeDrivergRPC service has no cleanup or reconciliation hook. The only way to release driver-side resources isDeleteSandbox, and the server never calls it when a sandbox disappears out-of-band.apply_deleted_lockedandprune_missing_sandboxonly clean the store — they never reach back into the driver.Every driver is affected:
A durable fix should live at the API boundary with two complementary pieces:
Reactive cleanup — add a
CleanupSandboxResourcesRPC (or makeDeleteSandboxidempotent/best-effort) thatapply_deleted_lockedcalls into the driver before dropping the store record.Startup/crash-recovery reconciliation — add a
ReconcileResourcesorListOwnedResourcesRPC so the server's periodic sweep can ask the driver "what resource families do you still have?" and prune anything that no longer maps to a known sandbox.Option 1 alone doesn't cover gateway crashes mid-create (the store record may never have existed), so both are needed. The VM driver's existing
cleanup_stale_tap_interfacesat startup is a driver-local version of option 2 — the need has been felt before but solved ad-hoc rather than at the trait level.krishicks commentedon Aug 14, 2026
politerealism commentedon Aug 28, 2026
I'd like to pick this up. Building on @feloy's driver-agnostic diagnosis above, I'll implement the two RPCs proposed in this thread at the
ComputeDriverservice boundary rather than a Podman-only patch:CleanupSandboxResources(or an idempotentDeleteSandbox) — called fromapply_deleted_lockedbefore the store record is dropped, so reactive cleanup happens driver-side instead of only in the store.ReconcileResources/ListOwnedResources— called from the periodic sweep so the server can ask each driver what it still holds and prune anything unmapped, covering crash-recovery cases where the deletion event itself was missed.I'll implement this for the Podman driver first, and design the contract so Docker, VM, and Kubernetes can adopt it as explicit follow-up work rather than needing a second redesign later.
One update to scope: #2245 has since merged, so the
openshell-proxy-auth-*secret family mentioned in the scope note above is no longer hypothetical — I checkedcrates/openshell-driver-podman/src/driver.rsdirectly and confirmedcleanup_sandbox_proxy_auth_secrethas the identical gap (only called from create-error cleanup anddelete_sandbox, never on out-of-band removal). So Podman's fix needs to reconcile three resource families, not two: workspace volume, token secret, and proxy-auth secret.Also worth noting: #2591 (server-managed expiration leases) explicitly defers to "driver-specific orphan cleanup" for resources left behind after a sandbox record is lost — so this reconciliation primitive already has a second consumer beyond this issue, which makes the case for the durable API-boundary fix stronger rather than a one-off patch.
Considered a Podman-only stopgap first, but decided against it — it'd be throwaway work once the real contract lands, and I'd rather put review effort into the durable fix directly. Happy to adjust if there's a reason to want an interim fix while the API design gets reviewed.
politerealism commentedon Aug 31, 2026
Update to my plan above — I traced the actual code paths before implementing and found a smaller, better fix than the two-RPC design I proposed.
DeleteSandboxis already idempotent for Podman and already does the full cleanup.crates/openshell-driver-podman/src/driver.rs'sdelete_sandboxalready reclaims the workspace volume, token secret, and proxy-auth secret even when the container is already gone — it doesn't error, it just cleans up and returnsdeleted: false. There's an existing test proving this (delete_sandbox_cleans_up_volume_when_container_is_already_gone).The actual bug is entirely server-side: I traced both places that discover a sandbox is gone out-of-band — the watch-path (
apply_deleted_locked) and the periodic sweep (remove_sandbox_record_if_version_locked, reached viaprune_missing_sandbox) — and neither one ever calls the driver'sDeleteSandboxRPC. Both only clean up store-side objects, through a shared helper,cleanup_sandbox_owned_records. The explicit, user-initiated delete path already callsdriver.delete_sandboxcorrectly and doesn't go through that helper, so this isn't a regression risk for normal deletes.Revised fix: add one
self.driver.call("driver.delete_sandbox", ...)invocation insidecleanup_sandbox_owned_records, using the same call pattern already used twice elsewhere in this file, made best-effort (log and continue on failure, since these paths already believe the sandbox is gone). No proto changes, no new RPCs.This also covers the mid-create crash reproduction, not just the out-of-band-removal one: the server writes the sandbox's store record via
put_ifbefore ever calling the driver'sCreateSandbox(compute/mod.rs:961-1015), so even in the "gateway killed between secret creation and container creation" case, a store record already exists. The existing periodic sweep already walks every store record and queries the driver viaGetSandbox— onceDeleteSandboxgets wired into the cleanup path that sweep already reaches, both reproduction scenarios close.Deliberately deferring the
ReconcileResources/ListOwnedResourcespiece. That RPC would matter for a genuinely different failure mode — a driver resource with no store record at all to reconcile against (e.g. the store record itself was lost or deleted outsideDeleteSandbox). That's real defense-in-depth, but it's not what this issue reproduces, and I'd rather not build it speculatively. Happy to open a follow-up issue for it if anyone has a concrete case where it's needed.politerealism commentedon Aug 31, 2026
Opened #3042 implementing the revised plan above. Also filed #3041 for a related-but-separate gap in the Docker driver's own
DeleteSandboxidempotency, surfaced while verifying this fix reaches every driver's leak, not just Podman's.fix(server): release driver-owned sandbox resources on out-of-band re…
fix(server): release driver-owned sandbox resources on out-of-band re…