bug: sandbox bind-mount ownership breaks after host reboot (unpinned rootless userns mapping, no launch-time reconcile) #2336

Summary

On the rootless Podman and Docker sandbox drivers, OpenShell attaches no user-namespace ID mapping to the container and mounts host state directories as plain (non-idmapped) rbinds. The host-side ownership of a bind-mounted state dir is therefore just the projection of a container-relative UID (e.g. sandbox) through whatever rootless subuid range the runtime happens to assign at create time. OpenShell never pins, persists, or reconciles that range. If the range shifts across a host reboot / runtime reconfig, previously-written inodes keep their old raw numbers and now resolve to unrelated host users — and OpenShell never repairs them.

Result: after a host reboot the sandbox's persisted state tree comes back owned by root / systemd-network / dnsmasq (group systemd-journal) with modes like drwx------ and -r--r--r--, and the sandbox process can no longer read/write its own state (readonly sqlite, EACCES). Reported downstream as NVIDIA/NemoClaw#6972 on a GB10 / Ubuntu 24.04 box: the agent could not recover the gateway and a rebuild's host-side backup could not read the mis-owned dirs.

Reproduction

1. Launch a sandbox (rootless Podman driver) with a bind-mounted persistent state dir under ~/.../mounts/<sandbox>/.
2. Let the sandbox write state (owned, on the host, by the runtime's rootless subuid projection of the container sandbox user).
3. Reboot the host (or otherwise cause the rootless subuid base / userns-remap to shift — daemon reconfig, storage//etc/subuid change).
4. Re-launch / recover the sandbox. Observe on the host: the state tree is now owned by unrelated low-numbered users (root, systemd-network, dnsmasq) with restrictive modes; inside the container the process hits EACCES / attempt to write a readonly database.

Root cause (code)

Confirmed against current main:

• No userns / idmap on the container spec. crates/openshell-driver-podman/src/container.rs ContainerSpec (~L174–222) has no userns/idmappings field and runs the supervisor as user: "0:0" (~L908). The Docker driver sets no UsernsMode and also runs 0:0 (crates/openshell-driver-docker/src/lib.rs ~L908). Mapping is fully delegated to the runtime default.
• Bind mounts are plain rbinds, not idmapped. Podman: crates/openshell-driver-podman/src/container.rs ~L562–585 builds options ["ro"|"rw","rbind"] (+ optional SELinux z/Z) — no idmap, no U. Docker: crates/openshell-driver-docker/src/lib.rs ~L1794–1829 emits source:target[:ro,z|Z] only.
• No subuid base is pinned or persisted. The only subuid handling is crates/openshell-driver-podman/src/driver.rs check_subuid_range() (~L807, called ~L242), which is warn-only — it neither allocates, chooses, nor records a base.
• Launch-time re-chown does not repair a persisted tree for these drivers. crates/openshell-supervisor-process/src/process.rs prepare_filesystem() (~L1147): read-write paths are chowned only when newly created (~L1183–1194, prepare_read_write_path returns true only on create), and the recursive /sandbox re-chown (~L1201–1207 → chown_sandbox_home ~L1105) runs only when OPENSHELL_SANDBOX_UID is set — which the rootless Podman/Docker path does not set (only K8s/VM drivers do). So an existing .hermes tree with a shifted mapping is never re-owned.
• No reboot/recreate reconcile exists anywhere for the bind-mounted tree.

Impact

• Permanent loss of access to sandbox state after a benign host reboot; user-visible as "impossible to recover the sandbox," data-loss on rebuild, and readonly-DB gateway failures.
• Security-adjacent: host ownership of sandbox-written files is non-deterministic and, on some runtimes, can collapse onto low host UIDs.

Proposed fix directions (for maintainer input)

1. Durable / deterministic (preferred, architectural): attach an idmapped mount to the host state binds, or pass explicit deterministic --uidmap/--gidmap to the container so the host-side ownership of the bind-mounted tree is stable and independent of the runtime's default rootless allocation across reboots. This pins the mapping OpenShell currently leaves implicit.
2. Mitigation (contained, self-healing at launch): extend the existing prepare_filesystem re-chown so the sandbox state tree is reconciled to the configured sandbox identity on every launch for the Podman/Docker drivers too (not only when OPENSHELL_SANDBOX_UID is injected). This makes a mis-owned tree readable again under the current mapping without changing the mapping model. It heals the symptom each start but does not pin the mapping.
3. Make check_subuid_range() fail-closed (or emit a clearer remediation) when the range is missing/ambiguous rather than warn-only.

Related: #1909 (rootless Podman UID-mapping issues on Fedora 44) touches adjacent territory but is a distinct scenario.

I'm happy to open a PR for the contained mitigation (#2) if that direction is acceptable, and defer the idmapped-mount design (#1) to maintainers.

Filed while triaging NVIDIA/NemoClaw#6972; root cause traced into OpenShell.