Repository navigation
feat(kubernetes): expose agent-sandbox operatingMode (suspend/resume) for idle scale-to-zero with PVC retention #2035
Description
Problem Statement
Disposable per-user sandboxes backed by durable PVCs (as proposed in open PR
#2034) give great continuity, but for lifecycle management Kubernetes OpenShell
exposes create/delete but no suspend/resume. There's no way to free a sandbox's**
**compute when idle while retaining its identity and PVC. For deployments with many
provisioned-but-often-idle users, keeping every pod running 24/7 is a large,
mostly-wasted compute cost; deleting on idle instead loses the sandbox identity
(and, without #2034, the data). We want: idle → free compute; next login → resume
with state intact.
This is the Kubernetes-driver realization of the general capability proposed in
#1823 (checkpoint/pause/resume), scoped concretely to what agent-sandbox already
implements.
Proposed Design
Surface agent-sandbox's existing suspend/resume capability through the OpenShell
Kubernetes driver and gateway/CLI — the lifecycle analog of how #2034 surfaced
pod-template/volume config through driver_config.kubernetes.
agent-sandbox already implements this in its Sandbox CRD and controller:
- v1beta1:
spec.operatingMode: Running | Suspended(defaultRunning). - v1alpha1: the equivalent is
spec.replicas(0= suspended; the API
conversion mapsSuspended ↔ replicas=0). - On Suspended, the controller deletes the backing Pod (frees CPU/memory)
while leaving theSandboxobject and its PVCs in place — PVCs are reconciled
independently and removed only when the Sandbox itself is deleted. Status surfaces
aSuspendedcondition (PodTerminated/PodNotTerminated). - On Running, the controller recreates the Pod and reattaches the same PVC(s).
What OpenShell would add:
- Driver: set
operatingMode(v1beta1) /replicas=0(v1alpha1) on the managed
Sandbox CR to suspend, and flip back to resume. - Gateway: keep the sandbox registered across a suspend (don't treat the absent
Pod as a dead sandbox) and re-route on resume when the Pod returns. - Interface: a lifecycle op (e.g.
openshell sandbox suspend|resume) and/or an
idle policy; resume triggered by the controlling app on session start. - Existing seam: OpenShell already defines a
StopSandboxRPC in the
compute-driver contract (proto/compute_driver.proto), but it is currently
unimplemented for the Kubernetes driver
(crates/openshell-driver-kubernetes/src/grpc.rs) — a natural hook for wiring
suspend, with resume as its counterpart. - Pairs with open PR #2034: that PR proposes the durable, caller-owned per-user PVC;
this gives the lifecycle to free its compute while keeping the data.
Alternatives Considered
- Always-on pods (status quo): simplest, but pays compute for every provisioned
user, not just active ones — expensive at scale. - Delete + recreate on idle/login: frees compute, but churns the sandbox identity
and pays a full cold-create each login; with #2034 the data survives, but
registration/orphan handling is messier than a first-class suspend. - In-place pod restart only: doesn't free compute.
- →
operatingModesuspend/resume is preferable: it's a first-class primitive
already modeled and implemented in agent-sandbox; OpenShell only needs to expose
and drive it.
Checklist
- I've reviewed existing issues and the architecture docs
- This is a design proposal, not a "please build this" request
Related: #1823 (general checkpoint/pause/resume design), #1551 (VM-driver
suspend/resume). Builds on open PR #2034 by @mjamiv.
Activity
mjamiv commentedon Jun 27, 2026
I reviewed the current implementation surface. My read is that this should get maintainer API review before a full code PR.
A narrow Kubernetes-driver patch for the existing internal
StopSandboxRPC is possible: it could patch the agent-sandbox resource tospec.operatingMode: Suspendedforv1beta1orspec.replicas: 0forv1alpha1. That would be a useful primitive, but it would not expose a user-facing suspend/resume flow by itself.The full feature crosses several public control-plane decisions:
SandboxPhaseneeds an explicitSuspendedstate;ResumeSandboxRPC, or whether resume is modeled some other way;I would split this after maintainer confirmation:
StopSandboxas agent-sandbox suspend, with unit coverage for thev1beta1andv1alpha1patch shapes.Happy to contribute the driver primitive once maintainers confirm the intended public lifecycle/API direction.
mjamiv commentedon Jul 1, 2026
@elezar - Putting this item on your radar as it relates to #2034 and #2086
TaylorMutch commentedon Jul 1, 2026
Suspend / resume overall is something I think we'd like to support. We have a number of other large changes in motion, but I think it would be ideal to have some consist expectation of how this might work across the current in-tree drivers rather than implemented for just Kubernetes.
johntmyers commentedon Jul 1, 2026
@mjamiv could you give some rough metrics on pod volume, sandbox life duration, etc you are working with?
mjamiv commentedon Jul 1, 2026
@johntmyers roughly, we are designing around one long-lived sandbox identity per user/agent, backed by durable PVC storage, with the pod treated as replaceable compute.
The long-run target is Kubernetes-scale operation for tens of thousands of sandboxes, with ~20k as the planning goal. That is why we want to move away from VM-style always-on pets toward durable identity/state plus fresh, disposable pods.
A few reasons this matters for us:
The main open design challenge is scheduled/cron-style work. Some agents need an "always-on" behavior for background jobs, but we still want the pod to stay fresh and be able to suspend/resume or rotate on a schedule. That probably needs an explicit lifecycle model rather than treating pod disappearance as failure.
So the target model is: durable sandbox identity/PVC, disposable pod compute, regular refresh from image, and a clear story for idle suspend/resume plus scheduled work.
krishicks commentedon Aug 14, 2026
github-actions commentedon Aug 29, 2026
This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.