feat(kubernetes): support emptyDir sandbox workspaces

[lunarwhite]

Summary

Kubernetes gateways can set workspace_volume = "empty_dir" to back /sandbox with an emptyDir capped at workspace_default_storage_size instead of a per-sandbox PVC, so sandboxes run on clusters without dynamic provisioning or a default StorageClass. /sandbox then lasts as long as the workload Pod. The default stays a PVC.

Related Issue

Closes #2787

Changes

• Kubernetes driver: a new workspace_volume setting, persistent_volume_claim (the default) or empty_dir. empty_dir drops the workspace claim template and adds a workspace emptyDir whose sizeLimit is workspace_default_storage_size. The /sandbox mount and the seed-workspace init container are unchanged. The standalone driver takes --workspace-volume / OPENSHELL_K8S_WORKSPACE_VOLUME. Helm needs no chart change: set gatewayConfig.openshell.drivers.kubernetes.workspace_volume.
• Tests: config parsing, the empty_dir render, and a check that the default render adds no Pod-level workspace volume.
• Docs: configuration.mdx, runtimes.mdx, kubernetes/setup.mdx, the driver README, and the debug-openshell-cluster skill.

Testing

• Checks appropriate to the affected code and behavior pass: cargo fmt --check, cargo clippy -p openshell-driver-kubernetes --all-targets -- -D warnings, cargo test -p openshell-driver-kubernetes, mise run markdown:lint, mise run docs
• Unit tests added/updated
• E2E tests added/updated (if applicable): none, because an empty_dir gateway fails the conformance sandbox-lifecycle scenario by design.

Manual e2e testing: no default StorageClass, before and after

Setup: kind v0.32.0, Kubernetes v1.35.5, Agent Sandbox v1.0.3

The images and CLI were built from this branch, the final commit changes only unit tests and doc wording. localhost/seeded-sandbox:2787 is nvcr.io/nvidia/base/ubuntu:24.04 plus /sandbox/from-image.txt.

OS=target/debug/openshell
HELM_FLAGS=(--namespace openshell --wait --timeout 5m \
  --values deploy/helm/openshell/ci/values-skaffold.yaml --set fullnameOverride=openshell \
  --set-string global.image.registry= --set-string global.image.tag=e2e-local)

Before (default PVC mode)

After the gateway's own claim binds, the default StorageClass is removed, and a new sandbox's workspace claim stays Pending.

$ helm install openshell deploy/helm/openshell --create-namespace "${HELM_FLAGS[@]}"
...
STATUS: deployed
REVISION: 1
$ kubectl annotate storageclass standard storageclass.kubernetes.io/is-default-class=false --overwrite
storageclass.storage.k8s.io/standard annotated
$ timeout 60 $OS sandbox create --name ws-pvc --from localhost/seeded-sandbox:2787 --detach

Created sandbox: ws-pvc

  [0.0s] Requesting compute...
$ kubectl -n openshell get pvc
NAME                         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
openshell-data-openshell-0   Bound     pvc-a08426a4-806a-4dda-8b3d-d4020f4be3f3   1Gi        RWO            standard       <u### Manual run: no default StorageClass, before and after

Environment: kind v0.32.0, Kubernetes v1.35.5, Agent Sandbox v1.0.3. The images and CLI were built from this branch at `0c2741014`; the final commit changes only unit tests and doc wording. `localhost/seeded-sandbox:2787` is `nvcr.io/nvidia/base/ubuntu:24.04` plus `/sandbox/from-image.txt`.

```shell
OS=target/debug/openshell
HELM_FLAGS=(--namespace openshell --wait --timeout 5m \
  --values deploy/helm/openshell/ci/values-skaffold.yaml --set fullnameOverride=openshell \
  --set-string global.image.registry= --set-string global.image.tag=e2e-local)

After (`empty_dir` on the same gateway)

The sandbox is Ready with no workspace claim, /sandbox holds the image's content, and stop/start re-seeds it.

$ helm upgrade openshell deploy/helm/openshell "${HELM_FLAGS[@]}" \
    --set-string 'gatewayConfig.openshell\.drivers\.kubernetes.workspace_volume=empty_dir'
...
STATUS: deployed
REVISION: 2
$ time timeout 300 $OS sandbox create --name ws-empty --from localhost/seeded-sandbox:2787 --detach

Created sandbox: ws-empty

  [0.0s] Requesting compute...
[0.0s] Sandbox allocated
[0.5s] Image pulled
[1.7s] Image pulled
[2.7s] Image pulled

real	2m48.763s
$ $OS sandbox get ws-empty
...
  Phase: Ready
$ kubectl -n openshell get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
openshell-data-openshell-0   Bound    pvc-a08426a4-806a-4dda-8b3d-d4020f4be3f3   1Gi        RWO            standard       <unset>                 13m
$ kubectl -n openshell get sandboxes.agents.x-k8s.io -o custom-columns='NAME:.metadata.name,CLAIM_TEMPLATES:.spec.volumeClaimTemplates'
NAME                CLAIM_TEMPLATES
default--ws-empty   <none>
$ kubectl -n openshell get pod default--ws-empty -o jsonpath='{.spec.volumes[?(@.name=="workspace")]}'
{"emptyDir":{"sizeLimit":"2Gi"},"name":"workspace"}
$ $OS sandbox exec --name ws-empty --no-tty -- sh -c 'echo marker > /sandbox/marker.txt && ls -A /sandbox'
.openshell-initialized
from-image.txt
marker.txt
$ $OS sandbox stop ws-empty
✓ Stopped sandbox ws-empty
$ $OS sandbox start ws-empty
✓ Started sandbox ws-empty
$ $OS sandbox exec --name ws-empty --no-tty -- ls -A /sandbox
.openshell-initialized
from-image.txt

The first empty_dir create after the gateway restart never became ready and was rolled back after five minutes, like the stalls above. The retry shown here succeeded.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[lunarwhite]

Hi @drew, may I request your review on this PR when you have the time? as you triaged/accepted the issue that it addresses.

[mrunalp]

I think we'd to support emptyDir workspaces in combination with PVCs.

[sjenning]

There is a catch in that if you stop the sandbox, then restart, your sandbox workspace is not persistent.  This is a violation of the existing start/stop contract.

[dimityrmirchev]

I second @mrunalp 's #4542 (comment).

We would like to support both scenarios in combination and also allow per sandbox override + sizing similarly to how #4473 proposes the implementation.

[kon-angelo]

I had already opened #4473 a couple days ago. I also added a few more things like:

• Per-sandbox workspace_volume override (via driver_config)
• Per-sandbox workspace_storage_size override
• workspace_max_storage_size ceiling (default 10Gi) with over-max rejection

[kon-angelo]

TLDR; I think it's important that the user or the operator can "on the fly" change the type of persistency he needs for a sandbox and not just rely on the gateway default. The changes in my PR are a reflection of that

[kon-angelo]

@sjenning that's why in my original issue in #2787 I refer to it as an "ephemeral" sandbox. It's willingly opting out of any such persistency

[lunarwhite]

Thanks for all your inputs. Given @kon-angelo is the issue reporter with specific uses cases in their mind, let's move the design discuss to #4473 and iterate then. I'll close mine to avoid dup efforts and review.