Skip to content

feat(driver-kubernetes): add empty_dir workspace volume with bounded size - #4473

Open
kon-angelo wants to merge 1 commit into
NVIDIA:mainfrom
kon-angelo:feat/2787-workspace-volume-empty-dir/kon-angelo
Open

kon-angelo wants to merge 1 commit into
NVIDIA:mainfrom
kon-angelo:feat/2787-workspace-volume-empty-dir/kon-angelo

Conversation

@kon-angelo

Copy link
Copy Markdown

Summary

Add a workspace_volume option to the Kubernetes compute driver so a gateway can back each sandbox's /sandbox with a size-capped emptyDir instead of a per-sandbox PVC. This lets sandboxes run on clusters that have no dynamic storage provisioning or no default StorageClass, and takes the PVC lifecycle (orphaned claims, reclaim policy) off the teardown path — kubelet reclaims the emptyDir with the Pod. Default is unchanged (persistent_volume_claim).

Related Issue

Closes #2787

Changes

  • workspace_volume gateway setting (persistent_volume_claim default | empty_dir); default preserves current behaviour byte-for-byte.
  • empty_dir omits volumeClaimTemplates on the Sandbox CR and mounts a pod-level emptyDir at /sandbox, with sizeLimit sourced from workspace_default_storage_size. The workspace-init container still seeds /sandbox from the image on every pod start (the volume is empty each start).
  • workspace_max_storage_size (default 10Gi) bounds per-sandbox size overrides so one sandbox cannot claim an unbounded workspace and starve the node; startup validation rejects a default above the maximum.
  • Per-sandbox overrides via driver_config.kubernetes.workspace_volume and workspace_storage_size, honoured only on gateways that set allow_driver_config. A requested size above the gateway maximum is rejected; workspace_storage_class applies to PVCs only.
  • Standalone driver env: OPENSHELL_K8S_WORKSPACE_VOLUME, OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE.
  • Small in-crate quantity parser (IEC/SI suffixes) for the size comparison; no new dependency.

Testing

  • Checks appropriate to the affected code and behavior pass (cargo test -p openshell-driver-kubernetes: 304 pass; cargo clippy clean; openshell-server builds)
  • Unit tests added/updated (serde, quantity parser, render diff for both modes, override precedence both directions, size→sizeLimit, size→PVC request, over-max rejection, unknown-value rejection)
  • E2E tests added/updated (if applicable)

Also verified manually on a local k3d cluster deployed with workspace_volume = empty_dir + allow_driver_config = true:

  • Sandbox reaches Ready with no workspace PVC; CR has no volumeClaimTemplates; Pod volume is emptyDir={"sizeLimit":"2Gi"}; workspace-init seeds /sandbox.
  • Per-sandbox workspace_storage_size: 5Gi → sizeLimit: 5Gi; 50Gi (> 10Gi max) → rejected with a clear error.
  • Marker written to /sandbox is gone after stop/start (ephemeral contract).

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Draft — remaining before ready for review: config docs (docs/how-it-works/gateways/configuration.mdx, docs/how-it-works/sandboxes/runtimes.mdx) are not yet updated for workspace_volume / workspace_max_storage_size; no Rust e2e lane added (an empty_dir gateway fails the default sandbox-lifecycle conformance by design).

…size

Add a `workspace_volume` option to the Kubernetes compute driver so a
gateway can back each sandbox's /sandbox with a size-capped emptyDir
instead of a per-sandbox PVC. This lets sandboxes run on clusters that
have no dynamic storage provisioning or no default StorageClass, and
removes the PVC lifecycle (orphaned claims, reclaim policy) from the
teardown path — kubelet reclaims the emptyDir with the Pod.

- workspace_volume (persistent_volume_claim | empty_dir); default
  persistent_volume_claim, so existing behaviour is unchanged.
- empty_dir omits volumeClaimTemplates on the Sandbox CR and mounts a
  pod-level emptyDir at /sandbox, sized from workspace_default_storage_size
  as the sizeLimit. The workspace-init container still seeds /sandbox from
  the image on every pod start (the volume is empty each start).
- workspace_max_storage_size (default 10Gi) bounds per-sandbox size
  overrides so a single sandbox cannot claim an unbounded workspace and
  starve the node; startup validation rejects a default above the maximum.
- Per-sandbox overrides via driver_config.kubernetes.workspace_volume and
  workspace_storage_size, honoured only on gateways that set
  allow_driver_config. A requested size above the gateway maximum is
  rejected; workspace_storage_class applies to PVCs only.
- Standalone driver env: OPENSHELL_K8S_WORKSPACE_VOLUME and
  OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE.

Refs: NVIDIA#2787
Signed-off-by: Konstantinos Angelopoulos <konstantinos.angelopoulos@sap.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@kon-angelo
kon-angelo marked this pull request as ready for review October 15, 2026 09:36
@kon-angelo

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(k8s): support ephemeral emptyDir-backed sandbox workspaces as an alternative to the default PVC

1 participant