From 2285718d17fe89265ed07e90671c9370d46c2d2d Mon Sep 17 00:00:00 2001 From: Konstantinos Angelopoulos Date: Tue, 13 Oct 2026 16:25:11 +0200 Subject: [PATCH] feat(driver-kubernetes): add empty_dir workspace volume with bounded size MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a `workspace_volume` option to the Kubernetes compute driver so a gateway can back each sandbox's /sandbox with a size-capped emptyDir instead of a per-sandbox PVC. This lets sandboxes run on clusters that have no dynamic storage provisioning or no default StorageClass, and removes the PVC lifecycle (orphaned claims, reclaim policy) from the teardown path — kubelet reclaims the emptyDir with the Pod. - workspace_volume (persistent_volume_claim | empty_dir); default persistent_volume_claim, so existing behaviour is unchanged. - empty_dir omits volumeClaimTemplates on the Sandbox CR and mounts a pod-level emptyDir at /sandbox, sized from workspace_default_storage_size as the sizeLimit. The workspace-init container still seeds /sandbox from the image on every pod start (the volume is empty each start). - workspace_max_storage_size (default 10Gi) bounds per-sandbox size overrides so a single sandbox cannot claim an unbounded workspace and starve the node; startup validation rejects a default above the maximum. - Per-sandbox overrides via driver_config.kubernetes.workspace_volume and workspace_storage_size, honoured only on gateways that set allow_driver_config. A requested size above the gateway maximum is rejected; workspace_storage_class applies to PVCs only. - Standalone driver env: OPENSHELL_K8S_WORKSPACE_VOLUME and OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE. Refs: https://github.com/NVIDIA/OpenShell/issues/2787 Signed-off-by: Konstantinos Angelopoulos --- crates/openshell-driver-kubernetes/README.md | 22 +- .../openshell-driver-kubernetes/src/config.rs | 276 ++++++++++++- .../openshell-driver-kubernetes/src/driver.rs | 368 +++++++++++++++++- crates/openshell-driver-kubernetes/src/lib.rs | 8 +- .../openshell-driver-kubernetes/src/main.rs | 13 + docs/how-it-works/gateways/configuration.mdx | 10 + docs/how-it-works/sandboxes/runtimes.mdx | 31 +- docs/kubernetes/setup.mdx | 2 +- skills/debug-openshell-cluster/SKILL.md | 2 +- 9 files changed, 702 insertions(+), 30 deletions(-) diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 9a9d6dca18..7685b467d2 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -198,9 +198,9 @@ errors, avoiding a gateway-side watch restart and its associated watch gap. ## Workspace Persistence -Sandbox pods use a PVC-backed `/sandbox` workspace. An init container seeds the -PVC from the image's original `/sandbox` contents on first start and writes a -sentinel so subsequent starts skip the copy. +Sandbox pods use a PVC-backed `/sandbox` workspace by default. An init +container seeds the PVC from the image's original `/sandbox` contents on first +start and writes a sentinel so subsequent starts skip the copy. This is a stopgap persistence model. It preserves user files across pod rescheduling but duplicates the base workspace and does not automatically apply @@ -225,6 +225,22 @@ runtime via `OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE` and that OpenShell provisions automatically; they have no effect when a `driver_config` mount attaches an existing PVC under `/sandbox`, which skips the default PVC. +Set `workspace_volume = "empty_dir"` (standalone driver: +`OPENSHELL_K8S_WORKSPACE_VOLUME`) to back `/sandbox` with an `emptyDir` capped +at `workspace_default_storage_size` instead of a PVC, for clusters without +dynamic provisioning. The Sandbox CR then carries no `volumeClaimTemplates`, +kubelet reclaims the workspace with the pod, and the init container re-seeds +`/sandbox` on every start, so stop and start or any other pod replacement +resets it. `workspace_storage_class` does not apply in this mode. + +`workspace_max_storage_size` (default `10Gi`, standalone driver: +`OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE`) bounds a per-sandbox workspace size +so one sandbox cannot claim an unbounded `emptyDir` and starve the node; +startup validation rejects a default above the maximum. On gateways with +`allow_driver_config = true`, a sandbox overrides the gateway default through +`driver_config.kubernetes.workspace_volume` and `workspace_storage_size`; a +requested size above the maximum is rejected. + ## Credentials, TLS, and Relay Both Pods set `automountServiceAccountToken: false`. The supervisor receives an diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index ccf130fab6..8ad978333b 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -114,6 +114,133 @@ impl FromStr for KubernetesControlConnection { } } +/// Default ceiling for a per-sandbox `workspace_storage_size` override. +/// +/// Caller requests above this are rejected so a single sandbox cannot claim an +/// unbounded `emptyDir` (and starve the node) or an oversized PVC. +pub const DEFAULT_WORKSPACE_MAX_STORAGE_SIZE: &str = "10Gi"; + +/// How the sandbox workspace (`/sandbox`) is backed on Kubernetes. +/// +/// - `PersistentVolumeClaim` (default) — a per-sandbox PVC declared via +/// `volumeClaimTemplates` on the `agents.x-k8s.io/Sandbox` CR. Survives pod +/// rescheduling, but requires dynamic provisioning (a `StorageClass`). +/// - `EmptyDir` — a size-capped `emptyDir` mounted at `/sandbox`. No PVC is +/// created, so it works on clusters without dynamic provisioning; the +/// workspace is reclaimed by kubelet when the Pod goes away. +/// +/// Set gateway-wide via `workspace_volume`, and per-sandbox via +/// `driver_config.kubernetes.workspace_volume` on gateways that enable +/// `allow_driver_config`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum WorkspaceVolume { + /// Per-sandbox PVC via `volumeClaimTemplates` on the Sandbox CR. + #[default] + PersistentVolumeClaim, + /// Size-capped `emptyDir` volume — no PVC, reclaimed with the Pod. + EmptyDir, +} + +impl std::fmt::Display for WorkspaceVolume { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::PersistentVolumeClaim => f.write_str("persistent_volume_claim"), + Self::EmptyDir => f.write_str("empty_dir"), + } + } +} + +impl FromStr for WorkspaceVolume { + type Err = String; + + fn from_str(s: &str) -> Result { + match s { + "persistent_volume_claim" => Ok(Self::PersistentVolumeClaim), + "empty_dir" => Ok(Self::EmptyDir), + other => Err(format!( + "unknown workspace volume '{other}'; expected 'persistent_volume_claim' or 'empty_dir'" + )), + } + } +} + +/// Parse a Kubernetes resource quantity into a byte count for comparison. +/// +/// Supports bare integers (bytes), IEC binary suffixes (`Ki Mi Gi Ti Pi Ei`), +/// and SI decimal suffixes (`k M G T P E`, plus the `m` milli suffix which +/// rounds up to at least one byte). This is deliberately a small subset of the +/// full quantity grammar — enough to bound a workspace size — and does not +/// accept exponent (`e`) notation. +#[allow( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + clippy::cast_precision_loss, + reason = "value is validated finite and non-negative, then clamped below u64::MAX before the cast" +)] +pub(crate) fn parse_quantity_bytes(quantity: &str) -> Result { + let trimmed = quantity.trim(); + if trimmed.is_empty() { + return Err("empty quantity".to_string()); + } + + let (number, multiplier) = split_quantity_suffix(trimmed); + if number.is_empty() { + return Err(format!("invalid quantity '{quantity}': missing number")); + } + + let value: f64 = number + .parse() + .map_err(|_| format!("invalid quantity '{quantity}': '{number}' is not a number"))?; + if value < 0.0 { + return Err(format!( + "invalid quantity '{quantity}': must not be negative" + )); + } + + // Round up so a fractional or milli quantity never resolves to zero bytes. + let bytes = (value * multiplier).ceil(); + if !bytes.is_finite() || bytes > u64::MAX as f64 { + return Err(format!("invalid quantity '{quantity}': out of range")); + } + Ok(bytes as u64) +} + +/// Split a quantity string into its numeric part and the byte multiplier its +/// suffix implies. +fn split_quantity_suffix(quantity: &str) -> (&str, f64) { + const KI: f64 = 1024.0; + const K: f64 = 1000.0; + let binary = [ + ("Ki", KI), + ("Mi", KI * KI), + ("Gi", KI * KI * KI), + ("Ti", KI * KI * KI * KI), + ("Pi", KI * KI * KI * KI * KI), + ("Ei", KI * KI * KI * KI * KI * KI), + ]; + for (suffix, multiplier) in binary { + if let Some(number) = quantity.strip_suffix(suffix) { + return (number, multiplier); + } + } + let decimal = [ + ("m", 0.001), + ("k", K), + ("M", K * K), + ("G", K * K * K), + ("T", K * K * K * K), + ("P", K * K * K * K * K), + ("E", K * K * K * K * K * K), + ]; + for (suffix, multiplier) in decimal { + if let Some(number) = quantity.strip_suffix(suffix) { + return (number, multiplier); + } + } + (quantity, 1.0) +} + /// Driver-owned requirements for the Kubernetes sandbox runtime. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(default, deny_unknown_fields)] @@ -314,11 +441,21 @@ pub struct KubernetesComputeConfig { pub host_gateway_ip: String, pub enable_user_namespaces: bool, pub workspace_default_storage_size: String, + /// How the sandbox workspace (`/sandbox`) is backed. `"persistent_volume_claim"` + /// (default) creates a per-sandbox PVC via `volumeClaimTemplates`; + /// `"empty_dir"` mounts a size-capped `emptyDir` instead, so sandboxes run + /// on clusters without dynamic storage provisioning. The workspace is then + /// reclaimed with the Pod and does not survive a reschedule or eviction. + pub workspace_volume: WorkspaceVolume, + /// Upper bound for a per-sandbox `workspace_storage_size` override. A caller + /// request above this is rejected. Bounds both the `emptyDir` `sizeLimit` + /// and the PVC storage request. Default `10Gi`. + pub workspace_max_storage_size: String, /// Kubernetes `StorageClass` name for the default workspace PVC. /// Empty string (default) = omit `storageClassName`, using the cluster's /// default `StorageClass`. Set this on clusters with no default /// `StorageClass`, otherwise the workspace PVC stays `Pending` and the - /// sandbox never starts. + /// sandbox never starts. Ignored when `workspace_volume = "empty_dir"`. pub workspace_storage_class: String, /// Default Kubernetes `runtimeClassName` for sandbox pods. /// Applied when a `CreateSandbox` request does not specify one. @@ -439,6 +576,8 @@ impl Default for KubernetesComputeConfig { host_gateway_ip: String::new(), enable_user_namespaces: false, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(), + workspace_volume: WorkspaceVolume::default(), + workspace_max_storage_size: DEFAULT_WORKSPACE_MAX_STORAGE_SIZE.to_string(), workspace_storage_class: String::new(), default_runtime_class_name: String::new(), allowed_pod_annotations: Vec::new(), @@ -458,6 +597,7 @@ impl KubernetesComputeConfig { return Err("supervisor_topology=workspace is not implemented".into()); } self.validate_workspace_mode()?; + self.validate_workspace_sizes()?; self.validate_provider_spiffe_workload_api_socket_path()?; self.validate_sandbox_identity_config()?; self.validate_proxy_uid()?; @@ -486,6 +626,47 @@ impl KubernetesComputeConfig { pod_metadata_key_allowed(&self.allowed_pod_labels, key) } + /// Reject workspace size settings that would never admit a sandbox: a + /// malformed quantity, or a default size already above the per-sandbox + /// maximum. Surfaces operator mistakes at startup instead of at first + /// `sandbox create`. + pub fn validate_workspace_sizes(&self) -> Result<(), String> { + let default_size = if self.workspace_default_storage_size.is_empty() { + DEFAULT_WORKSPACE_STORAGE_SIZE + } else { + &self.workspace_default_storage_size + }; + let default_bytes = parse_quantity_bytes(default_size) + .map_err(|err| format!("workspace_default_storage_size: {err}"))?; + + let max_size = if self.workspace_max_storage_size.is_empty() { + DEFAULT_WORKSPACE_MAX_STORAGE_SIZE + } else { + &self.workspace_max_storage_size + }; + let max_bytes = parse_quantity_bytes(max_size) + .map_err(|err| format!("workspace_max_storage_size: {err}"))?; + + if default_bytes > max_bytes { + return Err(format!( + "workspace_default_storage_size ('{default_size}') exceeds \ + workspace_max_storage_size ('{max_size}')" + )); + } + Ok(()) + } + + /// The effective per-sandbox maximum workspace size, resolving the + /// empty-string sentinel to the built-in default. + #[must_use] + pub fn effective_workspace_max_storage_size(&self) -> &str { + if self.workspace_max_storage_size.is_empty() { + DEFAULT_WORKSPACE_MAX_STORAGE_SIZE + } else { + &self.workspace_max_storage_size + } + } + /// Clamp `sa_token_ttl_secs` into the `[MIN_SA_TOKEN_TTL_SECS, /// MAX_SA_TOKEN_TTL_SECS]` range used by the projected-volume spec. /// Invalid (≤0) values fall back to the default 3600. @@ -1128,6 +1309,99 @@ mod tests { assert_eq!(cfg.workspace_storage_class, "fast-ssd"); } + #[test] + fn default_workspace_volume_is_persistent_volume_claim() { + let cfg = KubernetesComputeConfig::default(); + assert_eq!(cfg.workspace_volume, WorkspaceVolume::PersistentVolumeClaim); + } + + #[test] + fn default_workspace_max_storage_size_is_10gi() { + let cfg = KubernetesComputeConfig::default(); + assert_eq!( + cfg.workspace_max_storage_size, + DEFAULT_WORKSPACE_MAX_STORAGE_SIZE + ); + } + + #[test] + fn serde_override_workspace_volume_empty_dir() { + let json = serde_json::json!({ "workspace_volume": "empty_dir" }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.workspace_volume, WorkspaceVolume::EmptyDir); + } + + #[test] + fn serde_override_workspace_volume_persistent_volume_claim() { + let json = serde_json::json!({ "workspace_volume": "persistent_volume_claim" }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.workspace_volume, WorkspaceVolume::PersistentVolumeClaim); + } + + #[test] + fn serde_rejects_unknown_workspace_volume() { + let json = serde_json::json!({ "workspace_volume": "tmpfs" }); + let err = serde_json::from_value::(json).unwrap_err(); + assert!(err.to_string().contains("unknown variant")); + } + + #[test] + fn workspace_volume_display_round_trips() { + for volume in [ + WorkspaceVolume::PersistentVolumeClaim, + WorkspaceVolume::EmptyDir, + ] { + let rendered = volume.to_string(); + let parsed: WorkspaceVolume = rendered.parse().unwrap(); + assert_eq!(parsed, volume); + } + } + + #[test] + fn validate_workspace_sizes_accepts_default() { + KubernetesComputeConfig::default() + .validate_workspace_sizes() + .unwrap(); + } + + #[test] + fn validate_workspace_sizes_rejects_default_above_max() { + let cfg = KubernetesComputeConfig { + workspace_default_storage_size: "20Gi".to_string(), + workspace_max_storage_size: "10Gi".to_string(), + ..KubernetesComputeConfig::default() + }; + let err = cfg.validate_workspace_sizes().unwrap_err(); + assert!(err.contains("exceeds"), "unexpected error: {err}"); + } + + #[test] + fn validate_workspace_sizes_rejects_malformed_quantity() { + let cfg = KubernetesComputeConfig { + workspace_default_storage_size: "banana".to_string(), + ..KubernetesComputeConfig::default() + }; + assert!(cfg.validate_workspace_sizes().is_err()); + } + + #[test] + fn parse_quantity_bytes_handles_binary_and_decimal_suffixes() { + assert_eq!(parse_quantity_bytes("2Gi").unwrap(), 2 * 1024 * 1024 * 1024); + assert_eq!(parse_quantity_bytes("500Mi").unwrap(), 500 * 1024 * 1024); + assert_eq!(parse_quantity_bytes("1G").unwrap(), 1_000_000_000); + assert_eq!(parse_quantity_bytes("1000").unwrap(), 1000); + // Comparisons used by the per-sandbox clamp. + assert!(parse_quantity_bytes("5Gi").unwrap() < parse_quantity_bytes("10Gi").unwrap()); + assert!(parse_quantity_bytes("2000Mi").unwrap() > parse_quantity_bytes("1Gi").unwrap()); + } + + #[test] + fn parse_quantity_bytes_rejects_garbage() { + assert!(parse_quantity_bytes("").is_err()); + assert!(parse_quantity_bytes("abc").is_err()); + assert!(parse_quantity_bytes("-5Gi").is_err()); + } + #[test] fn serde_override_service_account_name() { let json = serde_json::json!({ diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 0a1acc7be2..f3e3510ce9 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -6,8 +6,8 @@ use crate::config::{ DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_SANDBOX_UID, DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesControlConnection, OperatorNamespaceAllowlist, - WorkspaceMode, is_dns_1123_label, managed_namespace, managed_namespace_prefix, - validate_managed_namespace_name, + WorkspaceMode, WorkspaceVolume, is_dns_1123_label, managed_namespace, managed_namespace_prefix, + parse_quantity_bytes, validate_managed_namespace_name, }; use crate::isolation::{ BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesControlPath, @@ -443,6 +443,16 @@ struct KubernetesSandboxDriverConfig { pod: KubernetesPodDriverConfig, containers: KubernetesDriverContainersConfig, volumes: Vec, + /// Per-sandbox override of the gateway-wide `workspace_volume`. `None` + /// (unset) uses the gateway default. Only honoured on gateways that enable + /// `allow_driver_config`. + #[serde(default)] + workspace_volume: Option, + /// Per-sandbox override of the workspace size — the `emptyDir` `sizeLimit` + /// or the PVC storage request. `None` uses `workspace_default_storage_size`. + /// Rejected when it exceeds the gateway `workspace_max_storage_size`. + #[serde(default)] + workspace_storage_size: Option, } impl KubernetesSandboxDriverConfig { @@ -890,6 +900,9 @@ impl KubernetesComputeDriver { config .validate_workspace_mode() .map_err(KubernetesDriverError::Precondition)?; + config + .validate_workspace_sizes() + .map_err(KubernetesDriverError::Precondition)?; config .validate_provider_spiffe_workload_api_socket_path() .map_err(KubernetesDriverError::Precondition)?; @@ -2124,6 +2137,8 @@ impl KubernetesComputeDriver { sandbox_id: &sandbox.id, enable_user_namespaces: self.config.enable_user_namespaces, workspace_default_storage_size: &self.config.workspace_default_storage_size, + workspace_volume: self.config.workspace_volume, + workspace_max_storage_size: self.config.effective_workspace_max_storage_size(), workspace_storage_class: &self.config.workspace_storage_class, default_runtime_class_name: &self.config.default_runtime_class_name, sandbox_uid: resolved_user_id, @@ -6297,17 +6312,21 @@ fn apply_supervisor_sandbox_runtime_boundary( /// /// This injects: /// 1. A volume mount on the agent container at `/sandbox`. -/// 2. An init container (same image) that seeds the PVC with the image's -/// original `/sandbox` contents on first use. +/// 2. An init container (same image) that seeds `/sandbox` with the image's +/// original contents on first use. +/// +/// In `persistent_volume_claim` mode the backing volume itself is **not** added +/// here — the Sandbox CRD controller creates a volume for each entry in +/// `volumeClaimTemplates` (following the `StatefulSet` convention). Adding one +/// here would create a duplicate volume name and fail pod validation. /// -/// The PVC volume itself is **not** added here — the Sandbox CRD controller -/// automatically creates a volume for each entry in `volumeClaimTemplates` -/// (following the `StatefulSet` convention). Adding one here would create a -/// duplicate volume name and fail pod validation. +/// In `empty_dir` mode there is no claim template, so this adds a size-capped +/// pod-level `emptyDir` volume. The `emptyDir` is empty on every pod start, so +/// the init container always re-seeds it from the image. /// -/// The init container mounts the PVC at a temporary path so it can still see -/// the image's `/sandbox` directory. It checks for a sentinel file and skips -/// the copy if the PVC was already initialised. +/// The init container mounts the volume at a temporary path so it can still see +/// the image's `/sandbox` directory. It checks for a sentinel file and skips +/// the copy if the volume was already initialised. #[allow(clippy::similar_names)] fn apply_workspace_persistence( pod_template: &mut serde_json::Value, @@ -6315,6 +6334,7 @@ fn apply_workspace_persistence( image_pull_policy: Option, sandbox_gid: Option, workspace_owner: Option<(u32, u32)>, + workspace: &ResolvedWorkspace, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -6331,6 +6351,24 @@ fn apply_workspace_persistence( } } + // In empty_dir mode, declare the size-capped workspace volume directly on + // the pod spec (kubelet allocates a per-pod `emptyDir` at pod start). In + // persistent_volume_claim mode the volume is instead injected by the + // agent-sandbox controller from `volumeClaimTemplates`, so adding one here + // would duplicate the volume name. + if workspace.volume == WorkspaceVolume::EmptyDir { + let volumes = spec + .entry("volumes") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut(); + if let Some(volumes) = volumes { + volumes.push(serde_json::json!({ + "name": WORKSPACE_VOLUME_NAME, + "emptyDir": { "sizeLimit": workspace.size } + })); + } + } + // 1. Add workspace volume mount to the agent container let containers = spec.get_mut("containers").and_then(|v| v.as_array_mut()); if let Some(containers) = containers { @@ -6467,6 +6505,54 @@ fn default_workspace_volume_claim_templates( }]) } +/// The workspace backing and size resolved for a specific sandbox, after +/// applying any per-sandbox `driver_config` override on top of the +/// gateway-wide defaults. +struct ResolvedWorkspace { + volume: WorkspaceVolume, + size: String, +} + +/// Resolve the effective workspace backing and size for a sandbox. +/// +/// A per-sandbox `driver_config` value wins over the gateway default. A +/// per-sandbox `workspace_storage_size` is rejected when it exceeds the gateway +/// `workspace_max_storage_size`, so a caller cannot claim an unbounded +/// `emptyDir` (and starve the node) or an oversized PVC. +fn resolve_workspace( + driver_config: &KubernetesSandboxDriverConfig, + params: &SandboxPodParams<'_>, +) -> Result { + let volume = driver_config + .workspace_volume + .unwrap_or(params.workspace_volume); + + let default_size = if params.workspace_default_storage_size.is_empty() { + DEFAULT_WORKSPACE_STORAGE_SIZE + } else { + params.workspace_default_storage_size + }; + + let size = match driver_config.workspace_storage_size.as_deref() { + Some(requested) => { + let requested_bytes = parse_quantity_bytes(requested) + .map_err(|err| format!("workspace_storage_size: {err}"))?; + let max_bytes = parse_quantity_bytes(params.workspace_max_storage_size) + .map_err(|err| format!("workspace_max_storage_size: {err}"))?; + if requested_bytes > max_bytes { + return Err(format!( + "workspace_storage_size '{requested}' exceeds the gateway maximum '{}'", + params.workspace_max_storage_size + )); + } + requested.to_string() + } + None => default_size.to_string(), + }; + + Ok(ResolvedWorkspace { volume, size }) +} + /// Parameters shared by `sandbox_to_k8s_spec` and `sandbox_template_to_k8s`. #[allow(clippy::struct_excessive_bools)] struct SandboxPodParams<'a> { @@ -6479,6 +6565,11 @@ struct SandboxPodParams<'a> { sandbox_id: &'a str, enable_user_namespaces: bool, workspace_default_storage_size: &'a str, + /// Gateway-wide default workspace backing. Overridable per-sandbox via + /// `driver_config.kubernetes.workspace_volume`. + workspace_volume: WorkspaceVolume, + /// Ceiling for a per-sandbox `workspace_storage_size` override. + workspace_max_storage_size: &'a str, workspace_storage_class: &'a str, default_runtime_class_name: &'a str, /// Resolved sandbox UID for supervisor `runAsUser` and env var. @@ -6503,6 +6594,8 @@ impl Default for SandboxPodParams<'_> { sandbox_id: "", enable_user_namespaces: false, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE, + workspace_volume: WorkspaceVolume::PersistentVolumeClaim, + workspace_max_storage_size: crate::config::DEFAULT_WORKSPACE_MAX_STORAGE_SIZE, workspace_storage_class: "", default_runtime_class_name: "", sandbox_uid: DEFAULT_SANDBOX_UID, @@ -6544,10 +6637,11 @@ fn sandbox_to_k8s_spec( params: &SandboxPodParams<'_>, ) -> Result { let driver_config = kubernetes_driver_config_for_spec(spec)?; + let workspace = resolve_workspace(&driver_config, params)?; let mut root = serde_json::Map::new(); // Determine early whether OpenShell should inject its default workspace - // PVC. Explicit Kubernetes driver-config mounts under /sandbox/ take + // volume. Explicit Kubernetes driver-config mounts under /sandbox/ take // ownership of workspace persistence. // We need this flag before building the podTemplate because the workspace // persistence transforms are applied inside sandbox_template_to_k8s. @@ -6565,6 +6659,7 @@ fn sandbox_to_k8s_spec( &pod_env, &driver_config, inject_workspace, + &workspace, params, ), ); @@ -6577,11 +6672,15 @@ fn sandbox_to_k8s_spec( } } - if inject_workspace { + // The workspace PVC is declared via `volumeClaimTemplates` only in + // persistent_volume_claim mode. In empty_dir mode the `emptyDir` volume is + // instead added to the podTemplate by `apply_workspace_persistence`, so the + // Sandbox CR carries no claim template and needs no StorageClass. + if inject_workspace && workspace.volume == WorkspaceVolume::PersistentVolumeClaim { root.insert( "volumeClaimTemplates".to_string(), default_workspace_volume_claim_templates( - params.workspace_default_storage_size, + &workspace.size, params.workspace_storage_class, ), ); @@ -6598,6 +6697,7 @@ fn sandbox_to_k8s_spec( &pod_env, &driver_config, inject_workspace, + &workspace, params, ), ); @@ -6619,12 +6719,15 @@ fn sandbox_template_to_k8s( let gpu_requirements = gpu.then_some(GpuResourceRequirements { count: None }); let driver_config = KubernetesSandboxDriverConfig::from_template(template) .expect("test Kubernetes driver_config should be valid"); + let workspace = resolve_workspace(&driver_config, params) + .expect("test workspace resolution should succeed"); sandbox_template_to_k8s_with_validated_config( template, gpu_requirements.as_ref(), spec_environment, &driver_config, inject_workspace, + &workspace, params, ) } @@ -6639,12 +6742,15 @@ fn sandbox_template_to_k8s_with_gpu_requirements( ) -> serde_json::Value { let driver_config = KubernetesSandboxDriverConfig::from_template(template) .expect("test Kubernetes driver_config should be valid"); + let workspace = resolve_workspace(&driver_config, params) + .expect("test workspace resolution should succeed"); sandbox_template_to_k8s_with_validated_config( template, gpu_requirements, spec_environment, &driver_config, inject_workspace, + &workspace, params, ) } @@ -6655,6 +6761,7 @@ fn sandbox_template_to_k8s_with_validated_config( spec_environment: &std::collections::HashMap, driver_config: &KubernetesSandboxDriverConfig, inject_workspace: bool, + workspace: &ResolvedWorkspace, params: &SandboxPodParams<'_>, ) -> serde_json::Value { let mut metadata = serde_json::Map::new(); @@ -6816,9 +6923,11 @@ fn sandbox_template_to_k8s_with_validated_config( apply_supervisor_sandbox_runtime_boundary(&mut result, params); - // Inject workspace persistence (init container + PVC volume mount) so - // that /sandbox data survives pod rescheduling. Skipped when the user - // provides custom storage through driver_config. + // Inject workspace persistence (init container + volume mount) so that + // /sandbox is seeded from the image on pod start. Skipped when the user + // provides custom storage through driver_config. In empty_dir mode the + // pod-level `emptyDir` volume is added here; in persistent_volume_claim + // mode the controller injects it from `volumeClaimTemplates`. if inject_workspace { apply_workspace_persistence( &mut result, @@ -6826,6 +6935,7 @@ fn sandbox_template_to_k8s_with_validated_config( params.image_pull_policy, None, Some((params.sandbox_uid, params.sandbox_gid)), + workspace, ); } @@ -8391,6 +8501,13 @@ mod tests { sandbox_to_k8s_spec(spec, params).expect("test Kubernetes driver_config should be valid") } + fn pvc_test_workspace() -> ResolvedWorkspace { + ResolvedWorkspace { + volume: WorkspaceVolume::PersistentVolumeClaim, + size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(), + } + } + fn kube_api_error(code: u16, message: &str) -> KubeError { KubeError::Api(kube::core::ErrorResponse { status: if code == 404 { @@ -10850,6 +10967,7 @@ mod tests { Some(crate::KubernetesImagePullPolicy::IfNotPresent), Some(1000), // sandbox_gid None, + &pvc_test_workspace(), ); // Init container @@ -10910,6 +11028,7 @@ mod tests { Some(crate::KubernetesImagePullPolicy::IfNotPresent), Some(1000), None, + &pvc_test_workspace(), ); let init_image = pod_template["spec"]["initContainers"][0]["image"] @@ -10938,6 +11057,7 @@ mod tests { Some(crate::KubernetesImagePullPolicy::Always), Some(1000), None, + &pvc_test_workspace(), ); let cmd = pod_template["spec"]["initContainers"][0]["command"] @@ -10996,6 +11116,220 @@ mod tests { ); } + // ----------------------------------------------------------------------- + // empty_dir workspace volume tests + // ----------------------------------------------------------------------- + + fn empty_dir_params<'a>() -> SandboxPodParams<'a> { + SandboxPodParams { + workspace_volume: WorkspaceVolume::EmptyDir, + ..SandboxPodParams::default() + } + } + + #[test] + fn empty_dir_omits_volume_claim_templates_and_declares_sized_emptydir() { + let spec = SandboxSpec { + template: Some(SandboxTemplate::default()), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + assert!( + cr["spec"].get("volumeClaimTemplates").is_none(), + "empty_dir must not emit volumeClaimTemplates on the Sandbox CR" + ); + + let volumes = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .expect("podTemplate must declare pod-level volumes"); + let workspace = volumes + .iter() + .find(|v| v["name"] == WORKSPACE_VOLUME_NAME) + .expect("empty_dir workspace volume must exist"); + assert_eq!( + workspace["emptyDir"]["sizeLimit"], DEFAULT_WORKSPACE_STORAGE_SIZE, + "emptyDir must be capped at the default workspace size" + ); + assert!( + workspace["emptyDir"].get("medium").is_none(), + "workspace emptyDir must be disk-backed, not tmpfs" + ); + } + + #[test] + fn empty_dir_still_seeds_via_init_container() { + let spec = SandboxSpec { + template: Some(SandboxTemplate::default()), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + let init_containers = cr["spec"]["podTemplate"]["spec"]["initContainers"] + .as_array() + .expect("empty_dir must still run init containers"); + assert!( + init_containers + .iter() + .any(|c| c["name"] == WORKSPACE_INIT_CONTAINER_NAME), + "empty_dir must still run the workspace-init container so /sandbox is seeded" + ); + } + + #[test] + fn default_pvc_mode_declares_no_pod_level_workspace_volume() { + let spec = SandboxSpec { + template: Some(SandboxTemplate::default()), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &SandboxPodParams::default()); + + let vcts = cr["spec"]["volumeClaimTemplates"] + .as_array() + .expect("PVC mode must emit volumeClaimTemplates"); + assert_eq!(vcts[0]["metadata"]["name"], WORKSPACE_VOLUME_NAME); + + // The pod-level workspace volume is injected by the controller from the + // claim template, not by apply_workspace_persistence. + let has_pod_volume = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .is_some_and(|vols| vols.iter().any(|v| v["name"] == WORKSPACE_VOLUME_NAME)); + assert!( + !has_pod_volume, + "PVC mode must not declare a pod-level workspace volume" + ); + } + + #[test] + fn per_sandbox_empty_dir_overrides_gateway_pvc_default() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_volume": "empty_dir" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + // Gateway default is PVC. + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &SandboxPodParams::default()); + + assert!( + cr["spec"].get("volumeClaimTemplates").is_none(), + "per-sandbox empty_dir must override the gateway PVC default" + ); + let has_pod_volume = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .is_some_and(|vols| { + vols.iter() + .any(|v| v["name"] == WORKSPACE_VOLUME_NAME && v["emptyDir"].is_object()) + }); + assert!( + has_pod_volume, + "per-sandbox empty_dir must render an emptyDir workspace volume" + ); + } + + #[test] + fn per_sandbox_pvc_overrides_gateway_empty_dir_default() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_volume": "persistent_volume_claim" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + assert!( + cr["spec"]["volumeClaimTemplates"].is_array(), + "per-sandbox persistent_volume_claim must override the gateway empty_dir default" + ); + } + + #[test] + fn per_sandbox_storage_size_sets_emptydir_size_limit() { + let template = SandboxTemplate { + driver_config: Some(json_struct(serde_json::json!({ + "workspace_volume": "empty_dir", + "workspace_storage_size": "5Gi" + }))), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + let volumes = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .expect("podTemplate must declare pod-level volumes"); + let workspace = volumes + .iter() + .find(|v| v["name"] == WORKSPACE_VOLUME_NAME) + .expect("workspace volume must exist"); + assert_eq!(workspace["emptyDir"]["sizeLimit"], "5Gi"); + } + + #[test] + fn per_sandbox_storage_size_sets_pvc_request() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_storage_size": "7Gi" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &SandboxPodParams::default()); + + let vcts = cr["spec"]["volumeClaimTemplates"] + .as_array() + .expect("PVC mode must emit volumeClaimTemplates"); + assert_eq!(vcts[0]["spec"]["resources"]["requests"]["storage"], "7Gi"); + } + + #[test] + fn per_sandbox_storage_size_above_max_is_rejected() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_storage_size": "50Gi" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + // Default max is 10Gi. + let err = sandbox_to_k8s_spec(Some(&spec), &SandboxPodParams::default()) + .expect_err("a size above the gateway maximum must be rejected"); + assert!( + err.contains("exceeds the gateway maximum"), + "unexpected error: {err}" + ); + } + + #[test] + fn driver_config_rejects_unknown_workspace_volume() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_volume": "tmpfs" }), + )), + ..SandboxTemplate::default() + }; + let err = KubernetesSandboxDriverConfig::from_template(&template) + .expect_err("unknown workspace_volume must be rejected"); + assert!(err.contains("unknown variant"), "unexpected error: {err}"); + } + // ----------------------------------------------------------------------- // User namespace tests // ----------------------------------------------------------------------- diff --git a/crates/openshell-driver-kubernetes/src/lib.rs b/crates/openshell-driver-kubernetes/src/lib.rs index 963935abef..d6971df8bd 100644 --- a/crates/openshell-driver-kubernetes/src/lib.rs +++ b/crates/openshell-driver-kubernetes/src/lib.rs @@ -11,10 +11,10 @@ mod resource_admission; mod sandbox_runtime; pub use config::{ - DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_STORAGE_SIZE, - KubernetesComputeConfig, KubernetesControlConnection, KubernetesImagePullPolicy, - KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, SupervisorTopology, WorkspaceMode, - managed_namespace_prefix, + DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_MAX_STORAGE_SIZE, + DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesControlConnection, + KubernetesImagePullPolicy, KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, + SupervisorTopology, WorkspaceMode, WorkspaceVolume, managed_namespace_prefix, }; pub use driver::{KubernetesComputeDriver, KubernetesDriverError}; pub use grpc::ComputeDriverService; diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index 30c7f899e7..2bef1f9607 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -314,6 +314,19 @@ async fn main() -> Result<()> { .unwrap_or_else(|_| { openshell_driver_kubernetes::DEFAULT_WORKSPACE_STORAGE_SIZE.to_string() }), + workspace_volume: std::env::var("OPENSHELL_K8S_WORKSPACE_VOLUME") + .ok() + .as_deref() + .map(str::parse) + .transpose() + .map_err(|err: String| { + miette::miette!("invalid OPENSHELL_K8S_WORKSPACE_VOLUME: {err}") + })? + .unwrap_or_default(), + workspace_max_storage_size: std::env::var("OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE") + .unwrap_or_else(|_| { + openshell_driver_kubernetes::DEFAULT_WORKSPACE_MAX_STORAGE_SIZE.to_string() + }), workspace_storage_class: std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS") .unwrap_or_default(), default_runtime_class_name: std::env::var("OPENSHELL_K8S_DEFAULT_RUNTIME_CLASS_NAME") diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 7f16929759..ab2e00ef5f 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -946,10 +946,20 @@ ssh_socket_path = "/run/openshell/ssh.sock" client_tls_secret_name = "openshell-client-tls" host_gateway_ip = "10.0.0.1" enable_user_namespaces = false +# Volume backing /sandbox: "persistent_volume_claim" (default) or "empty_dir". +# An emptyDir needs no StorageClass and is reclaimed with the pod, so it suits +# clusters without dynamic provisioning; its sizeLimit is workspace_default_storage_size +# and it does not survive a reschedule, stop, or eviction. +# workspace_volume = "empty_dir" workspace_default_storage_size = "10Gi" +# Upper bound for a per-sandbox workspace_storage_size override, bounding both +# the emptyDir sizeLimit and the PVC request. A request above this is rejected, +# and startup fails if workspace_default_storage_size exceeds it. +workspace_max_storage_size = "10Gi" # Kubernetes StorageClass for the workspace PVC. Empty (default) omits the # field, using the cluster's default StorageClass. Set this on clusters with no # default StorageClass, otherwise the workspace PVC stays Pending. +# Ignored when workspace_volume = "empty_dir". # workspace_storage_class = "fast-ssd" # Kubernetes RuntimeClass applied to sandbox pods when the API request does # not specify one. Empty (default) = omit the field, using the cluster default. diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index f56d194d63..7368757bfc 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -295,8 +295,10 @@ Only the OpenShell gateway and the Agent Sandbox controller should be able to ma | `client_tls_secret_name` | `server.tls.clientTlsSecretName` | Project or stage only `ca.crt` into supervisor Pods to authenticate the gateway. | | `sandbox_runtime_image` | `sandboxRuntime.image.*` | Override the sandbox runtime image. | | `supervisor_image` | `supervisor.image.*` | Override the supervisor image. | -| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Default workspace PVC size. | -| `workspace_storage_class` | `server.workspaceStorageClass` | `StorageClass` for workspace PVCs. Set this if the cluster has no default `StorageClass`. | +| `workspace_volume` | `gatewayConfig["openshell.drivers.kubernetes"].workspace_volume` | `persistent_volume_claim` (default) or `empty_dir`. `empty_dir` needs no `StorageClass`. | +| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Default workspace size: the PVC request, or the `empty_dir` size limit. | +| `workspace_max_storage_size` | `gatewayConfig["openshell.drivers.kubernetes"].workspace_max_storage_size` | Upper bound (default `10Gi`) for a per-sandbox `workspace_storage_size` override. | +| `workspace_storage_class` | `server.workspaceStorageClass` | `StorageClass` for workspace PVCs. Set this if the cluster has no default `StorageClass`. Ignored for `empty_dir`. | | `allowed_pod_annotations` | `gatewayConfig["openshell.drivers.kubernetes"].allowed_pod_annotations` | Pod annotation keys that sandbox templates may set, as exact keys or `/*` wildcards. Empty (default) rejects all template annotations. Keys under `openshell.ai` and `agents.x-k8s.io` are reserved and can't be allowlisted. | | `allowed_pod_labels` | `gatewayConfig["openshell.drivers.kubernetes"].allowed_pod_labels` | Pod label keys that sandbox templates may set, in the same format as `allowed_pod_annotations`. Empty (default) rejects all template labels. Labels select pods for NetworkPolicy rules and Services, so only allow keys that no other rule in the sandbox namespace selects on. | | `sandbox_uid` / `sandbox_gid` | `server.sandboxUid` / `server.sandboxGid` | Sandbox pod UID and GID. Takes priority over OpenShift SCC namespace annotations. `sandbox_gid` defaults to the UID. | @@ -323,7 +325,7 @@ Workspace storage settings cannot change after a sandbox is created. Delete and ### Kubernetes PVC Mounts -Mount existing PersistentVolumeClaims into the agent container through driver config. Any mount under `/sandbox` replaces the default workspace PVC for that sandbox. +Mount existing PersistentVolumeClaims into the agent container through driver config. Any mount under `/sandbox` replaces the default workspace volume for that sandbox. ```shell openshell sandbox create \ @@ -356,6 +358,29 @@ openshell sandbox create \ | `containers.agent.volume_mounts[].sub_path` | Optional relative path within the PVC. | | `containers.agent.volume_mounts[].read_only` | Defaults to `true`. Read-write mounts require a read-write volume. | +### Ephemeral Workspaces + +Set `workspace_volume = "empty_dir"` to back `/sandbox` with a size-capped `emptyDir` instead of a per-sandbox PVC. This runs on clusters with no dynamic provisioning or no default `StorageClass`, and kubelet reclaims the workspace with the pod. The init container still seeds `/sandbox` from the image on every pod start, so an `empty_dir` workspace does not survive a reschedule, stop, or eviction, and exceeding its size limit evicts the pod. + +On gateways with `allow_driver_config = true`, a sandbox can override the gateway default per creation through `driver_config.kubernetes`: + +```shell +openshell sandbox create \ + --from registry.example.com/team/claude-code:1.0 \ + --driver-config-json '{ + "kubernetes": { + "workspace_volume": "empty_dir", + "workspace_storage_size": "5Gi" + } + }' \ + -- claude +``` + +| Field | Description | +|---|---| +| `workspace_volume` | `persistent_volume_claim` or `empty_dir`. Overrides the gateway default for this sandbox. | +| `workspace_storage_size` | Workspace size: the `emptyDir` size limit or the PVC request. Rejected when it exceeds the gateway `workspace_max_storage_size` (default `10Gi`). | + ## Sandbox User Identity On Docker and Podman, set `process.run_as_user` and `process.run_as_group` in the creation policy to choose the sandbox user. Selectors accept `sandbox` or numeric IDs from `1` through `4294967294`. Kubernetes and MicroVM select the identity through driver configuration. When a field is unset, the driver supplies it: diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index ae0be7d027..fd78dae194 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -234,7 +234,7 @@ openshell status ## Upgrade OpenShell -To upgrade, rerun `helm upgrade --install` with the new `--version` and the values you installed with. Running sandboxes keep their supervisor and sandbox runtime images until they stop. When a stopped sandbox starts, it takes both images from the upgraded gateway and keeps its workload image and workspace. +To upgrade, rerun `helm upgrade --install` with the new `--version` and the values you installed with. Running sandboxes keep their supervisor and sandbox runtime images until they stop. When a stopped sandbox starts, it takes both images from the upgraded gateway and keeps its workload image and workspace PVC. If you override images, set `supervisor.image` and `sandboxRuntime.image` to the same release, or set only `global.image.tag`. An incompatible pair fails when a sandbox starts, and the supervisor log shows the cause while the start is in progress: diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 1dab79be80..1a3e4e7209 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -1270,7 +1270,7 @@ credential failures. | Docker GPU sandbox fails before startup | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` | | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod `; if the gateway PVC is unbound because the cluster has no default `StorageClass`, set `server.dataStorageClass` to a valid one, and size the claim with `server.dataStorageSize` | | `helm upgrade` fails on a forbidden update to the gateway StatefulSet after changing `server.dataStorageSize` or `server.dataStorageClass` | `volumeClaimTemplates` is immutable on an existing StatefulSet | `kubectl -n openshell delete statefulset openshell --cascade=orphan`, then re-upgrade; the orphaned PVC keeps its original size and class, so grow it with in-place PVC expansion or copy the data to a new claim | -| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` | +| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass`, or set gateway config `workspace_volume = "empty_dir"` if `/sandbox` need not survive stop and start | | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | | Mutating RPCs return `UNAVAILABLE` with "timed out waiting for a concurrent mutation" | Advisory-lock contention, a full lock pool on one replica (4 connections unless `database_lock_max_connections` is set), a slow PostgreSQL, or older gateways still running during an upgrade | `openshell_server_mutation_lock_timeouts_total`, `openshell_server_mutation_lock_connections_in_use` against its capacity, the `detail` field of `mutation lock acquisition timed out` in gateway logs, `mutation lock held longer than the lock wait timeout` warnings and `openshell_server_mutation_lock_hold_seconds`, the `pg_locks` query in Step 6 | | Mutating RPCs return `INTERNAL` with "could not open a mutation lock connection" | PostgreSQL out of connection slots, restarting, or unreachable | `openshell_server_mutation_lock_errors_total`, PostgreSQL logs, `SELECT count(*) FROM pg_stat_activity` against `max_connections`, `database_max_connections` and `database_lock_max_connections`, the connection sizing in the High Availability guide |