diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 9a9d6dca18..7685b467d2 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -198,9 +198,9 @@ errors, avoiding a gateway-side watch restart and its associated watch gap. ## Workspace Persistence -Sandbox pods use a PVC-backed `/sandbox` workspace. An init container seeds the -PVC from the image's original `/sandbox` contents on first start and writes a -sentinel so subsequent starts skip the copy. +Sandbox pods use a PVC-backed `/sandbox` workspace by default. An init +container seeds the PVC from the image's original `/sandbox` contents on first +start and writes a sentinel so subsequent starts skip the copy. This is a stopgap persistence model. It preserves user files across pod rescheduling but duplicates the base workspace and does not automatically apply @@ -225,6 +225,22 @@ runtime via `OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE` and that OpenShell provisions automatically; they have no effect when a `driver_config` mount attaches an existing PVC under `/sandbox`, which skips the default PVC. +Set `workspace_volume = "empty_dir"` (standalone driver: +`OPENSHELL_K8S_WORKSPACE_VOLUME`) to back `/sandbox` with an `emptyDir` capped +at `workspace_default_storage_size` instead of a PVC, for clusters without +dynamic provisioning. The Sandbox CR then carries no `volumeClaimTemplates`, +kubelet reclaims the workspace with the pod, and the init container re-seeds +`/sandbox` on every start, so stop and start or any other pod replacement +resets it. `workspace_storage_class` does not apply in this mode. + +`workspace_max_storage_size` (default `10Gi`, standalone driver: +`OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE`) bounds a per-sandbox workspace size +so one sandbox cannot claim an unbounded `emptyDir` and starve the node; +startup validation rejects a default above the maximum. On gateways with +`allow_driver_config = true`, a sandbox overrides the gateway default through +`driver_config.kubernetes.workspace_volume` and `workspace_storage_size`; a +requested size above the maximum is rejected. + ## Credentials, TLS, and Relay Both Pods set `automountServiceAccountToken: false`. The supervisor receives an diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index ccf130fab6..8ad978333b 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -114,6 +114,133 @@ impl FromStr for KubernetesControlConnection { } } +/// Default ceiling for a per-sandbox `workspace_storage_size` override. +/// +/// Caller requests above this are rejected so a single sandbox cannot claim an +/// unbounded `emptyDir` (and starve the node) or an oversized PVC. +pub const DEFAULT_WORKSPACE_MAX_STORAGE_SIZE: &str = "10Gi"; + +/// How the sandbox workspace (`/sandbox`) is backed on Kubernetes. +/// +/// - `PersistentVolumeClaim` (default) — a per-sandbox PVC declared via +/// `volumeClaimTemplates` on the `agents.x-k8s.io/Sandbox` CR. Survives pod +/// rescheduling, but requires dynamic provisioning (a `StorageClass`). +/// - `EmptyDir` — a size-capped `emptyDir` mounted at `/sandbox`. No PVC is +/// created, so it works on clusters without dynamic provisioning; the +/// workspace is reclaimed by kubelet when the Pod goes away. +/// +/// Set gateway-wide via `workspace_volume`, and per-sandbox via +/// `driver_config.kubernetes.workspace_volume` on gateways that enable +/// `allow_driver_config`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum WorkspaceVolume { + /// Per-sandbox PVC via `volumeClaimTemplates` on the Sandbox CR. + #[default] + PersistentVolumeClaim, + /// Size-capped `emptyDir` volume — no PVC, reclaimed with the Pod. + EmptyDir, +} + +impl std::fmt::Display for WorkspaceVolume { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::PersistentVolumeClaim => f.write_str("persistent_volume_claim"), + Self::EmptyDir => f.write_str("empty_dir"), + } + } +} + +impl FromStr for WorkspaceVolume { + type Err = String; + + fn from_str(s: &str) -> Result { + match s { + "persistent_volume_claim" => Ok(Self::PersistentVolumeClaim), + "empty_dir" => Ok(Self::EmptyDir), + other => Err(format!( + "unknown workspace volume '{other}'; expected 'persistent_volume_claim' or 'empty_dir'" + )), + } + } +} + +/// Parse a Kubernetes resource quantity into a byte count for comparison. +/// +/// Supports bare integers (bytes), IEC binary suffixes (`Ki Mi Gi Ti Pi Ei`), +/// and SI decimal suffixes (`k M G T P E`, plus the `m` milli suffix which +/// rounds up to at least one byte). This is deliberately a small subset of the +/// full quantity grammar — enough to bound a workspace size — and does not +/// accept exponent (`e`) notation. +#[allow( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + clippy::cast_precision_loss, + reason = "value is validated finite and non-negative, then clamped below u64::MAX before the cast" +)] +pub(crate) fn parse_quantity_bytes(quantity: &str) -> Result { + let trimmed = quantity.trim(); + if trimmed.is_empty() { + return Err("empty quantity".to_string()); + } + + let (number, multiplier) = split_quantity_suffix(trimmed); + if number.is_empty() { + return Err(format!("invalid quantity '{quantity}': missing number")); + } + + let value: f64 = number + .parse() + .map_err(|_| format!("invalid quantity '{quantity}': '{number}' is not a number"))?; + if value < 0.0 { + return Err(format!( + "invalid quantity '{quantity}': must not be negative" + )); + } + + // Round up so a fractional or milli quantity never resolves to zero bytes. + let bytes = (value * multiplier).ceil(); + if !bytes.is_finite() || bytes > u64::MAX as f64 { + return Err(format!("invalid quantity '{quantity}': out of range")); + } + Ok(bytes as u64) +} + +/// Split a quantity string into its numeric part and the byte multiplier its +/// suffix implies. +fn split_quantity_suffix(quantity: &str) -> (&str, f64) { + const KI: f64 = 1024.0; + const K: f64 = 1000.0; + let binary = [ + ("Ki", KI), + ("Mi", KI * KI), + ("Gi", KI * KI * KI), + ("Ti", KI * KI * KI * KI), + ("Pi", KI * KI * KI * KI * KI), + ("Ei", KI * KI * KI * KI * KI * KI), + ]; + for (suffix, multiplier) in binary { + if let Some(number) = quantity.strip_suffix(suffix) { + return (number, multiplier); + } + } + let decimal = [ + ("m", 0.001), + ("k", K), + ("M", K * K), + ("G", K * K * K), + ("T", K * K * K * K), + ("P", K * K * K * K * K), + ("E", K * K * K * K * K * K), + ]; + for (suffix, multiplier) in decimal { + if let Some(number) = quantity.strip_suffix(suffix) { + return (number, multiplier); + } + } + (quantity, 1.0) +} + /// Driver-owned requirements for the Kubernetes sandbox runtime. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(default, deny_unknown_fields)] @@ -314,11 +441,21 @@ pub struct KubernetesComputeConfig { pub host_gateway_ip: String, pub enable_user_namespaces: bool, pub workspace_default_storage_size: String, + /// How the sandbox workspace (`/sandbox`) is backed. `"persistent_volume_claim"` + /// (default) creates a per-sandbox PVC via `volumeClaimTemplates`; + /// `"empty_dir"` mounts a size-capped `emptyDir` instead, so sandboxes run + /// on clusters without dynamic storage provisioning. The workspace is then + /// reclaimed with the Pod and does not survive a reschedule or eviction. + pub workspace_volume: WorkspaceVolume, + /// Upper bound for a per-sandbox `workspace_storage_size` override. A caller + /// request above this is rejected. Bounds both the `emptyDir` `sizeLimit` + /// and the PVC storage request. Default `10Gi`. + pub workspace_max_storage_size: String, /// Kubernetes `StorageClass` name for the default workspace PVC. /// Empty string (default) = omit `storageClassName`, using the cluster's /// default `StorageClass`. Set this on clusters with no default /// `StorageClass`, otherwise the workspace PVC stays `Pending` and the - /// sandbox never starts. + /// sandbox never starts. Ignored when `workspace_volume = "empty_dir"`. pub workspace_storage_class: String, /// Default Kubernetes `runtimeClassName` for sandbox pods. /// Applied when a `CreateSandbox` request does not specify one. @@ -439,6 +576,8 @@ impl Default for KubernetesComputeConfig { host_gateway_ip: String::new(), enable_user_namespaces: false, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(), + workspace_volume: WorkspaceVolume::default(), + workspace_max_storage_size: DEFAULT_WORKSPACE_MAX_STORAGE_SIZE.to_string(), workspace_storage_class: String::new(), default_runtime_class_name: String::new(), allowed_pod_annotations: Vec::new(), @@ -458,6 +597,7 @@ impl KubernetesComputeConfig { return Err("supervisor_topology=workspace is not implemented".into()); } self.validate_workspace_mode()?; + self.validate_workspace_sizes()?; self.validate_provider_spiffe_workload_api_socket_path()?; self.validate_sandbox_identity_config()?; self.validate_proxy_uid()?; @@ -486,6 +626,47 @@ impl KubernetesComputeConfig { pod_metadata_key_allowed(&self.allowed_pod_labels, key) } + /// Reject workspace size settings that would never admit a sandbox: a + /// malformed quantity, or a default size already above the per-sandbox + /// maximum. Surfaces operator mistakes at startup instead of at first + /// `sandbox create`. + pub fn validate_workspace_sizes(&self) -> Result<(), String> { + let default_size = if self.workspace_default_storage_size.is_empty() { + DEFAULT_WORKSPACE_STORAGE_SIZE + } else { + &self.workspace_default_storage_size + }; + let default_bytes = parse_quantity_bytes(default_size) + .map_err(|err| format!("workspace_default_storage_size: {err}"))?; + + let max_size = if self.workspace_max_storage_size.is_empty() { + DEFAULT_WORKSPACE_MAX_STORAGE_SIZE + } else { + &self.workspace_max_storage_size + }; + let max_bytes = parse_quantity_bytes(max_size) + .map_err(|err| format!("workspace_max_storage_size: {err}"))?; + + if default_bytes > max_bytes { + return Err(format!( + "workspace_default_storage_size ('{default_size}') exceeds \ + workspace_max_storage_size ('{max_size}')" + )); + } + Ok(()) + } + + /// The effective per-sandbox maximum workspace size, resolving the + /// empty-string sentinel to the built-in default. + #[must_use] + pub fn effective_workspace_max_storage_size(&self) -> &str { + if self.workspace_max_storage_size.is_empty() { + DEFAULT_WORKSPACE_MAX_STORAGE_SIZE + } else { + &self.workspace_max_storage_size + } + } + /// Clamp `sa_token_ttl_secs` into the `[MIN_SA_TOKEN_TTL_SECS, /// MAX_SA_TOKEN_TTL_SECS]` range used by the projected-volume spec. /// Invalid (≤0) values fall back to the default 3600. @@ -1128,6 +1309,99 @@ mod tests { assert_eq!(cfg.workspace_storage_class, "fast-ssd"); } + #[test] + fn default_workspace_volume_is_persistent_volume_claim() { + let cfg = KubernetesComputeConfig::default(); + assert_eq!(cfg.workspace_volume, WorkspaceVolume::PersistentVolumeClaim); + } + + #[test] + fn default_workspace_max_storage_size_is_10gi() { + let cfg = KubernetesComputeConfig::default(); + assert_eq!( + cfg.workspace_max_storage_size, + DEFAULT_WORKSPACE_MAX_STORAGE_SIZE + ); + } + + #[test] + fn serde_override_workspace_volume_empty_dir() { + let json = serde_json::json!({ "workspace_volume": "empty_dir" }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.workspace_volume, WorkspaceVolume::EmptyDir); + } + + #[test] + fn serde_override_workspace_volume_persistent_volume_claim() { + let json = serde_json::json!({ "workspace_volume": "persistent_volume_claim" }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.workspace_volume, WorkspaceVolume::PersistentVolumeClaim); + } + + #[test] + fn serde_rejects_unknown_workspace_volume() { + let json = serde_json::json!({ "workspace_volume": "tmpfs" }); + let err = serde_json::from_value::(json).unwrap_err(); + assert!(err.to_string().contains("unknown variant")); + } + + #[test] + fn workspace_volume_display_round_trips() { + for volume in [ + WorkspaceVolume::PersistentVolumeClaim, + WorkspaceVolume::EmptyDir, + ] { + let rendered = volume.to_string(); + let parsed: WorkspaceVolume = rendered.parse().unwrap(); + assert_eq!(parsed, volume); + } + } + + #[test] + fn validate_workspace_sizes_accepts_default() { + KubernetesComputeConfig::default() + .validate_workspace_sizes() + .unwrap(); + } + + #[test] + fn validate_workspace_sizes_rejects_default_above_max() { + let cfg = KubernetesComputeConfig { + workspace_default_storage_size: "20Gi".to_string(), + workspace_max_storage_size: "10Gi".to_string(), + ..KubernetesComputeConfig::default() + }; + let err = cfg.validate_workspace_sizes().unwrap_err(); + assert!(err.contains("exceeds"), "unexpected error: {err}"); + } + + #[test] + fn validate_workspace_sizes_rejects_malformed_quantity() { + let cfg = KubernetesComputeConfig { + workspace_default_storage_size: "banana".to_string(), + ..KubernetesComputeConfig::default() + }; + assert!(cfg.validate_workspace_sizes().is_err()); + } + + #[test] + fn parse_quantity_bytes_handles_binary_and_decimal_suffixes() { + assert_eq!(parse_quantity_bytes("2Gi").unwrap(), 2 * 1024 * 1024 * 1024); + assert_eq!(parse_quantity_bytes("500Mi").unwrap(), 500 * 1024 * 1024); + assert_eq!(parse_quantity_bytes("1G").unwrap(), 1_000_000_000); + assert_eq!(parse_quantity_bytes("1000").unwrap(), 1000); + // Comparisons used by the per-sandbox clamp. + assert!(parse_quantity_bytes("5Gi").unwrap() < parse_quantity_bytes("10Gi").unwrap()); + assert!(parse_quantity_bytes("2000Mi").unwrap() > parse_quantity_bytes("1Gi").unwrap()); + } + + #[test] + fn parse_quantity_bytes_rejects_garbage() { + assert!(parse_quantity_bytes("").is_err()); + assert!(parse_quantity_bytes("abc").is_err()); + assert!(parse_quantity_bytes("-5Gi").is_err()); + } + #[test] fn serde_override_service_account_name() { let json = serde_json::json!({ diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 0a1acc7be2..f3e3510ce9 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -6,8 +6,8 @@ use crate::config::{ DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_SANDBOX_UID, DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesControlConnection, OperatorNamespaceAllowlist, - WorkspaceMode, is_dns_1123_label, managed_namespace, managed_namespace_prefix, - validate_managed_namespace_name, + WorkspaceMode, WorkspaceVolume, is_dns_1123_label, managed_namespace, managed_namespace_prefix, + parse_quantity_bytes, validate_managed_namespace_name, }; use crate::isolation::{ BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesControlPath, @@ -443,6 +443,16 @@ struct KubernetesSandboxDriverConfig { pod: KubernetesPodDriverConfig, containers: KubernetesDriverContainersConfig, volumes: Vec, + /// Per-sandbox override of the gateway-wide `workspace_volume`. `None` + /// (unset) uses the gateway default. Only honoured on gateways that enable + /// `allow_driver_config`. + #[serde(default)] + workspace_volume: Option, + /// Per-sandbox override of the workspace size — the `emptyDir` `sizeLimit` + /// or the PVC storage request. `None` uses `workspace_default_storage_size`. + /// Rejected when it exceeds the gateway `workspace_max_storage_size`. + #[serde(default)] + workspace_storage_size: Option, } impl KubernetesSandboxDriverConfig { @@ -890,6 +900,9 @@ impl KubernetesComputeDriver { config .validate_workspace_mode() .map_err(KubernetesDriverError::Precondition)?; + config + .validate_workspace_sizes() + .map_err(KubernetesDriverError::Precondition)?; config .validate_provider_spiffe_workload_api_socket_path() .map_err(KubernetesDriverError::Precondition)?; @@ -2124,6 +2137,8 @@ impl KubernetesComputeDriver { sandbox_id: &sandbox.id, enable_user_namespaces: self.config.enable_user_namespaces, workspace_default_storage_size: &self.config.workspace_default_storage_size, + workspace_volume: self.config.workspace_volume, + workspace_max_storage_size: self.config.effective_workspace_max_storage_size(), workspace_storage_class: &self.config.workspace_storage_class, default_runtime_class_name: &self.config.default_runtime_class_name, sandbox_uid: resolved_user_id, @@ -6297,17 +6312,21 @@ fn apply_supervisor_sandbox_runtime_boundary( /// /// This injects: /// 1. A volume mount on the agent container at `/sandbox`. -/// 2. An init container (same image) that seeds the PVC with the image's -/// original `/sandbox` contents on first use. +/// 2. An init container (same image) that seeds `/sandbox` with the image's +/// original contents on first use. +/// +/// In `persistent_volume_claim` mode the backing volume itself is **not** added +/// here — the Sandbox CRD controller creates a volume for each entry in +/// `volumeClaimTemplates` (following the `StatefulSet` convention). Adding one +/// here would create a duplicate volume name and fail pod validation. /// -/// The PVC volume itself is **not** added here — the Sandbox CRD controller -/// automatically creates a volume for each entry in `volumeClaimTemplates` -/// (following the `StatefulSet` convention). Adding one here would create a -/// duplicate volume name and fail pod validation. +/// In `empty_dir` mode there is no claim template, so this adds a size-capped +/// pod-level `emptyDir` volume. The `emptyDir` is empty on every pod start, so +/// the init container always re-seeds it from the image. /// -/// The init container mounts the PVC at a temporary path so it can still see -/// the image's `/sandbox` directory. It checks for a sentinel file and skips -/// the copy if the PVC was already initialised. +/// The init container mounts the volume at a temporary path so it can still see +/// the image's `/sandbox` directory. It checks for a sentinel file and skips +/// the copy if the volume was already initialised. #[allow(clippy::similar_names)] fn apply_workspace_persistence( pod_template: &mut serde_json::Value, @@ -6315,6 +6334,7 @@ fn apply_workspace_persistence( image_pull_policy: Option, sandbox_gid: Option, workspace_owner: Option<(u32, u32)>, + workspace: &ResolvedWorkspace, ) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; @@ -6331,6 +6351,24 @@ fn apply_workspace_persistence( } } + // In empty_dir mode, declare the size-capped workspace volume directly on + // the pod spec (kubelet allocates a per-pod `emptyDir` at pod start). In + // persistent_volume_claim mode the volume is instead injected by the + // agent-sandbox controller from `volumeClaimTemplates`, so adding one here + // would duplicate the volume name. + if workspace.volume == WorkspaceVolume::EmptyDir { + let volumes = spec + .entry("volumes") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut(); + if let Some(volumes) = volumes { + volumes.push(serde_json::json!({ + "name": WORKSPACE_VOLUME_NAME, + "emptyDir": { "sizeLimit": workspace.size } + })); + } + } + // 1. Add workspace volume mount to the agent container let containers = spec.get_mut("containers").and_then(|v| v.as_array_mut()); if let Some(containers) = containers { @@ -6467,6 +6505,54 @@ fn default_workspace_volume_claim_templates( }]) } +/// The workspace backing and size resolved for a specific sandbox, after +/// applying any per-sandbox `driver_config` override on top of the +/// gateway-wide defaults. +struct ResolvedWorkspace { + volume: WorkspaceVolume, + size: String, +} + +/// Resolve the effective workspace backing and size for a sandbox. +/// +/// A per-sandbox `driver_config` value wins over the gateway default. A +/// per-sandbox `workspace_storage_size` is rejected when it exceeds the gateway +/// `workspace_max_storage_size`, so a caller cannot claim an unbounded +/// `emptyDir` (and starve the node) or an oversized PVC. +fn resolve_workspace( + driver_config: &KubernetesSandboxDriverConfig, + params: &SandboxPodParams<'_>, +) -> Result { + let volume = driver_config + .workspace_volume + .unwrap_or(params.workspace_volume); + + let default_size = if params.workspace_default_storage_size.is_empty() { + DEFAULT_WORKSPACE_STORAGE_SIZE + } else { + params.workspace_default_storage_size + }; + + let size = match driver_config.workspace_storage_size.as_deref() { + Some(requested) => { + let requested_bytes = parse_quantity_bytes(requested) + .map_err(|err| format!("workspace_storage_size: {err}"))?; + let max_bytes = parse_quantity_bytes(params.workspace_max_storage_size) + .map_err(|err| format!("workspace_max_storage_size: {err}"))?; + if requested_bytes > max_bytes { + return Err(format!( + "workspace_storage_size '{requested}' exceeds the gateway maximum '{}'", + params.workspace_max_storage_size + )); + } + requested.to_string() + } + None => default_size.to_string(), + }; + + Ok(ResolvedWorkspace { volume, size }) +} + /// Parameters shared by `sandbox_to_k8s_spec` and `sandbox_template_to_k8s`. #[allow(clippy::struct_excessive_bools)] struct SandboxPodParams<'a> { @@ -6479,6 +6565,11 @@ struct SandboxPodParams<'a> { sandbox_id: &'a str, enable_user_namespaces: bool, workspace_default_storage_size: &'a str, + /// Gateway-wide default workspace backing. Overridable per-sandbox via + /// `driver_config.kubernetes.workspace_volume`. + workspace_volume: WorkspaceVolume, + /// Ceiling for a per-sandbox `workspace_storage_size` override. + workspace_max_storage_size: &'a str, workspace_storage_class: &'a str, default_runtime_class_name: &'a str, /// Resolved sandbox UID for supervisor `runAsUser` and env var. @@ -6503,6 +6594,8 @@ impl Default for SandboxPodParams<'_> { sandbox_id: "", enable_user_namespaces: false, workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE, + workspace_volume: WorkspaceVolume::PersistentVolumeClaim, + workspace_max_storage_size: crate::config::DEFAULT_WORKSPACE_MAX_STORAGE_SIZE, workspace_storage_class: "", default_runtime_class_name: "", sandbox_uid: DEFAULT_SANDBOX_UID, @@ -6544,10 +6637,11 @@ fn sandbox_to_k8s_spec( params: &SandboxPodParams<'_>, ) -> Result { let driver_config = kubernetes_driver_config_for_spec(spec)?; + let workspace = resolve_workspace(&driver_config, params)?; let mut root = serde_json::Map::new(); // Determine early whether OpenShell should inject its default workspace - // PVC. Explicit Kubernetes driver-config mounts under /sandbox/ take + // volume. Explicit Kubernetes driver-config mounts under /sandbox/ take // ownership of workspace persistence. // We need this flag before building the podTemplate because the workspace // persistence transforms are applied inside sandbox_template_to_k8s. @@ -6565,6 +6659,7 @@ fn sandbox_to_k8s_spec( &pod_env, &driver_config, inject_workspace, + &workspace, params, ), ); @@ -6577,11 +6672,15 @@ fn sandbox_to_k8s_spec( } } - if inject_workspace { + // The workspace PVC is declared via `volumeClaimTemplates` only in + // persistent_volume_claim mode. In empty_dir mode the `emptyDir` volume is + // instead added to the podTemplate by `apply_workspace_persistence`, so the + // Sandbox CR carries no claim template and needs no StorageClass. + if inject_workspace && workspace.volume == WorkspaceVolume::PersistentVolumeClaim { root.insert( "volumeClaimTemplates".to_string(), default_workspace_volume_claim_templates( - params.workspace_default_storage_size, + &workspace.size, params.workspace_storage_class, ), ); @@ -6598,6 +6697,7 @@ fn sandbox_to_k8s_spec( &pod_env, &driver_config, inject_workspace, + &workspace, params, ), ); @@ -6619,12 +6719,15 @@ fn sandbox_template_to_k8s( let gpu_requirements = gpu.then_some(GpuResourceRequirements { count: None }); let driver_config = KubernetesSandboxDriverConfig::from_template(template) .expect("test Kubernetes driver_config should be valid"); + let workspace = resolve_workspace(&driver_config, params) + .expect("test workspace resolution should succeed"); sandbox_template_to_k8s_with_validated_config( template, gpu_requirements.as_ref(), spec_environment, &driver_config, inject_workspace, + &workspace, params, ) } @@ -6639,12 +6742,15 @@ fn sandbox_template_to_k8s_with_gpu_requirements( ) -> serde_json::Value { let driver_config = KubernetesSandboxDriverConfig::from_template(template) .expect("test Kubernetes driver_config should be valid"); + let workspace = resolve_workspace(&driver_config, params) + .expect("test workspace resolution should succeed"); sandbox_template_to_k8s_with_validated_config( template, gpu_requirements, spec_environment, &driver_config, inject_workspace, + &workspace, params, ) } @@ -6655,6 +6761,7 @@ fn sandbox_template_to_k8s_with_validated_config( spec_environment: &std::collections::HashMap, driver_config: &KubernetesSandboxDriverConfig, inject_workspace: bool, + workspace: &ResolvedWorkspace, params: &SandboxPodParams<'_>, ) -> serde_json::Value { let mut metadata = serde_json::Map::new(); @@ -6816,9 +6923,11 @@ fn sandbox_template_to_k8s_with_validated_config( apply_supervisor_sandbox_runtime_boundary(&mut result, params); - // Inject workspace persistence (init container + PVC volume mount) so - // that /sandbox data survives pod rescheduling. Skipped when the user - // provides custom storage through driver_config. + // Inject workspace persistence (init container + volume mount) so that + // /sandbox is seeded from the image on pod start. Skipped when the user + // provides custom storage through driver_config. In empty_dir mode the + // pod-level `emptyDir` volume is added here; in persistent_volume_claim + // mode the controller injects it from `volumeClaimTemplates`. if inject_workspace { apply_workspace_persistence( &mut result, @@ -6826,6 +6935,7 @@ fn sandbox_template_to_k8s_with_validated_config( params.image_pull_policy, None, Some((params.sandbox_uid, params.sandbox_gid)), + workspace, ); } @@ -8391,6 +8501,13 @@ mod tests { sandbox_to_k8s_spec(spec, params).expect("test Kubernetes driver_config should be valid") } + fn pvc_test_workspace() -> ResolvedWorkspace { + ResolvedWorkspace { + volume: WorkspaceVolume::PersistentVolumeClaim, + size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(), + } + } + fn kube_api_error(code: u16, message: &str) -> KubeError { KubeError::Api(kube::core::ErrorResponse { status: if code == 404 { @@ -10850,6 +10967,7 @@ mod tests { Some(crate::KubernetesImagePullPolicy::IfNotPresent), Some(1000), // sandbox_gid None, + &pvc_test_workspace(), ); // Init container @@ -10910,6 +11028,7 @@ mod tests { Some(crate::KubernetesImagePullPolicy::IfNotPresent), Some(1000), None, + &pvc_test_workspace(), ); let init_image = pod_template["spec"]["initContainers"][0]["image"] @@ -10938,6 +11057,7 @@ mod tests { Some(crate::KubernetesImagePullPolicy::Always), Some(1000), None, + &pvc_test_workspace(), ); let cmd = pod_template["spec"]["initContainers"][0]["command"] @@ -10996,6 +11116,220 @@ mod tests { ); } + // ----------------------------------------------------------------------- + // empty_dir workspace volume tests + // ----------------------------------------------------------------------- + + fn empty_dir_params<'a>() -> SandboxPodParams<'a> { + SandboxPodParams { + workspace_volume: WorkspaceVolume::EmptyDir, + ..SandboxPodParams::default() + } + } + + #[test] + fn empty_dir_omits_volume_claim_templates_and_declares_sized_emptydir() { + let spec = SandboxSpec { + template: Some(SandboxTemplate::default()), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + assert!( + cr["spec"].get("volumeClaimTemplates").is_none(), + "empty_dir must not emit volumeClaimTemplates on the Sandbox CR" + ); + + let volumes = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .expect("podTemplate must declare pod-level volumes"); + let workspace = volumes + .iter() + .find(|v| v["name"] == WORKSPACE_VOLUME_NAME) + .expect("empty_dir workspace volume must exist"); + assert_eq!( + workspace["emptyDir"]["sizeLimit"], DEFAULT_WORKSPACE_STORAGE_SIZE, + "emptyDir must be capped at the default workspace size" + ); + assert!( + workspace["emptyDir"].get("medium").is_none(), + "workspace emptyDir must be disk-backed, not tmpfs" + ); + } + + #[test] + fn empty_dir_still_seeds_via_init_container() { + let spec = SandboxSpec { + template: Some(SandboxTemplate::default()), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + let init_containers = cr["spec"]["podTemplate"]["spec"]["initContainers"] + .as_array() + .expect("empty_dir must still run init containers"); + assert!( + init_containers + .iter() + .any(|c| c["name"] == WORKSPACE_INIT_CONTAINER_NAME), + "empty_dir must still run the workspace-init container so /sandbox is seeded" + ); + } + + #[test] + fn default_pvc_mode_declares_no_pod_level_workspace_volume() { + let spec = SandboxSpec { + template: Some(SandboxTemplate::default()), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &SandboxPodParams::default()); + + let vcts = cr["spec"]["volumeClaimTemplates"] + .as_array() + .expect("PVC mode must emit volumeClaimTemplates"); + assert_eq!(vcts[0]["metadata"]["name"], WORKSPACE_VOLUME_NAME); + + // The pod-level workspace volume is injected by the controller from the + // claim template, not by apply_workspace_persistence. + let has_pod_volume = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .is_some_and(|vols| vols.iter().any(|v| v["name"] == WORKSPACE_VOLUME_NAME)); + assert!( + !has_pod_volume, + "PVC mode must not declare a pod-level workspace volume" + ); + } + + #[test] + fn per_sandbox_empty_dir_overrides_gateway_pvc_default() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_volume": "empty_dir" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + // Gateway default is PVC. + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &SandboxPodParams::default()); + + assert!( + cr["spec"].get("volumeClaimTemplates").is_none(), + "per-sandbox empty_dir must override the gateway PVC default" + ); + let has_pod_volume = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .is_some_and(|vols| { + vols.iter() + .any(|v| v["name"] == WORKSPACE_VOLUME_NAME && v["emptyDir"].is_object()) + }); + assert!( + has_pod_volume, + "per-sandbox empty_dir must render an emptyDir workspace volume" + ); + } + + #[test] + fn per_sandbox_pvc_overrides_gateway_empty_dir_default() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_volume": "persistent_volume_claim" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + assert!( + cr["spec"]["volumeClaimTemplates"].is_array(), + "per-sandbox persistent_volume_claim must override the gateway empty_dir default" + ); + } + + #[test] + fn per_sandbox_storage_size_sets_emptydir_size_limit() { + let template = SandboxTemplate { + driver_config: Some(json_struct(serde_json::json!({ + "workspace_volume": "empty_dir", + "workspace_storage_size": "5Gi" + }))), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &empty_dir_params()); + + let volumes = cr["spec"]["podTemplate"]["spec"]["volumes"] + .as_array() + .expect("podTemplate must declare pod-level volumes"); + let workspace = volumes + .iter() + .find(|v| v["name"] == WORKSPACE_VOLUME_NAME) + .expect("workspace volume must exist"); + assert_eq!(workspace["emptyDir"]["sizeLimit"], "5Gi"); + } + + #[test] + fn per_sandbox_storage_size_sets_pvc_request() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_storage_size": "7Gi" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + let cr = sandbox_to_k8s_spec_for_test(Some(&spec), &SandboxPodParams::default()); + + let vcts = cr["spec"]["volumeClaimTemplates"] + .as_array() + .expect("PVC mode must emit volumeClaimTemplates"); + assert_eq!(vcts[0]["spec"]["resources"]["requests"]["storage"], "7Gi"); + } + + #[test] + fn per_sandbox_storage_size_above_max_is_rejected() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_storage_size": "50Gi" }), + )), + ..SandboxTemplate::default() + }; + let spec = SandboxSpec { + template: Some(template), + ..SandboxSpec::default() + }; + // Default max is 10Gi. + let err = sandbox_to_k8s_spec(Some(&spec), &SandboxPodParams::default()) + .expect_err("a size above the gateway maximum must be rejected"); + assert!( + err.contains("exceeds the gateway maximum"), + "unexpected error: {err}" + ); + } + + #[test] + fn driver_config_rejects_unknown_workspace_volume() { + let template = SandboxTemplate { + driver_config: Some(json_struct( + serde_json::json!({ "workspace_volume": "tmpfs" }), + )), + ..SandboxTemplate::default() + }; + let err = KubernetesSandboxDriverConfig::from_template(&template) + .expect_err("unknown workspace_volume must be rejected"); + assert!(err.contains("unknown variant"), "unexpected error: {err}"); + } + // ----------------------------------------------------------------------- // User namespace tests // ----------------------------------------------------------------------- diff --git a/crates/openshell-driver-kubernetes/src/lib.rs b/crates/openshell-driver-kubernetes/src/lib.rs index 963935abef..d6971df8bd 100644 --- a/crates/openshell-driver-kubernetes/src/lib.rs +++ b/crates/openshell-driver-kubernetes/src/lib.rs @@ -11,10 +11,10 @@ mod resource_admission; mod sandbox_runtime; pub use config::{ - DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_STORAGE_SIZE, - KubernetesComputeConfig, KubernetesControlConnection, KubernetesImagePullPolicy, - KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, SupervisorTopology, WorkspaceMode, - managed_namespace_prefix, + DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_MAX_STORAGE_SIZE, + DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesControlConnection, + KubernetesImagePullPolicy, KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, + SupervisorTopology, WorkspaceMode, WorkspaceVolume, managed_namespace_prefix, }; pub use driver::{KubernetesComputeDriver, KubernetesDriverError}; pub use grpc::ComputeDriverService; diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index 30c7f899e7..2bef1f9607 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -314,6 +314,19 @@ async fn main() -> Result<()> { .unwrap_or_else(|_| { openshell_driver_kubernetes::DEFAULT_WORKSPACE_STORAGE_SIZE.to_string() }), + workspace_volume: std::env::var("OPENSHELL_K8S_WORKSPACE_VOLUME") + .ok() + .as_deref() + .map(str::parse) + .transpose() + .map_err(|err: String| { + miette::miette!("invalid OPENSHELL_K8S_WORKSPACE_VOLUME: {err}") + })? + .unwrap_or_default(), + workspace_max_storage_size: std::env::var("OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE") + .unwrap_or_else(|_| { + openshell_driver_kubernetes::DEFAULT_WORKSPACE_MAX_STORAGE_SIZE.to_string() + }), workspace_storage_class: std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS") .unwrap_or_default(), default_runtime_class_name: std::env::var("OPENSHELL_K8S_DEFAULT_RUNTIME_CLASS_NAME") diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 7f16929759..ab2e00ef5f 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -946,10 +946,20 @@ ssh_socket_path = "/run/openshell/ssh.sock" client_tls_secret_name = "openshell-client-tls" host_gateway_ip = "10.0.0.1" enable_user_namespaces = false +# Volume backing /sandbox: "persistent_volume_claim" (default) or "empty_dir". +# An emptyDir needs no StorageClass and is reclaimed with the pod, so it suits +# clusters without dynamic provisioning; its sizeLimit is workspace_default_storage_size +# and it does not survive a reschedule, stop, or eviction. +# workspace_volume = "empty_dir" workspace_default_storage_size = "10Gi" +# Upper bound for a per-sandbox workspace_storage_size override, bounding both +# the emptyDir sizeLimit and the PVC request. A request above this is rejected, +# and startup fails if workspace_default_storage_size exceeds it. +workspace_max_storage_size = "10Gi" # Kubernetes StorageClass for the workspace PVC. Empty (default) omits the # field, using the cluster's default StorageClass. Set this on clusters with no # default StorageClass, otherwise the workspace PVC stays Pending. +# Ignored when workspace_volume = "empty_dir". # workspace_storage_class = "fast-ssd" # Kubernetes RuntimeClass applied to sandbox pods when the API request does # not specify one. Empty (default) = omit the field, using the cluster default. diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index f56d194d63..7368757bfc 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -295,8 +295,10 @@ Only the OpenShell gateway and the Agent Sandbox controller should be able to ma | `client_tls_secret_name` | `server.tls.clientTlsSecretName` | Project or stage only `ca.crt` into supervisor Pods to authenticate the gateway. | | `sandbox_runtime_image` | `sandboxRuntime.image.*` | Override the sandbox runtime image. | | `supervisor_image` | `supervisor.image.*` | Override the supervisor image. | -| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Default workspace PVC size. | -| `workspace_storage_class` | `server.workspaceStorageClass` | `StorageClass` for workspace PVCs. Set this if the cluster has no default `StorageClass`. | +| `workspace_volume` | `gatewayConfig["openshell.drivers.kubernetes"].workspace_volume` | `persistent_volume_claim` (default) or `empty_dir`. `empty_dir` needs no `StorageClass`. | +| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Default workspace size: the PVC request, or the `empty_dir` size limit. | +| `workspace_max_storage_size` | `gatewayConfig["openshell.drivers.kubernetes"].workspace_max_storage_size` | Upper bound (default `10Gi`) for a per-sandbox `workspace_storage_size` override. | +| `workspace_storage_class` | `server.workspaceStorageClass` | `StorageClass` for workspace PVCs. Set this if the cluster has no default `StorageClass`. Ignored for `empty_dir`. | | `allowed_pod_annotations` | `gatewayConfig["openshell.drivers.kubernetes"].allowed_pod_annotations` | Pod annotation keys that sandbox templates may set, as exact keys or `/*` wildcards. Empty (default) rejects all template annotations. Keys under `openshell.ai` and `agents.x-k8s.io` are reserved and can't be allowlisted. | | `allowed_pod_labels` | `gatewayConfig["openshell.drivers.kubernetes"].allowed_pod_labels` | Pod label keys that sandbox templates may set, in the same format as `allowed_pod_annotations`. Empty (default) rejects all template labels. Labels select pods for NetworkPolicy rules and Services, so only allow keys that no other rule in the sandbox namespace selects on. | | `sandbox_uid` / `sandbox_gid` | `server.sandboxUid` / `server.sandboxGid` | Sandbox pod UID and GID. Takes priority over OpenShift SCC namespace annotations. `sandbox_gid` defaults to the UID. | @@ -323,7 +325,7 @@ Workspace storage settings cannot change after a sandbox is created. Delete and ### Kubernetes PVC Mounts -Mount existing PersistentVolumeClaims into the agent container through driver config. Any mount under `/sandbox` replaces the default workspace PVC for that sandbox. +Mount existing PersistentVolumeClaims into the agent container through driver config. Any mount under `/sandbox` replaces the default workspace volume for that sandbox. ```shell openshell sandbox create \ @@ -356,6 +358,29 @@ openshell sandbox create \ | `containers.agent.volume_mounts[].sub_path` | Optional relative path within the PVC. | | `containers.agent.volume_mounts[].read_only` | Defaults to `true`. Read-write mounts require a read-write volume. | +### Ephemeral Workspaces + +Set `workspace_volume = "empty_dir"` to back `/sandbox` with a size-capped `emptyDir` instead of a per-sandbox PVC. This runs on clusters with no dynamic provisioning or no default `StorageClass`, and kubelet reclaims the workspace with the pod. The init container still seeds `/sandbox` from the image on every pod start, so an `empty_dir` workspace does not survive a reschedule, stop, or eviction, and exceeding its size limit evicts the pod. + +On gateways with `allow_driver_config = true`, a sandbox can override the gateway default per creation through `driver_config.kubernetes`: + +```shell +openshell sandbox create \ + --from registry.example.com/team/claude-code:1.0 \ + --driver-config-json '{ + "kubernetes": { + "workspace_volume": "empty_dir", + "workspace_storage_size": "5Gi" + } + }' \ + -- claude +``` + +| Field | Description | +|---|---| +| `workspace_volume` | `persistent_volume_claim` or `empty_dir`. Overrides the gateway default for this sandbox. | +| `workspace_storage_size` | Workspace size: the `emptyDir` size limit or the PVC request. Rejected when it exceeds the gateway `workspace_max_storage_size` (default `10Gi`). | + ## Sandbox User Identity On Docker and Podman, set `process.run_as_user` and `process.run_as_group` in the creation policy to choose the sandbox user. Selectors accept `sandbox` or numeric IDs from `1` through `4294967294`. Kubernetes and MicroVM select the identity through driver configuration. When a field is unset, the driver supplies it: diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index ae0be7d027..fd78dae194 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -234,7 +234,7 @@ openshell status ## Upgrade OpenShell -To upgrade, rerun `helm upgrade --install` with the new `--version` and the values you installed with. Running sandboxes keep their supervisor and sandbox runtime images until they stop. When a stopped sandbox starts, it takes both images from the upgraded gateway and keeps its workload image and workspace. +To upgrade, rerun `helm upgrade --install` with the new `--version` and the values you installed with. Running sandboxes keep their supervisor and sandbox runtime images until they stop. When a stopped sandbox starts, it takes both images from the upgraded gateway and keeps its workload image and workspace PVC. If you override images, set `supervisor.image` and `sandboxRuntime.image` to the same release, or set only `global.image.tag`. An incompatible pair fails when a sandbox starts, and the supervisor log shows the cause while the start is in progress: diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 1dab79be80..1a3e4e7209 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -1270,7 +1270,7 @@ credential failures. | Docker GPU sandbox fails before startup | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` | | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod `; if the gateway PVC is unbound because the cluster has no default `StorageClass`, set `server.dataStorageClass` to a valid one, and size the claim with `server.dataStorageSize` | | `helm upgrade` fails on a forbidden update to the gateway StatefulSet after changing `server.dataStorageSize` or `server.dataStorageClass` | `volumeClaimTemplates` is immutable on an existing StatefulSet | `kubectl -n openshell delete statefulset openshell --cascade=orphan`, then re-upgrade; the orphaned PVC keeps its original size and class, so grow it with in-place PVC expansion or copy the data to a new claim | -| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` | +| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass`, or set gateway config `workspace_volume = "empty_dir"` if `/sandbox` need not survive stop and start | | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | | Mutating RPCs return `UNAVAILABLE` with "timed out waiting for a concurrent mutation" | Advisory-lock contention, a full lock pool on one replica (4 connections unless `database_lock_max_connections` is set), a slow PostgreSQL, or older gateways still running during an upgrade | `openshell_server_mutation_lock_timeouts_total`, `openshell_server_mutation_lock_connections_in_use` against its capacity, the `detail` field of `mutation lock acquisition timed out` in gateway logs, `mutation lock held longer than the lock wait timeout` warnings and `openshell_server_mutation_lock_hold_seconds`, the `pg_locks` query in Step 6 | | Mutating RPCs return `INTERNAL` with "could not open a mutation lock connection" | PostgreSQL out of connection slots, restarting, or unreachable | `openshell_server_mutation_lock_errors_total`, PostgreSQL logs, `SELECT count(*) FROM pg_stat_activity` against `max_connections`, `database_max_connections` and `database_lock_max_connections`, the connection sizing in the High Availability guide |