feat(sandbox): initiate supervisor connections across compute drivers

.agents/skills/helm-dev-environment/SKILL.md

@@ -81,7 +81,16 @@ The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images
 and deploys the OpenShell Helm chart. The Kubernetes driver creates a
 capability-free workload Pod and a directly managed capability-free supervisor
 Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell
-workload Pod. The
+workload Pod. New sandbox generations use `runtime-initiated` control: the sandbox
+runtime dials its supervisor through `os-supervisor-<id>`. The legacy
+`os-boundary-<id>` Service keeps selecting the workload for old gateways during
+rolling upgrades. New gateways create both Services and bind the control-mode
+annotation to its runtime generation. The driver adds a per-sandbox
+`os-control-<id>` NetworkPolicy for that path only. Existing running generations
+keep their connection until replaced; stop/start and gateway recovery that
+recreates both Pods upgrade legacy generations automatically. Use compatible
+gateway, supervisor, and sandbox runtime images together; there is no legacy
+rollout flag. The
 `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`)
 generates gateway and CLI TLS secrets on first install. Supervisor Pods project
 only `ca.crt` and authenticate gateway RPCs with sandbox bearer tokens. User

crates/openshell-driver-docker/README.md

@@ -108,10 +108,16 @@ LSM decisions remain authoritative.
 | `network_mode = host` on the supervisor | Lets the trusted supervisor reach the gateway's primary loopback listener and originate approved upstream connections. |
 | `restart_policy = no` | Keeps canonical main-process exit terminal. |
 | `PidsLimit` | Applies the configured sandbox PID budget. Omit `sandbox_pids_limit` to use OpenShell's default. Explicit zero is invalid. |
-| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. |
+| Private named volumes | Separate volumes carry sandbox bootstrap keys, the runtime-initiated Unix socket, and supervisor-only credentials. The supervisor mounts bootstrap keys read-only and only the socket volume writable. |
 | In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. |
 | CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. |
 
+New containers dial the supervisor at `/.openshell/channel/control/supervisor/control.sock`.
+The runtime remains the TLS/gRPC server; the supervisor verifies its pinned
+identity and supplies the existing bearer credential. Saved descriptors retain
+the legacy Unix listener for existing containers, including across restart.
+New generations require matching supervisor and sandbox runtime binaries.
+
 ## Stop, Start, and Delete
 
 Stop terminates the supervisor companion and stops the workload container

crates/openshell-driver-docker/src/isolation.rs

@@ -109,7 +109,7 @@ impl DockerBoundarySpec {
                 auth_epoch: self.auth_epoch,
                 gateway_id: self.gateway_id,
                 verification_keys: self.verification_keys,
-                listener: BoundaryListener::Unix {
+                listener: BoundaryListener::RuntimeInitiatedUnix {
                     socket_path: self.listener_socket,
                     tls: self.sandbox_tls,
                 },
@@ -124,7 +124,7 @@ impl DockerBoundarySpec {
                 generation: self.generation,
                 session_id: self.session_id,
                 workload_identity: self.workload_identity,
-                transport: SandboxTransport::Unix {
+                transport: SandboxTransport::RuntimeInitiatedUnix {
                     socket_path: self.control_socket,
                 },
                 tls: self.supervisor_tls,

crates/openshell-driver-docker/src/lib.rs

@@ -107,7 +107,8 @@ const SUPERVISOR_UID: u32 = 65_534;
 const SUPERVISOR_GID: u32 = 65_534;
 const BOUNDARY_MOUNT_PATH: &str = "/.openshell/channel";
 const BOUNDARY_CONFIG_MOUNT_PATH: &str = "/.openshell/channel/sandbox/bootstrap.json";
-const BOUNDARY_SOCKET_MOUNT_PATH: &str = "/.openshell/channel/sandbox/control.sock";
+const BOUNDARY_CONTROL_MOUNT_PATH: &str = "/.openshell/channel/control";
+const BOUNDARY_SOCKET_MOUNT_PATH: &str = "/.openshell/channel/control/supervisor/control.sock";
 const BOUNDARY_CERTIFICATE_MOUNT_PATH: &str = "/.openshell/channel/sandbox/server.crt";
 const BOUNDARY_PRIVATE_KEY_MOUNT_PATH: &str = "/.openshell/channel/sandbox/server.key";
 const SUPERVISOR_STATE_MOUNT_PATH: &str = "/.openshell/supervisor";
@@ -1309,7 +1310,9 @@ impl DockerComputeDriver {
                             Status::unavailable("volume admission lookup failed")
                         }
                     })?;
-                    if name == docker_channel_volume_name_by_id(sandbox_id, &self.config) {
+                    if name == docker_channel_volume_name_by_id(sandbox_id, &self.config)
+                        || name == docker_control_volume_name_by_id(sandbox_id, &self.config)
+                    {
                         if volume.driver != "local"
                             || !volume.options.is_empty()
                             || volume.labels.get(LABEL_SANDBOX_ID) != Some(sandbox_id)
@@ -4126,28 +4129,36 @@ fn docker_supervisor_volume_name_by_id(
     format!("openshell-supervisor-{}", &digest[..32])
 }
 
+fn docker_control_volume_name_by_id(
+    sandbox_id: &str,
+    config: &DockerDriverRuntimeConfig,
+) -> String {
+    docker_channel_volume_name_by_id(sandbox_id, config).replacen(
+        "openshell-channel-",
+        "openshell-control-",
+        1,
+    )
+}
+
 async fn create_docker_channel_volume(
     docker: &Docker,
     sandbox: &DriverSandbox,
     config: &DockerDriverRuntimeConfig,
 ) -> Result<(), Status> {
-    create_docker_runtime_volume(
-        docker,
-        sandbox,
-        config,
+    let names = [
         docker_channel_volume_name(sandbox, config),
-    )
-    .await?;
-    if let Err(error) = create_docker_runtime_volume(
-        docker,
-        sandbox,
-        config,
         docker_supervisor_volume_name(sandbox, config),
-    )
-    .await
-    {
-        let _ = remove_docker_volume(docker, &docker_channel_volume_name(sandbox, config)).await;
-        return Err(error);
+        docker_control_volume_name_by_id(&sandbox.id, config),
+    ];
+    for (index, name) in names.iter().enumerate() {
+        if let Err(error) =
+            create_docker_runtime_volume(docker, sandbox, config, name.clone()).await
+        {
+            for created in &names[..index] {
+                let _ = remove_docker_volume(docker, created).await;
+            }
+            return Err(error);
+        }
     }
     Ok(())
 }
@@ -4213,6 +4224,11 @@ async fn remove_docker_channel_volume_by_id(
         docker,
         &docker_supervisor_volume_name_by_id(sandbox_id, config),
     )
+    .await?;
+    remove_docker_volume(
+        docker,
+        &docker_control_volume_name_by_id(sandbox_id, config),
+    )
     .await
 }
 
@@ -4472,6 +4488,8 @@ fn docker_sandbox_bundle_archive(
         identity.uid,
         identity.gid,
     )?;
+    // The control volume is read-only in the runtime container; the
+    // supervisor stager initializes it.
     append_docker_archive_file(
         &mut archive,
         ".openshell/runtime/openshell-sandbox",
@@ -4551,7 +4569,8 @@ async fn stage_docker_sandbox_bundle(
             bollard::body_full(Bytes::from(archive)),
         )
         .await
-        .map_err(|error| Status::internal(format!("stage Docker sandbox bundle: {error}")))
+        .map_err(|error| Status::internal(format!("stage Docker sandbox bundle: {error}")))?;
+    Ok(())
 }
 
 fn decode_docker_launch_authentication(
@@ -4964,6 +4983,32 @@ async fn stage_docker_supervisor_bundle(
             Some(RemoveContainerOptionsBuilder::default().force(true).build()),
         )
         .await;
+    let runtime_initiated = read_docker_runtime_descriptor(&sandbox.id, config).await?
+        .is_some_and(|descriptor| matches!(descriptor.transport, openshell_sandbox_backend::boundary_protocol::SandboxTransport::RuntimeInitiatedUnix { .. }));
+    let mut mounts = vec![Mount {
+        target: Some(SUPERVISOR_STATE_MOUNT_PATH.to_string()),
+        source: Some(docker_supervisor_volume_name(sandbox, config)),
+        typ: Some(MountTypeEnum::VOLUME),
+        read_only: Some(false),
+        volume_options: Some(MountVolumeOptions {
+            no_copy: Some(true),
+            ..Default::default()
+        }),
+        ..Default::default()
+    }];
+    if runtime_initiated {
+        mounts.push(Mount {
+            target: Some(BOUNDARY_CONTROL_MOUNT_PATH.to_string()),
+            source: Some(docker_control_volume_name_by_id(&sandbox.id, config)),
+            typ: Some(MountTypeEnum::VOLUME),
+            read_only: Some(false),
+            volume_options: Some(MountVolumeOptions {
+                no_copy: Some(true),
+                ..Default::default()
+            }),
+            ..Default::default()
+        });
+    }
     let created = docker
         .create_container(
             Some(
@@ -4985,17 +5030,7 @@ async fn stage_docker_supervisor_bundle(
                 )),
                 host_config: Some(HostConfig {
                     network_mode: Some("none".to_string()),
-                    mounts: Some(vec![Mount {
-                        target: Some(SUPERVISOR_STATE_MOUNT_PATH.to_string()),
-                        source: Some(docker_supervisor_volume_name(sandbox, config)),
-                        typ: Some(MountTypeEnum::VOLUME),
-                        read_only: Some(false),
-                        volume_options: Some(MountVolumeOptions {
-                            no_copy: Some(true),
-                            ..Default::default()
-                        }),
-                        ..Default::default()
-                    }]),
+                    mounts: Some(mounts),
                     cap_drop: Some(vec!["ALL".to_string()]),
                     security_opt: Some(vec!["no-new-privileges:true".to_string()]),
                     ..Default::default()
@@ -5013,14 +5048,50 @@ async fn stage_docker_supervisor_bundle(
         .path(SUPERVISOR_STATE_MOUNT_PATH)
         .copy_uidgid("true")
         .build();
-    let result = docker
-        .upload_to_container(
-            &created.id,
-            Some(options),
-            bollard::body_full(Bytes::from(archive)),
-        )
-        .await
-        .map_err(|error| Status::internal(format!("stage Docker supervisor bundle: {error}")));
+    let result: Result<(), Status> = async {
+        docker
+            .upload_to_container(
+                &created.id,
+                Some(options),
+                bollard::body_full(Bytes::from(archive)),
+            )
+            .await
+            .map_err(|error| {
+                Status::internal(format!("stage Docker supervisor bundle: {error}"))
+            })?;
+        if runtime_initiated {
+            // Initialize the socket volume using the supervisor's staging user.
+            // Docker preserves the existing volume root's ownership, so use
+            // a child directory created by the supervisor's staging user.
+            let mut archive = tar::Builder::new(Vec::new());
+            append_docker_archive_directory(
+                &mut archive,
+                "supervisor",
+                0o711,
+                SUPERVISOR_UID,
+                SUPERVISOR_GID,
+            )?;
+            let bytes = archive.into_inner().map_err(|error| {
+                Status::internal(format!("finish control socket archive: {error}"))
+            })?;
+            let options = UploadToContainerOptionsBuilder::default()
+                .path(BOUNDARY_CONTROL_MOUNT_PATH)
+                .copy_uidgid("true")
+                .build();
+            docker
+                .upload_to_container(
+                    &created.id,
+                    Some(options),
+                    bollard::body_full(Bytes::from(bytes)),
+                )
+                .await
+                .map_err(|error| {
+                    Status::internal(format!("stage Docker control socket volume: {error}"))
+                })?;
+        }
+        Ok(())
+    }
+    .await;
     let cleanup = docker
         .remove_container(
             &created.id,
@@ -5245,6 +5316,8 @@ async fn spawn_docker_control_process(
         &config.upstream_proxy,
         config.proxy_ca_bundle.is_some(),
     ));
+    let runtime_initiated = read_docker_runtime_descriptor(&sandbox.id, config).await?
+        .is_some_and(|descriptor| matches!(descriptor.transport, openshell_sandbox_backend::boundary_protocol::SandboxTransport::RuntimeInitiatedUnix { .. }));
     let mut supervisor_mounts = vec![
         Mount {
             target: Some(BOUNDARY_MOUNT_PATH.to_string()),
@@ -5269,6 +5342,20 @@ async fn spawn_docker_control_process(
             ..Default::default()
         },
     ];
+    if runtime_initiated {
+        supervisor_mounts.push(Mount {
+            target: Some(BOUNDARY_CONTROL_MOUNT_PATH.to_string()),
+            source: Some(docker_control_volume_name_by_id(&sandbox.id, config)),
+            typ: Some(MountTypeEnum::VOLUME),
+            read_only: Some(false),
+            volume_options: Some(MountVolumeOptions {
+                no_copy: Some(true),
+                ..Default::default()
+            }),
+            ..Default::default()
+        });
+    }
+
     if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() {
         let parent = socket.parent().ok_or_else(|| {
             Status::failed_precondition("provider SPIFFE socket has no parent directory")
@@ -5931,6 +6018,19 @@ fn build_container_create_body_for_image(
         }),
         ..Default::default()
     });
+    // The runtime only dials the supervisor-owned socket; it never writes here.
+    user_mounts.push(Mount {
+        target: Some(BOUNDARY_CONTROL_MOUNT_PATH.to_string()),
+        source: Some(docker_control_volume_name_by_id(&sandbox.id, config)),
+        typ: Some(MountTypeEnum::VOLUME),
+        read_only: Some(true),
+        volume_options: Some(MountVolumeOptions {
+            no_copy: Some(true),
+            ..Default::default()
+        }),
+        ..Default::default()
+    });
+
     let user_bind_strings = docker_driver_bind_strings(driver_config)?;
     let device_requests = gpu_device_ids.map(|device_ids| {
         vec![DeviceRequest {

crates/openshell-driver-docker/src/tests.rs

@@ -2056,7 +2056,7 @@ fn build_container_create_body_includes_driver_config_mounts() {
         .mounts
         .expect("driver config mounts should be set");
 
-    assert_eq!(mounts.len(), 3);
+    assert_eq!(mounts.len(), 4);
     assert_eq!(mounts[0].typ, Some(MountTypeEnum::VOLUME));
     assert_eq!(mounts[0].source.as_deref(), Some("work-nfs"));
     assert_eq!(mounts[0].target.as_deref(), Some("/sandbox/work"));

crates/openshell-driver-kubernetes/README.md

@@ -103,14 +103,70 @@ them. The cluster CNI must enforce ingress and egress `NetworkPolicy` for every
 sandbox namespace. Kubernetes accepts policy objects without confirming
 enforcement, so operators must verify CNI support before running sandboxes.
 
+### Control connection direction
+
+Every new runtime generation uses `runtime-initiated` control. The Sandbox
+annotation `openshell.ai/sandbox-runtime-control-connection` records the mode,
+bound to `openshell.ai/sandbox-runtime-control-generation`. A missing marker or
+a different current runtime generation identifies a legacy,
+`supervisor-initiated` generation, which keeps working until replaced. Stop/start
+upgrades it automatically: with both old Pods removed, the driver adds the
+supervisor Service and records the new mode and generation
+before releasing the replacement workload. Gateway recovery that recreates both
+Pods also upgrades the generation. Reconciliation never migrates a running
+generation. An unknown annotation value fails closed; there is no rollout flag.
+
+Only the dial direction differs. In both modes the sandbox runtime presents the
+generation-pinned TLS server identity and serves gRPC, and the supervisor
+verifies that identity, acts as the gRPC client, and sends the Sandbox Protocol
+JWT. The per-sandbox Service `os-boundary-<id>` always selects the paired
+workload Pod. A separate `os-supervisor-<id>` Service always selects the paired
+supervisor Pod. New gateways create both Services, retaining the legacy
+Service's UID, ClusterIP, and selector across restarts so an older gateway can
+still replace the generation during a rolling upgrade.
+
+For `runtime-initiated`:
+
+- The bootstrap Secret gives the runtime the Service ClusterIP and port. The
+  runtime retries with capped backoff until the supervisor listens, so neither
+  Pod waits on the other during bootstrap. The Service sets
+  `publishNotReadyAddresses` because the supervisor becomes Ready only after
+  the runtime attaches through it. The workload still cannot reach the control
+  port.
+- A per-sandbox `NetworkPolicy` `os-control-<id>` selects only this sandbox's
+  two Pods and admits one flow, from its workload Pod to its supervisor Pod on
+  `boundary_port` over TCP, as workload egress and supervisor ingress.
+  Supervisors may already be ingress-isolated by the managed SSH ingress policy;
+  when that policy is disabled, this one isolates them. Exec, file sync, and
+  logs need no other supervisor ingress because the supervisor opens those
+  connections. The CNI must still allow kubelet readiness probes. The
+  namespace-wide policies are unchanged.
+- The policy's UID and resourceVersion are bound into the boundary resource
+  claims and outer-fence evidence. Reconciliation suspends the workload if the
+  policy disappears, diverges, or changes generation, or if the active control
+  Service no longer selects the role the recorded mode requires.
+- Both Services and the control policy are owned by the Sandbox resource and are
+  deleted only with that sandbox.
+
+Deploy a gateway, supervisor image, and sandbox runtime image that all support
+`runtime-initiated` before using it. Older binaries reject the new descriptor
+shapes and fail closed.
+
+During a mixed-version rollout, an old gateway can restart a sandbox by
+creating its own compatible supervisor/runtime pair through the legacy
+Service. It preserves unknown annotations, so the mode marker is accepted only
+for its recorded generation. Do not replace an individual generation's
+supervisor or runtime binary with one that cannot read its descriptor.
+
 Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
 container stages the sandbox bootstrap into memory, and the sandbox removes it
 before starting untrusted code. The other Secret is mounted only by the
 supervisor; when `proxy_ca_bundle` is configured it also carries the operator's
 corporate proxy CA bundle, which the gateway reads from its own filesystem so
 the anchor stays in the gateway's trust domain rather than the sandbox
 namespace. The TLS channel binds the namespace, Sandbox CR, workload Pod,
-supervisor Pod, and shared network-policy identities. Stop deletes the workload
+supervisor Pod, and shared network-policy identities, plus the per-sandbox
+control policy for `runtime-initiated` sandboxes. Stop deletes the workload
 and supervisor Pods. Start rotates both Secrets and creates a new supervisor
 Pod before releasing a new workload Pod. It also renders the trusted init
 container again from gateway config, so a sandbox created by an older gateway