feat(sandbox): initiate supervisor connections across compute drivers

[drew]

Summary

New sandbox generations initiate their protected control connection to the supervisor across Kubernetes, Docker, Podman, and VM. Existing descriptors remain readable, and running legacy sandboxes retain their connection direction. This is an independent prerequisite to the shared-supervisor prototype in #4429.

Only connection establishment changes. The runtime remains the TLS/gRPC server, and the supervisor verifies its generation-pinned identity and supplies the existing bearer credential. No public RPC or operator configuration field is added.

Related Issue

Closes #4652

Changes

• Keep the runtime transport enum, validation, and connection handling in openshell-sandbox-backend. Operator-registered endpoints convert into backend-owned transports at the delegated launch boundary. Core and isolation-interface have no changes in this PR; legacy descriptor encodings remain unchanged.
• Share bounded handshake processing, authenticated spare connections, discovery, attachment, and reconnect handling across runtime-initiated TCP, Unix, and vsock transports.
• Kubernetes preserves os-boundary-<id> as the workload endpoint and adds os-supervisor-<id> for sandbox-initiated connections, with a narrowly scoped control NetworkPolicy. New gateways create both Services and preserve the legacy Service UID, ClusterIP, and selector across restart. The mode marker is bound to the runtime generation so reconciliation recognizes an older gateway’s replacement as legacy. Legacy sandboxes gain the supervisor endpoint when started by a new gateway; the driver no longer patches Service selectors.
• Docker and Podman keep workload networking disabled and use a separate writable socket volume. Bootstrap credentials remain read-only in the supervisor. Docker stages the socket directory under the supervisor identity; Podman accommodates differing rootless user-namespace mappings with a private sticky socket directory.
• Docker restarts retain the saved descriptor's direction. Podman records the direction in restart metadata and defaults missing metadata to legacy, preserving old container mounts and bootstrap behavior.
• Libkrun guests dial host CID 2 through a guest-initiated vsock-to-Unix mapping. QEMU guests dial a distinct host vsock port; the listener checks the expected guest CID before pinned TLS authentication. New VM boots select the new mode.
• MXC has no supervisor–Sandbox Protocol connection to reverse. Operator-registered external backend endpoints retain their existing contract. Runtime-initiated endpoint variants remain internal to driver-provisioned runtimes.
• Update crate implementation notes and relevant operator skill guidance. Add focused Kubernetes Service and generation-marker regression checks; no published docs changes.

Each new-mode generation requires compatible supervisor and sandbox runtime binaries; older binaries reject its new descriptors. Kubernetes retains separate fixed endpoints so older gateways can replace a generation with their own compatible pair during a rolling upgrade. Other drivers retain their existing rollback limitations. Connection establishment does not confer ownership or provide HA takeover.

Testing

Current Kubernetes Service compatibility change (c1c278611):

• Kubernetes driver: 298 library tests passed; driver and focused E2E target Clippy passed with warnings denied.
• Helm: 339 tests across 24 suites passed; repository pre-commit checks passed.
• Added endpoint/UID/ClusterIP preservation coverage to the existing runtime-reconciliation E2E, plus an ignored mixed-version lifecycle test requiring an old gateway fixture.
• Live Kubernetes lifecycle E2E passed (49.40 s): a stopped sandbox starts with the configured bootstrap image, gains a missing supervisor Service, retains its workspace, and preserves the legacy Service UID, ClusterIP, and workload selector.
• Mixed-version Kubernetes E2E passed (23.93 s): the patched gateway (c1c278611) and old gateway (35a3390db, with its matching supervisor/runtime images) run against the same PostgreSQL database. A sandbox created through the new gateway stops/starts through the old gateway, executes successfully, then stops/starts and executes through the new gateway again. Generation markers and the unchanged legacy Service are asserted. The new gateway Deployment has two Ready replicas during the test.

Earlier transport validation:

• Transport ownership refactor: 829 existing core, sandbox-backend, and supervisor library tests passed; automatic commit checks passed. No new tests added.

The transport ownership refactor preserves runtime behavior and serialization. The E2E results below were recorded before that type relocation.

• Workspace compile, Clippy with warnings denied, formatting, and automatic commit checks passed.
• Existing Docker library suite: 147 passed, two existing ignored tests. Existing Podman library suite: 245 passed after necessary fixture updates.
• Linux ARM64 libkrun conformance: 8/8 passed, including lifecycle, exec, file transfer, policy-local, and policy proposals.
• Podman conformance: 7/8 passed initially; the failed policy_local scenario passed an isolated rerun without source changes. The initial sandbox transitioned from Ready to Error with ContainerExited; its cause remains unestablished.
• Docker conformance: 8/8 passed after fixing socket directory ownership during staging.

The VM library suite passed 256 tests and failed the existing stopping_a_sandbox_lets_the_supervisor_drain_before_closing_liveness test. Its shell helper exits before the test reaches the changed transport code; the unchanged helper also exits in a standalone reproduction on this host. This failure remains visible; no assertion was weakened.

Before this extension, the rebased Kubernetes implementation passed 8/8 conformance and a full 100/100 Kubernetes E2E rerun with four existing skips. Kubernetes has not been rerun after this transport extension. QEMU is covered by Linux compilation; QEMU and macOS have not received live E2E validation here. CI will validate the pushed revision.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Crate implementation notes and related operator skill updated

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4656.docs.buildwithfern.com/openshell

[github-actions]

Label test:e2e applied, but pull-request/4656 does not exist yet. A maintainer needs to comment /ok to test 758a70e4b813b3f7ec502cc2cc2e107bccaf2a9d to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[github-actions]

Label test:e2e-kubernetes applied, but pull-request/4656 does not exist yet. A maintainer needs to comment /ok to test 758a70e4b813b3f7ec502cc2cc2e107bccaf2a9d to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[drew]

/ok to test b0041e4

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. state — No explicit risk acceptance found

v0.1.4 persisted only supervisor-initiated unix, tcp, or vsock descriptors, while v0.1.5 makes new generations persist runtime-initiated descriptors and redirects the Kubernetes boundary Service to the supervisor.

• Impact: After v0.1.5 creates or restarts a sandbox, rolling back to v0.1.4 leaves older supervisors, runtimes, and Kubernetes drivers unable to start or restart it without deletion.
• Action: The documented delete-or-roll-forward workaround does not address a patch train; keep supervisor-initiated state as the default or add a downgrade migration that restores v0.1.4-readable descriptors and Kubernetes Service selectors.
• PR: feat(sandbox): initiate supervisor connections across compute drivers #4656

Review details

• Execution: complete
• Candidate: PR #4656 (d42b5283c2b43be479d5258cfd5d47b9e76e3a5e)
• Baseline: 604b9f66f685c582114cd571ae3a10accec23ec2 (604b9f66f685c582114cd571ae3a10accec23ec2)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

2. helm — No explicit risk acceptance found

Legacy Kubernetes sandbox restart now patches its existing Service, while the baseline out-of-band RBAC contract required only create and get on Services.

• Impact: Upgrading with rbac.create=false and an unchanged custom Role causes stop/start or recovery of an existing sandbox to fail with a forbidden Service patch.
• Action: Avoid the new permission requirement in this patch train, or require and document adding patch on Services to out-of-band Roles before upgrade.
• PR: feat(sandbox): initiate supervisor connections across compute drivers #4656

Review details

• Execution: complete
• Candidate: PR #4656 (1cb80f7e88926d409001e750c4682879d66a0746)
• Baseline: 604b9f66f685c582114cd571ae3a10accec23ec2 (604b9f66f685c582114cd571ae3a10accec23ec2)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[pimlock]

One small comment.

I walked through backwards compatibility of this change and one thing potentially to look more into is partial downtime during rolling update for kube, where we have both old and new versions running. Added details from investigation here: https://gist.github.com/pimlock/0194dfa3319ce31a0759d037fada76a0

Another thing, but this one is probably okay, is downgrading gateway to older version (e.g. rollback) with Docker or Kube driver would break existing sandboxes. It looks like VM driver gateway could be rolled back without impact.

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. configuration — Risk acceptance could not be verified

Existing sandbox_runtime_image and supervisor_image pins remain valid TOML, but new Docker and Podman containers now receive runtime-initiated descriptors that v0.1.4 binaries reject.

• Impact: After upgrading only the gateway, creating a sandbox with pinned v0.1.4 runtime images fails until both images are upgraded together.
• Action: Keep supervisor-initiated descriptors for older configured images or add version negotiation, and document upgrading both image pins before the gateway.
• PR: feat(sandbox): initiate supervisor connections across compute drivers #4656

2. state — Risk acceptance could not be verified

v0.1.4 persisted only supervisor-initiated descriptors, while v0.1.5 persists runtime-initiated descriptors and redirects new Kubernetes generations to the supervisor.

• Impact: Rolling back after v0.1.5 creates or restarts a sandbox leaves v0.1.4 unable to start or restart that sandbox without deletion.
• Action: Keep supervisor-initiated state as the patch-train default or provide a downgrade migration restoring v0.1.4-readable descriptors and Kubernetes Service selectors.
• PR: feat(sandbox): initiate supervisor connections across compute drivers #4656

3. helm — Risk acceptance could not be verified

Restarting a legacy Kubernetes sandbox now patches its existing Service, whereas the baseline out-of-band RBAC contract granted only create and get on Services.

• Impact: Upgrading with rbac.create=false or a pre-created unchanged ClusterRole causes stop/start or recovery of an existing sandbox to fail with a forbidden Service patch.
• Action: Avoid the new permission requirement in this patch train or document and require adding patch on Services to every out-of-band Role before upgrade.
• PR: feat(sandbox): initiate supervisor connections across compute drivers #4656

Review details

• Execution: complete
• Candidate: PR #4656 (cce73772ae1d8a1f50cf2d1243c77c0dac7201af)
• Baseline: 604b9f66f685c582114cd571ae3a10accec23ec2 (604b9f66f685c582114cd571ae3a10accec23ec2)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[drew]

@pimlock fixed this in c1c2786. We now keep os-boundary-<id> pointing at the workload and use a separate os-supervisor-<id> Service for the new connection direction. We also bind the mode marker to the runtime generation so an old gateway restarting a sandbox doesn't leave stale metadata.

Tested with old and new gateways sharing a database: new gateway creates → old gateway restarts → new gateway restarts, with exec working after each restart and the legacy Service unchanged. Both that test and the lifecycle E2E passed.

This fixes the Kubernetes rolling-upgrade issue; the broader downgrade limitations still apply- but I think that's ok for now.