feat(snap): ship the VM driver in the snap and use it by default

.agents/skills/test-release-canary/SKILL.md

@@ -16,8 +16,8 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti
 | `macos` | `macos-latest-xlarge` | Installs the dev Homebrew artifacts, reaches the VM gateway, and creates, executes in, and deletes a sandbox. |
 | `ubuntu-deb` | `ubuntu-latest` | Installs the dev Debian package, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. |
 | `fedora` | `fedora:latest` container | Installs the dev RPM packages, reaches the Podman gateway, and creates, executes in, and deletes a sandbox. |
-| `ubuntu-snap-system-docker` | `ubuntu-latest` | Uses `install.sh` to install the snap from `latest/edge`, reuses system Docker, verifies the packaged prover version and a local policy boundary check, reaches the Docker gateway, creates, executes in, and deletes a sandbox, and verifies that the Docker snap is not installed. |
-| `ubuntu-snap-docker-preflight` | `ubuntu-latest` | Verifies that `install.sh` rejects the OpenShell snap path when Docker is absent or supplied by the Docker snap, without installing OpenShell. |
+| `ubuntu-snap-vm` | `linux-amd64-cpu8` | Uses `install.sh` to install the snap from `latest/edge` without Docker, verifies the VM driver, KVM interface, e2fsprogs tools, and packaged prover, then creates, executes in, and deletes a VM sandbox. |
+| `ubuntu-snap-docker` | `ubuntu-latest` | Installs the snap from `latest/edge` with system Docker preinstalled, selects Docker through the user service environment, verifies the system `:docker` connection, then creates, executes in, and deletes a Docker sandbox. |
 | `kubernetes` | `ubuntu-latest` + kind | Installs the dev Helm chart, reaches the in-cluster gateway, and creates, executes in, and deletes a sandbox using the published runtime images. |
 
 All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject
@@ -27,10 +27,11 @@ does not contribute to product usage metrics.
 
 The workflow sets `OPENSHELL_VERSION=dev` for every `install.sh` job. Positive
 jobs consume the rolling dev release produced by the triggering workflow. The
-system-Docker snap lane tracks `latest/edge`; the missing-Docker lane exits
-before installing a snap. The Debian and Kubernetes CLI lanes remove snapd so
-they continue to exercise the dev Debian package. Kubernetes pins the matching
-`0.0.0-dev` chart and `:dev` images.
+snap VM and Docker lanes track `latest/edge`. The VM lane verifies the snap
+works on its own, while the Docker lane verifies an explicit
+`OPENSHELL_COMPUTE_DRIVER` override. The Debian and Kubernetes CLI lanes remove
+snapd so they continue to exercise the dev Debian package. Kubernetes pins the
+matching `0.0.0-dev` chart and `:dev` images.
 
 RPM package installation also has tmachine conformance coverage in
 `Branch E2E Checks` (with `test:e2e`), `Release Dev`, `Release`, and `Pre-release`. Those
@@ -143,9 +144,9 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i
 | `macos`/`ubuntu-deb`/`fedora` job fails on `install.sh` | Dev release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
 | Sandbox create or exec fails | Published sandbox and supervisor artifacts are missing, incompatible, or cannot establish the protected runtime channel. | Gateway logs plus Docker, Podman, VM, snap, or Kubernetes runtime diagnostics for the job. |
 | `macos`/`ubuntu-deb`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. |
-| `ubuntu-snap-system-docker` fails during `install.sh` | System Docker was unavailable to the runner user, the edge revision or automatic interfaces were unavailable, or the user gateway did not become reachable. | Failure diagnostics dump system Docker, snap service/connection/change state, user and legacy gateway journals, snap logs, and port 17670 listeners. |
-| `ubuntu-snap-system-docker` fails during the prover checks | The prover artifact is missing or packaged for the wrong architecture, `openshell.prover` is not exposed or confined to read the test policies, or its solver linkage is not runnable. | The `Install and check status` and `Check a policy boundary with the snap prover` steps, plus `snap info openshell` and `snap connections openshell`. |
-| `ubuntu-snap-docker-preflight` unexpectedly succeeds | The installer no longer fails before installing the OpenShell snap when Docker is absent or supplied by the Docker snap. | Inspect `install.log`, `docker-snap.log`, `snap list`, and snapd changes. |
+| `ubuntu-snap-vm` fails during `install.sh` or sandbox creation | The edge revision is unavailable, KVM is inaccessible, the VM driver or e2fsprogs is missing, or the user gateway did not become reachable. | Failure diagnostics dump `/dev/kvm` ACLs, snap service/connection/change state, user and legacy gateway journals, snap logs, and port 17670 listeners. |
+| `ubuntu-snap-vm` fails during the prover checks | The prover artifact is missing or packaged for the wrong architecture, `openshell.prover` is not exposed or confined to read the test policies, or its solver linkage is not runnable. | The `Install and check status` and `Check a policy boundary with the snap prover` steps, plus `snap info openshell` and `snap connections openshell`. |
+| `ubuntu-snap-docker` fails during installation or sandbox creation | System Docker is unavailable, the service environment did not preserve the Docker selector, the snap Docker plug did not connect to `:docker`, or confinement blocked the Docker sandbox. | Failure diagnostics include Docker status and logs, the user manager environment, snap connections and changes, gateway journals, and port 17670 listeners. |
 | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |
 | `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. |
 | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. |

.github/workflows/release-build.yml

@@ -210,7 +210,7 @@ jobs:
 
   build-snap:
     name: Build Snap
-    needs: [compute-versions, build-binaries]
+    needs: [compute-versions, build-binaries, build-vm-driver]
     uses: ./.github/workflows/snap-package.yml
     with:
       checkout-ref: ${{ needs.compute-versions.outputs.source_sha }}

.github/workflows/release-canary.yml

@@ -191,10 +191,10 @@ jobs:
         run: |
           docker rm -f "${FEDORA_CANARY_CONTAINER}" >/dev/null 2>&1 || true
 
-  ubuntu-snap-system-docker:
-    name: Ubuntu snap with system Docker
+  ubuntu-snap-vm:
+    name: Ubuntu snap with VM driver
     if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
-    runs-on: ubuntu-latest
+    runs-on: linux-amd64-cpu8
     env:
       OPENSHELL_INSTALL_METHOD: snap
     timeout-minutes: 20
@@ -203,20 +203,20 @@ jobs:
         run: |
           set -euo pipefail
           sudo apt-get update
-          sudo apt-get install -y snapd
+          sudo apt-get install -y acl snapd
           sudo systemctl enable --now snapd.socket
           sudo systemctl start snapd
           sudo snap wait system seed.loaded
 
-      - name: Ensure Docker
+      - name: Ensure KVM
         run: |
           set -euo pipefail
-          if ! command -v docker >/dev/null 2>&1; then
-            sudo apt-get update
-            sudo apt-get install -y docker.io
-          fi
-          sudo systemctl start docker || sudo service docker start
-          docker info
+          sudo systemctl stop docker.service docker.socket 2>/dev/null || true
+          sudo apt-get -y --purge remove docker-ce docker-ce-cli containerd.io \
+            docker-buildx-plugin docker-compose-plugin docker.io 2>/dev/null || true
+          ! command -v docker >/dev/null 2>&1
+          test -e /dev/kvm
+          sudo setfacl -m "u:${USER}:rw" /dev/kvm
 
       - name: Install and check status
         run: |
@@ -229,15 +229,12 @@ jobs:
           systemctl --user set-environment \
             "OPENSHELL_TELEMETRY_ENABLED=${OPENSHELL_TELEMETRY_ENABLED}"
           curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh |
-            DOCKER_HOST=unix:///tmp/nonexistent-docker.sock DOCKER_CONTEXT=nonexistent sh
+            sh
           sudo snap list openshell
-          if sudo snap list docker >/dev/null 2>&1; then
-            echo "install.sh unexpectedly installed the Docker snap" >&2
-            exit 1
-          fi
           snap info openshell | grep -E '^tracking: +latest/edge$'
-          docker info
-          sudo snap connections openshell | grep -E '^docker +openshell:docker +:docker +'
+          sudo snap connections openshell | grep -E '^kvm +openshell:kvm +:kvm +'
+          snap run --shell openshell.user-gateway -c \
+            'test -x "$SNAP/bin/openshell-driver-vm" && command -v mke2fs && command -v debugfs && command -v e2fsck'
           openshell --version
           openshell.prover --version
           snap services openshell
@@ -262,6 +259,12 @@ jobs:
           ! systemctl is-active --quiet snap.openshell.system-gateway.service
           journalctl --user -u snap.openshell.user-gateway.service --no-pager |
             grep -F "mTLS user authentication enabled"
+          invocation_id=$(systemctl --user show snap.openshell.user-gateway.service \
+            --property=InvocationID --value)
+          test -n "$invocation_id"
+          journalctl --user "_SYSTEMD_INVOCATION_ID=${invocation_id}" --no-pager |
+            grep -F 'Using compute driver' |
+            grep -F 'driver=vm'
           openshell gateway list | grep -F "https://127.0.0.1:17670"
           openshell status
 
@@ -299,9 +302,8 @@ jobs:
         if: failure()
         run: |
           set +e
-          docker info
-          sudo systemctl status docker.service --no-pager
-          sudo journalctl -b -u docker.service --no-pager -n 300
+          stat /dev/kvm
+          getfacl /dev/kvm
           sudo snap services openshell
           sudo snap connections openshell
           sudo snap changes
@@ -313,8 +315,8 @@ jobs:
           snap logs openshell.user-gateway -n=300
           sudo ss -ltnp '( sport = :17670 )'
 
-  ubuntu-snap-docker-preflight:
-    name: Ubuntu snap Docker preflight
+  ubuntu-snap-docker:
+    name: Ubuntu snap with Docker driver
     if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
     runs-on: ubuntu-latest
     env:
@@ -330,50 +332,76 @@ jobs:
           sudo systemctl start snapd
           sudo snap wait system seed.loaded
 
-      - name: Remove system Docker
+      - name: Ensure system Docker
         run: |
           set -euo pipefail
-          sudo systemctl stop docker.service docker.socket 2>/dev/null || true
-          sudo apt-get -y --purge remove docker-ce docker-ce-cli containerd.io \
-            docker-buildx-plugin docker-compose-plugin docker.io 2>/dev/null || true
-          ! command -v docker >/dev/null 2>&1
+          if ! command -v docker >/dev/null 2>&1; then
+            sudo apt-get install -y docker.io
+          fi
+          sudo systemctl start docker || sudo service docker start
+          if sudo snap list docker >/dev/null 2>&1; then
+            echo "the Docker snap must not provide the canary daemon" >&2
+            exit 1
+          fi
+          docker info
 
-      - name: Verify installer rejects missing Docker
+      - name: Select Docker for the snap gateway
         run: |
           set -euo pipefail
-          curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh -o install.sh
-          if OPENSHELL_VERSION=dev sh install.sh >install.log 2>&1; then
-            echo "install.sh unexpectedly succeeded without Docker" >&2
-            cat install.log >&2
-            exit 1
-          fi
-          cat install.log
-          grep -F "Docker is required before installing the OpenShell snap" install.log
-          ! sudo snap list docker >/dev/null 2>&1
-          ! sudo snap list openshell >/dev/null 2>&1
+          systemctl --user set-environment \
+            OPENSHELL_COMPUTE_DRIVER=docker \
+            "OPENSHELL_TELEMETRY_ENABLED=${OPENSHELL_TELEMETRY_ENABLED}"
+          systemctl --user show-environment |
+            grep -Fx 'OPENSHELL_COMPUTE_DRIVER=docker'
 
-      - name: Verify installer rejects Docker snap
+      - name: Install and check status
         run: |
           set -euo pipefail
-          sudo snap install docker
-          if OPENSHELL_VERSION=dev sh install.sh >docker-snap.log 2>&1; then
-            echo "install.sh unexpectedly succeeded with the Docker snap" >&2
-            cat docker-snap.log >&2
-            exit 1
-          fi
-          cat docker-snap.log
-          grep -F "the Docker snap is not currently compatible with OpenShell" docker-snap.log
-          sudo snap list docker >/dev/null
-          ! sudo snap list openshell >/dev/null 2>&1
+          curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh |
+            sh
+          sudo snap list openshell
+          snap info openshell | grep -E '^tracking: +latest/edge$'
+          sudo snap connections openshell |
+            grep -E '^docker +openshell:docker +:docker +'
+          docker info
+          snap services openshell
+          systemctl --user is-enabled --quiet snap.openshell.user-gateway.service
+          systemctl --user is-active snap.openshell.user-gateway.service
+          ! systemctl is-enabled --quiet snap.openshell.system-gateway.service
+          ! systemctl is-active --quiet snap.openshell.system-gateway.service
+          journalctl --user -u snap.openshell.user-gateway.service --no-pager |
+            grep -F 'Using compute driver' |
+            grep -F 'driver=docker'
+          openshell gateway list | grep -F 'https://127.0.0.1:17670'
+          openshell status
 
-      - name: Collect snap diagnostics
+      - name: Create and exercise a sandbox
+        run: |
+          set -euo pipefail
+          sandbox="rc-docker-${GITHUB_RUN_ID}"
+          openshell sandbox create --name "$sandbox" --detach
+          openshell sandbox exec --name "$sandbox" --no-tty -- true
+          openshell sandbox delete "$sandbox"
+
+      - name: Collect snap and Docker diagnostics
         if: failure()
         run: |
           set +e
-          cat install.log
-          cat docker-snap.log
+          docker info
+          sudo systemctl status docker.service --no-pager
+          sudo journalctl -b -u docker.service --no-pager -n 300
+          systemctl --user show-environment |
+            grep -E '^OPENSHELL_(COMPUTE_DRIVER|TELEMETRY_ENABLED)='
+          sudo snap services openshell
+          sudo snap connections openshell
           sudo snap changes
+          systemctl --user status snap.openshell.user-gateway.service --no-pager
+          journalctl --user -u snap.openshell.user-gateway.service --no-pager -n 300
+          sudo systemctl status snap.openshell.system-gateway.service --no-pager
+          sudo journalctl -b -u snap.openshell.system-gateway.service --no-pager -n 300
           sudo journalctl -b -u snapd.service --no-pager -n 300
+          snap logs openshell.user-gateway -n=300
+          sudo ss -ltnp '( sport = :17670 )'
 
   kubernetes:
     name: Kubernetes Helm (kind)

.github/workflows/release-dev.yml

@@ -247,7 +247,7 @@ jobs:
 
   build-snap:
     name: Build snap
-    needs: [compute-versions, build-binaries]
+    needs: [compute-versions, build-binaries, build-vm-driver]
     uses: ./.github/workflows/snap-package.yml
     with:
       checkout-ref: ${{ github.sha }}

.github/workflows/snap-package.yml

@@ -88,13 +88,20 @@ jobs:
           name: openshell-sandbox-${{ matrix.rust_arch }}-unknown-linux-musl
           path: prebuilt/sandbox
 
+      - name: Download prebuilt VM driver
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: openshell-driver-vm-${{ matrix.rust_arch }}-unknown-linux-gnu
+          path: prebuilt/driver-vm
+
       - name: Configure prebuilt binaries
         run: |
           set -euo pipefail
           chmod +x prebuilt/cli/openshell
           chmod +x prebuilt/prover/openshell-prover
           chmod +x prebuilt/gateway/openshell-gateway
           chmod +x prebuilt/sandbox/openshell-sandbox
+          chmod +x prebuilt/driver-vm/openshell-driver-vm
           ls -laR prebuilt/
 
       - name: Prepare snap build directory
@@ -106,6 +113,7 @@ jobs:
           cp prebuilt/prover/openshell-prover snap/prebuilt/openshell-prover
           cp prebuilt/gateway/openshell-gateway snap/prebuilt/openshell-gateway
           cp prebuilt/sandbox/openshell-sandbox snap/prebuilt/openshell-sandbox
+          cp prebuilt/driver-vm/openshell-driver-vm snap/prebuilt/openshell-driver-vm
 
           cp tasks/scripts/snap-gateway-wrapper.sh snap/prebuilt/openshell-gateway-wrapper
           cp LICENSE snap/prebuilt/

CI.md

@@ -645,7 +645,7 @@ These workflows run after merge to publish dev/tagged artifacts and verify them.
 | `.github/workflows/release-qualification.yml` | Reusable qualification suites and result aggregation. Runs security scans, conformance and feature integration, Docker and VM E2E, protobuf compatibility, and advisory compatibility review. |
 | `.github/workflows/snap-package.yml` | Builds snap and component artifacts for Release Dev, Release, and Pre-release without Store credentials or publication. |
 | `.github/workflows/snap-publish.yml` | Uploads existing snap and component artifacts to the Store without rebuilding. Release Dev calls it directly for `latest/edge` after snap builds and requires it to succeed before creating the dev release. Release calls it through `release-publish.yml` for `latest/stable` after qualification and release publication succeed. |
-| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts in the `macos`, `ubuntu-deb`, `ubuntu-snap-system-docker`, `fedora`, and `kubernetes` (kind + Helm) jobs. Each job reaches its gateway and creates, exercises, and deletes a sandbox. The `ubuntu-snap-system-docker` job verifies a compatible system Docker lifecycle, while `ubuntu-snap-docker-preflight` tests fail-fast behavior when Docker is absent or supplied by the Docker snap. The positive snap lane also runs a local policy containment check with the packaged prover. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref <branch>`). See the `test-release-canary` skill for the playbook and local kind reproduction. |
+| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts in the `macos`, `ubuntu-deb`, `ubuntu-snap-vm`, `ubuntu-snap-docker`, `fedora`, and `kubernetes` (kind + Helm) jobs. The snap lanes verify both the standalone VM driver and an explicit Docker driver selection. Each job reaches its gateway and creates, exercises, and deletes a sandbox. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref <branch>`). See the `test-release-canary` skill for the playbook and local kind reproduction. |
 
 ## Required status contexts