diff --git a/.agents/skills/test-release-canary/SKILL.md b/.agents/skills/test-release-canary/SKILL.md index 5081c85e50..0b952efb2f 100644 --- a/.agents/skills/test-release-canary/SKILL.md +++ b/.agents/skills/test-release-canary/SKILL.md @@ -16,8 +16,9 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti | `macos` | `macos-latest-xlarge` | Installs the dev Homebrew artifacts, reaches the VM gateway, and creates, executes in, and deletes a sandbox. | | `ubuntu-deb` | `ubuntu-latest` | Installs the dev Debian package, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. | | `fedora` | `fedora:latest` container | Installs the dev RPM packages, reaches the Podman gateway, and creates, executes in, and deletes a sandbox. | -| `ubuntu-snap-system-docker` | `ubuntu-latest` | Uses `install.sh` to install the snap from `latest/edge`, reuses system Docker, verifies the packaged prover version and a local policy boundary check, reaches the Docker gateway, creates, executes in, and deletes a sandbox, and verifies that the Docker snap is not installed. | -| `ubuntu-snap-docker-preflight` | `ubuntu-latest` | Verifies that `install.sh` rejects the OpenShell snap path when Docker is absent or supplied by the Docker snap, without installing OpenShell. | +| `ubuntu-snap-vm` | `linux-amd64-cpu8` | Uses `install.sh` to install the snap from `latest/edge` without Docker, verifies the VM driver, KVM interface, e2fsprogs tools, and packaged prover, then creates, executes in, and deletes a VM sandbox. | +| `ubuntu-snap-docker` | `ubuntu-latest` | Installs the snap from `latest/edge` with system Docker preinstalled, selects Docker through the user service environment, verifies the system `:docker` connection, then creates, executes in, and deletes a Docker sandbox. | +| `ubuntu-snap-podman` | `ubuntu-26.04` | Installs the snap from `latest/edge` with rootless Podman 5 preinstalled, selects Podman through the user service environment, verifies the system `:podman` connection and rootless user socket, then creates, executes in, and deletes a Podman sandbox. | | `kubernetes` | `ubuntu-latest` + kind | Installs the dev Helm chart, reaches the in-cluster gateway, and creates, executes in, and deletes a sandbox using the published runtime images. | All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject @@ -27,10 +28,11 @@ does not contribute to product usage metrics. The workflow sets `OPENSHELL_VERSION=dev` for every `install.sh` job. Positive jobs consume the rolling dev release produced by the triggering workflow. The -system-Docker snap lane tracks `latest/edge`; the missing-Docker lane exits -before installing a snap. The Debian and Kubernetes CLI lanes remove snapd so -they continue to exercise the dev Debian package. Kubernetes pins the matching -`0.0.0-dev` chart and `:dev` images. +snap VM, Docker, and Podman lanes track `latest/edge`. The VM lane verifies the +snap works on its own, while the Docker and Podman lanes verify explicit +`OPENSHELL_COMPUTE_DRIVER` overrides. The Debian and Kubernetes CLI lanes remove +snapd so they continue to exercise the dev Debian package. Kubernetes pins the +matching `0.0.0-dev` chart and `:dev` images. RPM package installation also has tmachine conformance coverage in `Branch E2E Checks` (with `test:e2e`), `Release Dev`, `Release`, and `Pre-release`. Those @@ -143,9 +145,10 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i | `macos`/`ubuntu-deb`/`fedora` job fails on `install.sh` | Dev release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | | Sandbox create or exec fails | Published sandbox and supervisor artifacts are missing, incompatible, or cannot establish the protected runtime channel. | Gateway logs plus Docker, Podman, VM, snap, or Kubernetes runtime diagnostics for the job. | | `macos`/`ubuntu-deb`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. | -| `ubuntu-snap-system-docker` fails during `install.sh` | System Docker was unavailable to the runner user, the edge revision or automatic interfaces were unavailable, or the user gateway did not become reachable. | Failure diagnostics dump system Docker, snap service/connection/change state, user and legacy gateway journals, snap logs, and port 17670 listeners. | -| `ubuntu-snap-system-docker` fails during the prover checks | The prover artifact is missing or packaged for the wrong architecture, `openshell.prover` is not exposed or confined to read the test policies, or its solver linkage is not runnable. | The `Install and check status` and `Check a policy boundary with the snap prover` steps, plus `snap info openshell` and `snap connections openshell`. | -| `ubuntu-snap-docker-preflight` unexpectedly succeeds | The installer no longer fails before installing the OpenShell snap when Docker is absent or supplied by the Docker snap. | Inspect `install.log`, `docker-snap.log`, `snap list`, and snapd changes. | +| `ubuntu-snap-vm` fails during `install.sh` or sandbox creation | The edge revision is unavailable, KVM is inaccessible, the VM driver or e2fsprogs is missing, or the user gateway did not become reachable. | Failure diagnostics dump `/dev/kvm` ACLs, snap service/connection/change state, user and legacy gateway journals, snap logs, and port 17670 listeners. | +| `ubuntu-snap-vm` fails during the prover checks | The prover artifact is missing or packaged for the wrong architecture, `openshell.prover` is not exposed or confined to read the test policies, or its solver linkage is not runnable. | The `Install and check status` and `Check a policy boundary with the snap prover` steps, plus `snap info openshell` and `snap connections openshell`. | +| `ubuntu-snap-docker` fails during installation or sandbox creation | System Docker is unavailable, the service environment did not preserve the Docker selector, the snap Docker plug did not connect to `:docker`, or confinement blocked the Docker sandbox. | Failure diagnostics include Docker status and logs, the user manager environment, snap connections and changes, gateway journals, and port 17670 listeners. | +| `ubuntu-snap-podman` fails during installation or sandbox creation | Rootless Podman or its user socket is unavailable, the service environment did not preserve the Podman selector, the snap Podman plug did not connect to `:podman`, or confinement blocked the Podman sandbox. | Failure diagnostics include Podman system and container state, user socket/service logs, the user manager environment, snap connections and changes, gateway journals, kernel audit denials, and port 17670 listeners. | | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. | | `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. | | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. | diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml index e92a05df87..159eb5c41a 100644 --- a/.github/workflows/release-build.yml +++ b/.github/workflows/release-build.yml @@ -210,7 +210,7 @@ jobs: build-snap: name: Build Snap - needs: [compute-versions, build-binaries] + needs: [compute-versions, build-binaries, build-vm-driver] uses: ./.github/workflows/snap-package.yml with: checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index 514a7bc8a8..4cd0082e51 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -191,10 +191,10 @@ jobs: run: | docker rm -f "${FEDORA_CANARY_CONTAINER}" >/dev/null 2>&1 || true - ubuntu-snap-system-docker: - name: Ubuntu snap with system Docker + ubuntu-snap-vm: + name: Ubuntu snap with VM driver if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} - runs-on: ubuntu-latest + runs-on: linux-amd64-cpu8 env: OPENSHELL_INSTALL_METHOD: snap timeout-minutes: 20 @@ -203,20 +203,20 @@ jobs: run: | set -euo pipefail sudo apt-get update - sudo apt-get install -y snapd + sudo apt-get install -y acl snapd sudo systemctl enable --now snapd.socket sudo systemctl start snapd sudo snap wait system seed.loaded - - name: Ensure Docker + - name: Ensure KVM run: | set -euo pipefail - if ! command -v docker >/dev/null 2>&1; then - sudo apt-get update - sudo apt-get install -y docker.io - fi - sudo systemctl start docker || sudo service docker start - docker info + sudo systemctl stop docker.service docker.socket 2>/dev/null || true + sudo apt-get -y --purge remove docker-ce docker-ce-cli containerd.io \ + docker-buildx-plugin docker-compose-plugin docker.io 2>/dev/null || true + ! command -v docker >/dev/null 2>&1 + test -e /dev/kvm + sudo setfacl -m "u:${USER}:rw" /dev/kvm - name: Install and check status run: | @@ -229,15 +229,12 @@ jobs: systemctl --user set-environment \ "OPENSHELL_TELEMETRY_ENABLED=${OPENSHELL_TELEMETRY_ENABLED}" curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | - DOCKER_HOST=unix:///tmp/nonexistent-docker.sock DOCKER_CONTEXT=nonexistent sh + sh sudo snap list openshell - if sudo snap list docker >/dev/null 2>&1; then - echo "install.sh unexpectedly installed the Docker snap" >&2 - exit 1 - fi snap info openshell | grep -E '^tracking: +latest/edge$' - docker info - sudo snap connections openshell | grep -E '^docker +openshell:docker +:docker +' + sudo snap connections openshell | grep -E '^kvm +openshell:kvm +:kvm +' + snap run --shell openshell.user-gateway -c \ + 'test -x "$SNAP/bin/openshell-driver-vm" && command -v mke2fs && command -v debugfs && command -v e2fsck' openshell --version openshell.prover --version snap services openshell @@ -262,6 +259,12 @@ jobs: ! systemctl is-active --quiet snap.openshell.system-gateway.service journalctl --user -u snap.openshell.user-gateway.service --no-pager | grep -F "mTLS user authentication enabled" + invocation_id=$(systemctl --user show snap.openshell.user-gateway.service \ + --property=InvocationID --value) + test -n "$invocation_id" + journalctl --user "_SYSTEMD_INVOCATION_ID=${invocation_id}" --no-pager | + grep -F 'Using compute driver' | + grep -F 'driver=vm' openshell gateway list | grep -F "https://127.0.0.1:17670" openshell status @@ -296,12 +299,99 @@ jobs: openshell sandbox delete "$sandbox" - name: Collect snap diagnostics + if: failure() + run: | + set +e + stat /dev/kvm + getfacl /dev/kvm + sudo snap services openshell + sudo snap connections openshell + sudo snap changes + systemctl --user status snap.openshell.user-gateway.service --no-pager + journalctl --user -u snap.openshell.user-gateway.service --no-pager -n 300 + sudo systemctl status snap.openshell.system-gateway.service --no-pager + sudo journalctl -b -u snap.openshell.system-gateway.service --no-pager -n 300 + sudo journalctl -b -u snapd.service --no-pager -n 300 + snap logs openshell.user-gateway -n=300 + sudo ss -ltnp '( sport = :17670 )' + + ubuntu-snap-docker: + name: Ubuntu snap with Docker driver + if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} + runs-on: ubuntu-latest + env: + OPENSHELL_INSTALL_METHOD: snap + timeout-minutes: 20 + steps: + - name: Install snapd + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y snapd + sudo systemctl enable --now snapd.socket + sudo systemctl start snapd + sudo snap wait system seed.loaded + + - name: Ensure system Docker + run: | + set -euo pipefail + if ! command -v docker >/dev/null 2>&1; then + sudo apt-get install -y docker.io + fi + sudo systemctl start docker || sudo service docker start + if sudo snap list docker >/dev/null 2>&1; then + echo "the Docker snap must not provide the canary daemon" >&2 + exit 1 + fi + docker info + + - name: Select Docker for the snap gateway + run: | + set -euo pipefail + systemctl --user set-environment \ + OPENSHELL_COMPUTE_DRIVER=docker \ + "OPENSHELL_TELEMETRY_ENABLED=${OPENSHELL_TELEMETRY_ENABLED}" + systemctl --user show-environment | + grep -Fx 'OPENSHELL_COMPUTE_DRIVER=docker' + + - name: Install and check status + run: | + set -euo pipefail + curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | + sh + sudo snap list openshell + snap info openshell | grep -E '^tracking: +latest/edge$' + sudo snap connections openshell | + grep -E '^docker +openshell:docker +:docker +' + docker info + snap services openshell + systemctl --user is-enabled --quiet snap.openshell.user-gateway.service + systemctl --user is-active snap.openshell.user-gateway.service + ! systemctl is-enabled --quiet snap.openshell.system-gateway.service + ! systemctl is-active --quiet snap.openshell.system-gateway.service + journalctl --user -u snap.openshell.user-gateway.service --no-pager | + grep -F 'Using compute driver' | + grep -F 'driver=docker' + openshell gateway list | grep -F 'https://127.0.0.1:17670' + openshell status + + - name: Create and exercise a sandbox + run: | + set -euo pipefail + sandbox="rc-docker-${GITHUB_RUN_ID}" + openshell sandbox create --name "$sandbox" --detach + openshell sandbox exec --name "$sandbox" --no-tty -- true + openshell sandbox delete "$sandbox" + + - name: Collect snap and Docker diagnostics if: failure() run: | set +e docker info sudo systemctl status docker.service --no-pager sudo journalctl -b -u docker.service --no-pager -n 300 + systemctl --user show-environment | + grep -E '^OPENSHELL_(COMPUTE_DRIVER|TELEMETRY_ENABLED)=' sudo snap services openshell sudo snap connections openshell sudo snap changes @@ -313,10 +403,10 @@ jobs: snap logs openshell.user-gateway -n=300 sudo ss -ltnp '( sport = :17670 )' - ubuntu-snap-docker-preflight: - name: Ubuntu snap Docker preflight + ubuntu-snap-podman: + name: Ubuntu snap with Podman driver if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} - runs-on: ubuntu-latest + runs-on: ubuntu-26.04 env: OPENSHELL_INSTALL_METHOD: snap timeout-minutes: 20 @@ -330,50 +420,110 @@ jobs: sudo systemctl start snapd sudo snap wait system seed.loaded - - name: Remove system Docker + - name: Ensure rootless Podman run: | set -euo pipefail sudo systemctl stop docker.service docker.socket 2>/dev/null || true sudo apt-get -y --purge remove docker-ce docker-ce-cli containerd.io \ docker-buildx-plugin docker-compose-plugin docker.io 2>/dev/null || true ! command -v docker >/dev/null 2>&1 + sudo apt-get install -y podman + systemctl --user enable --now podman.socket + case "$(podman version --format '{{.Client.Version}}')" in + 5.*) ;; + *) echo "Podman 5.x is required" >&2; exit 1 ;; + esac + test "$(podman info --format '{{.Host.Security.Rootless}}')" = true + test "$(podman info --format '{{.Host.CgroupsVersion}}')" = v2 + socket="/run/user/$(id -u)/podman/podman.sock" + test -S "$socket" + podman --url "unix://$socket" info >/dev/null + + - name: Select Podman for the snap gateway + run: | + set -euo pipefail + systemctl --user set-environment \ + OPENSHELL_COMPUTE_DRIVER=podman \ + "OPENSHELL_TELEMETRY_ENABLED=${OPENSHELL_TELEMETRY_ENABLED}" + systemctl --user show-environment | + grep -Fx 'OPENSHELL_COMPUTE_DRIVER=podman' - - name: Verify installer rejects missing Docker + - name: Install and check status run: | set -euo pipefail - curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh -o install.sh - if OPENSHELL_VERSION=dev sh install.sh >install.log 2>&1; then - echo "install.sh unexpectedly succeeded without Docker" >&2 - cat install.log >&2 - exit 1 - fi - cat install.log - grep -F "Docker is required before installing the OpenShell snap" install.log - ! sudo snap list docker >/dev/null 2>&1 - ! sudo snap list openshell >/dev/null 2>&1 + curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | + sh + sudo snap list openshell + snap info openshell | grep -E '^tracking: +latest/edge$' + sudo snap connections openshell | + grep -E '^podman +openshell:podman +:podman +' + podman info + snap services openshell + systemctl --user is-enabled --quiet snap.openshell.user-gateway.service + systemctl --user is-active snap.openshell.user-gateway.service + ! systemctl is-enabled --quiet snap.openshell.system-gateway.service + ! systemctl is-active --quiet snap.openshell.system-gateway.service + journalctl --user -u snap.openshell.user-gateway.service --no-pager | + grep -F 'Using compute driver' | + grep -F 'driver=podman' + openshell gateway list | grep -F 'https://127.0.0.1:17670' + openshell gateway info --output json | + jq -e '(.compute_drivers | length) == 1 and .compute_drivers[0].name == "podman" and .compute_drivers[0].capabilities.driver_name == "podman"' + openshell status - - name: Verify installer rejects Docker snap + - name: Create and exercise a sandbox run: | set -euo pipefail - sudo snap install docker - if OPENSHELL_VERSION=dev sh install.sh >docker-snap.log 2>&1; then - echo "install.sh unexpectedly succeeded with the Docker snap" >&2 - cat docker-snap.log >&2 + sandbox="rc-podman-${GITHUB_RUN_ID}" + openshell sandbox create --name "$sandbox" --detach + podman ps -aq \ + --filter 'label=openshell.managed=true' \ + --filter "label=openshell.ai/sandbox-name=${sandbox}" | + grep -q . + openshell sandbox exec --name "$sandbox" --no-tty -- true + openshell sandbox delete "$sandbox" + for _ in $(seq 1 60); do + if ! podman ps -aq \ + --filter 'label=openshell.managed=true' \ + --filter "label=openshell.ai/sandbox-name=${sandbox}" | + grep -q .; then + exit 0 + fi + sleep 1 + done + if podman ps -aq \ + --filter 'label=openshell.managed=true' \ + --filter "label=openshell.ai/sandbox-name=${sandbox}" | + grep -q .; then + echo "Podman resources remain after deleting ${sandbox}" >&2 exit 1 fi - cat docker-snap.log - grep -F "the Docker snap is not currently compatible with OpenShell" docker-snap.log - sudo snap list docker >/dev/null - ! sudo snap list openshell >/dev/null 2>&1 - - name: Collect snap diagnostics + - name: Collect snap and Podman diagnostics if: failure() run: | set +e - cat install.log - cat docker-snap.log + podman version + podman info --debug + podman ps --all + podman network ls + podman volume ls + systemctl --user status podman.socket podman.service --no-pager + journalctl --user -u podman.socket -u podman.service --no-pager -n 300 + stat "/run/user/$(id -u)/podman/podman.sock" + systemctl --user show-environment | + grep -E '^OPENSHELL_(COMPUTE_DRIVER|TELEMETRY_ENABLED)=' + sudo snap services openshell + sudo snap connections openshell sudo snap changes + systemctl --user status snap.openshell.user-gateway.service --no-pager + journalctl --user -u snap.openshell.user-gateway.service --no-pager -n 300 + sudo systemctl status snap.openshell.system-gateway.service --no-pager + sudo journalctl -b -u snap.openshell.system-gateway.service --no-pager -n 300 sudo journalctl -b -u snapd.service --no-pager -n 300 + snap logs openshell.user-gateway -n=300 + sudo journalctl -k --no-pager -n 300 | grep -E 'apparmor=.*DENIED|audit' + sudo ss -ltnp '( sport = :17670 )' kubernetes: name: Kubernetes Helm (kind) diff --git a/.github/workflows/release-dev.yml b/.github/workflows/release-dev.yml index aec77972ee..15eb398172 100644 --- a/.github/workflows/release-dev.yml +++ b/.github/workflows/release-dev.yml @@ -247,7 +247,7 @@ jobs: build-snap: name: Build snap - needs: [compute-versions, build-binaries] + needs: [compute-versions, build-binaries, build-vm-driver] uses: ./.github/workflows/snap-package.yml with: checkout-ref: ${{ github.sha }} diff --git a/.github/workflows/snap-package.yml b/.github/workflows/snap-package.yml index 2d30f85141..834cf1cb02 100644 --- a/.github/workflows/snap-package.yml +++ b/.github/workflows/snap-package.yml @@ -88,6 +88,12 @@ jobs: name: openshell-sandbox-${{ matrix.rust_arch }}-unknown-linux-musl path: prebuilt/sandbox + - name: Download prebuilt VM driver + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-driver-vm-${{ matrix.rust_arch }}-unknown-linux-gnu + path: prebuilt/driver-vm + - name: Configure prebuilt binaries run: | set -euo pipefail @@ -95,6 +101,7 @@ jobs: chmod +x prebuilt/prover/openshell-prover chmod +x prebuilt/gateway/openshell-gateway chmod +x prebuilt/sandbox/openshell-sandbox + chmod +x prebuilt/driver-vm/openshell-driver-vm ls -laR prebuilt/ - name: Prepare snap build directory @@ -106,6 +113,7 @@ jobs: cp prebuilt/prover/openshell-prover snap/prebuilt/openshell-prover cp prebuilt/gateway/openshell-gateway snap/prebuilt/openshell-gateway cp prebuilt/sandbox/openshell-sandbox snap/prebuilt/openshell-sandbox + cp prebuilt/driver-vm/openshell-driver-vm snap/prebuilt/openshell-driver-vm cp tasks/scripts/snap-gateway-wrapper.sh snap/prebuilt/openshell-gateway-wrapper cp LICENSE snap/prebuilt/ diff --git a/CI.md b/CI.md index b647d73001..5c5b115ad5 100644 --- a/CI.md +++ b/CI.md @@ -645,7 +645,7 @@ These workflows run after merge to publish dev/tagged artifacts and verify them. | `.github/workflows/release-qualification.yml` | Reusable qualification suites and result aggregation. Runs security scans, conformance and feature integration, Docker and VM E2E, protobuf compatibility, and advisory compatibility review. | | `.github/workflows/snap-package.yml` | Builds snap and component artifacts for Release Dev, Release, and Pre-release without Store credentials or publication. | | `.github/workflows/snap-publish.yml` | Uploads existing snap and component artifacts to the Store without rebuilding. Release Dev calls it directly for `latest/edge` after snap builds and requires it to succeed before creating the dev release. Release calls it through `release-publish.yml` for `latest/stable` after qualification and release publication succeed. | -| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts in the `macos`, `ubuntu-deb`, `ubuntu-snap-system-docker`, `fedora`, and `kubernetes` (kind + Helm) jobs. Each job reaches its gateway and creates, exercises, and deletes a sandbox. The `ubuntu-snap-system-docker` job verifies a compatible system Docker lifecycle, while `ubuntu-snap-docker-preflight` tests fail-fast behavior when Docker is absent or supplied by the Docker snap. The positive snap lane also runs a local policy containment check with the packaged prover. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref `). See the `test-release-canary` skill for the playbook and local kind reproduction. | +| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts in the `macos`, `ubuntu-deb`, `ubuntu-snap-vm`, `ubuntu-snap-docker`, `ubuntu-snap-podman`, `fedora`, and `kubernetes` (kind + Helm) jobs. The snap lanes verify the standalone VM driver plus explicit Docker and rootless Podman selections. Each job reaches its gateway and creates, exercises, and deletes a sandbox. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref `). See the `test-release-canary` skill for the playbook and local kind reproduction. | ## Required status contexts diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index deb4ca9dae..82111abe1b 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -325,6 +325,11 @@ includes `openshell-gateway` and `openshell-driver-vm`, but leaves `OPENSHELL_COMPUTE_DRIVER` unset so the gateway uses its normal runtime auto-detection. Set `OPENSHELL_COMPUTE_DRIVER=vm` to force the VM driver. +The OpenShell snap also bundles `openshell-driver-vm` and e2fsprogs. Its gateway +wrapper selects `vm` when neither the service environment nor gateway TOML sets +`compute_driver`. The confined VM path supports CPU-only libkrun sandboxes and +requires the `kvm` interface plus read/write access to `/dev/kvm`. + On RPM-family Linux x86_64 and aarch64 systems, `install.sh` installs the `openshell` and `openshell-gateway` RPM packages from the selected release tag. The RPM gateway package is configured for the Podman driver. diff --git a/docs/about/installation.mdx b/docs/about/installation.mdx index ff5af03dab..3c55aaac04 100644 --- a/docs/about/installation.mdx +++ b/docs/about/installation.mdx @@ -119,8 +119,6 @@ sudo loginctl enable-linger $USER ## Snap -The snap requires Docker Engine installed from your distribution or Docker's package repository. The Docker snap is not compatible. - The snap does not migrate existing Debian, RPM, or Homebrew installs. Remove any existing installation first, then either rerun the `install.sh` script with `OPENSHELL_INSTALL_METHOD=snap OPENSHELL_ACK_BREAKING_UPGRADE=1`, or install the snap directly: ```shell @@ -143,12 +141,36 @@ The global command prevents automatic activation for other users but does not st systemctl --user disable --now snap.openshell.user-gateway.service ``` -The snap installs the standalone policy prover as `openshell.prover`. The -`openshell-prover` alias requires Snap Store approval and may not be available. -The prover reads local policy files through the `home` interface and does not -connect to the gateway. +The snap bundles the VM compute driver and selects it when the gateway user has read/write access to `/dev/kvm`. Without that access, the gateway tries to detect another accessible runtime, such as Docker. If none is available, the gateway fails to start until KVM access is restored or `compute_driver` selects an available runtime. Active desktop sessions commonly receive KVM access through a systemd-logind ACL. On systems where this is not the case, add the gateway user to the `kvm` group and start a new login session: + +```shell +sudo usermod -aG kvm "$USER" +``` + +The installer connects the `kvm` interface and warns without blocking installation when the user cannot access `/dev/kvm`. + +The snap installs the standalone policy prover as `openshell.prover`. The `openshell-prover` alias requires Snap Store approval and may not be available. The prover reads local policy files through the `home` interface and does not connect to the gateway. + +The gateway runs at `https://127.0.0.1:17670`. It reads `~/snap/openshell/common/.config/openshell/gateway.toml` when that file exists. When neither the service environment nor this file selects a compute driver, the snap selects `vm` if the gateway user can read and write `/dev/kvm`; otherwise the gateway tries to detect another available runtime. To use a different installed driver, set `compute_driver` explicitly; for example: + +```toml +[openshell] +version = 2 + +[openshell.gateway] +compute_driver = "docker" +``` + +To use rootless Podman, install Podman 5.x, verify that the host uses cgroups v2, and enable its API socket from the same user account that runs the snap gateway: + +```shell +podman info +systemctl --user enable --now podman.socket +``` + +Then set `compute_driver = "podman"` in the same gateway table and restart the gateway with `systemctl restart --user snap.openshell.user-gateway.service`. The snap does not install, configure, or start Podman. -The gateway runs at `https://127.0.0.1:17670`. It reads `~/snap/openshell/common/.config/openshell/gateway.toml` when that file exists. Its database defaults to `~/snap/openshell/common/.local/state/openshell/gateway/openshell.db`, and its TLS material lives under `~/snap/openshell/common/.local/state/openshell/tls`. Register it from the same user account: +OpenShell's database defaults to `~/snap/openshell/common/.local/state/openshell/gateway/openshell.db`, and its TLS material lives under `~/snap/openshell/common/.local/state/openshell/tls`. Register it from the same user account: ```shell snap services openshell.user-gateway @@ -236,19 +258,18 @@ openshell status Set `old_registration` to the name shown by `openshell gateway list`, and omit the variable assignment and remove command when no registration exists. If a transition is interrupted while both services are disabled, select `gateway-mode=user` or `gateway-mode=system` to resume it. Keep the TLS backup until the selected gateway works and can create a sandbox. -To install a locally built snap, connect its interfaces manually. The user gateway may reach systemd's start limit before Docker is connected, so reset the failed unit and restart it from the affected user session after connecting the interfaces: +To install a locally built snap, connect its interfaces manually: ```shell sudo snap install ./openshell_*.snap --dangerous sudo snap connect openshell:log-observe sudo snap connect openshell:system-observe +sudo snap connect openshell:kvm sudo snap connect openshell:docker :docker -systemctl --user reset-failed snap.openshell.user-gateway.service +sudo snap connect openshell:podman :podman systemctl --user enable --now snap.openshell.user-gateway.service ``` -For an installation upgraded from a legacy system gateway, use `sudo systemctl reset-failed snap.openshell.system-gateway.service` and `sudo snap restart openshell.system-gateway` instead. - ## Kubernetes Deploy the gateway to a cluster with the OpenShell Helm chart. See [Kubernetes Setup](/kubernetes/setup). diff --git a/docs/about/support-matrix.mdx b/docs/about/support-matrix.mdx index 155cf355a6..57277e00ce 100644 --- a/docs/about/support-matrix.mdx +++ b/docs/about/support-matrix.mdx @@ -129,6 +129,10 @@ The gateway can manage sandboxes through several runtimes. | Kubernetes | Supported through the [OpenShell Helm chart](https://github.com/NVIDIA/OpenShell/blob/main/deploy/helm/openshell/README.md). | Requires a Kubernetes cluster supplied by the operator. | | MicroVM | Supported for VM-backed sandboxes. | Uses the VM compute driver and libkrun-based runtime. | +The snap bundles MicroVM support and selects it when no compute driver is set +in the service environment or gateway TOML. Snap VM support is CPU-only; GPU +VFIO passthrough is not available under snap confinement. + ## Software Prerequisites Install the software for the compute driver you use: diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index 42c47f0d0b..26a3f679dc 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -37,7 +37,7 @@ compute_driver = "docker" Every gateway TOML file must start with the `[openshell]` table and `version = 2`, or `openshell-gateway config preflight` rejects it with `missing_version`. The remaining examples on this page show only the tables they change. -When `compute_driver` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. The VM driver is never auto-detected. +When `compute_driver` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. The VM driver is never auto-detected by the gateway binary. The snap wrapper supplies `vm` as its package default when neither the service environment nor gateway TOML selects a driver. Configure driver-specific values, such as images, endpoints, and sizing, under `[openshell.drivers.]`. See the [Gateway Configuration File](/how-it-works/gateways/configuration) reference for every option. @@ -214,7 +214,8 @@ Podman also supports `image` mounts, which mount a container image read-only at The MicroVM driver runs each sandbox in its own lightweight VM. It requires host virtualization: Apple Hypervisor on macOS or KVM on Linux. -The VM driver is opt-in. Enable it in the gateway TOML file: +The VM driver is opt-in for ordinary gateway packages. The snap bundles it and +uses it by default. Enable it explicitly elsewhere with gateway TOML: ```toml [openshell.gateway] diff --git a/install.sh b/install.sh index f552289c43..10c7883bf7 100755 --- a/install.sh +++ b/install.sh @@ -74,9 +74,8 @@ NOTES: OPENSHELL_INSTALL_METHOD=snap to install the OpenShell snap instead; hosts that already have the OpenShell snap keep refreshing it. snap installs use latest/stable by default and latest/edge for dev, and do not support - explicit release tags or prereleases. The OpenShell snap requires a running - Docker Engine installed from a system package or Docker's package - repository. The Docker snap is not currently compatible with OpenShell. + explicit release tags or prereleases. The snap uses its bundled VM driver + by default; select a different compute driver in gateway.toml if needed. macOS installs the release Homebrew formula on Apple Silicon and starts a brew services-backed local gateway. EOF @@ -1262,46 +1261,27 @@ openshell_snap_channel() { esac } -wait_for_docker_daemon() { - _timeout="${OPENSHELL_INSTALL_DOCKER_TIMEOUT:-30}" - _elapsed=0 - _last_output="" - - info "waiting for Docker daemon at /var/run/docker.sock to become reachable..." - while [ "$_elapsed" -lt "$_timeout" ]; do - if _last_output="$(as_root env -u DOCKER_HOST -u DOCKER_CONTEXT docker --host unix:///var/run/docker.sock info 2>&1)"; then - info "Docker daemon at /var/run/docker.sock is reachable" - return 0 - fi - sleep 1 - _elapsed=$((_elapsed + 1)) - done - - [ -z "$_last_output" ] || printf '%s\n' "$_last_output" >&2 - if snap list docker >/dev/null 2>&1; then - snap services docker >&2 || true - snap changes >&2 || true - fi - error "Docker daemon at /var/run/docker.sock did not become reachable within ${_timeout}s" +snap_kvm_device() { + printf '%s\n' /dev/kvm } -wait_for_user_docker_daemon() { - _timeout="${OPENSHELL_INSTALL_DOCKER_TIMEOUT:-30}" - _elapsed=0 - _last_output="" - - info "waiting for Docker daemon at /var/run/docker.sock to become reachable as ${TARGET_USER}..." - while [ "$_elapsed" -lt "$_timeout" ]; do - if _last_output="$(as_target_user env -u DOCKER_HOST -u DOCKER_CONTEXT docker --host unix:///var/run/docker.sock info 2>&1)"; then - info "Docker daemon at /var/run/docker.sock is reachable as ${TARGET_USER}" - return 0 - fi - sleep 1 - _elapsed=$((_elapsed + 1)) - done +check_snap_kvm_access() { + _kvm_device="$(snap_kvm_device)" + if as_target_user sh -c 'test -r "$1" && test -w "$1"' sh "$_kvm_device"; then + info "${TARGET_USER} can access ${_kvm_device}" + return 0 + fi - [ -z "$_last_output" ] || printf '%s\n' "$_last_output" >&2 - error "Docker daemon at /var/run/docker.sock did not become reachable as ${TARGET_USER} within ${_timeout}s. Ensure ${TARGET_USER} can access the Docker socket, then log out and back in after changing group membership." + warn "${TARGET_USER} cannot access ${_kvm_device}, so the default VM compute driver cannot create sandboxes." + if [ -e "$_kvm_device" ] && getent group kvm >/dev/null 2>&1; then + warn "To use the VM driver, run: sudo usermod -aG kvm \"${TARGET_USER}\"" + warn "Then log out and back in so the new group membership takes effect." + elif [ ! -e "$_kvm_device" ]; then + warn "${_kvm_device} is unavailable; enable host virtualization before using the VM driver." + else + warn "The kvm group is unavailable; grant ${TARGET_USER} read/write access to ${_kvm_device} before using the VM driver." + fi + warn "Alternatively, set compute_driver to another installed driver, such as docker or podman, in the snap gateway TOML configuration." } snap_gateway_mode() { @@ -1468,21 +1448,6 @@ install_linux_snap() { error "OpenShell snap gateway mode is disabled. Complete the gateway transition with 'sudo snap set openshell gateway-mode=user' or 'sudo snap set openshell gateway-mode=system', then rerun the installer." fi - if snap list docker >/dev/null 2>&1; then - error "the Docker snap is not currently compatible with OpenShell because its AppArmor confinement prevents OpenShell's hardened containers from starting. -Remove the Docker snap and install Docker Engine from a system package or Docker's package repository, then rerun this installer." - fi - if ! has_cmd docker; then - error "Docker is required before installing the OpenShell snap. -Install Docker Engine from a system package or Docker's package repository, then rerun this installer. The Docker snap is not currently compatible with OpenShell." - fi - info "using existing Docker installation" - if [ "$_existing_mode" = user ]; then - wait_for_user_docker_daemon - else - wait_for_docker_daemon - fi - _channel="$(openshell_snap_channel)" if [ "$_existing_snap" = true ]; then info "refreshing OpenShell snap from ${_channel}..." @@ -1493,6 +1458,9 @@ Install Docker Engine from a system package or Docker's package repository, then fi _gateway_mode="$(snap_gateway_mode)" + if [ "$_gateway_mode" = user ]; then + check_snap_kvm_access + fi if [ "$_gateway_mode" = system ]; then warn "restarting the OpenShell gateway to use the refreshed snap; active sandbox sessions will be interrupted" as_root snap restart openshell.system-gateway diff --git a/nix/test-guest/README.md b/nix/test-guest/README.md index 155b95305c..8d6eaede24 100644 --- a/nix/test-guest/README.md +++ b/nix/test-guest/README.md @@ -78,10 +78,10 @@ The root [`flake.nix`](../../flake.nix) exposes this directory as the `test-gues | Fedora 44 | No | Yes | Yes | Yes | `.rpm` | | Rocky Linux 9 | Yes | Yes | No | Yes | `.rpm` | -The `snapd` configuration is available for Ubuntu and prepares snapd for snap -Store installation experiments. Combine it with `docker` to reproduce the -system-Docker canary, or use it alone to verify that `install.sh` fails safely -when Docker is absent. +The `snapd` configuration is available for Ubuntu and prepares snapd for +testing the OpenShell snap. It does not install the OpenShell snap itself. +OpenShell snap with VM driver qualification remains in the bare-host +`ubuntu-snap-vm` release canary. `podman-rootless` configures the explicit rootless Podman guest setup used by OpenShell tests. It supports Fedora and Ubuntu 26.04 or later. Ubuntu adds the diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 1dab79be80..5fef984f2c 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -1225,6 +1225,36 @@ For `vm.mounts` (virtiofs host bind mounts) failures, also verify: - The sandbox launches on libkrun. GPU sandboxes and QEMU launches do not support VM mounts. - On Linux with a non-root gateway, file creation in a read-write mount fails unless `sandbox_uid`/`sandbox_gid` match the gateway user's UID/GID. Compare `id -u`/`id -g` of the gateway user with the configured values. +For snap gateways, also check the package default, interface, and effective +device access. A healthy gateway or successful `openshell status` does not prove +that its user can launch a VM: + +```bash +snap connections openshell | grep kvm +test -r /dev/kvm && test -w /dev/kvm +getfacl /dev/kvm +snap run --shell openshell.user-gateway -c 'command -v mke2fs; command -v debugfs; command -v e2fsck' +``` + +When `gateway.toml` has no `compute_driver`, the snap wrapper selects `vm`. +Set `[openshell.gateway].compute_driver` explicitly to choose another installed +driver. Snap VM support is CPU-only and does not include VFIO passthrough. + +For a snap user gateway configured with rootless Podman, verify the privileged +interface connection and the same user's API socket before debugging sandbox +startup: + +```bash +snap connections openshell | grep podman +systemctl --user status podman.socket podman.service +test -S "/run/user/$(id -u)/podman/podman.sock" +podman info --format '{{.Host.Security.Rootless}} {{.Host.CgroupsVersion}}' +journalctl --user -u snap.openshell.user-gateway.service --no-pager | grep 'driver=podman' +``` + +The `podman` interface grants the user gateway access to that user's Podman API. +The snap does not install Podman or start `podman.socket`. + Then run: ```bash diff --git a/snapcraft.yaml b/snapcraft.yaml index 5278343d45..b3d358b243 100644 --- a/snapcraft.yaml +++ b/snapcraft.yaml @@ -18,11 +18,19 @@ description: | **Setup instructions** - 1. Install and start Docker using your preferred package source before - installing the OpenShell snap. Use a system package or Docker's package - repository. The Docker snap is not currently compatible with OpenShell. + 1. The snap uses the VM compute driver when the gateway user has read/write + access to /dev/kvm, unless `compute_driver` is set in gateway.toml or + OPENSHELL_COMPUTE_DRIVER is set in the service environment. Without that + access, the gateway tries to detect another runtime and fails to start if + none is available. Desktop sessions commonly receive access + automatically; otherwise add the gateway user to the kvm group and start + a new login session: - Snap Store installations automatically connect the required interfaces. + sudo usermod -aG kvm "$USER" + + To use a different installed driver, such as Docker or rootless Podman, + set `compute_driver` under `[openshell.gateway]` in gateway.toml. Start + `podman.socket` as the gateway user before selecting Podman. Enable and start the gateway from the single user account that will run it: @@ -42,13 +50,6 @@ description: | This does not stop gateways already running in other user sessions. Those users must disable and stop the service from their own sessions. - If the OpenShell snap was installed before Docker was running, its user - service may have hit the systemd start limit. Resolve this from the - affected user's session after Docker is available: - - systemctl --user reset-failed snap.openshell.user-gateway.service - systemctl --user restart snap.openshell.user-gateway.service - 2. Register the per-user gateway after it provisions its mTLS credentials: snap services openshell.user-gateway @@ -87,10 +88,10 @@ description: | base: core24 grade: stable confinement: strict -# The system :docker slot was added in snapd 2.76, so this is the minimum -# required version. Snapd 2.77 allowed manual connection to that slot without -# snap-declaration overrides from the store, such as when installing a locally- -# built snap, but that is not required for installations from the snap store. +# The system :docker slot and podman interface were added in snapd 2.76, so +# this is the minimum required version. Snapd 2.77 allowed manual connection +# to :docker without snap-declaration overrides from the store, such as when +# installing a locally-built snap, but that is not required for store installs. assumes: [snapd2.76] license: Apache-2.0 website: https://docs.nvidia.com/openshell/latest/index.html @@ -156,6 +157,7 @@ apps: OPENSHELL_LOCAL_TLS_DIR: "$SNAP_COMMON/tls" plugs: - docker + - kvm - log-observe - network - network-bind @@ -174,9 +176,11 @@ apps: XDG_STATE_HOME: "$SNAP_USER_COMMON/.local/state" plugs: - docker + - kvm - log-observe - network - network-bind + - podman - system-observe parts: @@ -190,7 +194,7 @@ parts: set -euo pipefail MISSING=() - for bin in openshell openshell-prover openshell-gateway openshell-sandbox openshell-gateway-wrapper; do + for bin in openshell openshell-prover openshell-gateway openshell-sandbox openshell-driver-vm openshell-gateway-wrapper; do if [ ! -f "$CRAFT_PART_SRC/$bin" ]; then MISSING+=("$bin") fi @@ -214,6 +218,8 @@ parts: "$CRAFT_PART_INSTALL/bin/openshell-gateway" install -D -m 0755 "$CRAFT_PART_SRC/openshell-sandbox" \ "$CRAFT_PART_INSTALL/bin/openshell-sandbox" + install -D -m 0755 "$CRAFT_PART_SRC/openshell-driver-vm" \ + "$CRAFT_PART_INSTALL/bin/openshell-driver-vm" install -D -m 0755 "$CRAFT_PART_SRC/openshell-gateway-wrapper" \ "$CRAFT_PART_INSTALL/bin/openshell-gateway-wrapper" install -D -m 0644 "$CRAFT_PART_SRC/meta/gui/term.desktop" \ @@ -235,3 +241,10 @@ parts: - openssh-client prime: - usr/bin/ssh + + e2fsprogs: + # The VM driver formats and modifies file-backed ext4 disks with these + # tools. Vendor them because strict snaps cannot execute host sbin tools. + plugin: nil + stage-packages: + - e2fsprogs diff --git a/tasks/scripts/snap-gateway-wrapper.sh b/tasks/scripts/snap-gateway-wrapper.sh index 38ac1f6a9c..df37473443 100755 --- a/tasks/scripts/snap-gateway-wrapper.sh +++ b/tasks/scripts/snap-gateway-wrapper.sh @@ -14,6 +14,45 @@ then export OPENSHELL_GATEWAY_CONFIG="$OPENSHELL_SNAP_CONFIG_FILE" fi +# The snap prefers the bundled VM driver when KVM is usable. Preserve an +# explicit service environment selector, then a selector in the effective +# gateway TOML. +if [ -z "${OPENSHELL_COMPUTE_DRIVER+x}" ] \ + && [ -z "${OPENSHELL_DRIVERS+x}" ] +then + kvm_device=/dev/kvm + config_file="" + expect_config_path=false + for arg in "$@"; do + if [ "$expect_config_path" = true ]; then + config_file="$arg" + expect_config_path=false + continue + fi + case "$arg" in + --config) expect_config_path=true ;; + --config=*) config_file=${arg#--config=} ;; + esac + done + if [ -z "$config_file" ]; then + config_file="${OPENSHELL_GATEWAY_CONFIG:-}" + fi + if [ -z "$config_file" ] && [ -n "${XDG_CONFIG_HOME:-}" ]; then + config_file="${XDG_CONFIG_HOME}/openshell/gateway.toml" + fi + if { [ -z "$config_file" ] \ + || [ ! -f "$config_file" ] \ + || ! grep -Eq \ + -e "^[[:space:]]*(compute_driver|\"compute_driver\"|'compute_driver')[[:space:]]*=" \ + -e "^[[:space:]]*(openshell|\"openshell\"|'openshell')[[:space:]]*\.[[:space:]]*(gateway|\"gateway\"|'gateway')[[:space:]]*\.[[:space:]]*(compute_driver|\"compute_driver\"|'compute_driver')[[:space:]]*=" \ + "$config_file"; } \ + && [ -r "$kvm_device" ] \ + && [ -w "$kvm_device" ] + then + export OPENSHELL_COMPUTE_DRIVER=vm + fi +fi + if [ -z "${OPENSHELL_LOCAL_TLS_DIR:-}" ]; then if [ -z "${XDG_STATE_HOME:-}" ]; then echo "openshell-gateway: OPENSHELL_LOCAL_TLS_DIR or XDG_STATE_HOME is required" >&2 diff --git a/tasks/scripts/test-install-sh.sh b/tasks/scripts/test-install-sh.sh index 82bfb9427c..9376dbc0a8 100755 --- a/tasks/scripts/test-install-sh.sh +++ b/tasks/scripts/test-install-sh.sh @@ -265,17 +265,15 @@ rm -f "$out" "$err" assert_snap_install_flow() { local name=$1 - local docker_present=$2 - local openshell_present=$3 - local requested_version=$4 - local expected=$5 + local openshell_present=$2 + local requested_version=$3 + local expected=$4 local calls ( has_cmd() { case "$1" in snap) return 0 ;; - docker) [ "$docker_present" = "1" ] ;; *) command -v "$1" >/dev/null 2>&1 ;; esac } @@ -297,8 +295,7 @@ assert_snap_install_flow() { } as_target_user() { printf 'target:%s\n' "$*"; } set_linux_target_runtime_dir() { :; } - wait_for_docker_daemon() { printf '%s\n' "wait:docker"; } - wait_for_user_docker_daemon() { printf '%s\n' "wait:user-docker"; } + check_snap_kvm_access() { printf '%s\n' "check:kvm-access"; } snap_gateway_mode() { if [ "$openshell_present" = 1 ]; then printf '%s\n' system @@ -332,7 +329,7 @@ assert_snap_user_refresh_flow() { ( has_cmd() { case "$1" in - snap | docker) return 0 ;; + snap) return 0 ;; *) command -v "$1" >/dev/null 2>&1 ;; esac } @@ -354,7 +351,7 @@ assert_snap_user_refresh_flow() { } as_target_user() { printf 'target:%s\n' "$*"; } set_linux_target_runtime_dir() { :; } - wait_for_user_docker_daemon() { printf '%s\n' "wait:user-docker"; } + check_snap_kvm_access() { printf '%s\n' "check:kvm-access"; } snap_gateway_mode() { printf '%s\n' user; } wait_for_user_snap_gateway_listener() { printf '%s\n' "wait:user-gateway-listener"; } register_user_snap_gateway() { printf '%s\n' "register:user-gateway"; } @@ -364,8 +361,8 @@ assert_snap_user_refresh_flow() { install_linux_snap ) >"$out" calls="$(cat "$out")" - expected="wait:user-docker -root:snap refresh openshell --channel=latest/stable + expected="root:snap refresh openshell --channel=latest/stable +check:kvm-access root:systemctl --global disable snap.openshell.user-gateway.service target:systemctl --user daemon-reload target:systemctl --user enable --now snap.openshell.user-gateway.service @@ -381,10 +378,10 @@ wait:gateway-status" } assert_snap_install_flow \ - "existing Docker is reused" \ - 1 0 "" \ - "wait:user-docker -root:snap install openshell --channel=latest/stable + "fresh snap defaults to VM without Docker" \ + 0 "" \ + "root:snap install openshell --channel=latest/stable +check:kvm-access root:systemctl --global disable snap.openshell.user-gateway.service target:systemctl --user daemon-reload target:systemctl --user enable --now snap.openshell.user-gateway.service @@ -396,9 +393,8 @@ assert_snap_user_refresh_flow assert_snap_install_flow \ "existing OpenShell snap is refreshed" \ - 1 1 "" \ - "wait:docker -root:snap refresh openshell --channel=latest/stable + 1 "" \ + "root:snap refresh openshell --channel=latest/stable root:snap restart openshell.system-gateway wait:system-gateway-listener register:system-gateway @@ -445,167 +441,39 @@ if grep -Fq "disabled mode reached" "$err"; then exit 1 fi -assert_snap_install_rejected() { - local name=$1 - local docker_present=$2 - local docker_snap_present=$3 - local expected=$4 - - if ( - has_cmd() { - case "$1" in - snap) return 0 ;; - docker) [ "$docker_present" = "1" ] ;; - *) command -v "$1" >/dev/null 2>&1 ;; - esac - } - snap() { - if [ "${1:-}:${2:-}" = list:docker ]; then - [ "$docker_snap_present" = "1" ] - else - echo "FAIL: ${name}: unexpected snap command: $*" >&2 - return 99 - fi - } - as_root() { - echo "FAIL: ${name}: installation reached root command: $*" >&2 - return 99 - } - set_linux_target_runtime_dir() { :; } - install_linux_snap - ) >"$out" 2>"$err"; then - echo "FAIL: ${name}: snap installation should have been rejected" >&2 - exit 1 - fi - if ! grep -Fq "$expected" "$err"; then - echo "FAIL: ${name}: missing rejection message" >&2 - cat "$err" >&2 || true - exit 1 - fi - if grep -Fq "installation reached root command" "$err"; then - cat "$err" >&2 || true - exit 1 - fi -} - -assert_snap_install_rejected \ - "missing Docker" \ - 0 0 \ - "Docker is required before installing the OpenShell snap" - -assert_snap_install_rejected \ - "Docker snap" \ - 1 1 \ - "the Docker snap is not currently compatible with OpenShell" - -attempts_file="${tmpdir}/docker-attempts" -root_probes_file="${tmpdir}/docker-root-probes" -printf '0\n' >"$attempts_file" -printf '0\n' >"$root_probes_file" if ! ( - env() { - if [ "$*" != "-u DOCKER_HOST -u DOCKER_CONTEXT docker --host unix:///var/run/docker.sock info" ]; then - echo "unexpected Docker readiness probe: $*" >&2 - return 1 - fi - attempts="$(cat "$attempts_file")" - attempts=$((attempts + 1)) - printf '%s\n' "$attempts" >"$attempts_file" - [ "$attempts" -ge 3 ] - } - as_root() { - root_probes="$(cat "$root_probes_file")" - printf '%s\n' "$((root_probes + 1))" >"$root_probes_file" - "$@" - } - sleep() { :; } - info() { :; } - OPENSHELL_INSTALL_DOCKER_TIMEOUT=3 wait_for_docker_daemon -) >"$out" 2>"$err"; then - echo "FAIL: Docker readiness should succeed after retries" >&2 - cat "$err" >&2 || true - exit 1 -fi -if [ "$(cat "$attempts_file")" != "3" ]; then - echo "FAIL: Docker readiness did not retry three times" >&2 - exit 1 -fi -if [ "$(cat "$root_probes_file")" != "3" ]; then - echo "FAIL: Docker readiness probes did not run through as_root" >&2 - exit 1 -fi - -if ( - env() { return 1; } - as_root() { "$@"; } - snap() { return 1; } - sleep() { :; } + as_target_user() { return 0; } info() { :; } - OPENSHELL_INSTALL_DOCKER_TIMEOUT=2 wait_for_docker_daemon + warn() { echo "FAIL: accessible KVM produced warning: $*" >&2; return 1; } + TARGET_USER=test-user + check_snap_kvm_access ) >"$out" 2>"$err"; then - echo "FAIL: Docker readiness timeout should fail" >&2 - exit 1 -fi -if ! grep -Fq "Docker daemon at /var/run/docker.sock did not become reachable within 2s" "$err"; then - echo "FAIL: missing Docker readiness timeout message" >&2 + echo "FAIL: target-user KVM access should be accepted" >&2 cat "$err" >&2 || true exit 1 fi -user_attempts_file="${tmpdir}/docker-user-attempts" -user_probes_file="${tmpdir}/docker-user-probes" -printf '0\n' >"$user_attempts_file" -printf '0\n' >"$user_probes_file" if ! ( - env() { - if [ "$*" != "-u DOCKER_HOST -u DOCKER_CONTEXT docker --host unix:///var/run/docker.sock info" ]; then - echo "unexpected user Docker readiness probe: $*" >&2 - return 1 - fi - attempts="$(cat "$user_attempts_file")" - attempts=$((attempts + 1)) - printf '%s\n' "$attempts" >"$user_attempts_file" - [ "$attempts" -ge 3 ] - } - as_target_user() { - user_probes="$(cat "$user_probes_file")" - printf '%s\n' "$((user_probes + 1))" >"$user_probes_file" - "$@" - } - sleep() { :; } + as_target_user() { return 1; } + getent() { return 0; } + snap_kvm_device() { printf '%s\n' "${tmpdir}/test-kvm"; } info() { :; } + warn() { printf '%s\n' "$*"; } + : >"${tmpdir}/test-kvm" TARGET_USER=test-user - export DOCKER_HOST=tcp://remote.example:2376 - export DOCKER_CONTEXT=remote - OPENSHELL_INSTALL_DOCKER_TIMEOUT=3 wait_for_user_docker_daemon + check_snap_kvm_access ) >"$out" 2>"$err"; then - echo "FAIL: user Docker readiness should succeed after retries" >&2 - cat "$err" >&2 || true - exit 1 -fi -if [ "$(cat "$user_attempts_file")" != "3" ]; then - echo "FAIL: user Docker readiness did not retry three times" >&2 + echo "FAIL: missing target-user KVM access must be non-fatal" >&2 exit 1 fi -if [ "$(cat "$user_probes_file")" != "3" ]; then - echo "FAIL: user Docker readiness probes did not run through as_target_user" >&2 +if ! grep -Fq 'sudo usermod -aG kvm "test-user"' "$out" || \ + ! grep -Fq 'set compute_driver to another installed driver, such as docker or podman' "$out"; then + echo "FAIL: missing KVM access guidance" >&2 + cat "$out" >&2 exit 1 fi - -if ( - env() { return 1; } - as_target_user() { "$@"; } - sleep() { :; } - info() { :; } - TARGET_USER=test-user - OPENSHELL_INSTALL_DOCKER_TIMEOUT=2 wait_for_user_docker_daemon -) >"$out" 2>"$err"; then - echo "FAIL: user Docker readiness timeout should fail" >&2 - exit 1 -fi -if ! grep -Fq "Docker daemon at /var/run/docker.sock did not become reachable as test-user within 2s" "$err"; then - echo "FAIL: missing user Docker readiness timeout message" >&2 - cat "$err" >&2 || true +if grep -Fq 'usermod ' <(declare -f check_snap_kvm_access | grep -v 'warn '); then + echo "FAIL: KVM access check must not modify group membership" >&2 exit 1 fi diff --git a/tasks/scripts/test-packaging-assets.sh b/tasks/scripts/test-packaging-assets.sh index e056d3121e..f846f4750a 100755 --- a/tasks/scripts/test-packaging-assets.sh +++ b/tasks/scripts/test-packaging-assets.sh @@ -138,13 +138,20 @@ assert_contains "$package_deb" "\$pkgroot/usr/lib/systemd/user/openshell-gateway assert_contains "$snap_wrapper" '[ -z "${OPENSHELL_GATEWAY_CONFIG:-}" ]' assert_contains "$snap_wrapper" '[ -e "$OPENSHELL_SNAP_CONFIG_FILE" ] || [ -L "$OPENSHELL_SNAP_CONFIG_FILE" ]' assert_contains "$snap_wrapper" 'export OPENSHELL_GATEWAY_CONFIG="$OPENSHELL_SNAP_CONFIG_FILE"' +assert_contains "$snap_wrapper" '[ -z "${OPENSHELL_COMPUTE_DRIVER+x}" ]' +assert_contains "$snap_wrapper" '[ -z "${OPENSHELL_DRIVERS+x}" ]' +assert_contains "$snap_wrapper" "(compute_driver|\\\"compute_driver\\\"|'compute_driver')" +assert_contains "$snap_wrapper" "(openshell|\\\"openshell\\\"|'openshell')" +assert_contains "$snap_wrapper" '[ -r "$kvm_device" ]' +assert_contains "$snap_wrapper" '[ -w "$kvm_device" ]' +assert_contains "$snap_wrapper" 'export OPENSHELL_COMPUTE_DRIVER=vm' assert_contains "$snap_wrapper" 'config preflight -- "$@"' assert_not_contains "$snap_wrapper" "CANONICAL_CONFIG_FILE" bash "$ROOT/tasks/scripts/test-snap-gateway-wrapper.sh" "$snap_wrapper" # Store installs autoconnect all required interfaces and require snapd 2.76 for -# the system Docker slot. Manual connection for locally-built snaps requires -# snapd 2.77. +# the system Docker slot and Podman interface. Manual connection to :docker for +# locally-built snaps requires snapd 2.77. assert_contains "$snapcraft" "assumes: [snapd2.76]" for snap_file in \ "$snapcraft" \ @@ -160,6 +167,10 @@ if [[ -e "${ROOT}/snap/hooks/connect-plug-docker" ]]; then echo "FAIL: obsolete snap Docker connection hook must not exist" >&2 exit 1 fi +if [[ -e "${ROOT}/snap/hooks/connect-plug-podman" ]]; then + echo "FAIL: snap Podman connection hook must not exist" >&2 + exit 1 +fi if [[ ! -x "$snap_install_hook" ]]; then echo "FAIL: snap install hook must be executable" >&2 exit 1 @@ -208,8 +219,34 @@ bash "$ROOT/tasks/scripts/test-snap-post-refresh-hook.sh" "$snap_post_refresh_ho assert_contains "$snap_workflow" 'name: openshell-prover-${{ matrix.rust_arch }}-unknown-linux-musl' assert_contains "$snap_workflow" 'chmod +x prebuilt/prover/openshell-prover' assert_contains "$snap_workflow" 'cp prebuilt/prover/openshell-prover snap/prebuilt/openshell-prover' -assert_contains "$snapcraft" 'for bin in openshell openshell-prover openshell-gateway openshell-sandbox openshell-gateway-wrapper; do' +assert_contains "$snap_workflow" 'name: openshell-driver-vm-${{ matrix.rust_arch }}-unknown-linux-gnu' +assert_contains "$snap_workflow" 'chmod +x prebuilt/driver-vm/openshell-driver-vm' +assert_contains "$snap_workflow" 'cp prebuilt/driver-vm/openshell-driver-vm snap/prebuilt/openshell-driver-vm' +assert_contains "$snapcraft" 'for bin in openshell openshell-prover openshell-gateway openshell-sandbox openshell-driver-vm openshell-gateway-wrapper; do' assert_contains "$snapcraft" '"$CRAFT_PART_INSTALL/bin/openshell-prover"' +assert_contains "$snapcraft" '"$CRAFT_PART_INSTALL/bin/openshell-driver-vm"' +assert_not_contains "$snapcraft" '/usr/libexec/openshell:' +assert_not_contains "$snapcraft" 'symlink: $SNAP/usr/libexec/openshell' +assert_contains "$snapcraft" ' - e2fsprogs' +if [[ $(grep -c '^ - kvm$' "$snapcraft") -ne 2 ]]; then + echo "FAIL: both snap gateway apps must plug kvm" >&2 + exit 1 +fi +if [[ $(grep -c '^ - podman$' "$snapcraft") -ne 1 ]]; then + echo "FAIL: only the snap user gateway must plug podman" >&2 + exit 1 +fi +if ! awk ' + /^ system-gateway:$/ { app = "system"; next } + /^ user-gateway:$/ { app = "user"; next } + /^ [[:alnum:]_-]+:$/ { app = "" } + app == "system" && /^ - podman$/ { system_podman = 1 } + app == "user" && /^ - podman$/ { user_podman = 1 } + END { exit !(!system_podman && user_podman) } +' "$snapcraft"; then + echo "FAIL: podman must be plugged only by the snap user gateway" >&2 + exit 1 +fi if ! awk ' /^ prover:$/ { in_prover = 1; next } in_prover && /^ [[:alnum:]_-]+:$/ { finished = 1; exit } @@ -228,8 +265,11 @@ fi assert_not_contains "$snap_install_docs" "snap connect openshell:home" assert_not_contains "$snap_install_docs" "snap connect openshell:network" assert_not_contains "$snap_install_docs" "snap connect openshell:network-bind" +assert_contains "$snap_install_docs" "snap connect openshell:kvm" assert_contains "$snap_install_docs" "snap connect openshell:docker :docker" -assert_contains "$snap_install_docs" "systemctl --user reset-failed snap.openshell.user-gateway.service" +assert_contains "$snap_install_docs" "snap connect openshell:podman :podman" +assert_contains "$snap_install_docs" 'compute_driver = "docker"' +assert_contains "$snap_install_docs" 'compute_driver = "podman"' assert_contains "$snap_install_docs" "systemctl --user enable --now snap.openshell.user-gateway.service" assert_contains "$snapcraft" "systemctl --user enable --now snap.openshell.user-gateway.service" assert_contains "$ROOT/install.sh" "systemctl --user enable --now snap.openshell.user-gateway.service" @@ -258,11 +298,72 @@ assert_contains "$snap_install_docs" "sudo snap set openshell gateway-mode=syste assert_contains "$snapcraft" "gateway-mode=disable" assert_not_contains "$snap_install_docs" "snap services --user" assert_contains "$snap_canary" "install.sh | sh" -assert_contains "$snap_canary" "ubuntu-snap-system-docker:" -assert_contains "$snap_canary" "ubuntu-snap-docker-preflight:" +assert_contains "$snap_canary" "ubuntu-snap-vm:" +assert_contains "$snap_canary" "ubuntu-snap-docker:" +assert_contains "$snap_canary" "ubuntu-snap-podman:" +assert_contains "$snap_canary" "OPENSHELL_COMPUTE_DRIVER=docker" +assert_contains "$snap_canary" "OPENSHELL_COMPUTE_DRIVER=podman" +assert_contains "$snap_canary" "^docker +openshell:docker +:docker +" +assert_contains "$snap_canary" "^podman +openshell:podman +:podman +" +assert_contains "$snap_canary" "openshell-driver-vm" +assert_contains "$snap_canary" "snap connections openshell" assert_contains "$snap_canary" "openshell.prover check" assert_not_contains "$snap_canary" "--dangerous" -assert_not_contains "$snap_canary" "snap connect openshell:docker" +assert_not_contains "$snap_canary" "snap connect openshell:podman" +assert_not_contains "$snap_canary" "ubuntu-snap-docker-preflight:" +if ! awk ' + /^ ubuntu-snap-vm:$/ { in_job = 1; next } + in_job && /^ [[:alnum:]_-]+:$/ { in_job = 0 } + in_job && /journalctl --user .*_SYSTEMD_INVOCATION_ID=/ { checked_journal = 1 } + in_job && checked_journal && /grep -F '\''Using compute driver'\''/ { checked_log = 1 } + in_job && checked_log && /grep -F '\''driver=vm'\''/ { checked_driver = 1 } + in_job && /openshell sandbox create/ { + if (!checked_driver) premature_sandbox = 1 + created_sandbox = 1 + } + END { exit !(checked_journal && checked_log && checked_driver && created_sandbox && !premature_sandbox) } +' "$snap_canary"; then + echo "FAIL: snap VM canary must verify VM selection before creating a sandbox" >&2 + exit 1 +fi +if ! awk ' + /^ ubuntu-snap-docker:$/ { in_job = 1; next } + in_job && /^ [[:alnum:]_-]+:$/ { in_job = 0 } + in_job && /systemctl --user set-environment/ { set_env = 1 } + in_job && set_env && /grep -Fx '\''OPENSHELL_COMPUTE_DRIVER=docker'\''/ { checked_env = 1 } + in_job && checked_env && /grep -E '\''\^docker \+openshell:docker \+:docker \+'\''/ { checked_plug = 1 } + in_job && checked_plug && /grep -F '\''Using compute driver'\''/ { checked_log = 1 } + in_job && checked_log && /grep -F '\''driver=docker'\''/ { checked_driver = 1 } + in_job && checked_driver && /openshell sandbox create/ { created_sandbox = 1 } + END { exit !(set_env && checked_env && checked_plug && checked_log && checked_driver && created_sandbox) } +' "$snap_canary"; then + echo "FAIL: snap Docker canary must select and verify Docker before creating a sandbox" >&2 + exit 1 +fi +if ! awk ' + /^ ubuntu-snap-podman:$/ { in_job = 1; next } + in_job && /^ [[:alnum:]_-]+:$/ { in_job = 0 } + in_job && /runs-on: ubuntu-26\.04/ { correct_runner = 1 } + in_job && /apt-get install -y podman/ { installed_podman = 1 } + in_job && /systemctl --user enable --now podman\.socket/ { started_socket = 1 } + in_job && /Host\.Security\.Rootless/ { checked_rootless = 1 } + in_job && /Host\.CgroupsVersion/ { checked_cgroups = 1 } + in_job && /systemctl --user set-environment/ { set_env = 1 } + in_job && set_env && /grep -Fx '\''OPENSHELL_COMPUTE_DRIVER=podman'\''/ { checked_env = 1 } + in_job && checked_env && /grep -E '\''\^podman \+openshell:podman \+:podman \+'\''/ { checked_plug = 1 } + in_job && checked_plug && /grep -F '\''Using compute driver'\''/ { checked_log = 1 } + in_job && checked_log && /grep -F '\''driver=podman'\''/ { checked_driver = 1 } + in_job && checked_driver && /gateway info --output json/ { checked_info = 1 } + in_job && checked_info && /openshell sandbox create/ { created_sandbox = 1 } + in_job && created_sandbox && /label=openshell\.managed=true/ { checked_owner = 1 } + in_job && checked_owner && /openshell sandbox exec/ { executed_sandbox = 1 } + in_job && executed_sandbox && /openshell sandbox delete/ { deleted_sandbox = 1 } + in_job && deleted_sandbox && /Podman resources remain after deleting/ { checked_cleanup = 1 } + END { exit !(correct_runner && installed_podman && started_socket && checked_rootless && checked_cgroups && set_env && checked_env && checked_plug && checked_log && checked_driver && checked_info && created_sandbox && checked_owner && executed_sandbox && deleted_sandbox && checked_cleanup) } +' "$snap_canary"; then + echo "FAIL: snap Podman canary must select and verify Podman before creating a sandbox" >&2 + exit 1 +fi if ! awk '/config preflight/ { seen = 1 } /generate-certs/ { exit !seen }' "$service"; then echo "FAIL: Debian preflight must precede certificate generation" >&2 exit 1 diff --git a/tasks/scripts/test-snap-gateway-wrapper.sh b/tasks/scripts/test-snap-gateway-wrapper.sh index 532bbe3e6d..3c08edd509 100755 --- a/tasks/scripts/test-snap-gateway-wrapper.sh +++ b/tasks/scripts/test-snap-gateway-wrapper.sh @@ -3,8 +3,9 @@ # SPDX-License-Identifier: Apache-2.0 set -euo pipefail +unset OPENSHELL_COMPUTE_DRIVER OPENSHELL_DRIVERS -wrapper=${1:?Usage: test-snap-gateway-wrapper.sh } +wrapper_source=${1:?Usage: test-snap-gateway-wrapper.sh } work=$(mktemp -d "${TMPDIR:-/tmp}/openshell snap wrapper.XXXXXX") trap 'rm -rf "$work"' EXIT @@ -14,13 +15,34 @@ log="$work/calls" expected="$work/expected" mkdir -p "$snap/bin" "$common" +# Use regular files as deterministic stand-ins for /dev/kvm, including +# inaccessible and single-permission cases. +kvm_device="$work/kvm" +: >"$kvm_device" +read_only_kvm="$work/read-only-kvm" +write_only_kvm="$work/write-only-kvm" +: >"$read_only_kvm" +: >"$write_only_kvm" +chmod 0400 "$read_only_kvm" +chmod 0200 "$write_only_kvm" +wrapper="$work/openshell-gateway-wrapper" +unavailable_wrapper="$work/openshell-gateway-wrapper-no-kvm" +read_only_wrapper="$work/openshell-gateway-wrapper-read-only-kvm" +write_only_wrapper="$work/openshell-gateway-wrapper-write-only-kvm" +sed "s|kvm_device=/dev/kvm|kvm_device='$kvm_device'|" "$wrapper_source" >"$wrapper" +sed "s|kvm_device=/dev/kvm|kvm_device='$work/missing-kvm'|" "$wrapper_source" >"$unavailable_wrapper" +sed "s|kvm_device=/dev/kvm|kvm_device='$read_only_kvm'|" "$wrapper_source" >"$read_only_wrapper" +sed "s|kvm_device=/dev/kvm|kvm_device='$write_only_kvm'|" "$wrapper_source" >"$write_only_wrapper" +chmod +x "$wrapper" "$unavailable_wrapper" "$read_only_wrapper" "$write_only_wrapper" + cat >"$snap/bin/openshell-gateway" <<'EOF' #!/bin/sh -printf '%s|config=%s|db=%s|tls=%s\n' \ +printf '%s|config=%s|db=%s|tls=%s|driver=%s\n' \ "$*" \ "${OPENSHELL_GATEWAY_CONFIG:-}" \ "${OPENSHELL_DB_URL:-}" \ - "${OPENSHELL_LOCAL_TLS_DIR:-}" >>"$FAKE_GATEWAY_LOG" + "${OPENSHELL_LOCAL_TLS_DIR:-}" \ + "${OPENSHELL_COMPUTE_DRIVER:-}" >>"$FAKE_GATEWAY_LOG" if [ "${1:-}:${2:-}" = config:preflight ] && [ "${FAKE_PREFLIGHT_FAIL:-}" = 1 ]; then exit 42 fi @@ -51,23 +73,98 @@ assert_log() { # normal config discovery. : >"$log" run_system_wrapper --trace -assert_log "config preflight -- --trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls -generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls ---trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls" +assert_log "config preflight -- --trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm +--trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm" + +# Without read/write KVM access, leave selection unset so the gateway can run +# its normal driver detection. +: >"$log" +wrapper=$unavailable_wrapper run_system_wrapper --trace +assert_log "config preflight -- --trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" + +if [ "$(id -u)" -ne 0 ]; then + for restricted_wrapper in "$read_only_wrapper" "$write_only_wrapper"; do + : >"$log" + wrapper=$restricted_wrapper run_system_wrapper --trace + assert_log "config preflight -- --trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" + done +fi # An existing compatibility config is exposed through the standard gateway # config environment variable for both preflight and startup. printf 'valid schema-v2\n' >"$common/gateway.toml" : >"$log" run_system_wrapper --trace -assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls -generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls ---trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls" +assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm +--trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm" + +# An explicit TOML selector remains authoritative, including indentation. +printf '[openshell.gateway]\n compute_driver = "docker"\n' >"$common/gateway.toml" +: >"$log" +run_system_wrapper --trace +assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" + +# An explicit TOML selector remains authoritative without KVM access. +: >"$log" +wrapper=$unavailable_wrapper run_system_wrapper --trace +assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" + +# Best-effort TOML inspection recognizes quoted keys and dotted key syntax. +for selector in \ + ' "compute_driver" = "docker"' \ + " 'compute_driver' = \"docker\"" \ + 'openshell.gateway.compute_driver = "docker"' \ + '"openshell"."gateway"."compute_driver" = "docker"' \ + "openshell . \"gateway\" . 'compute_driver' = \"docker\"" +do + printf '%s\n' "$selector" >"$common/gateway.toml" + : >"$log" + run_system_wrapper --trace + assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" +done + +# The deprecated environment selector remains authoritative during migration. +rm "$common/gateway.toml" +: >"$log" +OPENSHELL_DRIVERS=docker run_system_wrapper --trace +assert_log "config preflight -- --trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--trace|config=|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" + +printf '[openshell.gateway]\ncompute_driver = "docker"\n' >"$common/gateway.toml" + +# A service-level environment selector takes precedence without reading TOML. +: >"$log" +OPENSHELL_COMPUTE_DRIVER=docker run_system_wrapper --trace +assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=docker +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=docker +--trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=docker" + +# An explicit environment selector remains authoritative without KVM access. +: >"$log" +OPENSHELL_COMPUTE_DRIVER=docker wrapper=$unavailable_wrapper run_system_wrapper --trace +assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=docker +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=docker +--trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=docker" # An operator-provided environment path takes precedence over the snap # compatibility path, while CLI arguments are replayed unchanged. override="$work/override.toml" printf 'operator override\n' >"$override" +cli_config="$work/cli.toml" +printf '[openshell.gateway]\ncompute_driver = "docker"\n' >"$cli_config" : >"$log" env \ SNAP="$snap" \ @@ -77,10 +174,18 @@ env \ OPENSHELL_DB_URL="sqlite:$common/gateway.db?mode=rwc" \ OPENSHELL_LOCAL_TLS_DIR="$common/tls" \ FAKE_GATEWAY_LOG="$log" \ - "$wrapper" --config "$work/cli.toml" --trace -assert_log "config preflight -- --config $work/cli.toml --trace|config=$override|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls -generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$override|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls ---config $work/cli.toml --trace|config=$override|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls" + "$wrapper" --config "$cli_config" --trace +assert_log "config preflight -- --config $cli_config --trace|config=$override|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$override|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--config $cli_config --trace|config=$override|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" + +# An explicitly empty environment selector is still preserved for gateway +# validation instead of being replaced with the snap default. +: >"$log" +OPENSHELL_COMPUTE_DRIVER= run_system_wrapper --trace +assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +generate-certs --output-dir $common/tls --server-san host.openshell.internal|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver= +--trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=" # Broken compatibility symlinks are selected so preflight fails closed rather # than silently falling back to defaults. @@ -91,7 +196,7 @@ if FAKE_PREFLIGHT_FAIL=1 run_system_wrapper --trace; then echo "FAIL: broken compatibility symlink reached certificate generation" >&2 exit 1 fi -assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls" +assert_log "config preflight -- --trace|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm" # Preflight failure prevents certificate generation and gateway startup. : >"$log" @@ -99,7 +204,7 @@ if FAKE_PREFLIGHT_FAIL=1 run_system_wrapper --config --; then echo "FAIL: preflight failure reached certificate generation" >&2 exit 1 fi -assert_log "config preflight -- --config --|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls" +assert_log "config preflight -- --config --|config=$common/gateway.toml|db=sqlite:$common/gateway.db?mode=rwc|tls=$common/tls|driver=vm" # User mode supplies only XDG roots. The wrapper derives TLS state, leaves the # database unset for OpenShell's native default, and lets OpenShell discover @@ -117,9 +222,9 @@ env -u OPENSHELL_GATEWAY_CONFIG \ XDG_STATE_HOME="$user_common/.local/state" \ FAKE_GATEWAY_LOG="$log" \ "$wrapper" --trace -assert_log "config preflight -- --trace|config=|db=|tls=$user_tls -generate-certs --output-dir $user_tls --server-san host.openshell.internal|config=|db=|tls=$user_tls ---trace|config=|db=|tls=$user_tls" +assert_log "config preflight -- --trace|config=|db=|tls=$user_tls|driver=vm +generate-certs --output-dir $user_tls --server-san host.openshell.internal|config=|db=|tls=$user_tls|driver=vm +--trace|config=|db=|tls=$user_tls|driver=vm" if env -u OPENSHELL_LOCAL_TLS_DIR -u XDG_STATE_HOME \ SNAP="$snap" FAKE_GATEWAY_LOG="$log" "$wrapper" --trace \