bug(supervisor): policy v2 never acknowledged after PushSandboxLogs INTERNAL error (v0.0.112 regression) #3117

Description

Actual behavior: After upgrading from v0.0.111 to v0.0.112, openshell policy update --wait times out on every sandbox. The supervisor reports loading policy v1 (the default), but never acknowledges policy v2 (the incremental update with endpoints and binaries). The CLI polls GetSandboxPolicyStatus for ~15 seconds and exits with code 124 (timeout).

Expected behavior: The supervisor should receive the v2 policy via UpdateConfig, compile it, and report it as loaded via ReportPolicyStatus. The --wait flag should return successfully.

Working version: v0.0.111-rhaiv.0
Broken version: v0.0.112-rhaiv.0 (built from commits between v0.0.111 and the v0.0.112 tag)

Reproduction Steps

Tested in CI (GitHub Actions, ubuntu-latest, privileged container) using the agentic-ci E2E harness. The failure is 100% reproducible across multiple runs.

Minimal reproducer (requires v0.0.112 CLI, gateway, and supervisor)

#!/bin/bash
set -euo pipefail

# 1. Start podman API socket
XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
mkdir -p "$XDG_RUNTIME_DIR/podman"
podman system service --time=0 "unix://$XDG_RUNTIME_DIR/podman/podman.sock" &
sleep 2

# 2. Write gateway config
mkdir -p ~/.config/openshell
cat > ~/.config/openshell/gateway.toml <<'TOML'
[openshell]
version = 1

[openshell.gateway]
bind_address = "0.0.0.0:17670"
compute_drivers = ["podman"]

[openshell.drivers.podman]
supervisor_image = "quay.io/opendatahub/odh-openshell-supervisor:v0.0.112-rhaiv.0"
TOML

# 3. Generate TLS certs and start gateway
mkdir -p ~/.local/state/openshell/tls
openshell-gateway generate-certs \
    --output-dir ~/.local/state/openshell/tls \
    --server-san host.openshell.internal

openshell-gateway --db-url sqlite::memory: --log-level info \
    > ~/.local/state/openshell/gateway.log 2>&1 &
sleep 3

# 4. Register gateway
openshell gateway add https://localhost:17670 --local --name ci

# 5. Wait for gateway health
for i in $(seq 1 30); do
    openshell status >/dev/null 2>&1 && break
    sleep 2
done

# 6. Create sandbox with a short-lived command
openshell sandbox create \
    --name ci \
    --no-tty \
    --no-auto-providers \
    --from quay.io/aipcc/agentic-ci/claude-sandbox:latest \
    -- true

# 7. Apply policy update -- THIS TIMES OUT
openshell policy update --wait \
    --binary /usr/local/bin/claude \
    --add-endpoint "github.com:443:full" \
    --add-endpoint "*.github.com:443:full" \
    ci

# Expected: exits 0 after supervisor reports policy v2 loaded
# Actual:   "Policy version 2 submitted" then
#           "Timeout waiting for policy version 2 to load" (exit 124)

What the gateway log shows

The full sequence from the gateway log (timestamps relative to sandbox creation):

T+0.0s  CreateSandbox: pulling supervisor image (policy=missing)
T+9.2s  ReportPolicyStatus: version=1 status=loaded        ← supervisor loads default policy
T+9.3s  ConnectSupervisor: session accepted                 ← supervisor connects
T+9.3s  PushSandboxLogs: rpc.grpc.status_code=13 ERROR     ← INTERNAL error
T+9.3s  UpdateConfig: merged 9 incremental policy ops [version:v2 hash:afc22bc0...]
        (each add-endpoint logged individually, all version:v2)
T+9.3s  UpdateConfig: response status=200
T+10s   GetSandboxPolicyStatus: 200  (CLI polling)
T+11s   GetSandboxPolicyStatus: 200  (CLI polling)
...     (polls every ~1s for 15 seconds)
T+25s   GetSandboxPolicyStatus: 200  (CLI polling)
        CLI exits 124: "Timeout waiting for policy version 2 to load"

Key observations:

1. The supervisor reports v1 loaded and connects successfully
2. PushSandboxLogs returns gRPC status 13 (INTERNAL) immediately after ConnectSupervisor
3. The gateway merges v2 into UpdateConfig and responds 200
4. The supervisor never calls ReportPolicyStatus for v2
5. The CLI polls GetSandboxPolicyStatus but never sees v2 acknowledged
6. This pattern repeats identically on every subsequent sandbox creation in the same session

Suspicious commits between v0.0.111 and v0.0.112

fix(providers): honor configured profile sources in sandboxes (#2878)
feat(supervisor): expose sandbox name to middleware request context (#2771)
fix(policy): validate the applicable advisor candidate (#2850)
fix(provider): isolate unbound static credentials (#2862)
feat(provider): support for SPIFFE backed token exchange (#1970)

Environment

• OS: Ubuntu (GitHub Actions runner, privileged container)
• OpenShell CLI: v0.0.112-rhaiv.0 (from quay.io/opendatahub/odh-openshell-cli:v0.0.112-rhaiv.0)
• OpenShell Gateway: v0.0.112-rhaiv.0 (from quay.io/opendatahub/odh-openshell-gateway:v0.0.112-rhaiv.0)
• Supervisor: v0.0.112-rhaiv.0 (from quay.io/opendatahub/odh-openshell-supervisor:v0.0.112-rhaiv.0)
• Podman: 5.8.4
• Container driver: podman (rootful)
• Sandbox image: Hummingbird-based agentic image

CI Runs Showing the Failure

• https://github.com/opendatahub-io/agentic-ci/actions/runs/33572215462/job/100068852805 (v0.0.112, 46 pass / 14 fail)
• https://github.com/opendatahub-io/agentic-ci/actions/runs/33572053578/job/100068328955 (v0.0.112, 46 pass / 14 fail)
• https://github.com/opendatahub-io/agentic-ci/actions/runs/33565191005/job/100060648399 (v0.0.112 from PR Security hardening: remediate aardvark/codex scanner findings #350, same failure)

All 14 failures are agent-run tests that go through openshell policy update --wait. All 46 static sandbox checks (binary presence, plugin setup, user/workdir config) pass.

Related Issues

• [Bug] Supervisor does not acknowledge newer sandbox policy revisions with an unchanged policy hash #2518 — Supervisor does not acknowledge newer sandbox policy revisions with an unchanged policy hash (fixed in fix(sandbox): acknowledge unchanged policy revisions #2557, different root cause)
• [Bug] UpdateConfig does not wake sandbox watchers after an atomic policy revision commit #2517 — UpdateConfig does not wake sandbox watchers after an atomic policy revision commit (fixed, different root cause)
• bug: driver-controlled runtimes do not apply live policy revisions #3077 — Driver-controlled runtimes do not apply live policy revisions (MXC-specific, different driver)