Skip to content

fix(sandbox): acknowledge initial policy load and expose SDK labels #2159

Closed
#2170
Closed
fix(sandbox): acknowledge initial policy load and expose SDK labels#2159
#2170

Description

@KyleZheng1284
KyleZheng1284
opened on Jul 7, 2026

Summary

Two OpenShell lifecycle/API gaps block the policy-bound per-job sandbox integration in NVIDIA-AI-Blueprints/aiq#298 for AI-Q 2.2:

  1. A newly created sandbox can reach Ready and expose an effective policy while the same revision remains Pending indefinitely.
  2. The public Python SDK cannot send gateway-object labels or list sandboxes by label selector, although the protobuf supports both.

AI-Q intentionally fails closed instead of exposing execution without authoritative policy attestation.

Confirmed policy-state behavior

Reproduced on macOS/Apple Silicon with Docker Desktop:

OpenShell Creation path Result
0.0.72 clean gateway diagnostic Ready; effective policy present; matching revision remains Pending; current_policy_version=0
0.0.77 CLI directly, without AI-Q Ready; revision 2 is Effective via policy get; revision 2 remains Pending via policy list; revision 1 is Superseded

The state remained stable rather than transitioning to LOADED. The policy itself was successfully constructed and applied. landlock.compatibility: best_effort is therefore not an attestation bypass or explanation; it controls filesystem-enforcement compatibility.

Linux has not yet been reproduced. The observed behavior is confirmed on the macOS/Docker Desktop path; source inspection suggests shared lifecycle logic, but a Linux run is required to establish platform scope.

Minimal reproduction

  1. Create a sandbox with an explicit policy.
  2. Wait until the sandbox reaches Ready.
  3. Compare policy get <sandbox> -o json, policy list <sandbox>, and the sandbox's current_policy_version.
  4. Observe an effective positive revision while that same revision remains Pending.
  5. Delete the sandbox and verify it is absent.

Expected contract

After initial policy construction:

  • report the exact sandbox-scoped revision as LOADED;
  • report FAILED with a load error when construction fails;
  • never leave a successfully applied initial revision permanently Pending;
  • continue reporting later hot-reload revisions normally.

Suspected source path

Inspection of v0.0.72 and v0.0.74 found:

  • load_policy() constructs the initial OpaEngine;
  • the first run_policy_poll_loop() iteration records config_revision and policy_hash;
  • ReportPolicyStatus(LOADED) is sent only from the later policy_changed branch;
  • an unchanged initial hash never enters that branch.

The same runtime symptom is present in 0.0.77; its exact source path should be confirmed against that tag.

Python SDK label/selector gap

CreateSandboxRequest supports request-level labels and ListSandboxes supports label_selector, but the public Python lifecycle API does not expose them:

  • SandboxClient.create(), create_session(), and Sandbox(...) cannot forward gateway labels;
  • SandboxClient.list() cannot forward a selector;
  • SandboxRef does not retain gateway metadata labels.

Template labels reach Docker/runtime resources, but gateway selectors inspect gateway-object metadata. Consequently, a Python-created AI-Q sandbox is not returned by:

openshell sandbox list --selector aiq=deep-research

Proposed fix

Initial policy acknowledgement

  • Preserve the fetched sandbox policy version/source through initial construction.
  • Report initial success or failure exactly once.
  • Track the acknowledged version so later polling does not duplicate it.
  • Preserve existing hot-reload behavior.

Python SDK

Add optional, source-compatible parameters:

  • name and labels for creation/session/Sandbox;
  • label_selector for listing;
  • immutable gateway labels on SandboxRef.

The lifecycle and SDK fixes may land separately, but AI-Q requires both in a tagged release.

Acceptance criteria

  • Initial success and failure acknowledgement tests.
  • Later policy-revision/hot-reload regression tests.
  • Python request-label and selector-forwarding tests.
  • Docker E2E: create → Ready → LOADED → selector membership → execute → delete.
  • Direct lookup and selector both confirm deletion.
  • Tagged release with compatible CLI, Python SDK, and gateway versions.
  • AI-Q macOS demo validation with explicit best_effort.
  • AI-Q Linux production validation with hard_requirement.

Downstream impact

This issue blocks AI-Q #298. It blocks only the OpenShell-specific live acceptance for stacked artifact-lifecycle PR AI-Q #305; #305's provider-independent artifact-storage implementation can be restacked onto develop and proceed separately. AI-Q will not treat Pending as success, weaken attestation, or pin an unreleased fork.

A focused implementation PR can be prepared after maintainer triage and the repository's human state:agent-ready gate.

Activity

  1. github-actions
    added
    state:triage-neededOpened without agent diagnostics and needs triage
    on Jul 7, 2026

  2. KyleZheng1284 commented on Jul 7, 2026

    @KyleZheng1284
    KyleZheng1284
    on Jul 7, 2026
    ContributorAuthor

    Cross-project impact: blocks AI-Q 2.2 OpenShell integration

    This issue currently blocks NVIDIA-AI-Blueprints/aiq#298, which integrates policy-bound, per-job OpenShell sandboxes for the AI-Q 2.2 release. The AI-Q PR has been marked BLOCKED and will continue to fail closed rather than expose execution before authoritative policy attestation succeeds.

    There is now an additional reproduction using the OpenShell 0.0.77 CLI directly, without AI-Q, on macOS/Docker Desktop:

    • the sandbox reaches Ready;
    • openshell policy get reports revision 2 as Effective;
    • openshell policy list leaves revision 2 Pending, with revision 1 Superseded;
    • AI-Q's subsequent timeout is therefore a consequence of the unresolved authoritative status, not its sandbox-creation implementation.

    This also shows that upgrading from the earlier 0.0.72 reproduction to 0.0.77 does not by itself resolve the policy-state contradiction. best_effort is not being treated as an attestation bypass; it only changes Landlock compatibility.

    For AI-Q to unblock, we need:

    1. confirmation of the intended authoritative success contract for Effective + Pending;
    2. initial policy construction to report LOADED or FAILED exactly once if Pending is not intended;
    3. public Python SDK support for request-level labels and selector-based listing;
    4. a tagged release containing the fixes so AI-Q can pin matching CLI/SDK/gateway versions;
    5. successful macOS demo and Linux hard_requirement live acceptance.

    A focused upstream implementation PR can be prepared after maintainer triage and the repository's human state:agent-ready gate. A Linux CLI-only reproduction is still needed to establish whether the observed runtime manifestation is limited to macOS/Docker Desktop or is platform-independent.

  3. johntmyers commented on Jul 7, 2026

    @johntmyers
    johntmyers
    on Jul 7, 2026
    Collaborator

    @KyleZheng1284 I'm unable to reproduce. I see a different gap where current version was 1 vs 0 between the policy get and sandbox list commands but I'm not seeing the Pending state. Will post a more thorough analysis shortly.

  4. johntmyers commented on Jul 7, 2026

    @johntmyers
    johntmyers
    on Jul 7, 2026
    Collaborator

    📋 triage-agent

    Triage Assessment

    Classification: bug-confirmed

    Summary

    I reproduced the initial policy acknowledgement problem on local docker-dev gateway 0.0.78-dev.6+ga7271169. The exact revision 2 Pending state reproduces when the initial policy is sparse enough for sandbox-side baseline enrichment to sync an updated policy back to the gateway during startup. The Python SDK label/selector API gap also reproduces: public SDK methods reject labels, name, and label_selector, and template labels do not satisfy gateway object selectors.

    Reproduction

    Using examples/policy-advisor/sandbox-policy.yaml:

    openshell sandbox create --name issue-2159-enrich-repro \
  --policy examples/policy-advisor/sandbox-policy.yaml \
  --label enrich-2159=true \
  --no-tty -- true

    Observed after the sandbox reached Ready:

    openshell policy get issue-2159-enrich-repro -o json

    returned version: 2, active_version: 2, status: effective.

    openshell policy list issue-2159-enrich-repro --limit 50

    returned revision 2 Pending and revision 1 Superseded.

    openshell policy get issue-2159-enrich-repro --rev 2 -o json

    returned status: pending, active_version: 0.

    openshell sandbox list --selector enrich-2159=true -o json

    returned current_policy_version: 0.

    Sandbox logs showed the policy was actually used successfully: CONFIG:ENRICHED, Landlock applied, and the workload process launched.

    Root Cause

    current_policy_version lives on SandboxStatus, not in policy history. Hot reload updates it only through ReportPolicyStatus(Loaded).

    Relevant paths:

    crates/openshell-server/src/grpc/policy.rs:1133-1179: GetSandboxConfig lazily backfills spec.policy to revision 1 and marks it loaded, but does not update Sandbox.status.current_policy_version.

    crates/openshell-sandbox/src/lib.rs:1456-1465: sandbox startup may enrich the fetched policy and sync it back to the gateway.

    crates/openshell-server/src/grpc/policy.rs:1879-1891: that sync uses UpdateConfig, which inserts revision 2 as pending and supersedes older revisions.

    crates/openshell-sandbox/src/lib.rs:1634-1668: the poll loop seeds current_policy_hash from the first poll, so the already-current enriched revision is not treated as a policy change.

    crates/openshell-sandbox/src/lib.rs:1775-1782: ReportPolicyStatus is only called inside the later policy_changed branch.

    crates/openshell-server/src/grpc/policy.rs:2051-2069: ReportPolicyStatus(Loaded) is the path that updates Sandbox.status.current_policy_version.

    Why Commands Disagree

    openshell sandbox get prints Revision from GetSandboxConfig.version (crates/openshell-cli/src/run.rs:2635-2711).

    openshell sandbox list -o json prints Sandbox.status.current_policy_version (crates/openshell-cli/src/run.rs:3427-3438).

    openshell policy get without --rev labels the current config as effective and sets active_version equal to that config version (crates/openshell-cli/src/run.rs:7062-7123).

    openshell policy get --rev uses GetSandboxPolicyStatus, whose active_version comes from sandbox.current_policy_version() (crates/openshell-server/src/grpc/policy.rs:1927-1936).

    Recommendation

    Fix in the sandbox runtime rather than only in server backfill. After initial policy construction succeeds, the sandbox should report the actual initial sandbox-scoped policy version exactly once via ReportPolicyStatus(Loaded). If initial construction fails after a persisted revision exists, report ReportPolicyStatus(Failed) with the load error. Then seed the poll loop with the acknowledged version/hash so existing hot-reload behavior does not duplicate reports.

  5. KyleZheng1284
    added 3 commits that reference this issue on Jul 8, 2026

    test(sandbox): add sparse-policy revision-2 acknowledgement e2e

    cb3efa5

    test(sandbox): add sparse-policy revision-2 acknowledgement e2e

    eacda4d

    test(sandbox): add sparse-policy revision-2 acknowledgement e2e

    e7c3790
  6. johntmyers
    added a commit that references this issue on Jul 9, 2026

    fix(sandbox): acknowledge initial policy revision; expose SDK labels/…

    ff9af8e
  7. johntmyers
    closed this as completedin #2170on Jul 9, 2026
  8. AjayThorve
    added a commit that references this issue on Jul 14, 2026

    feat: isolate and attest OpenShell jobs (#298)

    e7abd3d
  9. Manushpm8
    added a commit that references this issue on Jul 16, 2026

    feat: isolate and attest OpenShell jobs (NVIDIA-AI-Blueprints#298)

    17e1adb
  10. rodbutters
    added a commit that references this issue on Aug 24, 2026

    Chore/resync 20260820 (#4)

    dab1dee
  11. purp
    removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions