Skip to content

bug(podman): verify managed volume ownership before sandbox cleanup #4111

Open

Description

@elezar

User Story

While reviewing OpenShell PR #3981, I identified a gap in managed-volume cleanup. I need sandbox deletion to preserve Podman volumes that do not belong to the sandbox being deleted, including when the workload container is already absent.

This was identified during code review. No live data loss or runtime reproduction is claimed.

Problem Statement

The Podman driver checks ownership labels before adopting an existing managed volume during creation, but deletes managed workspace and channel volumes by their generated names without checking their ownership labels.

delete_sandbox attempts to remove openshell-channel-{sandbox_id} and openshell-sandbox-{sandbox_id}-workspace. The workspace removal also occurs when the workload-container lookup returns no container. PodmanClient::remove_volume validates the name and sends a DELETE request directly; it does not inspect the volume.

If an unused, unrelated volume occupies either generated name, sandbox cleanup can delete it despite missing or conflicting sandbox ownership labels. This gap predates #3981; the PR's UID/GID creation options do not introduce it.

Impact / Why This Matters

An unrelated volume that collides with a generated managed-volume name can lose its stored data during sandbox cleanup. Name uniqueness is currently the only deletion safeguard for these volumes. Operators would need to avoid collisions or manually verify names before cleanup; deletion itself does not enforce the ownership boundary already used during provisioning.

Acceptance Criteria

  • Managed workspace and channel cleanup checks sandbox ownership labels before requesting volume deletion.
  • Missing or mismatching ownership labels preserve the volume and produce a clear diagnostic.
  • Correctly labeled managed volumes remain cleanable when the workload container is already absent.
  • Missing volumes remain a successful, idempotent no-op.
  • Filesystem UID/GID options are not required to match for deletion: legacy volumes with empty options and correctly labeled volumes with stale ownership settings remain cleanable.
  • Regression coverage verifies that no DELETE request is issued for an unrelated same-name volume, both with and without a workload container.

Reproduction Steps / Proposed Validation

Runtime reproduction has not been performed. The following is a proposed OpenShell driver regression scenario, using the existing Podman API stub rather than installing additional tools:

  1. Configure the OpenShell Podman driver's existing test fixture for a sandbox ID, with an unused workspace or channel volume at its generated managed-volume name. Give that volume no ownership labels, or an openshell.ai/sandbox-id label naming a different sandbox.
  2. Arrange for the workload-container lookup to return no container.
  3. Invoke the driver's delete_sandbox for that sandbox ID and capture its API requests.
  4. The current implementation requests DELETE for the generated volume name without an inspect/ownership check. The expected behavior is to preserve the unrelated volume.
  5. Repeat with a present workload container and with correctly labeled or missing managed volumes to verify preservation, ordinary cleanup, and idempotence.

Environment

Agent Investigation

  • Base delete_sandbox implementation removes managed volumes by generated names, including the no-workload-container path.
  • Base client volume operations show creation ownership checks and direct deletion without inspection.
  • The existing delete_sandbox_cleans_up_volume_when_container_is_already_gone test asserts the direct DELETE request; extend coverage with mismatched ownership metadata.
  • A scoped implementation could retain low-level remove_volume and introduce remove_owned_volume(name, sandbox_id) or an equivalent managed-cleanup check. Ownership here means OpenShell resource labels, not filesystem UID/GID.
  • Searched open and closed issues for Podman volume ownership, Podman volume deletion, and volume "unrelated". No matching issue was found. Related closed issue bug(driver-podman): per-sandbox secrets and volumes are orphaned when a container disappears without DeleteSandbox #2352 concerns orphaned resources when cleanup is not invoked; this issue concerns preserving unrelated resources when cleanup is invoked.

Activity

  1. aetherxeg-source commented on Oct 2, 2026

    @aetherxeg-source

    A generated resource name should not itself confer deletion authority.

    The invariant I would use is:

    "name match != ownership proof"

    For managed-volume cleanup, the deletion path should consume authoritative ownership evidence from the actual volume:

    "resolved volume -> ownership labels -> expected sandbox identity/generation -> delete authorization"

    A particularly important negative regression:

    1. create unrelated volume V using the generated name expected for sandbox S;
    2. give V missing or conflicting ownership metadata;
    3. delete S;
    4. verify no DELETE request for V is emitted.

    Then repeat after the workload container is already absent.

    This catches the dangerous case where loss of one piece of runtime state causes the cleanup path to fall back to weaker name-based authority.

    Related reference implementation / design work: AETHER X GLOBAL ExecSurface — https://github.com/AETHERXGLOBAL/execsurface

  2. added 2 commits that reference this issue on Oct 4, 2026
    2e8b94f
    ce958eb
  3. lunarwhite commented on Oct 13, 2026

    @lunarwhite
    Contributor

    🏗️ build-plan

    Outcome

    On Podman, sandbox cleanup removes the managed channel and workspace volumes only when they belong to the sandbox, so an unrelated volume at either name survives delete and failed creates. One PR closes #4111.

    Design

    • Delete inspects the channel and workspace volumes before each existing removal and removes a volume only when its openshell.ai/sandbox-id label equals the sandbox ID (AC1). Nothing else is compared: delete receives only the sandbox ID (proto/compute_driver.proto:437-442), and options must not matter (AC5).
    • A missing volume is a no-op (AC4). Any other volume is kept with a gateway-log warning naming the sandbox, volume, and reason (AC2), and delete still succeeds, as it does today when a volume removal fails.
    • Create rollback removes the workspace volume only when this attempt created or adopted it, as it already does for the channel (crates/openshell-driver-podman/src/driver.rs:1158-1160).

    Tests

    • Driver unit tests through the existing Podman API stub; new tests assert the complete request log:
      • Without a workload container (the failed-create path), an unlabeled volume at one managed name and another sandbox's volume at the other get no DELETE (AC2, AC6).
      • With a workload container, an unowned volume at a custom-WORKDIR sandbox's workspace name gets no DELETE, and the channel volume is still removed (AC6).
      • The existing delete tests gain inspect responses and cover owned volumes with owner options, an absent workload, and a missing volume (AC3-AC5).
      • A failed custom-WORKDIR create sends no request for the workspace volume name.
    • No new e2e test: every Podman e2e create already reads these labels back from real Podman (crates/openshell-driver-podman/src/client.rs:795-800). The Podman lane under test:e2e must pass.

    Documentation

    crates/openshell-driver-podman/README.md: one sentence in the lifecycle section (:122-125) saying that delete keeps, and logs, a channel or workspace volume whose label does not name the sandbox. No published page covers Podman volume deletion.

  4. lunarwhite commented on Oct 13, 2026

    @lunarwhite
    Contributor

    I plan to raise a PR based on above plan, will wait for some time to see if any review comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions