Skip to content

openshell sandbox create --timeout=? #2913

Closed as not planned
Closed as not planned
openshell sandbox create --timeout=?#2913
Labels
state:needs-infoAssessment needs specific evidence or reproduction detailsstate:staleInactive item at risk of automatic closure.

Description

@hpcpony
hpcpony
opened on Aug 24, 2026

User Story

For whatever reason some of my "sandbox create" (using podman) take longer than 30 seconds and thus timeout.

Problem Statement

For whatever reason some of my "sandbox create" (using podman) take longer than 30 seconds and thus timeout.

Impact / Why This Matters

it's awkward trying to download images by hand; try running multiple times to get one to work; debug whether it's just a time issue or some other problem.

Proposed Design

add "--timeout=?" to sandbox create to override 30 second default.

Acceptance Criteria

observe that sandbox create could wait longer for podman to create the container

Alternatives Considered

none

Agent Investigation

No response

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Activity

  1. github-actions
    added
    state:triage-neededOpened without agent diagnostics and needs triage
    on Aug 24, 2026

  2. elezar commented on Aug 25, 2026

    @elezar
    elezar
    on Aug 25, 2026
    Member

    📋 triage-agent

    Classification: needs-information

    Current OpenShell source does not have a fixed 30-second sandbox create timeout: provisioning uses a 300-second idle timeout, reset by recognized provisioning progress events. It has an undocumented OPENSHELL_PROVISION_TIMEOUT process-level override, but no per-invocation sandbox create --timeout option.

    Please provide:

    1. Output of openshell version, gateway version, OS, Podman version, and the exact sandbox create command/image.
    2. The complete timeout/error output and approximate elapsed time.
    3. Whether the delay is an image pull, container creation, supervisor readiness, or an interval with no progress output.
    4. Whether a current release reproduces it.
    5. Whether the requested flag should control the current idle timeout, a total provisioning deadline, or both.

    That evidence will tell us whether this is a regression/driver failure or a request to expose the existing timeout control as a documented per-command option.

  3. elezar
    added
    state:needs-infoAssessment needs specific evidence or reproduction details
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Aug 25, 2026

  4. hpcpony commented on Aug 27, 2026

    @hpcpony
    hpcpony
    on Aug 27, 2026
    Author

    (I'm still working on 3 and 4...)

    1. Output of openshell version, gateway version, OS, Podman version, and the exact sandbox create command/image.

    openshell 0.0.109
    openshell-gateway 0.0.109
    Linux 5.14.0-687.38.1.el9_8.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Aug 9 21:18:40 EDT 2026
    podman version 5.8.2
    openshell sandbox create --name test --policy policy-test.yml

    1. The complete timeout/error output and approximate elapsed time.

    This seems to be the relevant part....
    .
    .
    .
    2026-08-26T18:40:57.170750Z INFO request{method=POST path="/openshell.v1.OpenShell/CreateSandbox" request_id="347e78e4-d344-4a2c-84d9-fe303a43be9e" otel.name=openshell.v1.OpenShell/CreateSandbox otel.kind="server" rpc.system="grpc" rpc.service="openshell.v1.OpenShell" rpc.method="CreateSandbox"}:driver{otel.name="driver.create_sandbox" otel.kind="client" driver.name=podman sandbox.id="7e0e3fea-c0a7-4820-a0e2-934bd016ba58"}: openshell_driver_podman::driver: Ensuring supervisor image image=ghcr.io/nvidia/openshell/supervisor:8d67250a5d17348eb96c4fa46226b06d8041f2ba policy="missing"

    2026-08-26T18:40:57.204810Z INFO request{method=POST path="/openshell.v1.OpenShell/CreateSandbox" request_id="347e78e4-d344-4a2c-84d9-fe303a43be9e" otel.name=openshell.v1.OpenShell/CreateSandbox otel.kind="server" rpc.system="grpc" rpc.service="openshell.v1.OpenShell" rpc.method="CreateSandbox"}:driver{otel.name="driver.create_sandbox" otel.kind="client" driver.name=podman sandbox.id="7e0e3fea-c0a7-4820-a0e2-934bd016ba58"}: openshell_driver_podman::driver: Ensuring sandbox image image=ghcr.io/nvidia/openshell-community/sandboxes/base:latest policy=missing

    2026-08-26T18:41:17.521806Z INFO refresh{otel.name="refresh.provider_credentials" watched_count=0 due_count=0}: openshell_server::provider_refresh: provider credential refresh worker sweep watched_count=0 due_count=0 rotation_requested_count=0

    2026-08-26T18:41:27.459614Z WARN request{method=POST path="/openshell.v1.OpenShell/CreateSandbox" request_id="347e78e4-d344-4a2c-84d9-fe303a43be9e" otel.name=openshell.v1.OpenShell/CreateSandbox otel.kind="server" rpc.system="grpc" rpc.service="openshell.v1.OpenShell" rpc.method="CreateSandbox"}:driver{otel.name="driver.create_sandbox" otel.kind="client" driver.name=podman sandbox.id="7e0e3fea-c0a7-4820-a0e2-934bd016ba58"}: openshell_driver_podman::driver: Failed to start container; cleaning up sandbox_name=test error=timeout after 30s

    2026-08-26T18:41:27.630360Z WARN openshell_driver_podman::watcher: Failed to inspect container after event container_id=d7cb05183296bc7fd902f8f01a128527698b53255430662ca89e9fdd1cc354cd error=timeout after 30s

    2026-08-26T18:41:48.457322Z INFO request{method=POST path="/openshell.v1.OpenShell/GetSandboxConfig" request_id="46822149-c589-4ebc-840a-345cc04fff61" otel.name=openshell.v1.OpenShell/GetSandboxConfig otel.kind="server" rpc.system="grpc" rpc.service="openshell.v1.OpenShell" rpc.method="GetSandboxConfig"}: openshell_server::grpc::policy: GetSandboxConfig served from spec (backfilled version 1) sandbox_id=7e0e3fea-c0a7-4820-a0e2-934bd016ba58
    .
    .
    .

    1. Whether the delay is an image pull, container creation, supervisor readiness, or an interval with no progress output.

    TBD

    1. Whether a current release reproduces it.

    TBD

    1. Whether the requested flag should control the current idle timeout, a total provisioning deadline, or both.

    I don't fully appreciate the issue so I have no preference.

  5. hpcpony commented on Aug 29, 2026

    @hpcpony
    hpcpony
    on Aug 29, 2026
    Author
    1. I am unable to tell what's executing during the 30 seconds. From what I can see from podman debug here's what it's doing:
      .
  .
  .
DEBU[2061] rootlessport: time="2026-08-29T12:35:57-04:00" level=info msg="Starting parent driver" 
DEBU[2061] rootlessport: time="2026-08-29T12:35:57-04:00" level=info msg="opaque=map[builtin.readypipepath:/scr/openshell/TMP/rootlessport2538444085/.bp-ready.pipe builtin.socketpath:/scr/openshell/TMP/rootlessport2538444085/.bp.sock]" 
DEBU[2061] rootlessport: time="2026-08-29T12:35:57-04:00" level=info msg="Starting child driver in child netns (\"/proc/self/exe\" [rootlessport-child])" 
DEBU[2061] rootlessport: time="2026-08-29T12:35:57-04:00" level=info msg="Waiting for initComplete" 
DEBU[2061] rootlessport: time="2026-08-29T12:35:57-04:00" level=info msg="initComplete is closed; parent and child established the communication channel"
time="2026-08-29T12:35:57-04:00" level=info msg="Exposing ports [{ 2222 35377 1 tcp}]" 
DEBU[2061] rootlessport: time="2026-08-29T12:35:57-04:00" level=info msg=Ready 
DEBU[2061] rootlessport is ready                        

   <- 30 second "hang"?
   <- I think buildah-copier is what's running during the 30 seconds and continues after the timeout
   <- resume as soon an openshell-gateway times out?

DEBU[2079] IdleTracker:new 3m+0h/176t connection(s)      X-Reference-Id=0x1f7cecb0e008
DEBU[2079] IdleTracker:active 3m+0h/177t connection(s)   X-Reference-Id=0x1f7cecb0e008
DEBU[2090] IdleTracker:new 4m+0h/177t connection(s)      X-Reference-Id=0x1f7cedc022b8
DEBU[2090] IdleTracker:active 4m+0h/178t connection(s)   X-Reference-Id=0x1f7cedc022b8
DEBU[2090] Starting parallel job on container fb641c1561288314e1b54f916d09be39a025433e0fc8638b2b063b29479c6ed9 
DEBU[2090] Removing container fb641c1561288314e1b54f916d09be39a025433e0fc8638b2b063b29479c6ed9 
   .
   .
   .

    I think podman continues on after the timeout and sometimes I even see a running container (often the status is "Stopping (starting)"):

    [OpenShell] podman ps -a
CONTAINER ID  IMAGE                                                     COMMAND               CREATED         STATUS      PORTS                    NAMES
fb641c156128  ghcr.io/nvidia/openshell-community/sandboxes/base:latest  --workdir /sandbo...  15 minutes ago  Created     0.0.0.0:35377->2222/tcp  openshell-default--test2-137ee510-61a5-4c29-98b9-6bcc98a6cdf0

    But openshell knows nothing about it:

    [OpenShell] openshell sandbox list
No sandboxes found.
    1. Yes (0.0.113)

    As for the other suggestion... I tried setting OPENSHELL_PROVISION_TIMEOUT to 120 seconds but it still times out at 30.

    2026-08-29T15:49:37.846863Z  WARN request{method=POST path="/openshell.v1.OpenShell/CreateSandbox" request_id="6f49ea8c-9817-42c1-b4a8-0d5cc666156c" otel.name=openshell.v1.OpenShell/CreateSandbox otel.kind="server" rpc.system="grpc" rpc.service="openshell.v1.OpenShell" rpc.method="CreateSandbox"}:driver{otel.name="driver.create_sandbox" otel.kind="client" driver.name=podman sandbox.id="5adcb50f-d1c8-42ac-a8eb-7095cb7467da"}:driver_rpc{otel.name="driver.create_sandbox" otel.kind="server" rpc.system="grpc" rpc.service="openshell.compute.v1.ComputeDriver" rpc.method="create_sandbox"}:podman.create_sandbox{otel.name="podman.create_sandbox" sandbox.id=5adcb50f-d1c8-42ac-a8eb-7095cb7467da sandbox.name=test2}: openshell_driver_podman::driver: Failed to start container; cleaning up sandbox_name=test2 error=timeout after 30s

2026-08-29T15:49:38.252944Z  WARN openshell_driver_podman::watcher: Failed to inspect container after event container_id=80788264253f2015405268ed07483109da533a56b2e0d1bd29913169fe720934 error=timeout after 30s

  6. github-actions commented on Sep 13, 2026

    @github-actions
    github-actionsbot
    on Sep 13, 2026 – with GitHub Actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  7. github-actions
    added
    state:staleInactive item at risk of automatic closure.
    on Sep 13, 2026

  8. github-actions commented on Oct 13, 2026

    @github-actions
    github-actionsbot
    on Oct 13, 2026 – with GitHub Actions

    Closing this issue after 14 days with no activity since it was marked stale. Reopen it if the work is still relevant.

  9. github-actions
    closed this as not plannedon Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:needs-infoAssessment needs specific evidence or reproduction detailsstate:staleInactive item at risk of automatic closure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions