Skip to content

fix(server): consistent sandbox access after the main process exits - #4950

Open
pimlock wants to merge 4 commits into
mainfrom
4949-sandbox-access-after-exit/pimlock
Open

pimlock wants to merge 4 commits into
mainfrom
4949-sandbox-access-after-exit/pimlock

Conversation

@pimlock

@pimlock pimlock commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

When a retained sandbox's main process exits, the supervisor stays connected so you can still get into the sandbox. SSH, file transfer and port forwarding already worked in that state. Exec and exposed services did not, so openshell sandbox exec -- cat result.txt failed on a sandbox you could download the same file from. On a gateway with several replicas, the post-exit check also only looked at the replica that received the request, so access worked or failed depending on the load balancer.

This PR applies one rule to every way into a sandbox. A Ready sandbox is reachable. A Completed or Error sandbox is reachable while its supervisor is connected to any replica. Every other phase is refused. The gateway makes the call, and the CLI no longer rejects Completed or Error on its own. #4949 has the full set of rules and a lifecycle diagram. New ways into a sandbox should use the same check.

Related Issue

Closes #4949

Changes

  • Exec, SSH sessions, port forwarding and exposed services share one replica-aware reachability check in the gateway.
  • The CLI sends exec and service forwards for Completed and Error sandboxes to the gateway.
  • The docs and CLI skill now say what you can still do after a retained sandbox's main process exits, and that --no-keep sandboxes are deleted.
  • A new sandbox-retention conformance scenario checks both cases, for exit codes 0 and 3. A --no-keep sandbox is deleted. A retained sandbox records the exit code and still accepts exec.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

The conformance scenarios ran against a local Docker gateway built from this branch. I didn't test Kubernetes, Podman or a real multi-replica gateway live.

Still broken and tracked in #4949: after a Docker gateway restart, Completed and Error sandboxes lose their supervisor and can't be reached.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

After the main process exits, the supervisor keeps its access plane, and
SSH and port forwarding already reached Completed and Error sandboxes
while it stayed connected. Exec still required Ready in both the CLI
and the gateway, so it was refused on the same sandboxes.

Exec, SSH sessions, and TCP forwarding now share one gate: a sandbox is
reachable when it is Ready, or Completed/Error with a supervisor session
on this replica or a fresh owner record naming a peer replica. The
previous check only looked at the local session map, so on a
multi-replica gateway a request landing on a non-owner replica was
refused even though relay opening routes to the owner.

The CLI exec pre-check and the service forward recheck accept Ready,
Completed, and Error and leave the supervisor decision to the gateway.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…upervisor

Exposed-service HTTP routing still required Ready while port forwarding
to the same sandbox kept working after the main process exited. Service
routing now uses the shared relay reachability gate, so a retained
Completed or Error sandbox serves its exposed services while its
supervisor stays connected.

Document what stays reachable after a retained sandbox's main process
exits, and that --no-keep sandboxes are deleted once it does.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Add a sandbox-retention scenario that runs for exit codes 0 and 3:
an ephemeral (--no-keep) sandbox returns the exit code and is deleted,
and a retained sandbox records Completed or Error with the exit code
and still accepts exec.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@github-actions

Copy link
Copy Markdown

Add a phase diagram to the Sandbox Lifecycle section and describe what
happens after the main process exits. A --no-keep sandbox is deleted. A
retained sandbox keeps its compute and still accepts exec, file
transfer, port forwarding and exposed services while its supervisor is
connected. sandbox connect fails because it attaches to the exited main
process, so the docs point to sandbox exec --tty for a shell.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: consistent sandbox access after the main process exits

1 participant