docs(rfc): add RFC 0018 for multi-sandbox and HA supervisors

[FrostGod]

Summary

Adds RFC 0018, which proposes four ways to deploy supervisors: per workspace, per node, per sandbox, and per gateway. It sets out one shared set of safety rules for replacing a failed supervisor (fencing), and a plan to compare the four models and choose one.

Related Issue

Refs #4311. This PR only adds the RFC and changes no code. It also builds on #3661 and #2385.

Changes

• Adds rfc/0018-multi-sandbox-ha-supervisors/README.md with goals, the four proposed models, and the shared rules (fenced recovery, ownership epoch, leases).
• Describes how the models behave on failure and how to compare them, and lists the risks, alternatives, and open questions.

Testing

Docs-only change, so there are no unit or E2E tests. The RFC links, the Mermaid diagrams, and the headings are checked by eye. The full mise run pre-commit has not completed on this branch: the first run failed on a local uv environment error unrelated to the RFC, and the rerun did not start.

• Checks appropriate to the affected code and behavior pass
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mayawang]

Thanks for opening this. Some input from the Agent Substrate side, mostly on one case the RFC doesn't cover yet: a backend suspending a sandbox on purpose.

Suspended sandboxes

On Agent Substrate an idle sandbox is suspended and uses no compute, and it can resume later, possibly on a different node. The RFC's resume path is about replacing a failed supervisor. For a backend-initiated suspend, we'd hope for the following:

• While a sandbox is suspended, it holds no supervisor session. Otherwise sleeping sandboxes still take supervisor capacity, and reconnect attempts wake them back up (feat(supervisor): hold reconnect while a sandbox is suspended by its backend #4352).
• On resume, the sandbox can be assigned to any healthy replica, with fresh authority. The fenced recovery flow in section 3 looks close: suspend could be a planned release, and resume a new assignment.
• The runtime's reconnect window doesn't count time spent suspended. In our prototype it only started counting after resume, which worked well.
• Binary-identity pins and policy revisions survive a suspend, in line with section 4. On a backend that snapshots memory, they come back with the sandbox.

Placement

Our lean is a pool that isn't tied to nodes: one HA deployment per install, or per gateway group, scaled horizontally, with sessions assignable to any replica. Per node fits a suspend-capable backend poorly, because sandboxes move between nodes on resume and many nodes would mostly hold sleeping sandboxes. Per workspace tends to split capacity.

Capacity

If suspended sandboxes release their sessions, the number to size supervisors for on a backend like ours is active sandboxes, not total sandboxes.

Happy to walk through what we saw in our prototype, and to review as the comparison of the four models takes shape.

[FrostGod]

#4377 (comment)

will take this into consideration and update the RFC, thanks @mayawang

[EmilienM]

The direction makes sense to me. The main gaps are things that exist today and would stop or still authorize the old supervisor (driver reconciler, sandbox bearer, launch Secrets, namespace fence), plus how the lease gets renewed. Details inline.

[FrostGod]

Thanks @mayawang. Suspend isn't supported in OpenShell yet, so I've listed backend-initiated suspend (#4352) as a separate non-goal, but the design leaves scope for it as a planned release plus a fresh assignment on wake. I also noted that per-node fits poorly with backends that move sandboxes, and that capacity should be sized by active sandboxes.

[EmilienM]

Almost there, thanks for the quick turnaround. 9 of the 13 threads are fixed in the text, so go ahead and resolve those.

Two small things before I approve:

• L147, see inline.
• The four items you're leaving for the prototype (per-node binding, per-replica vs per-sandbox lease, launch material delivery, lease renewal path) aren't in the RFC yet. Could you list them under Open questions so they don't get lost?

Also I think this would be a feature behind a k8s feature gate, maybe worth documenting.

[EmilienM]

Today a PostgreSQL outage doesn't stop running sandboxes (RFC 0012 treats gateway loss as a reconnect, not supervisor loss). Can you scope the lease to resume-enabled runs here, like we discussed in the lease thread? Otherwise a long DB outage stops every opted-in sandbox.

[FrostGod]

Done: the lease applies only to runs with resume enabled and is checked only for takeovers. The runtime's freeze-on-disconnect is the fence, so a PostgreSQL outage never stops running sandboxes; only takeovers wait.

[grs]

The JWT-SVIDs should be short lived. Might it make sense for the supervisor to pull them rather than for the sandbox to push them? I.e. when the supervisor needs a new JWT-SVID, it makes a call to the sandbox which then retrieves the new token from the SPIFFE workload API and returns it?

[dhirajsb]

Agreed. FrostGod/OpenShell#2 proposes on-demand retrieval of short-lived, audience-specific JWT-SVIDs, with caches bounded by expiry and sandbox/run/ownership. Authorized middleware can use the same interface to inject an SVID into a configured X-Workload-SVID header for an admitted destination, replacing any agent-supplied value.

[EmilienM]

Another iteration needed, mostly for the new section 5 (cc @dhirajsb). Details inline. L147 and the open questions from my last review still stand.

[EmilienM]

With config push (#4320, the only mode after #4498), policy and provider credentials flow down the already-open session, so rotating the token (L105) no longer cuts the old owner off from them. Worth a line here: the epoch CAS closes any older-epoch session, and this step becomes a pushed bootstrap that the candidate acks on a session at the reserved epoch (config only, no relays) before activation.

[FrostGod]

Added a paragraph covering both config modes. In poll mode the new token ID cuts off the old owner. In push mode the epoch CAS also closes every older-epoch session. In both, the new supervisor acknowledges a config-only setup push at the reserved epoch before activation.

[EmilienM]

This puts the Workload API socket in the Pod where the agent runs, with the same UID as the runtime, so attestation can't keep the agent from minting JWT-SVIDs for any audience. The K8s driver strips spiffe-workload-api from the workload Pod today on purpose. Could the shared supervisor fetch them instead (SPIRE's Delegated Identity API)? I'd also scope this section to shared models, since per-sandbox already works. Small thing: nothing in the Sandbox Pod is a gRPC client today, the supervisor dials the runtime.

[grs]

SPIRE's Delegated Identity API works at the Agent scope i.e. at node scope on kubernetes. So you can't - as far as I know - allow a given supervisor to only retrieve SVIDs for the sandbox pods it is serving.

Though not ideal, the agent requesting SVIDs for any given audience is not in itself necessarily a problem. The SVID merely proves the identity of the workload that retrieved it. It is for accessed services to verify the SVIDs and determine if that workload is indeed authorized. The network restrictions also only allow the agent to communicate out through the supervisor, so it is only approved endpoints that can be reached.

Another problem with having the Workload API mounted on the sandbox pod though is when kata containers are used as the SPIRE socket isn't accessible from inside the VM.

The SPIFFE Broker API could be an option but it is still 'experimental'.

[grs]

Could the current broker in the supervisor be extended to prevent the agent from making connect) calls to domain sockets (while allowing itself to do so)? That could restrict the agent from accessing the Workload API, but would also prevent use of any other domain sockets. Is that too restrictive? Are there legitimate cases where an agent should be allowed to access a socket?

(Still does not solve the kata problem though)

[FrostGod]

Blocking all Unix sockets would break things agents rely on, such as language servers, tmux and ssh-agent. Blocking only the SPIFFE socket isn't safe either, because the agent can change the socket path after we check it. Simplest is to keep the socket out of the agent's view, as we do today, and let the runtime or supervisor fetch SVIDs for it. We'll pick the approach during implementation.

let me get to the kata part

[dhirajsb]

Agreed: keep the Workload API socket outside the agent's view and leave the acquisition backend to implementation. FrostGod/OpenShell#2 scopes section 5 to shared supervisors, preserves the supervisor-initiated runtime channel, and flags node-wide delegation for separate authorization review.

Kata's host-to-guest UDS limitation is independent of supervisor topology. For Teleport, a bot sidecar inside the Kata VM can expose a guest-local socket, provided issuance is sandbox-scoped and the socket, credentials, and configuration are isolated from the agent. This remains a deployment option to validate.

[EmilienM]

The gateway needs this check too. validate_supervisor_jwt_svid checks issuer, audience and trust domain, but not that sub is this sandbox. That's OK while each supervisor can only fetch its own SVID. Once one process holds many, a mix-up can hand one sandbox's credentials to another, so let's have the gateway match sub against the authenticated sandbox.

[dhirajsb]

Agreed. FrostGod/OpenShell#2 requires the gateway to independently validate the SVID and match sub to the authenticated sandbox using trusted control-plane state, with use bound to the current run and ownership epoch.

[FrostGod]

will be addressing the comments today, was not online yesterday