refactor(compute): define a uniform gateway callback endpoint contract #2215

Summary

Define a uniform semantic contract for the gateway callback endpoint used by sandbox supervisors, and separate it from gateway listener ownership, driver-specific route materialization, and credential delivery.

This issue intentionally does not select a protobuf shape or delivery mechanism. Its purpose is to reach maintainer agreement on the boundary that any implementation must satisfy across built-in and external compute drivers.

Problem

Every supervisor-backed compute driver must arrange for its sandbox supervisor to connect back to the gateway through OPENSHELL_ENDPOINT. The compute-driver protocol does not currently define where this callback URI comes from, what it means, or which component owns making it reachable.

The gap is filled differently for each built-in driver:

This makes callback reachability a built-in construction concern rather than part of the common driver contract. It also encourages Docker-like listener exceptions to be copied into other in-tree drivers, as #2111 demonstrates.

Docker and Podman are the immediate compatibility baseline. A design is not sufficient if it only enables external or future drivers while leaving either local container driver on a separate callback-endpoint or listener path. Their route materialization may remain runtime-specific, but the endpoint semantics and gateway ownership boundary must be shared.

Terminology

• Compute-driver control endpoint: the Unix domain socket or transport implementing ComputeDriver; the gateway initiates these calls.
• Gateway callback endpoint: the HTTP(S) URI used by the sandbox supervisor; the sandbox initiates these calls.
• Advertised callback endpoint: the URI supplied to a driver.
• Effective callback endpoint: the workload-visible URI ultimately injected as OPENSHELL_ENDPOINT after any permitted transformation.
• Gateway listener: a local address bound by the gateway process.
• Route materialization: driver-specific DNS, host alias, NAT, forwarding, Service, bridge, or proxy work that connects the effective endpoint to a listener.
• Credential delivery: sandbox JWT and TLS material; separate from endpoint selection.
• Listener authorization: the gateway/operator decision permitting a process to bind a network address.

callback endpoint is preferable to grpc_endpoint: the same gateway endpoint carries policy/configuration, inference bundles, logs, relay traffic, token operations, and supervisor control streams. It is not the compute-driver control endpoint.

Technical findings

Shared protocol gap

• proto/compute_driver.proto:18-56 defines the service and an identity-only GetCapabilities response.
• proto/compute_driver.proto:79-97 carries the per-sandbox JWT but no callback endpoint.
• ValidateSandboxCreateRequest and CreateSandboxRequest only carry DriverSandbox (proto/compute_driver.proto:210-239).
• External-driver configuration accepts only socket_path (crates/openshell-server/src/compute/driver_config.rs:98-113).
• RFC 0001 expects driver APIs to use version and capability negotiation (rfc/0001-core-architecture/README.md:179-183), but the current handshake cannot negotiate this feature.

Gateway listener coupling

• Driver construction occurs before listeners bind (crates/openshell-server/src/lib.rs:249-265).
• ComputeRuntime stores extra bind addresses outside the shared driver API (crates/openshell-server/src/compute/mod.rs:260-324).
• Docker's concrete method is called before trait erasure (crates/openshell-server/src/compute/mod.rs:339-369).
• The primary and driver-derived listeners all serve the same MultiplexService with the same TLS configuration (crates/openshell-server/src/lib.rs:384-394,525-692).
• Sandbox JWT principals are method-allowlisted within that shared service (crates/openshell-server/src/multiplex.rs:596-615); this is distinct from client-certificate authentication.

#2111's proposed Podman path also performs a bind-and-drop preflight before the gateway later binds the address for real. That is a time-of-check/time-of-use window and leaves authoritative bind failure ownership unclear.

Driver-specific behavior

• Docker route derivation and listener reporting: crates/openshell-driver-docker/src/lib.rs:353-447; explicit URI authority rewrite: :2422-2432.
• Podman endpoint derivation: crates/openshell-driver-podman/src/driver.rs:265-287; protected environment injection: container.rs:347-400.
• Kubernetes Service URI derivation: deploy/helm/openshell/templates/_helpers.tpl:138-144; driver config rendering: gateway-config.yaml:112-120.
• VM/libkrun and VM/QEMU rewrites: crates/openshell-driver-vm/src/driver.rs:871-877,4031-4070.
• The VM firewall extracts only Url::port(), so a URI without an explicit port can become port 0 in the TAP rule.
• The supervisor derives plaintext versus mTLS behavior from the URI scheme (crates/openshell-core/src/grpc_client.rs:125-175); the URI host is also the effective TLS/SNI identity.

architecture/compute-runtimes.md:33 is stale: it says Docker uses host networking, while the implementation uses a managed bridge and, on native Linux, an extra listener.

Minimum contract to define

The chosen design should settle these semantics independently of its protobuf or configuration shape:

1. The callback endpoint is an absolute http:// or https:// URI.
2. Port semantics are unambiguous; decide whether an explicit nonzero port is mandatory.
3. Legal URI components are defined, including base path, userinfo, query, and fragment behavior.
4. The endpoint contains no credentials.
5. Its effective authority is stable for the sandbox lifetime unless an update mechanism is defined.
6. HTTPS authority, SNI, CA trust, and server certificate SANs agree.
7. Sandbox JWT identity remains distinct from transport mTLS.
8. User/template environment cannot override the driver-injected endpoint.
9. Listener authorization is not implied by route materialization or access to the driver control socket.
10. Explicit configuration provenance is retained; a driver does not silently reinterpret an operator URI without defined semantics.
11. Failures distinguish invalid URI, unsupported topology, unauthorized listener request, bind failure, and post-start reachability failure.
12. The same semantic contract works for built-in and external drivers.
13. Drivers that do not launch a supervisor can represent that capability explicitly.
14. The endpoint's global, per-driver, or per-sandbox scope is defined before multi-driver routing lands.
15. Startup ordering is defined, including whether dynamic or port-zero listeners can be advertised.
16. Existing sandboxes can reconnect across gateway or driver restart.
17. Docker and Podman both implement the contract without retaining separate in-process listener semantics.

Responsibility boundaries

The design should keep these responsibilities separate:

• The gateway or deployment selects and serves an advertised endpoint, issues sandbox identity, and authorizes listeners.
• The compute driver validates and materializes a runtime-specific route, injects the effective endpoint, and provisions runtime-appropriate credential references or material.
• The supervisor parses the URI, applies scheme/TLS semantics, authenticates with sandbox-scoped credentials, and reconnects without changing the endpoint's meaning.

Neutral alternatives

No alternative is preferred by this issue.

1. Deployment/operator-owned endpoint and listeners: drivers receive equivalent static configuration and only inject/materialize it.
2. Endpoint in each create request: add a typed descriptor to DriverSandboxSpec or CreateSandboxRequest.
3. Driver initialization RPC: provide deployment callback information once per driver connection.
4. Gateway-advertised endpoint candidates: provide an ordered set from which the driver chooses a materializable route.
5. Authorized driver-reported listener requirements: formalize the Docker pattern through a policy-constrained common contract.
6. Driver-owned forwarder or NAT: keep a fixed gateway listener and let the runtime map a sandbox-visible address to it.
7. Dedicated callback-only listener/API: expose only supervisor-callable methods on a separate surface.
8. Standard logical hostname: require runtimes to materialize a conventional name such as host.openshell.internal.

These alternatives have different HA, security, startup, compatibility, and deployment consequences and may not be mutually exclusive. Listener negotiation, if retained, should be evaluated as a separately authorized capability rather than an implicit consequence of endpoint advertisement.

Affected components

Scope assessment

• Issue type: refactor
• Complexity: High
• Confidence: Medium
• Estimated implementation surface: 15–25 files, depending on the selected protocol and migration model

The missing boundary is clear. Complexity remains high because a uniform implementation crosses gateway startup/listening, protocol compatibility, all four current drivers, supervisor TLS semantics, packaging, Helm, and end-to-end coverage.

Risks and open decisions

Security

All current listeners expose the same multiplexed gateway service. Application authorization limits sandbox JWT methods, but an extra listener still expands the TLS/parser/router and user API attack surface. A remote driver's control-socket access must not automatically authorize raw host binds.

The URI itself has no direct SELinux/AppArmor impact. Particular implementations can introduce labeling, socket ownership, capabilities, firewall/NAT, user-namespace, and TLS-secret concerns; #1909 remains relevant to containerized Podman.

High availability

A replica-local endpoint or listener is unsuitable for reconnect after replica failure. A stable Service or load balancer helps, but session affinity/shared state and the readiness work in #1951 remain related design constraints.

Configuration and compatibility

bind_address cannot be mechanically converted to a sandbox-visible URI: wildcard and loopback addresses, NAT, Service DNS, TLS identity, and container namespaces differ. Migration must cover all four driver grpc_endpoint fields, Helm server.grpcEndpoint, Docker's concrete listener hook, and older external drivers.

Human decisions required

1. Is the authoritative endpoint global, per driver, or per sandbox?
2. Is it one endpoint or ordered candidates?
3. May a driver rewrite an explicit authority, or only make it routable?
4. Does listener negotiation belong in this contract, a separately authorized contract, or nowhere?
5. Does the callback use the full gateway surface or a callback-only surface?
6. Is delivery out of band, initialization-time, or per-create?
7. What capability/version negotiation is required for existing external drivers?
8. Must the URI include an explicit nonzero port, and which other URI components are valid?
9. How are dynamic listener ports and driver-before-listener startup ordering handled?
10. What stable endpoint is required for HA and supervisor reconnect?
11. What happens to running sandboxes if the endpoint changes?
12. Is VM/QEMU's TAP-host listener requirement supported behavior or a separate bug?
13. What migration/deprecation path applies to grpc_endpoint and Docker's hook?

Test impact

A selected implementation should cover:

• URI validation, TLS identity, default/explicit ports, and endpoint provenance.
• Protocol compatibility and capability negotiation for older external drivers.
• A fake external driver receiving the same callback contract as a built-in.
• Listener authorization, de-duplication, wildcard coverage, and atomic bind failure.
• Per-driver environment injection and route transformation.
• HTTP, HTTPS, mTLS, and sandbox JWT authorization as distinct concerns.
• Restart/HA behavior for existing supervisors.
• E2E on native Linux Docker bridge, Docker Desktop, containerized Docker, rootless Podman pasta/bridge, Podman machine, containerized Podman on SELinux-enforcing Fedora, Kubernetes Service/ingress/HA, VM libkrun/gvproxy, VM QEMU/TAP/GPU, and a fake external driver.
• Future Apple/container and Windows drivers when they exist; no Apple/container compute driver exists on current main.

Documentation impact

Likely updates include architecture/compute-runtimes.md, architecture/gateway.md, driver networking documentation, docs/reference/gateway-config.mdx, sandbox compute-driver documentation, Helm values/templates/tests, packaging configuration, and external-driver protocol documentation.

Related work

• bug: containerized Podman gateway broken on Fedora 44 (SELinux, UID mapping, port binding, state path) #1909 — containerized Podman networking, SELinux, and port binding failures
• refactor: make sandbox readiness gateway-owned across compute drivers #1951 — gateway-owned readiness and supervisor-session semantics
• refactor: align driver ownership of gateway callback listeners #1952 / refactor(core): remove unused extra bind addresses #2059 — prior gateway callback listener ownership work
• refactor(compute): move driver-specifics into a dedicated crate #1999 — moving driver-specific behavior out of the gateway crate
• Spike: should the built-in compute drivers move to the external (out-of-process) model? #2061 — externalizing built-in drivers and removing in-process hooks
• fix(podman): diagnostics for containerized rootless socket + subuid preflight #2111 — proposed Podman callback hardening that exposes the missing common boundary
• fix(podman): deliver sandbox JWTs as secrets #2156 — sandbox JWT delivery, relevant to keeping credentials separate from endpoint selection

Expected outcome

Maintainers agree on the terminology, responsibility boundaries, minimum semantics, and required decisions above. A follow-on implementation issue can then select a protocol shape and migration sequence without adding another in-tree special case.