Kubernetes Operator

Problem Statement

OpenShell's Kubernetes deployment path currently uses the gateway as the user-facing control plane and depends on the Kubernetes Agent Sandbox controller for runtime sandbox pods. As Kubernetes users adopt OpenShell, we need to decide whether an OpenShell Kubernetes Operator should exist, what responsibilities it should own, and how it should relate to the existing gateway and Agent Sandbox CRD.

This matters because platform teams may expect Kubernetes-native installation, reconciliation, status, and declarative sandbox workflows. At the same time, the gateway already owns OpenShell API behavior, credentials, inference configuration, policy attachment, sandbox lifecycle APIs, logs, watch streams, and client integrations through the CLI, SDKs, and TUI. An operator could complement that model, overlap with it, or replace parts of it in Kubernetes environments.

We should collect user, platform team, and contributor feedback before committing to a specific operator shape.

Proposed Design

Explore an OpenShell Kubernetes Operator with two possible directions:

OpenShell Deployment Operator

The operator could install and manage an OpenShell gateway inside a Kubernetes cluster. It may handle:

Gateway installation and upgrades.
Gateway configuration.
Kubernetes, Docker, or VM-backed compute driver configuration.
TLS, service exposure, ingress, and authentication wiring.
Integration with existing secret management workflows.
Gateway health and readiness through Kubernetes status conditions.

OpenShell Sandbox Custom Resource Operator

The operator could introduce an OpenShell-owned custom resource for declarative sandbox management. This resource may let users or platform teams define:

Sandbox runtime requirements.
Images, commands, environment variables, resources, and volumes.
OpenShell sandbox policies.
Lifecycle behavior such as TTLs, restarts, cleanup, and termination.
Status exposed through .status.
Integration with OpenShell audit logging, policy enforcement, provider configuration, and inference routing.
Reusable sandbox classes or templates for platform-managed environments.

Existing Agent Sandbox CRD Boundary

OpenShell already depends on the Kubernetes Agent Sandbox CRD in Kubernetes deployments today. That resource is also named Sandbox, but it lives in the agents.x-k8s.io API group and is reconciled by the Agent Sandbox controller into Kubernetes pods.

The existing Agent Sandbox CRD is a runtime-level Kubernetes primitive used by the OpenShell Kubernetes compute driver. A proposed OpenShell sandbox custom resource may be a different API surface: a user- or platform-facing OpenShell object that includes OpenShell policy references, inference configuration, provider wiring, lifecycle semantics, audit expectations, and gateway integration.

The design should disambiguate these two concepts before proposing a CRD shape:

Is the OpenShell operator expected to expose the existing Agent Sandbox CRD directly?
Should OpenShell introduce a separate CRD with a distinct name or API group, such as an OpenShell-owned sandbox request/resource?
If both exist, does the OpenShell CRD own and generate Agent Sandbox resources, or does the gateway continue to do that?
Which object is the source of truth for lifecycle, status, policy attachment, and deletion?
How should naming avoid confusing agents.x-k8s.io/Sandbox with any OpenShell-specific sandbox resource?

Operator And Gateway Boundary

The operator and current OpenShell gateway likely have overlapping responsibilities, especially around sandbox lifecycle, configuration, status, policy attachment, and infrastructure integration. It may be valid for some deployments to use an operator or a gateway, but not both.

The design should clarify when users interact with the Kubernetes Operator directly versus when they use the existing gateway API, CLI, SDKs, or TUI. In particular, we should decide whether the operator is primarily:

A Kubernetes-native deployment and reconciliation layer for the gateway.
A Kubernetes-native frontend for sandbox lifecycle management.
A replacement for some gateway responsibilities inside Kubernetes environments.
A complementary controller that delegates most runtime behavior to the gateway.

Questions For Feedback

Would you use an operator mainly to deploy OpenShell, to manage sandboxes, or both?
What Kubernetes-native workflows should this support?
What should the first version of the operator include?
Should proposed OpenShell sandbox resources be user-facing, platform-team-facing, or both?
How should a proposed OpenShell sandbox resource relate to the existing Agent Sandbox CRD?
What fields would you expect on an OpenShell sandbox custom resource?
How should policies be referenced or embedded?
How should credentials, provider configuration, and inference routing be handled?
What status, events, and observability would be required?
Are there existing operator patterns or tools we should align with?

Non-Goals For Now

Committing to a specific CRD shape.
Renaming, replacing, or exposing the existing Agent Sandbox CRD without a clear design.
Deciding whether the operator replaces Helm-based deployment.
Implementing the operator before collecting operator requirements.
Supporting every OpenShell runtime mode in the first version.

Alternatives Considered

Keep Helm plus gateway as the only Kubernetes user-facing workflow. This avoids new API surface, but may not satisfy platform teams that expect operator-managed lifecycle, status, upgrades, and reconciliation.
Expose the existing agents.x-k8s.io/Sandbox CRD directly as the user-facing sandbox API. This avoids a second sandbox resource, but may leak a runtime-level primitive that does not model OpenShell policy, credentials, inference routing, audit behavior, or gateway integrations.
Introduce a separate OpenShell-owned sandbox CRD. This can provide a clearer OpenShell API, but creates another resource to reconcile and requires a firm source-of-truth model with the gateway and Agent Sandbox controller.
Build only a deployment operator first. This keeps v1 focused on cluster installation and upgrades, but leaves declarative sandbox lifecycle management to the gateway API, CLI, SDKs, and TUI.

Agent Investigation

The local repo already documents and uses the existing Agent Sandbox CRD:

deploy/kube/manifests/agent-sandbox.yaml defines sandboxes.agents.x-k8s.io with kind: Sandbox.
deploy/helm/openshell/README.md says the Kubernetes Agent Sandbox CRDs and controller must be installed before deploying OpenShell.
architecture/gateway.md describes the gateway as the OpenShell control plane and notes that Kubernetes sandbox authentication verifies the pod's controlling Sandbox ownerReference against the live Sandbox CR UID.
crates/openshell-driver-kubernetes/ contains the Kubernetes compute driver that creates and watches Agent Sandbox resources.

These findings make the operator design a boundary and API-shape question, not just a request to add a new controller.

Proposed Outcome

Use this issue to gather requirements and decide whether to create one or more follow-up design issues for:

An OpenShell deployment operator.
An OpenShell sandbox CRD and sandbox lifecycle controller.
Shared operator infrastructure, status reporting, policy integration, and docs.

We operate an OpenShell Kubernetes Operator. Sharing feedback based on that experience.

Personas

Before answering the questions — a framing that shaped most of our design decisions. The issue treats "platform teams" and "users" as two groups. In practice we found three distinct personas with different trust levels:

Platform Admin (cluster operator): "Run OpenShell reliably, securely, at scale."

Deploys the operator and its dependencies
Manages CRDs, webhooks, cluster-level defaults (supervisor delivery method, image pull policies)
Scopes Tenant Admins via namespace-level RBAC — but does not decide which providers or policies exist within a tenant's domain

Tenant Admin (DevOps / team lead): "Configure the sandbox environment for my team."

Authors security policies (filesystem, network, process restrictions)
Manages the credential plane (which API keys are available, how they're scoped)
Sets infrastructure parameters (images, RuntimeClass, resource limits)

Sandbox Consumer (end user / developer / service): "Give me an isolated environment."

Does not care about infrastructure, security policy, or credentials
Two sub-cases worth distinguishing:
- A service requesting a sandbox (knows OpenShell, knows how to create a CR — e.g. a chatbot backend spinning up an agent sandbox for each session)
- An end user of that service (doesn't know OpenShell exists — e.g. someone chatting with the bot)
In K8s API terms, the "Sandbox Consumer" is always the service or human creating the CR — the end user behind it is invisible to the operator

These personas should be answered for differently. A "user-facing Sandbox resource" means different things to the Sandbox Consumer (who just wants kubectl apply -f my-sandbox.yaml with a policy reference) versus the Tenant Admin (who defines what that policy allows).

Questions

Would you use an operator mainly to deploy OpenShell, to manage sandboxes, or both?

Both. A deployment-only operator becomes insufficient quickly — platform teams appreciate a declarative sandbox lifecycle from CI pipelines, GitOps tools, and custom controllers. Shipping both in a single operator with a clear boundary feels like the better approach.

What Kubernetes-native workflows should this support?

The primary value is providing a Kubernetes-native control plane for OpenShell. With CRDs for sandboxes, policies, and providers, native K8s RBAC applies directly:

Tenant Admin controls which policies and providers are available, who can create sandboxes within their domain
Platform Admin scopes Tenant Admins via namespace-level RBAC
No separate authorization layer required

Workflows that follow naturally:

Declarative sandbox lifecycle from GitOps tools, CI pipelines, or any K8s API client — no orchestration layer needed beyond kubectl apply
Policy-as-code (providers and policies are a natural fit for declarative spec — they change infrequently and benefit from version control)

The operator should also make minimal assumptions about the underlying infrastructure — particularly around networking and security. Enterprise K8s clusters vary widely in how traffic enters and exits (Ingress, Gateway API, NodePort, LoadBalancer), whether a service mesh is present (Istio, Linkerd, mTLS between services), how network policies are enforced, and what cluster-level security standards apply (PodSecurityStandards, AppArmor/seccomp defaults). Sandboxes — or at least the proxy component — need to talk to other cluster services and external APIs, and must play well with existing network and security infrastructure.

Today this is complicated by the secondary network namespace architecture — service mesh sidecars expect a single network namespace per pod, which conflicts with the current model. In that context, #1305 (removing the secondary network namespace and splitting the proxy from the supervisor/agent) is important — it's what makes coexistence with meshes and standard NetworkPolicy infrastructure possible.

In a Kubernetes deployment, the Sandbox Consumer interacts through kubectl and the K8s API. The TUI and direct gRPC should not be required.

What should the first version include?

At minimum:

Installation and upgrades of the gateway and the OpenShell control plane
NetworkPolicy generation for gateway and sandbox pods
Agent-sandbox-controller installation (could be externalized — it has its own release lifecycle — but debatable)

However — deploying the gateway in K8s without additional native capabilities is limiting. A Kubernetes-native frontend for sandbox lifecycle management is necessary for the operator to deliver value beyond what Helm already provides.

This connects to the boundary question posed in the issue. In my view the operator is:

A replacement for some gateway responsibilities inside Kubernetes environments.

Specifically, it replaces the gateway's control-plane half. The gateway today conflates:

Control-plane: providers, policies, sandbox lifecycle CRUD, persistence
Data-plane: supervisor sessions, relay byte-streaming, exec/forward/SSH, service routing

These are fundamentally different concerns — and Kubernetes already has a native separation between them. The operator absorbs the control plane:

CRDs become the persistence layer (etcd replaces SQLite/Postgres)
RBAC replaces custom authorization
Reconciliation loops replace the gateway's CRUD endpoints

The gateway shrinks to its data-plane role — relay, streaming, supervisor sessions — and becomes a slimmer runtime service that the operator deploys and manages.

This mirrors how Kubernetes itself separates concerns: API server + controllers own declarative state; kubelets and kube-proxy own runtime execution. An OpenShell operator should follow the same pattern.

The Sandbox CRD shape depends on how the agents.x-k8s.io relationship resolves (see below), so it could be phased slightly later. Policies and Providers are natural first-class CRDs from the start — the Tenant Admin persona needs them immediately.

Should proposed sandbox resources be user-facing, platform-team-facing, or both?

Both, layered by persona:

Sandbox resource — Sandbox-Consumer-facing. RBAC controls who can create them.
Policy resource — Tenant-Admin-facing. Defines the security boundary sandboxes must reference.
Provider resource — Tenant-Admin-facing. Makes credentials available within a trust domain.
Gateway resource — Platform-Admin-facing. Defines the deployment.

The Sandbox Consumer picks from pre-approved policies and providers by name. The Tenant Admin authors those. The Platform Admin doesn't touch either.

This is the platform-centric multi-tenant view. In single-player or developer-centric deployments (one person wearing all hats), the rigid separation is optional — but the system should still support:

Tenant Admin policies as enforceable baselines (even in single-player, a default policy should apply)
Self-service within those bounds (the developer can configure their own sandbox without a separate admin approving each request)

Key point: the gateway should not be the sole entrypoint for sandbox creation. In enterprise deployments where frontend services are separate systems, those services should create sandboxes directly via the K8s API — as simple as creating a CR.

How should a proposed OpenShell sandbox resource relate to the existing Agent Sandbox CRD?

This is the most difficult design question and relates directly to #1680.

The tension:

ASB resources have capabilities (warm pools, scale-to-zero) that OpenShell will increasingly want to leverage
OpenShell exposes concepts (policies, inference routing, provider attachment, audit) with no representation in the ASB CRD
Every new ASB feature will require integration work; every OpenShell-specific field has no upstream home

The exact layering needs careful design. What is clear: the user-facing API should be OpenShell's, not the raw ASB primitive.

What fields would you expect on an OpenShell sandbox custom resource?

Spec:

gatewayRef — which gateway manages this sandbox
policyRef — reference to a reusable policy resource (see policy question below)
image — sandbox container image
entrypoint — override command (immutable after creation)
environment — environment variables for the supervisor
providers — list of Provider names to attach
gpu — GPU resource request

Pod-level fields (runtimeClassName, imagePullSecrets, resource requests, published ports, volumes) depend on the driver_config design in #1589 landing first — those should follow whatever shape that takes.

Status:

phase — Pending / Provisioning / Ready / Error / Deleting
sandboxID — UUID assigned by the gateway
conditions — e.g. Synced, GatewayReachable, PolicyLoaded, UpstreamSandboxReady
observedGeneration — required for GitOps health checks (ArgoCD/Flux use this)

How should policies be referenced or embedded?

Referenced, not embedded. A dedicated reusable policy resource that sandboxes point to by name. The Tenant Admin defines the security boundary (filesystem access, process identity, network rules); the Sandbox Consumer just references it.

Benefits:

Tenant Admins manage policies independently of Sandbox Consumers
Policy updates propagate to running sandboxes for mutable fields (network rules)
Validation webhook prevents deletion of a policy still referenced by active sandboxes
Per-field immutability constraints: process identity is immutable after creation; network rules are freely updatable; filesystem paths are additive-only
Opens the door for the policy advisor to surface suggestions as proposed modifications

How should credentials, provider configuration, and inference routing be handled?

A dedicated Provider CRD referencing a Kubernetes Secret:

spec.type — provider type (anthropic, openai, nvidia, etc.), immutable
spec.credentialsSecretRef — references a Secret (all data keys become credential entries)
spec.config — non-secret configuration (model overrides, base URLs)

This is a Tenant Admin resource. The Sandbox Consumer references providers by name — never sees credential values, never touches Secrets directly. The operator watches the referenced Secret and re-syncs on credential rotation without requiring a Provider CR edit. Standard K8s secret management (external-secrets-operator, vault-agent) integrates without additional work.

Are there existing operator patterns or tools we should align with?

controller-runtime with standard patterns:

Finalizer-based lifecycle for guaranteed cleanup
CreateOrUpdate for idempotent reconciliation
Config hash annotations on pod templates for rolling restarts on config changes
Leader election for the cluster-wide operator
Watches on owned resources for immediate reaction (not polling)
Compatible with existing cluster infrastructure: security standards (PodSecurityStandards, AppArmor, seccomp), common upstream controllers (cert-manager), and networking/service-mesh integrations

Additional Recommendations

Naming: Do not reuse the kind Sandbox in the same cluster as agents.x-k8s.io/Sandbox. Use OpenShellSandbox or a distinct API group. Tooling confusion (kubectl, k9s, ArgoCD resource trees) is significant in practice.
Hardened nodes: The K8s compute driver needs to set AppArmor and seccomp profiles on sandbox pods, not just capabilities. On hardened distributions (Garden Linux, Bottlerocket), sandbox creation fails without Unconfined profiles due to network namespace setup. Related: #1650 is important — removing CAP_SYS_ADMIN is the correct direction for secure K8s deployments.
Entrypoint: Expose command/entrypoint in the sandbox creation API. Hardcoding sleep infinity forces admission webhook workarounds for CI and batch workloads.
driver_config is mandatory: Private registries (imagePullSecrets) and alternative runtimes (runtimeClassName for GPU/kata/gVisor) are standard in enterprise K8s. These need first-class representation in the compute driver — #1589 is the right direction and a prerequisite for a serious operator.
Pod selection contract: Document the agents.x-k8s.io/sandbox-name-hash label as stable API. Operators writing NetworkPolicies for sandbox pods need a reliable, documented selector — replicating internal hashing logic is fragile.

A Note on Deployment Profiles

The control-plane / data-plane separation above naturally leads to two deployment profiles:

Gateway-fronted (resembling the current k3s setup): the gateway exposes the user-facing API (TUI/CLI/SDK over gRPC). The operator handles deployment and upgrades but does not replace the gateway's control-plane role. The gateway retains both control-plane and data-plane responsibilities.
Operator-fronted (OpenShell as an embeddable system): the operator/CRDs are the user-facing API. Frontends and services create Sandbox CRs via the K8s API. The gateway is reduced to data-plane only (relay, streaming, supervisor sessions). The Sandbox Consumer (often an automated system, not a human) creates Sandbox CRs directly.

The operator design could try to accommodate both. The difference is how far the control-plane/data-plane decomposition goes — in gateway-fronted mode the gateway keeps both hats; in operator-fronted mode the operator takes the control-plane hat entirely.

Feedback from a user here planning on utilising a sizeable amount of openshell + nemoclaw sandboxes in our company (Platform Engineer)

I would really appreciate having an openshellsandbox CRD natively in the controller, in my org we have adopted nemoClaw + Openshell and we had to make a wrapper controller around it to achieve nice gitops workflow to spawn up instances.

continuing to utilize agent-sandbox under the hood is fine, as its an official SIGS project thus will align with industry standards

I made a draft MR above implementing this feature and it works when testing but I guess this topic would involve a lot more discussion and probably several tweaks/improvements to this MR.

I am happy to get involved in this and potentially be a contributor.

I have made a vouch request
#1829

@drew

@drew a couple of answers/recommendation for your questions. The use cases we are looking for is enterprise production deployment of OpenShell on Kubernetes and integration with NVIDIA Blueprints at scale. At high level and for first implementation, OpenShell should use a Kubernetes-native controller-gateway design. The OpenShell Kubernetes deployment should run one OpenShell control-plane service that has two roles:

It acts as the Kubernetes controller/operator.
It acts as the OpenShell gateway/runtime API.

Of course there are pros and cons for this design mainly API endpoint scale at a different rate with different trigger metrics, but it simplifies the architecture as opposed to having one micro service for the operator and one for the gateway and many for the agent sandboxes. There are many example operators in the industry that adopted this pattern: Argo CD, Prometheus Operator and cert-manager.

Should the operator deploy OpenShell, manage sandboxes, or both?
Both, in one OpenShell controller-gateway service.
The Kubernetes deployment should run a single OpenShell control-plane service that:

Acts as the Kubernetes controller/operator
Acts as the OpenShell gateway/runtime API
It should watch OpenShell sandbox CRDs such as:

apiVersion: openshell.ai/v1alpha1
kind: OpenShellSandbox

What Kubernetes-native workflows should this support?
It should support a simple YAML-first workflow centered on one main CRD:

kind: OpenShellSandbox

A user should be able to create a sandbox with one Kubernetes object:

apiVersion: openshell.ai/v1alpha1
kind: OpenShellSandbox
metadata:
  ......

The OpenShell controller-gateway should:

watch OpenShellSandbox objects
validate requested image/resources/providers/policy
deny requests the user or namespace is not allowed to use
create agents.x-k8s.io/Sandbox
serve runtime gateway APIs for the sandbox
update OpenShellSandbox status
clean up child resources on deletion

Multi-tenancy should use normal Kubernetes controls:

namespaces
RBAC
ResourceQuota
LimitRange
NetworkPolicy
admission validation

What should the first version include?
The first version should be small and focused with one micro service installed by helm that combines both gateway endpoint function and operator.

Should OpenShell sandbox resources be user-facing, platform-team-facing, or both?

Both. OpenShellSandbox should be usable by both platform teams and application/user teams, but with different permissions and policy boundaries.
Platform teams should be able to create sandboxes for managed services, shared demos, CI systems, etc
Application/user teams should be able to create sandboxes in namespaces where they have permission.
The difference is what each persona is allowed to do.

Platform team:
  can install/configure the controller-gateway
  can grant RBAC
  can set namespace quotas and policies
  can define allowed registries, providers, resources, and runtime settings
  can create OpenShellSandbox objects for shared/platform workloads

Application/user team:
  can create OpenShellSandbox objects only in allowed namespaces
  can use only allowed images, resources, providers, policies, and secrets
  cannot change controller-gateway configuration
  cannot bypass namespace or admission policy

If a user requests something outside their allowed boundary, the controller-gateway should reject it or mark it rejected:

status:
  phase: Rejected
  conditions:
    - type: Accepted
      status: "False"
      reason: ProviderNotAllowed
      message: "Provider ai-inference is not allowed in namespace team-a"

How should OpenShellSandbox relate to the Agent Sandbox CRD?
OpenShellSandbox should be the OpenShell-facing CRD abstracting Sandbox CRD. Each operator reconciles its own CRD. agents.x-k8s.io/Sandbox should not be exposed by OpenShell operator.
Tradeoff: OpenShell must stay compatible with Agent Sandbox. That means OpenShell must track Agent Sandbox CRD versions and schema changes and track the api version. Which also means deployment and runtime api version checking and compatibility.

How should policies be referenced or embedded?
While embedded is easier to understand and require fewer CRDs, it makes more sense to use reference when resource parameters grow large.

How should credentials, provider configuration, and inference routing be handled?
For v1, keep them inline in OpenShellSandbox, but never embed raw secret values.
Use Kubernetes Secret references for credential material.
Example:

apiVersion: openshell.ai/v1alpha1
kind: OpenShellSandbox
metadata:
  name: teama-openclaw
  namespace: team-a
spec:
  image: <image-url>

  providers:
    - name: nvidia-inference
      type: openai
      credentialSecretRef:
        name: teama-api-key
        key: api-key
      config:
        baseUrl: https://integrate.api.teama.com/v1

The controller-gateway should read only referenced Secrets in the sandbox namespace and validate the user is allowed to reference those Secrets.

We're looking at Kubernetes-native integration from the perspective of platform teams deploying OpenShell as part of larger agent orchestration systems. Our interest is in what the OpenShellSandbox CRD could represent beyond infrastructure lifecycle. This is the real tension: What actually is the scope of OpenShell itself ?

The Boundary Is Already Blurry

The existing comments frame the operator as managing sandbox infrastructure: images, resources, lifecycle, policies. But OpenShell itself already crosses the line between agent-agnostic infrastructure and agent-aware configuration in several places. Provider credentials aren't simple env var passthrough; they use placeholder-based injection with multiple refresh strategies, proxy-level credential rewriting, and provider profiles categorized by agent concern (INFERENCE, AGENT, SOURCE_CONTROL, MESSAGING). Inference routing via inference.local transparently intercepts agent API calls and routes through centrally managed model configuration. The proxy understands MCP at a semantic level and enforces per-tool-name policies. A built-in skill system installs policy advisor capabilities into sandboxes, teaching agents to self-manage network policy through structured proposals with formal verification.

OpenShell already knows it's running agents, not just containers. The question for the operator is whether the Kubernetes CRD should reflect that, or stop at the infrastructure boundary and leave agent-aware configuration to the gateway API. If OpenShell doesn't provide a declarative Kubernetes surface for this, out-of-tree platforms will build their own CRDs on top of OpenShell's primitives. They will do that regardless. The question is whether OpenShell wants to shape that space.

What the Extended CRD Could Look Like

If the scope extends into agent configuration, the CRD becomes the declarative surface for the full OpenShell experience:

Skills: Declared on the CR, referencing a ConfigMap, a skill catalog, or inline definitions. The operator gets skills into the sandbox (mount, upload via gateway API, keep in sync on updates). This is the core declarative-to-imperative bridge problem.

MCP server connections: Remote MCP servers (SSE, Streamable HTTP) declared with endpoint URLs and credential references. These are independent of the sandbox image since the connection goes over the network through the proxy. The operator translates these into gateway-side MCP policy rules and handles credential rotation by watching the referenced Secrets.

Observability wiring: OTel collector endpoints, tracing configuration, and environment variables declared on the CR rather than scattered across Helm values and ConfigMaps.

Status as discoverability: If OpenShell integrates with A2A eventually, the agent card (metadata, capabilities, endpoint URL) could be exposed in .status, making sandboxes discoverable by other agents through standard Kubernetes API queries.

apiVersion: openshell.ai/v1alpha1
kind: OpenShellSandbox
metadata:
  name: my-agent
  namespace: team-ml
spec:
  gatewayRef:
    name: default-gateway
  policyRef:
    name: team-default-policy
  image: nvcr.io/nvidia/openshell:latest

  # Agent capabilities (if scope extends beyond infrastructure)
  skills:
    - name: web-search
      sourceRef:
        configMapRef: { name: skill-web-search }
    - name: code-review
      catalogRef: { name: corporate-catalog, version: "1.2" }

  # Remote MCP servers (no sandbox image dependency)
  mcpServers:
    - name: github
      transport: streamable-http
      url: https://mcp.github.internal/sse
      credentialRef:
        secretRef: { name: github-token, key: token }
    - name: jira
      transport: sse
      url: https://mcp-gateway.corp.internal/jira
      credentialRef:
        secretRef: { name: jira-oauth, key: access-token }

  observability:
    tracing:
      endpoint: otel-collector.monitoring:4317

  environment:
    - name: PROJECT_ID
      value: alpha
    - name: API_KEY
      valueFrom:
        secretKeyRef: { name: my-secret, key: value }

status:
  phase: Ready
  sandboxID: abc-123
  conditions:
    - type: SkillsSynced
      status: "True"
    - type: MCPConnected
      status: "True"
  agentCard:
    name: my-agent
    capabilities: ["code-execution", "web-search"]
    url: "https://..."
  observedGeneration: 3

Multi-Tenancy and the Multi-Player RFC

This design should be considered alongside the multi-player RFC (#1980). Both Kubernetes namespace modes from that RFC (managed and operator) map OpenShell workspaces to K8s namespaces, differing only in who provisions the namespace.

For control plane operations (who can create/delete OpenShellSandbox CRs, manage providers and policies), K8s RBAC is a natural fit. For data plane operations (exec into a running sandbox, stream relay output, share sessions), the gateway needs its own authorization model since K8s RBAC doesn't reach into gRPC streaming endpoints. The multi-player RFC's role model covers this. Both are needed, with clear boundaries.

The Declarative-to-Imperative Bridge

If the operator extends into agent configuration, the hardest engineering problem isn't deployment or lifecycle. It's continuously reconciling declarative CRD state with the gateway's imperative API. Skills change, MCP connections come and go, credentials rotate, policies update. The operator must detect drift between the CR's desired state and what the gateway actually has configured, and reconcile without disrupting running sessions.

This is what would make the operator more than a Helm replacement: an active reconciliation controller translating Kubernetes-native desired state into gateway API calls, continuously.

Answers to the Specific Questions

Would you use an operator mainly to deploy OpenShell, to manage sandboxes, or both?
Both. Deployment is table stakes. The value question is how far sandbox management extends: infrastructure only, or agent configuration too.

What Kubernetes-native workflows should this support?
At minimum: declarative sandbox lifecycle via GitOps. If scope extends: skills, MCP connections, credentials, observability, and policies all declared on the CR and reconciled by the operator.

What should the first version include?
Gateway deployment, OpenShellSandbox CRD with policy and provider references, basic status reporting, and the reconciliation loop against the gateway API. Agent configuration fields (skills, MCP) could follow once the bridge pattern is proven.

Should proposed sandbox resources be user-facing, platform-team-facing, or both?
Both, scoped by K8s namespace RBAC for control plane operations.

How should a proposed OpenShell sandbox resource relate to the existing Agent Sandbox CRD?
OpenShellSandbox is a higher-level abstraction that generates Agent Sandbox resources under the hood. Users never interact with agents.x-k8s.io/Sandbox directly. +1 on using OpenShellSandbox to avoid tooling confusion.

What fields would you expect?
See the spec sketch above. Infrastructure fields (image, resources, policyRef, gatewayRef, environment) are clear. Agent configuration fields (skills, mcpServers, observability, agentCard in status) depend on scope decisions.

How should policies be referenced or embedded?
Referenced.

How should credentials, provider configuration, and inference routing be handled?
Providers as configuration resources referencing K8s Secrets. The operator resolves Secret references at runtime and watches for rotation. Never stores raw credential material on the CR.

What status, events, and observability would be required?
Phase, conditions, observedGeneration, sandboxID. If agent configuration is in scope: SkillsSynced, MCPConnected conditions, and optionally agentCard for A2A discoverability.

Are there existing operator patterns or tools we should align with?
controller-runtime with standard patterns. If agent configuration is included, the reconciliation loop needs reliable drift detection against the gateway's imperative API, which is more complex than typical CRD-to-resource controllers.

docs: clarify multi-gateway is not multi-tenancy, link upstream issues

@kon-angelo

We've built this as an out-of-tree operator, now usable: https://github.com/lensapp/openshell-k8s-operator. Feedback and testers welcome.

The design choice that answers most of the disambiguation questions here: it's a thin front-end, not a second control plane. CRDs in a distinct group (openshell.lenshq.io/v1alpha1) — OpenShellSandbox, OpenShellProvider, OpenShellPolicy, OpenShellWorkspace — translate into gateway gRPC calls and mirror gateway state into .status. So:

Separate CRDs, not agents.x-k8s.io/Sandbox re-exposed; we never reuse the kind name Sandbox.
The gateway stays the source of truth — its compute driver still owns the agents.x-k8s.io/Sandbox object.
Additive, not either/or: the operator authenticates to the gateway as an OIDC client; the Helm chart bundles gateway + issuer + operator, or targets a BYO gateway.

It maps onto the three-persona model @kon-angelo described, via native RBAC. In testing it also grew a webhook that confines kubectl exec into the privileged sandbox pods, plus CEL immutability on identity fields. It reuses OpenShell's own crates pinned to an exact rev (v0.0.90), so it tracks upstream rather than forking.

On the in-tree / controller-gateway direction (@nakfour): no stake in staying out-of-tree. Keeping the gateway as authority made the operator thin; if OpenShell converges on an in-tree operator, happy for this to be a reference, a migration source, or superseded.

@nakfour

Perspective from the OpenShift / OLM packaging side. Most of the thread has debated what the operator does; I want to add the constraint set that shows up when you try to ship it as an OperatorHub-installable operator, because it turns out to disambiguate a couple of the open questions rather than just adding requirements on top of them.

To be clear up front: I am not proposing a separate "OpenShift operator". OpenShift is a packaging and admission target for whatever operator this issue converges on. But the packaging constraints are load-bearing enough that I think they belong in the design discussion rather than after it.

1. OLM has an opinion on the control-plane / data-plane question

@nakfour proposes one service that is both controller and gateway/runtime API, citing Argo CD, Prometheus Operator, and cert-manager. Worth noting that all three of those are counter-examples to the merged shape: the Prometheus Operator does not become Prometheus, cert-manager's controller is separate from its webhook and CA injector, and Argo CD's operator deploys the Argo components rather than serving the Argo API itself.

That split is not stylistic under OLM. The operator's Deployment is created and owned by OLM from the ClusterServiceVersion install.spec.deployments, which means:

OLM controls its replica count, and upgrades roll it by replacing the CSV. A byte-streaming data plane (relay, exec, forward, SSH) being torn down by a catalog-driven operator upgrade is a materially different failure mode than a reconciler being restarted.
The operator scales on the number of reconciled objects. The gateway data plane scales on concurrent sandbox sessions and bytes. Fusing them puts two unrelated scaling curves in an OLM-managed Deployment that the cluster admin is not supposed to hand-edit.
Leader election is expected for the controller and wrong for the data plane.

This doesn't decide how far the control-plane decomposition goes — @kon-angelo's gateway-fronted vs operator-fronted framing still stands. It does mean that whichever direction wins, the reconciler and the streaming data plane should be separate Deployments. @jakolehm's thin front-end shape is the one that survives this constraint without further work, and the operator-fronted variant survives it too as long as the shrunken gateway stays its own workload.

2. Agent Sandbox as an OLM dependency

sandboxes.agents.x-k8s.io is installed today by applying the upstream manifest (docs/kubernetes/setup.mdx:36). OLM cannot express "depend on a CRD owned by a controller that isn't packaged as an operator" — required CRDs in a CSV resolve against other bundles in a catalog, not against kubectl apply. So there are exactly three options and it's worth picking one deliberately:

Declare sandboxes.agents.x-k8s.io as a required CRD and document that Agent Sandbox must be installed first. Install fails with a resolution error until it is. Honest, but a poor OperatorHub experience.
Ship an OpenShell-maintained bundle of the Agent Sandbox controller and declare a real OLM dependency. Means taking on its release cadence, which @kon-angelo already flagged as debatable.
Have the operator install and manage the Agent Sandbox controller itself. Conflicts with clusters that already run it for other workloads.

I'd argue for (1) in v1 with a clear precondition check surfaced as a status condition, but the choice needs to be explicit because it's visible to every OperatorHub user on day one.

3. The current OpenShift path cannot be packaged as-is

This is the concrete blocker, and it's independent of the CRD debate. docs/kubernetes/openshift.mdx documents the supported OpenShift install as: pre-create the namespace, oc adm policy add-scc-to-user privileged -z openshell-sandbox, and --set server.disableTls=true. It carries an explicit experimental warning.

An operator whose sandboxes require the privileged SCC will not pass Red Hat operator certification, and in most enterprise OpenShift clusters it will not get past the platform team regardless of certification. privileged is the SCC of last resort; requesting it for the workload pods rather than the operator is a hard stop in a lot of shops.

Looking at what the driver actually asks for, privileged looks broader than necessary:

Combined topology requests SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYSLOG on the agent container (crates/openshell-driver-kubernetes/src/driver.rs:3572).
Sidecar topology (SupervisorTopology::Sidecar, crates/openshell-driver-kubernetes/src/config.rs:66) puts network supervision in a sidecar requesting NET_ADMIN, NET_RAW, CHOWN, FOWNER (driver.rs:2846) and leaves the agent container as a low-capability wrapper.
The AppArmor profile is already configurable (RuntimeDefault / Unconfined / Localhost/<name>), which is the right shape for hardened nodes — this addresses part of @kon-angelo's point 2.

That reads like a named custom SCC shipped by the operator, not privileged: a small capability set, RunAsAny for UID/fsGroup so OpenShift admission still assigns them, and the sidecar topology as the OpenShift default so the blast radius sits in the proxy sidecar rather than the agent container. Two things would need resolving to get there: the combined-topology CAP_SETPCAP gap in #2751, and the hardcoded UID/fsGroup that #2697 and #2707 already cover.

The server.disableTls=true requirement is the other half. Passthrough Route support already exists in the chart (openshiftRoute.*, values.yaml:523) and #2466 landed cert-manager external issuer support, so the TLS-on path is reachable today — it just isn't the documented default. An operator should only ever install the TLS-on configuration.

I think this is the real v1 prerequisite, ahead of any CRD work. If OpenShell runs on OpenShift only under privileged with TLS off, the operator has nothing certifiable to package no matter which CRD shape wins.

4. Disconnected installs constrain the CRD, not just the bundle

docs/kubernetes/setup.mdx:42 currently describes air-gapped installs as a manual mirroring exercise across four images: the Agent Sandbox controller, image.repository (gateway), server.supervisorImage, and server.sandboxImage.

OLM's answer is relatedImages in the CSV with every image pinned by digest, so oc-mirror can resolve the full closure and ImageDigestMirrorSet can rewrite them. The gateway, supervisor, and controller images are all pinnable that way. The sandbox image is not — it's per-sandbox and user-supplied, so it can never appear in relatedImages.

That has a CRD consequence worth deciding now rather than later: a disconnected cluster needs the operator to validate sandbox images against an admin-controlled allowlist and to surface a clear rejection when a user references an unmirrorable image. That's a Tenant-Admin-facing concern in @kon-angelo's model and it argues for image policy living on a platform-owned resource rather than being free-form on the Sandbox CR. It also lines up with @nakfour's Rejected phase / Accepted: False condition suggestion — ImageNotMirrored is exactly that shape.

5. Install modes map onto the three-persona model

@kon-angelo's Platform Admin / Tenant Admin / Sandbox Consumer split translates cleanly onto OLM install modes, which is a useful consistency check on the CRD layering:

AllNamespaces — Platform Admin installs the operator once; Gateway (or equivalent) is cluster-scoped or lives in the operator's namespace.
OwnNamespace / SingleNamespace — Tenant Admin owns Policy and Provider in their namespace.
Sandbox Consumer creates namespaced Sandbox CRs, gated by native RBAC.

If any of the proposed resources can't be cleanly assigned to one of those tiers, that's a signal the layering isn't right yet. Concretely, it means Policy and Provider should be namespaced with an optional cluster-scoped default, not cluster-scoped only.

6. Capability levels as shared vocabulary for "what should v1 include"

The question "what should the first version include" has had several good answers with no common yardstick. The Operator Framework capability levels are one, and they're what OperatorHub displays:

L1 Basic Install — gateway + dependencies installed and configured. This is roughly Helm parity and, per the thread's consensus, not enough on its own.
L2 Seamless Upgrades — version upgrades handled, which is where OLM earns its place over helm upgrade.
L3 Full Lifecycle — sandbox/policy/provider lifecycle, backup of gateway state.
L4 Deep Insights — metrics, alerts, status conditions. Connects to feat(observability): OpenTelemetry export surface for gateway operators #2507 and feat(server): metrics instrumentation #909.
L5 Auto Pilot — auto-scaling, auto-remediation, auto-tuning.

L2 is the honest floor for shipping to OperatorHub at all, and L3 for the declarative sandbox workflow this thread is actually asking for. Naming a target level would make "what's in v1" concrete without pre-committing to a CRD shape.

Suggested sequencing

Make OpenShell installable on OpenShift under a named custom SCC with TLS enabled — resolving Kubernetes driver never requests CAP_SETPCAP, so combined-topology sandboxes crash-loop with EPERM on privilege drop #2751, feat(helm): expose sandbox_uid / sandbox_gid as first-class Helm values #2697, Support low numeric UID and GID values. #2707 and promoting the sidecar topology as the OpenShift default. Independent of the operator debate, and a prerequisite for any certifiable bundle.
Decide the Agent Sandbox dependency posture (§2), since it's user-visible from the first install.
Whatever operator shape wins, keep the reconciler and the streaming data plane as separate Deployments (§1).
Add image-policy validation to the platform-facing resource so disconnected clusters work (§4).

No strong preference between in-tree and out-of-tree — the SCC and TLS work in (1) is needed either way.

@flg77

@flg77 thanks for your input, but you add a lot of OpenShift specific concerns that goes beyond a vanilla Kubernetes operator (the topic of this issue). I think its worth to create a separate issue for OpenShift specific concerns, like the OpenShift installation issues you address (which are only slightly connected to the operator discussion at hand). Your comment is also a bit hard to consume by humans, maybe you can summarize in your own words, what an pure Kubernetes Operator for the control and data plane should look like ? That would be very helpful. We are still in an early discussion phase, so we are not yet there for a concrete implementation plan. Let's keep the discussion on a personal level (agent assisted, but with a human focus, and with some empathy for the readers).

@drew

Hi @drew - I realize the issue title is "Kubernetes Operator" but it seems like an immediate gap for K8s installations is a CRD for an OpenShell Sandbox with a controller for lifecycle management. As one of the comments suggests, the controller can be a thin layer above the existing API. Would it make sense to split that out and tackle it separately from an operator that installs and manages OpenShell? Is an RFC the right next step?

Navigation Menu

Repository navigation

Kubernetes Operator #1719

Problem Statement

Proposed Design

OpenShell Deployment Operator

OpenShell Sandbox Custom Resource Operator

Existing Agent Sandbox CRD Boundary

Operator And Gateway Boundary

Questions For Feedback

Non-Goals For Now

Alternatives Considered

Agent Investigation

Proposed Outcome

Personas

Questions

Additional Recommendations

A Note on Deployment Profiles

The Boundary Is Already Blurry

What the Extended CRD Could Look Like

Multi-Tenancy and the Multi-Player RFC

The Declarative-to-Imperative Bridge

Answers to the Specific Questions

1. OLM has an opinion on the control-plane / data-plane question

2. Agent Sandbox as an OLM dependency

3. The current OpenShift path cannot be packaged as-is

4. Disconnected installs constrain the CRD, not just the bundle

5. Install modes map onto the three-persona model

6. Capability levels as shared vocabulary for "what should v1 include"

Suggested sequencing

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Kubernetes Operator #1719

Description

Problem Statement

Proposed Design

OpenShell Deployment Operator

OpenShell Sandbox Custom Resource Operator

Existing Agent Sandbox CRD Boundary

Operator And Gateway Boundary

Questions For Feedback

Non-Goals For Now

Alternatives Considered

Agent Investigation

Proposed Outcome

Activity

kon-angelo commented on Jun 5, 2026

Personas

Questions

Additional Recommendations

A Note on Deployment Profiles

bjw123 commented on Jun 9, 2026

nakfour commented on Jun 25, 2026

rhuss commented on Jul 2, 2026

The Boundary Is Already Blurry

What the Extended CRD Could Look Like

Multi-Tenancy and the Multi-Player RFC

The Declarative-to-Imperative Bridge

Answers to the Specific Questions

jakolehm commented on Jul 24, 2026

flg77 commented on Aug 19, 2026

1. OLM has an opinion on the control-plane / data-plane question

2. Agent Sandbox as an OLM dependency

3. The current OpenShift path cannot be packaged as-is

4. Disconnected installs constrain the CRD, not just the bundle

5. Install modes map onto the three-persona model

6. Capability levels as shared vocabulary for "what should v1 include"

Suggested sequencing

rhuss commented on Aug 19, 2026

JimBugwadia commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions