Concept: an abstract runtime for boxes #109

Status: draft. Base: main at 3a16674.

1. Summary

Today, the orchestrator runs boxes on Docker only. This concept makes the runtime replaceable, so that boxes can also
run on Kubernetes. The code outside the runtime does not know which runtime it uses.

The central change is the access to the files of a box. Today, the orchestrator reads and writes them on its own
filesystem. On Kubernetes this is not possible, so a file service in the box does this work, on both runtimes. Many
other operations also move into the box or into the box image, so that they work the same way on both runtimes. The
runtime itself keeps only the parts that are different by nature: the container or pod lifecycle, the storage, the
network isolation, the exec transport, the image lifecycle and the inventory.

2. Background

PR #94 added Kubernetes as a second runtime. It was closed after a review, for these reasons:

• Review, attachments and agent sets did not work on Kubernetes, because they read the files of a box on the
  filesystem of the orchestrator.
• The runtime interface was a copy of docker.ts. On Kubernetes, some of its methods were stubs or threw an error.
• Code outside the runtime checked cfg.RUNTIME, and logins had a separate runtime of their own.
• On Kubernetes, opening the review started a stopped box. With Docker, it did not.
• Some operations worked differently on the two runtimes without a need to. For example, Docker read the process list
  with docker top, and Kubernetes ran ps in the box.

An inventory of main shows two facts that this concept builds on:

• No host binary runs against the files of a box. Git, ps, kill, the terminal and the adapter all run in the box
  through docker exec. The only host git is the skill repository pull, and it works on a global directory.
• The orchestrator reads and writes the files of a box with Node fs calls. This is in four places:
  workspaces.ts (create, check, delete), agents.ts (agent set materialization), attachments.ts with app.ts
  (upload and download), and review/* (tree, repository discovery, file read and write, REVIEW.md). Disk usage is
  a Node walk.

The first fact stays as it is. The second fact must change.

3. Goals and non-goals

Goals:

• Boxes run on Docker or on Kubernetes. A deployment selects one runtime.
• Each feature works the same way on both runtimes. Where it does not, this concept names the difference.
• CI tests both runtimes for each pull request.
• Each step of the implementation is useful on Docker alone.

Non-goals:

• More than one user. Boxes is a tool for one person. The concept must not prevent more users later: the runtime
  keys everything by the box id and a deployment label, and it has no global state other than the configuration. An
  owner of a box would be a field in the database, not a runtime concept.
• A smooth upgrade. A running box can need a restart after an update of the orchestrator (see 8.3).
• Private box images. Boxes does not support them today: the Docker pull sends no credentials. On Kubernetes, the
  pull secrets of the cluster apply (see 11).

4. Assumptions

On Kubernetes, the concept assumes only this about storage:

• Each box gets one persistent volume claim from the default storage class of the cluster.
• One pod at a time mounts it. This pod is the box pod.
• The orchestrator gets one volume of its own for boxes.db, the lock file and the skill repository checkouts.

The concept does not assume ReadWriteMany, a volume that two pods share, or pods on one node. The orchestrator never
mounts the storage of a box.

On Docker, the storage stays as it is: directories in the data volume, bind mounted into the box container.

On Kubernetes, the concept assumes this about the network:

• The cluster has a CNI that enforces NetworkPolicy, for ingress and for egress. A policy is only an object in the
  API, and a cluster whose CNI does not enforce it gives the boxes no isolation, without an error anywhere.
• CI uses a kind cluster with such a CNI, and one test confirms that a box cannot reach another box, the orchestrator,
  or the control port of the proxy.

5. Principles

1. The interface describes box operations, not Docker operations. "Make the box reach the proxy and nothing else" is a
   box operation. "Connect the proxy container to a network" is a Docker detail.
2. When an operation can work the same way on both runtimes, it works that way on both, and it is not part of the
   runtime. Each difference is a place where a feature can break on one runtime only.
3. One factory selects the runtime at boot. No other code reads cfg.RUNTIME.
4. The runtime has no knowledge of threads, reviews or credentials. It knows boxes, their storage, their processes and
   the box image.

6. Architecture

flowchart TB
    subgraph orch["orchestrator"]
        core["boxes, gateway, review,<br>attachments, agent sets, login"]
        rt["Runtime interface"]
        docker["Docker runtime"]
        k8s["Kubernetes runtime"]
    end

    subgraph box["box (container or pod)"]
        files["file service"]
        adapter["adapters"]
        term["terminal (tmux)"]
        storage[("workspace, home, Nix store")]
    end

    core -- "lifecycle, exec, spawn" --> rt
    rt --> docker
    rt --> k8s
    core -. "file operations<br>over a spawned stream" .-> files
    files --> storage
    adapter --> storage
    term --> storage

The orchestrator reaches a box only through the runtime: it starts and stops the box, and it runs processes in it. All
work on the files of a box is done by processes in the box.

What each runtime does in its own way:

What both runtimes do in the same way, and what is thus not part of the runtime:

7. The runtime interface

Names are a draft:

interface Runtime {
  /** Creates the storage of a new box. Fails when it exists. */
  createStorage(boxId: string): Promise<void>;
  /** Whether the storage of a box exists. */
  storageExists(boxId: string): Promise<boolean>;
  removeStorage(boxId: string): Promise<void>;

  /**
   * Creates or repairs the container definition and the isolation of a box. Idempotent. On a running box, it repairs
   * only the isolation and never changes the container.
   */
  ensure(box: BoxSpec): Promise<void>;
  /** Starts a box. Resolves when an exec in the box works. */
  start(boxId: string): Promise<void>;
  stop(boxId: string): Promise<void>;
  /** Removes the container and the isolation. The storage stays. */
  remove(boxId: string): Promise<void>;
  /** `unknown` when the runtime does not answer. It counts as live, so a failed read stops nothing. */
  state(boxId: string): Promise<'running' | 'stopped' | 'unknown'>;

  /** The image a box runs, as an image id or digest. */
  imageOf(boxId: string): Promise<string | null>;
  /** Makes the image that BOX_IMAGE names available, and removes old images where the runtime must. */
  refreshImage(): Promise<void>;

  /**
   * All objects of this deployment, storage and login containers included, for the boot reconcile and the orphan
   * sweep.
   */
  inventory(): Promise<RuntimeObject[]>;
  removeObject(obj: RuntimeObject): Promise<void>;

  /** A short command as the box user. Output is capped, a timeout applies. */
  run(boxId: string, cmd: string[], opts?: RunOptions): Promise<RunOutput>;
  /** A long-lived process with stdio, with or without a TTY. Can end at any time. */
  spawn(boxId: string, cmd: string[], opts: SpawnOptions): Promise<ProcessStream>;

  /** Starts a login container: short-lived, from the box image, with internet access and no storage. */
  startLogin(spec: LoginSpec): Promise<Login>;

  /** The URL of the front door of the egress proxy, as a box sees it. */
  proxyUrl(): string;
  /** Runtime facts for the details page, for example the network and the subnet. */
  details(boxId: string): Promise<Array<[string, string]>>;
  /** Throws when the runtime cannot be reached. */
  health(): Promise<void>;
  /** The images of the orchestrator, the proxy and a new box, for the image footer. */
  deploymentImages(): Promise<DeploymentImages>;
}

BoxSpec carries the id, the environment and the limits. It does not carry the image, because each runtime uses
BOX_IMAGE (see 11). It does not carry the mounts, because they are the same for every box: /workspace,
/home/agent and /nix. Each runtime makes them from its own storage.

The core builds the full environment of a box as a plain record: the credential placeholders, the git identity, the
proxy variables from proxyUrl, and the CA certificate with the variables that point to it. Each runtime only converts
the record to its own format. The environment holds no secret, because the egress proxy replaces the placeholders. On
Kubernetes, it is thus a plain env in the pod spec, and no Secret object is necessary.

The runtime keys everything by the box id and keeps no database of its own. The columns container_id,
network_name, subnet and image of the boxes table go away with a migration. The Docker runtime finds the values
itself: the container by its name box-<id>, the network by its name, a free subnet at a create from the subnets of
the existing box networks, and the image with imageOf. A create that is not complete is the row status creating,
as today. After docker network prune, a stopped box gets a new subnet at its next start. Nothing depends on a fixed
subnet.

state has no value for a missing container. On Docker, an exited container and a container that is gone are both
stopped, and ensure creates a missing one again before the next start. On Kubernetes, a pod in the phase Running
is running, a pod in Pending counts as running because a start is in progress, and no pod or a pod in a terminal
phase is stopped.

spawn serves the adapters, the terminal and the file service. The contract of its stream is in 12.

A login container is not a box, so the box methods do not apply to it. The Login handle has its own spawn, with
the same stream contract, and its own remove. A box id and a login can thus not be mixed up.

The image footer of the box list shows three images with digest, build time and size. For the orchestrator and the
proxy, it shows the image that runs now. For the box, it shows the image that a new box gets (see 11).

• Docker: the orchestrator inspects its own container and the container EGRESS_PROXY_CONTAINER.
• Kubernetes: the orchestrator reads the imageID of its own pod, which it finds by the pod name from the downward API,
  and the imageID of the proxy pod.

8. File access

8.1 The file service

All access of the orchestrator to the files of a box goes through one file service. This is a long-lived process
in the box. The orchestrator starts it with spawn, as it starts an adapter, and talks to it over its standard input
and output. The same file service is used on Docker and on Kubernetes. There is one code path and one set of tests.

The file service is a fixed command in the box image, boxes-files, as the adapters are. The image and the
orchestrator come from one repository, so the name is a contract of the repository. The orchestrator starts the file
service as the first exec after start, because the agent set install (9.2) needs it before the box is ready. After
the connection ended, the next file operation starts it again.

Operations, as a draft:

The rules of the operations stay as they are in review/fs.ts, review/tree.ts and review/repos.ts:

• Paths: the root and the target are resolved with realpath, a symlink in the last component is refused, and a new
  file must have its parent inside the root.
• read detects a binary file by a NUL byte, and gives the real size.
• write keeps the file mode of the old file.
• list leaves out symlinks. findRepos also accepts a .git that is a file.

The orchestrator keeps everything that is not file access: the per-box lock around REVIEW.md, the parsing, the drift
logic, the caches and the HTTP routes.

Wire protocol. The orchestrator and the file service speak HTTP/2 over the standard input and output of the
process. Node's http2 module accepts any byte stream as its connection, on the client side and on the server side,
so neither side needs a port, a network path or a dependency. Each operation is one request: the method and the path
name the operation and the file, the body carries the bytes, and the usual status codes report the result, for
example 404 for a missing file and 409 for a hash that does not match. The hash of a file travels in the ETag
header, and a write sends the expected hash in If-Match.

Each request is its own HTTP/2 stream, so a long download and a short stat run at the same time on one connection.
The flow control of HTTP/2 applies to each stream. A slow reader stops the writer, so a large file stays on disk and
does not go into memory on either side. This holds only when both sides pipe streams. A body that is written in one
call is held in memory first, and Node ends such a stream when it is larger than the session memory limit. The
orchestrator therefore pipes a download into the HTTP reply to the browser, and an upload from the request into the
file service. A test with a 300 MiB file in each direction and a reader at 20 MiB/s kept the memory of both processes
flat.

8.2 Properties

• Rights. The file service runs as the agent account. Files it creates belong to the agent, so the chown calls of
  the orchestrator go away. The orchestrator no longer reads a tree that the agent controls with its own rights. The
  containment check of review/fs.ts stays in the file service, but only to keep the review inside /workspace, not as
  a protection of the orchestrator.
• Replies are untrusted. The agent has the rights of the file service and can replace it. The orchestrator
  therefore treats the file service as it treats an adapter: it applies the caps (the list size, the read cap, the
  attachment size) on its own side and ends a request that exceeds them, it validates the shape of each reply before
  it uses it, it never turns a reply into a path, a command or a URL of its own, and each request has a timeout.
• One process per box. The file service does not start a process for each operation.
• Size. The caps of today stay: an attachment (MAX_ATTACHMENT_MB, default 25 MiB) and REVIEW.md (8 MiB) are
  held in memory by the orchestrator, as today. A raw download has no cap and is piped end to end.
• End of the connection. The next call starts the file service again. A request that is in progress when the
  connection ends fails. The caller retries a read once. A write is not retried: when the first attempt arrived, the retry finds a
  changed hash and reports a conflict that is not real. The caller reports the error.

8.3 Version

The file service is part of the box image. The image and the orchestrator are built from the same repository, but a
running box keeps its image until it stops, so the orchestrator can be newer. The file service therefore states its
protocol version in a header of each response. On a mismatch, the orchestrator refuses the file operations. Older
versions are not supported.

A restart moves the box to the current image only when BOX_IMAGE resolves to a new image. With a pinned tag, after
a failed pull on Docker, or with IfNotPresent on kind, the restart gives the same image again. The error therefore
names the version it found and the version it needs, and says that the box needs an image with the current file
service. When the last pull on Docker failed, the error says so too. The box gets the status error with this
message, as a box with missing storage does (9.1), because without the file service the agent set is not installed
and the box does not become ready.

8.4 Disk usage

The orchestrator asks the file service for the size before each stop, and stores the value with its time. Nothing in a
stopped box changes, so the value stays correct. A running box is measured again at most every 15 minutes, as today. A
box that ends without a stop by the orchestrator, for example after an out-of-memory kill, keeps its last value.

The measurement before a stop has the timeout of a run. A walk over a large Nix store can take longer on slow
storage. When the walk does not finish in time, the stop goes on, and the box keeps its last value with its old time.
The dashboard shows that time, so the user sees that the value is old.

8.5 Stopped boxes

A file operation starts a stopped box, on both runtimes. Git already works like this. The effect is small:

With a stopped box, the orchestrator can thus only read the stored disk usage and delete the box.

9. The box lifecycle

9.1 Storage

The runtime creates, checks and removes the storage of a box. It never gives the orchestrator a path.

• Docker: the directories workspaces/<id>, homes/<id> and nix/<id> in the data volume, bind mounted.
• Kubernetes: one claim per box, mounted with subPath at /workspace, /home/agent and /nix. New subPath
  directories must be writable for BOX_UID. The plan is securityContext.fsGroup.

The storage is created only when a box is created. When the storage of an existing box is missing, the box is marked
as an error, and the storage is not created again: a new empty home would erase the transcripts of the agent without a
warning. This is the behavior of requireDirectories today for the workspace and the home.

The Nix store is the exception. It holds only installed packages, and nix lays out an empty store at first use, so a
missing one is created again at each start, as today. On Docker, this is an empty directory that belongs to the agent.
On Kubernetes, a missing claim is a missing storage, and a missing nix directory inside the claim is made by the
kubelet at the mount.

9.2 Start and the ready state

The inputs that a box gets at its start are:

• Environment variables, as today: the credential placeholders, the git identity, the URL of the egress proxy and
  the CA certificate. They are fixed when the container or the pod is created.
• The setup of the home. Today, the entrypoint prepares the home at each start: it writes the CA bundle, adds the
  CA to the trust store of Chromium, sets up the browser CLI, and writes the git identity and the credential helpers.
  These steps move into one command of the image, boxes-setup, with the same shell code. The orchestrator runs it
  with run after the file service is connected. The orchestrator knows only the name, as it knows the adapter names.
  The image decides what its setup does. A failed step is an error of the box with its output, where today it is a
  warning in the container log.
• The agent set. The orchestrator no longer writes agents/<id>, and the box has no /boxes/agent mount. After the
  setup, the orchestrator sends the merged set as a tar stream to installAgentSet of the file service. The installer
  does what the entrypoint does today: it removes what the last install recorded in ~/.boxes/managed, checks each
  manifest line, and installs the new entries. The entrypoint also installs the skills of the image (nix,
  share-app, diagrams and the skill of the browser CLI), and a skill of the same name in the set wins. These
  installs read the manifest, so they move into the installer too, and both happen in one step. As today, a change
  to a set reaches a box at its next start, and a running box keeps what it has.

A box is ready when:

1. the runtime has started it, and an exec works,
2. the file service is connected,
3. boxes-setup has run,
4. the agent set is installed.

The orchestrator drives this order, so nothing writes into the home at the same time as the installer. The entrypoint
does only two things: it seeds the home (9.3) and it holds the container.

The adapters, the terminal and the review wait for "ready", not for "started". On Kubernetes, start must wait for the
pod to run: scheduling, attaching the volume and pulling the image can take longer than the retries of an adapter
spawn.

9.3 The home seed

A new home must get the files of the home of the image. The mount hides the original, so the image keeps a copy of its
home outside /home/agent. The copy is a few kilobytes: the shipped home holds only the files of /etc/skel (from
useradd -m), some empty directories and .claude/settings.json.

The entrypoint seeds the home: when the marker file ~/.boxes/seeded does not exist, it copies the files and writes
the marker. A home that the agent emptied thus stays empty. The entrypoint runs as the agent, so the files belong to
BOX_UID without a chown. This also solves #99. The root helper container of today goes away.

9.4 Stop and removal

Before a stop, the orchestrator measures the disk usage (8.4). On Docker, a stop stops the container. On Kubernetes, a
stop deletes the pod, and the claim stays.

On Kubernetes, stop deletes the pod with the grace period of today, 10 seconds, and returns only when the pod no
longer exists. This is the contract of Docker: after stop, the box is really stopped. start first deletes a pod of
the box that still exists, for example a terminal pod after an out-of-memory kill or a pod in Terminating after an
interrupted stop, and waits until it is gone. Only then does it create the new pod. The new pod thus never competes
with the old one for the volume. The detach of the volume can still take a moment, and the new pod waits in Pending
for it.

On Kubernetes, a box is a bare pod with restartPolicy: Never, the counterpart of RestartPolicy: no on Docker. The
orchestrator alone creates and deletes the pod, and no controller makes a pod on its own. When the container ends by
itself, for example by an out-of-memory kill, the kubelet does not restart it. The pod stays in a terminal phase with
its reason, the orchestrator sees it as stopped, and the next start deletes the old pod and creates a new one. An
eviction by the kubelet is the same case: the pod stays as Failed, and the next start places the box where there is
room.

A node failure is a known limitation. The control plane cannot know whether the lost node still writes to the volume,
so it does not detach it, and the pod stays in Terminating. The box cannot start on another node until then. The
orchestrator must not force-delete the pod, because that does not release the volume and risks the data. The way out
belongs to the cluster: the operator, or an automation of the cluster, sets the out-of-service taint on the dead node,
and the control plane then deletes the pod and detaches the volume. After that, the box is stopped and starts again
on another node.

When a box is deleted, the orchestrator calls remove and then removeStorage.

9.5 Reconcile and orphan sweep

At boot, the orchestrator compares its database rows with the inventory of the runtime: it adopts live boxes, marks
missing ones as stopped, and marks an interrupted create as an error. The periodic sweep removes each object of the
inventory that has no live row. The inventory includes the storage, so a storage left by a failed delete is removed
too. The guard against a sweep of a wrongly mounted data volume stays.

The inventory also includes the login containers, with their creation time. A login container stays behind when the
orchestrator stops during a login. The sweep removes each login container that is older than 15 minutes, as today.

10. Network isolation

The contract:

A process in a box can open connections to the front door of the egress proxy, and to nothing else. It finds the
front door at the URL that the runtime gives. Nothing reaches a box from the network.

The proxy does not identify a box. It has one policy for all boxes, keyed by the destination host, and it does not
read the source address. So the network alone keeps the boxes apart from each other and from the orchestrator.

• Docker: an internal network per box, with the proxy attached under the alias proxy. The subnet pool is an internal
  detail of the Docker runtime.
• Kubernetes: the box pods, the proxy and the orchestrator are in one pod network, so three NetworkPolicy objects
  do the work of the Docker networks. The box policy allows egress only to the proxy pods on port 3128, and to the
  cluster DNS when the URL is a name, and it denies all ingress. The orchestrator needs no ingress to a box, because
  it uses exec through the API server. The proxy policy allows ingress on port 3128 from the box pods and on port 3129
  from the orchestrator pod only. The orchestrator policy is in 14. The proxy is a Service.

The orchestrator reaches the control channel of the proxy at a URL from the configuration (EGRESS_CONTROL_URL,
default http://boxes-egress-proxy:3129), not at a container name. On Docker, the control channel is safe because it
binds to an address on the Compose network, which the box networks cannot reach. On Kubernetes, the box pods and the
proxy are in one pod network, so the bind address does not protect it. The policy must therefore allow port 3128 only.

The isolation of a running box can break. On Docker, a proxy container that Compose creates again has no connection to
the box networks. On Kubernetes, a policy can be deleted. A periodic loop therefore calls ensure for each running box.
On Docker, this connects the proxy to the box network again. On Kubernetes, it applies the policy again. A box where
ensure fails shows as a warning in the dashboard, as today.

Each call of ensure runs in the per-box queue of the orchestrator, as a start does. A box can stop between the check
of the loop and the call, and ensure can then create the container again. In the queue, this cannot overlap with a
start of the same box.

The details page shows the network name and the subnet today. These become runtime facts from details.

11. The image lifecycle

The image lifecycle is a runtime matter. Each runtime does it in its own way. The contract:

When a box starts, it runs the image that BOX_IMAGE names at that time. A running box keeps its image.

The image refresh loop calls refreshImage. On Kubernetes, this does nothing.

Before each start, the orchestrator calls ensure and then start. On Docker, ensure creates the network and the
container again when they are missing, for example after docker container prune. It also creates the container again
when its image is not the current image of BOX_IMAGE. Both happen only when the box does not run. On Kubernetes,
ensure only applies the policy, because each start creates a new pod. The core thus has no image comparison and no
case for a missing container.

On Kubernetes, imagePullPolicy is a setting. The default is Always. CI with kind and local development use
IfNotPresent, because their images are loaded into the cluster and are in no registry. The pull secrets of the cluster
apply, so the orchestrator needs no registry credentials.

Consequences on Kubernetes: a box start needs the registry, and two boxes started at different times can run different
images. The orchestrator sees this in imageOf.

On Kubernetes, the kubelet resolves the tag, so the orchestrator knows a digest only from a pod that started. The image
footer therefore shows the image that the last box start got. When the tag moves after that start, the footer shows the
old build until the next box starts. This is a known difference from Docker.

12. Long-lived streams

The contract: spawn returns a stream that can end at any time, and the caller must cope with it.

On Docker, an exec stream has no time limit. On Kubernetes, it goes through the API server and the kubelet to the
container runtime, and can end:

• The container runtime closes a stream that is idle. containerd does this after 4 hours by default
  (stream_idle_timeout). CRI-O has no limit by default. The kubelet setting streamingConnectionIdleTimeout has no
  effect; it was deprecated in 2025 (kubernetes PR 131992).
• A load balancer in front of the API server can have a shorter limit.
• A restart of the API server ends all streams.

An idle timeout does not end a stream with traffic. A running turn sends traffic.

How each user of spawn copes:

• Adapter: this works today. An ended exec drops the connection, and the next message spawns the adapter again and
  loads the sessions. An idle timeout can only cut a turn that waits longer than 4 hours for a permission. This is
  accepted.
• Terminal: today, an ended exec closes the WebSocket. With this concept, the orchestrator opens a new exec and
  attaches to the same tmux session again, while the WebSocket stays open. tmux keeps the state. Each terminal
  connection has its own tmux client session, grouped with the shared session of the box, and that client session
  outlives a cut stream but not a shell that the user ended. The reattach therefore uses tmux attach to the client
  session by its name, never a new session. After an ended exec, the orchestrator waits one second and attaches. When
  the session exists, the user sees the same screen. When tmux reports no such session, the user ended the shell, and
  the WebSocket closes with "shell ended", as today. When the box is not running, the WebSocket closes with "box
  stopped": opening a terminal starts a stopped box, as today, but a reattach never does. When the spawn fails for
  another reason, the orchestrator tries once more after five seconds and then closes the WebSocket with the error.
• File service: see 8.2.
• Login: a login flow is short and has its own timeout (LOGIN_TIMEOUT_MS). An ended stream fails the flow, and
  the user starts it again.

An open terminal tab keeps the idle reaper from stopping a box. This is the same on Docker, and it is not part of this
concept.

13. Logins

A login runs a CLI of a harness, for example claude setup-token, in a short-lived container from the box image. Today,
login.ts has a separate LoginRuntime for this. With this concept, it uses startLogin of the runtime, and spawn
and remove of the Login handle.

A login container has the opposite network contract of a box: it reaches the public internet directly, not through
the proxy, and it has no box storage. It does not use the proxy because the proxy refuses a real credential that is
not the placeholder on an intercepted host, and a login exists to obtain one. Its home is temporary, so the credential
material ends with the login container.

• Docker: the default bridge network and a tmpfs home, as today. The bridge can also reach the host and the LAN, and
  Docker gives no way to filter that for one container. This is a known difference.
• Kubernetes: a pod with an emptyDir home in memory, and a NetworkPolicy that allows egress to the internet except
  the private ranges that the proxy already blocks as destinations (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
  100.64.0.0/10, 169.254.0.0/16 and the IPv6 counterparts), plus the cluster DNS. The pod and Service ranges of
  nearly every cluster are inside these, so no setting for them is necessary. A cluster with public pod addresses is
  not covered. The policy denies all ingress.

14. Kubernetes deployment

The Kubernetes runtime and its manifests must also meet these requirements:

• Box pods and login pods set automountServiceAccountToken: false and enableServiceLinks: false.
• Box pods meet the restricted Pod Security Standard: non-root, no privilege escalation, all capabilities dropped,
  read-only root filesystem, seccompProfile: RuntimeDefault. The Docker container has the same hardening today.
• The resource limits BOX_MEM_LIMIT and BOX_CPUS apply to the box pod. /tmp is an emptyDir in memory with
  the size of today's tmpfs.
• BOX_PIDS_LIMIT applies on Docker only. A pod spec has no stable field for a pids limit. The limit is the kubelet
  setting podPidsLimit, which applies to every pod on the node, and the manifests document it as a requirement for
  the node, with the Docker default of 512 per pod, because a fork bomb in a box must not take the node down. A field
  per pod, spec.resources.limits.pids, exists as alpha (KEP-6063, feature gate PerPodPIDLimit). Once it is
  stable, the runtime sets it from BOX_PIDS_LIMIT. The implementation opens a ticket that tracks the status of the
  KEP.
• The box pod is Burstable: its requests are smaller than its limits, because most boxes are idle most of the time,
  and the scheduler packs them by their idle use. The requests are the settings BOX_MEM_REQUEST (default 1 GiB) and
  BOX_CPU_REQUEST (default 0.5 CPU). An idle box with a connected adapter and the file service uses about 400 MiB
  and far below a tenth of a CPU. Docker has no requests and ignores the two settings. A box that uses more than its
  request can be evicted when the node runs out of memory (9.4), and a node can take more boxes than its memory holds
  at their full limits. That is the trade the operator makes with the two settings.
• A NetworkPolicy lets only the ingress controller reach the orchestrator API. The API has no authentication of its
  own, on both runtimes, and the deployment puts authentication in front of it. On Docker, this is the reverse proxy
  of the operator, as today. On Kubernetes, the Ingress carries it, for example an auth annotation of the ingress
  controller, or the Ingress is not exposed and the user reaches the API with a port forward. The manifests ship the
  Ingress with a placeholder for the authentication, and the documentation says that an Ingress without it is not
  safe. The WebSocket upgrade on /ws passes through it.
• The orchestrator is one replica with the Recreate strategy, because of the lock on its data directory.
• The orchestrator has the RBAC rights for pods, pods/exec, claims and network policies in its namespace, and no
  other rights.

15. Tests

• The file service is plain Node on a directory. Its unit tests need no runtime.
• One conformance test suite for the Runtime interface runs against a real Docker and a real kind cluster. It
  checks:
  • storage: create, exists, remove;
  • lifecycle: ensure, start, stop, state, remove;
  • run, and spawn with and without a TTY;
  • the file service end to end;
  • that BOX_UID can write the three mounts of a new box;
  • isolation: the proxy front door can be reached; the internet, the control port, the API server and the
    orchestrator cannot;
  • inventory.
• kind v0.24.0 and later enforces NetworkPolicy with its default CNI (kube-network-policies). Calico is not
  necessary in CI.
• tests/smoke-test.sh gets a Kubernetes variant that runs in CI, as the Docker variant does.

16. Risks and items to test

17. Implementation steps

Each step is a separate pull request. Steps 1 to 8 are useful on Docker alone.

1. Process list with ps in the box; remove docker top. The ps asks for pid,ppid,etimes,args, so the age of a
   process stays, which the work list of the dashboard shows. The kill and the background probe then share one
   command.
2. Proxy URL from the runtime, control channel from EGRESS_CONTROL_URL.
3. The terminal attaches again when its exec ends.
4. The file service in the box image, used for the review reads. Then the writes and the attachments. Then the disk
   usage. These can be two or three pull requests.
5. The agent set through installAgentSet, and the home setup through boxes-setup; remove agents/<id> and the
   /boxes/agent mount. The ready state.
6. The home seed in the entrypoint; remove the helper container.
7. The Runtime interface with the Docker implementation only. Storage, login, subnet pool, proxy attachment,
   container repair and the image footer move behind it. The columns container_id, network_name, subnet and
   image go away with a migration. No change of behavior.
8. Conformance tests against Docker in CI.
9. The Kubernetes implementation, the manifests, and kind in CI.