feat(gateway): consistent driver usage after auto-detection #4844

User Story

A user had Docker installed when they installed OpenShell, and they started using OpenShell without setting an explicit compute_driver in the gateway configuration TOML. They created various sandboxes and the gateway always auto-detected and used the docker driver.

Later, they happened to install Podman. Since the gateway prioritizes podman over docker in driver auto-detection, the gateway now auto-selects podman, meaning it can no longer find the existing sandboxes the user had created, and those sandbox records are pruned. The user didn't expect to lose their existing sandboxes just because they installed a seemingly-unrelated service on their system.

Problem Statement

If gateway.toml does not have a compute_driver defined and there is no environment variable override, then the OpenShell gateway auto-detects supported drivers and selects one each time it starts. It does not cache or pin the driver that it previously selected. That means that if the state of the system happens to be different when the gateway service later (re)starts, it may choose a different driver, and thus prune sandbox records which were created when the previous driver was used.

Impact / Why This Matters

For users who have not set an explicit compute_driver value in their gateway.toml, they risk having the gateway auto-select a different compute driver and thus prune their existing sandbox records unexpectedly.

Two situations where this could happen would be:

• The user had Docker already installed, they installed OpenShell and it auto-detected the docker driver, then later they happened to install Podman, which OpenShell prioritizes above Docker in driver auto-detection, so they lose their existing sandbox records
• The gateway service starts before before the previously-used compute backend is ready, e.g. the gateway service races with the Docker or Podman service starting, so doesn't detect that one or the other is supported, and chooses a different driver, so the existing sandbox records for the previous driver are pruned

In the former case, all that would be needed is preference for the previous auto-detected driver when multiple are supported. The latter is different and more difficult to work around, since the issue is that the previously-used driver is (temporarily) not detected as supported.

Another consideration is that if we want to change the precedence of compute drivers in the future, such as by making the VM driver the default, making the gateway favor auto-selecting the same driver it previously selected would allow that migration without users losing sandbox records in the process.

Proposed Design

From the user's perspective, the gateway should choose the same compute driver it previously selected via auto-detection. If the OPENSHELL_COMPUTE_DRIVER environment variable is set or compute_driver is defined, it should use those (as it currently does). But if the gateway falls back to auto-detection, users should still expect their sandbox records to exist, so the gateway needs to auto-select the same driver it previously used, when possible.

I see two potential solutions:

1. When the gateway starts and falls back to compute driver auto-detection because there's no existing setting or environment variable, write the result as compute_driver in gateway.toml

This is the more explicit option. After auto-selecting a driver, the gateway will always use that same driver in the future (except for environment variable overrides, etc.) It is clear to the user what driver the gateway chose, and they can change that selection by editing the configuration value, as documented. This way there's no chance of the gateway auto-selecting another driver later and pruning sandbox records.

The con is that if the system state changes in a way which prevents the gateway from using the driver that was previously auto-detected and set in the config value, the gateway will fail to work until the user manually edits the config file. This could occur if e.g. Podman was previously used but it is uninstalled for some reason. If the gateway didn't write the selection to gateway.toml then it could use auto-detection to choose some other supported driver, though it would then prune existing sandbox records.

2. When the gateway starts and falls back back to compute driver auto-detection because there's no existing setting or environment variable, write the result to a separate driver selection cache file and make a best-effort attempt to use that driver in the future

If OPENSHELL_COMPUTE_DRIVER is set or gateway.toml defines compute_driver, then use that, regardless of auto-detection or a previously-cached value. But in the event that neither of those are set, do the current driver auto-detection/selection, and cache that selection to a file alongside the gateway DB. Subsequently when falling back to driver auto-selection, try to use the same driver which was previously cached. Don't write or use a cached value if the environment variable or toml value is explicitly defined.

This is more implicit than approach (1), as caching the driver auto-selection decision is not something we expect users to be aware of or change, and it's not clear via gateway.toml which driver the gateway auto-selected, so the user must figure this out if they want to set the same value explicitly in gateway.toml.

Another key difference is the "best effort" approach. If the cached driver is no longer supported, then the gateway should auto-select a new supported one, and write that selection to the cache. This has the pro of ending up with a working gateway in the event that the previously-used driver is permanently unsupported (docker/podman uninstalled, or lost /dev/kvm access). The con is that then temporary failure to reach compute backends (docker/podman service still starting as gateway starts) could cause the gateway to auto-select a different driver and prune all the existing sandbox records, where if it just waited a bit it could have used the same driver as previously and maintained all sandboxes.

The "best effort" part could be removed and we could handle this cached value more like the explicit gateway.toml setting where the gateway always uses the cached driver (in the absence of OPENSHELL_COMPUTE_DRIVER and compute_driver), but the issue is then if the cached driver becomes unsupported for some reason, it's less clear how users should identify which driver the gateway is trying to use, and how to tell it to use a different one. So I favor either doing "best effort" with the cached driver, or explicitly setting the driver in gateway.toml.

Acceptance Criteria

• Assuming OPENSHELL_COMPUTE_DRIVER and compute_driver are not specified, when an existing gateway previously used a particular auto-selected compute driver and that driver remains supported, the gateway should continue to auto-select that compute driver, even if other higher-precedence drivers become supported

Alternatives Considered

I proposed two alternatives in the Proposed Design section. I slightly prefer just writing the compute_driver setting to gateway.toml following auto-selection, this feels like the simplest and clearest.

Agent Investigation

No response

Checklist

• I've reviewed existing issues and the published docs
• This is a design proposal, not a "please build this" request