refactor: clean up gateway.toml config schema inconsistencies #2792

Problem Statement

The gateway.toml configuration schema has accumulated 21 distinct inconsistencies across the gateway core and its four compute-driver tables (kubernetes, docker, podman, vm): misplaced fields (Kubernetes-specific settings living at the gateway level), inconsistent field naming for the same concept across drivers, overloaded 0 sentinel values instead of Option<T>, duplicated TLS fields with no practical use for the duplication, and feature parity gaps (proxy config, AppArmor, SPIFFE support present on some drivers but arbitrarily missing from others). This makes the config surface harder to document, harder to reason about, and easy to misconfigure. This issue proposes a structured cleanup.

Technical Context

[openshell.gateway] holds both true gateway concerns (listeners, TLS, auth) and driver-inheritable defaults (image names, TLS paths, K8s-only settings) that flow into driver tables via an allowlist-based inheritance mechanism. Because only one compute driver is active per gateway process, several fields designed for "shared defaults across drivers" provide no real value and only add indirection. Driver tables also evolved independently, so equivalent concepts (SSH socket path, gRPC endpoint, image pull policy, PID limits) have different names, types, and default-value conventions per driver.

Affected Components

Technical Investigation

Architecture Overview

Config loading happens in crates/openshell-server/src/config_file.rs:

1. TOML is parsed into ConfigFile (schema version validated against SCHEMA_VERSION = 1; secrets like database_url are rejected if present in the file).
2. driver_table() (config_file.rs:331-360) merges [openshell.gateway] defaults into the active driver's table.
3. Each driver has an allowlist of inheritable keys, inheritable_keys() (config_file.rs:362-386):
   • Kubernetes: namespace, default_image, supervisor_image, client_tls_secret_name, service_account_name, host_gateway_ip, enable_user_namespaces, sa_token_ttl_secs
   • Docker: sandbox_namespace, default_image, supervisor_image, host_gateway_ip, guest_tls_ca, guest_tls_cert, guest_tls_key
   • Podman: default_image, supervisor_image, host_gateway_ip, guest_tls_ca, guest_tls_cert, guest_tls_key
   • VM: default_image, guest_tls_ca, guest_tls_cert, guest_tls_key
   • Remote/custom drivers: no inheritance
4. gateway_inherited_value() (config_file.rs:388-402) copies the value, and this is where a single gateway key (sandbox_namespace) maps to different meanings per driver (K8s namespace vs. Docker sandbox_namespace/label) - a source of confusion in its own right.
5. Precedence overall: CLI flag > OPENSHELL_* env var > TOML file > built-in default (documented in docs/reference/gateway-config.mdx:16-19).

Because only one compute driver runs per gateway process (despite compute_drivers being typed as Vec<String>, config_file.rs:115 / config.rs:480), the inheritance model exists mainly to avoid repeating a handful of values, but has become a source of misplaced ownership (Kubernetes-only settings sitting at the "shared" gateway level) rather than genuine cross-driver sharing.

Code References

Current Behavior

• 0-overload pattern appears in at least 3 independent fields (gateway_jwt.ttl_secs, sandbox_pids_limit in both Docker and Podman, health_check_interval_secs in Podman), each reimplementing the same "0 means special case" convention with no shared idiom, and no way to distinguish "unset" from "explicitly zero."
• Inheritance key remapping: the same gateway key can mean different things per driver (sandbox_namespace maps to K8s namespace vs. Docker label), which is confusing when reading gateway.toml in isolation.
• TLS field triplication: guest_tls_ca/cert/key exist at the gateway level (for inheritance) and are redundantly redeclared in Docker, Podman, and VM driver structs, even though only one driver is ever active.
• Image pull policy has three different representations across drivers with no shared validation, so typos in Kubernetes/Docker configs are only caught (or not caught) at runtime by the container runtime itself, not at config-parse time.
• Feature parity gaps: proxy config (Podman, Kubernetes only), AppArmor (Kubernetes only), SPIFFE (Kubernetes only) are runtime-agnostic capabilities arbitrarily scoped to specific drivers, apparently because that's where they were first implemented rather than because of a technical constraint.

What Would Need to Change

Config file schema (config_file.rs)

• Remove Kubernetes-only fields (sandbox_namespace, service_account_name, enable_user_namespaces) from GatewayFileSection; move to [openshell.drivers.kubernetes] only, updating inheritable_keys() accordingly.
• Resolve compute_drivers: Vec<String> vs. single-active-driver reality - either narrow the type to a scalar or explicitly validate/document the "first wins" semantics.
• Add serde aliases for any renamed keys to support a migration window.

Core config types (config.rs)

• Convert 0-overloaded fields to Option<T> (gateway_jwt.ttl_secs, sandbox_pids_limit in Docker/Podman, health_check_interval_secs in Podman), with None meaning the "special" behavior and explicit validation rejecting nonsensical Some(0) where relevant.
• Rename GatewayInterceptorConfig.max_response_bytes to align with middleware's max_body_bytes (or vice versa) for naming consistency.

Driver configs

• Docker: rename sandbox_namespace to sandbox_label; adopt a shared typed ImagePullPolicy enum; add proxy config fields matching Podman; add app_armor_profile.
• Podman: rename sandbox_ssh_socket_path to ssh_socket_path.
• VM: rename openshell_endpoint to grpc_endpoint; reconcile sandbox_uid default with Kubernetes' 1000; add proxy config fields; add app_armor_profile.
• Kubernetes: no field changes needed, but should be the source of truth for shared types/fields being generalized (typed pull policy, SPIFFE, AppArmor, proxy).

Cross-cutting

• Decide TLS field ownership: consolidate guest_tls_ca/cert/key to gateway-level only, removing the redundant per-driver copies (recommended, since only one driver is active and the paths are host-side).
• Decide grpc_endpoint/openshell_endpoint ownership: move toward gateway-computed/auto-injected endpoint rather than per-driver configuration, with an explicit override escape hatch.
• Generalize SPIFFE workload socket support beyond Kubernetes (Docker/Podman via host socket bind-mount, VM via virtio-vsock).
• Update docs/reference/gateway-config.mdx for every renamed, moved, or retyped field (currently documents ~250+ TOML examples touching these fields).
• Update Helm chart values/templates and packaged default configs (deploy/rpm/gateway.toml.default, Homebrew default config) that reference any changed field.

Alternative Approaches Considered

• Alias lifetime: keep serde aliases for renamed fields permanently vs. a deprecation window (e.g., 2-3 releases) vs. a hard break tied to a major version. This needs a human/product decision, not an agent one.
• 0-overload replacement: Option<T> (idiomatic Rust, recommended) vs. sentinel constants (e.g., u64::MAX) vs. separate enable_* boolean fields. Option<T> is recommended for clarity, but changes TOML ergonomics (must omit the key rather than write = 0).
• TLS field ownership: gateway-only (recommended - single active driver, host-side paths, no case where drivers need different values) vs. keeping per-driver copies for hypothetical future multi-driver support.
• grpc_endpoint auto-detection: full auto-injection by the gateway vs. keeping explicit per-driver config as an opt-in override when auto-detection is insufficient (e.g., custom network topologies). A pure auto-detect-only approach risks removing power-user control drivers currently rely on.
• compute_drivers cardinality: whether OpenShell's roadmap ever intends multi-driver support. If never, collapse to a scalar field; if it's a real future possibility, keep the Vec type but tighten validation/docs now.

Patterns to Follow

• Existing driver validate() methods (e.g., Podman's validate_tls_config(), validate_proxy_config() in podman/config.rs) are the established place to add validation for new Option<T> semantics or shared enum values.
• Podman's ImagePullPolicy enum (podman/config.rs:25-57) is the best existing example of a typed pull-policy and should be promoted to openshell-core as the shared base type, with driver-specific extension (e.g., Podman's Newer variant) layered on top.
• Serde #[serde(alias = "...")] is the established idiom in this codebase for backward-compatible renames.

Proposed Approach

Stage the cleanup rather than attempting all 21 issues in one PR. A natural split: (1) low-risk pure field renames with serde aliases (#6, #12, #13), (2) Option<T> conversions for 0-overloaded fields with validation (#5, #7, #10, #11), (3) feature-parity additions copying existing Kubernetes/Podman patterns to Docker/VM (#9, #15, #18, #21), and (4) architectural moves requiring the most human input (#1-3, #4, #16, #17, #19, #20 - field ownership changes, endpoint auto-detection, shared enum promotion, TLS consolidation). Each stage should update docs/reference/gateway-config.mdx, Helm chart values, and packaged default configs in the same PR, and ship serde aliases for any renamed/relocated key to avoid breaking existing deployments outright.

Scope Assessment

• Complexity: High (overall) - individual issues range from Low (pure renames) to High (field ownership moves, endpoint auto-detection)
• Confidence: Medium - the technical mapping is well understood and file:line-verified, but several design decisions (alias lifetime, TLS/endpoint ownership, compute_drivers cardinality) require human/product input before implementation
• Estimated files to change: ~10-15 core files (config.rs, config_file.rs, 4 driver config modules) plus docs/reference/gateway-config.mdx, Helm chart values/templates, and packaged default configs (RPM, Homebrew)
• Issue type: refactor

Risks & Open Questions

• Backward compatibility is the dominant risk. OpenShell has real deployments with existing gateway.toml files (hand-written, RPM/Homebrew defaults, Helm-generated). Field renames and moves are breaking changes unless serde aliases are added; type changes (0 to Option<T>) change what's valid TOML syntax for a field.
• VM sandbox_uid default change (10001 to 1000) is a semantic behavior change, not just a rename - could affect file ownership expectations in already-running VM sandboxes. Needs explicit call-out in release notes if changed.
• grpc_endpoint auto-detection could remove power-user control for custom network topologies if implemented as auto-detect-only rather than auto-detect-with-override.
• compute_drivers Vec vs. scalar depends on whether multi-driver support is ever on the roadmap - a product decision, not a technical one.
• Helm chart and packaged defaults (deploy/rpm/gateway.toml.default, Homebrew config) must be updated in lockstep with any renamed/moved field, or upgrades will silently stop applying settings.
• Alias retention window (how long to support old field names before removal) needs a human decision - likely tied to release/versioning policy.

Disposition Readiness

• State: state:validated
• Assessment: The investigation confirms all 21 issues against current source with exact file:line references, maps the inheritance mechanism, and surfaces the key design decisions and backward-compatibility risks needed for a human accept/decline call. No further investigation is needed to make that decision.
• Missing evidence: None for disposition. Implementation-time decisions (alias lifetime, TLS/endpoint ownership model, compute_drivers cardinality) are flagged above and should be resolved during build-from-issue planning or via explicit human direction before implementation begins.

Test Considerations

• crates/openshell-server/src/config_file.rs has 30+ existing tests including driver-table inheritance tests (~lines 729-858) that will need updates for any moved/renamed inheritable keys.
• crates/openshell-core/src/config.rs has 40+ tests covering compute driver kind parsing, gateway JWT defaults, interceptor failure policies - default-value tests will need updates for 0 to Option<T> conversions.
• Kubernetes driver config has 56 tests (kubernetes/config.rs), Podman has 31 (podman/config.rs) covering TLS validation, proxy config, runtime limits - new tests needed for any fields ported to Docker/VM.
• A packaged-default "contract test" exists (referenced around config_file.rs:860) validating RPM/Homebrew default configs parse correctly - must be updated alongside any schema change.
• Any staged PR should include: unit tests for new Option<T> validation logic, alias-parsing tests (old name still works, new name works, both present is an error or new-name-wins), and updated inheritance tests for any relocated keys.
• docs/reference/gateway-config.mdx should be treated as a documentation contract - every schema example in the ~1051-line file touching a changed field must be updated in the same PR as the code change.

Created by spike investigation. state:validated means the issue is ready for human disposition; state:needs-info means specific evidence is still required. A human applies state:accepted or places the issue on the roadmap if OpenShell should pursue the work. To queue unattended agent planning, a human applies agent:plan-requested; a direct request to an agent does not require that label.