feat: FIPS 140-3 compliance path for TLS and SSH crypto stack #900

Problem Statement

OpenShell's TLS and SSH subsystems use non-FIPS-validated cryptographic libraries and default to non-FIPS-approved algorithms. On FIPS-enabled clusters (common in government, defense, and regulated-industry Kubernetes deployments), this makes OpenShell non-compliant from an audit perspective -- even though the kernel does not block the operations at runtime.

The sandbox has been split into two crates: openshell-supervisor-process (SSH, process lifecycle) and openshell-supervisor-network (proxy, L7, TLS MITM). References below reflect this split.

Specifically:

• TLS (all connections): Uses ring 0.17 via rustls. ring has no FIPS validation path. Default cipher suites include ChaCha20-Poly1305 and X25519 key exchange, neither of which are FIPS-approved.
• SSH (sandbox transport): Uses russh 0.62 with a mix of aws-lc-rs, ed25519-dalek, and curve25519-dalek. The sandbox SSH server hardcodes Ed25519 host keys (openshell-supervisor-process/src/ssh.rs:52). Default negotiation prefers ChaCha20-Poly1305 and Curve25519 key exchange.
• PKI (certificate generation): Uses rcgen 0.13 backed by ring. The default algorithm (ECDSA P-256) is FIPS-approved, but the implementation module is not validated.
• Credential encryption (at rest): The openshell-driver-db-credstore crate performs envelope encryption using ring directly (AES-256-GCM via ring::aead, key material via ring::rand::SystemRandom). This is the most audit-exposed crypto surface outside of TLS.
• JWT signing: Sandbox JWTs use EdDSA (Ed25519) for signing and validation. Ed25519 is approved only under FIPS 186-5 with a validated module.

FIPS-enabled RHEL 9 / OpenShift 4.x clusters enforce FIPS 140-3 via system-wide crypto policies. Processes using non-validated crypto modules fail compliance audits regardless of the algorithms selected. There are no existing FIPS-related issues in the tracker.

This is complementary to #899 (Platform mode / restricted SCC support) -- FIPS clusters are a subset of the managed Kubernetes deployments that issue addresses.

Proposed Design

Add a workspace-level fips Cargo feature flag that switches the crypto backend from ring to aws-lc-rs in FIPS mode (CMVP certificate #4631), restricts algorithm negotiation to FIPS-approved algorithms only, and documents the SSH layer's validation gap.

Phase 1: Feature-flagged FIPS for TLS + PKI + credential encryption + EdDSA JWT decision

Crypto provider switch -- Five binary-facing sites install the rustls CryptoProvider and would switch based on the feature flag:

// Current (all five sites):
rustls::crypto::ring::default_provider().install_default()

// With --features fips:
rustls::crypto::aws_lc_rs::default_provider().install_default()

The five sites are:

• openshell-server/src/cli.rs:222
• openshell-cli/src/main.rs:2154
• openshell-sandbox/src/main.rs:522 (debug-rpc)
• openshell-sandbox/src/main.rs:568 (main supervisor)
• openshell-sdk/src/transport.rs:186 (SDK crate)

Additionally, openshell-server/src/tls.rs:21 has a direct ring import (use rustls::crypto::ring::sign;) that needs editing even after the provider swap.

Workspace dependency changes:

# Current:
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] }
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] }
rcgen = { version = "0.13", features = ["crypto"] }

# With fips feature:
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "aws_lc_rs", "fips"] }
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
rcgen = { version = "0.13", default-features = false, features = ["aws_lc_rs", "pem"] }

TLS cipher suite restriction -- In FIPS mode, configure the provider to exclude ChaCha20-Poly1305 cipher suites and X25519 key exchange, allowing only:

• TLS 1.3: TLS13_AES_256_GCM_SHA384, TLS13_AES_128_GCM_SHA256
• TLS 1.2: TLS_ECDHE_ECDSA_WITH_AES_*_GCM_SHA*, TLS_ECDHE_RSA_WITH_AES_*_GCM_SHA*
• Key exchange: ECDH-P256, ECDH-P384 (no X25519)

SSH algorithm restriction -- Change the sandbox SSH server's host key from Ed25519 to ECDSA-P256 and configure russh::server::Config::preferred / russh::client::Config::preferred to exclude non-FIPS algorithms:

• Host keys: ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, rsa-sha2-256, rsa-sha2-512 (no ssh-ed25519)
• Key exchange: ecdh-sha2-nistp256, ecdh-sha2-nistp384, diffie-hellman-group14-sha256, diffie-hellman-group16-sha512 (no Curve25519, no post-quantum mlkem)
• Ciphers: aes256-gcm@openssh.com, aes128-gcm@openssh.com, aes256-ctr, aes128-ctr (no ChaCha20-Poly1305)

Credential encryption backend switch -- The openshell-driver-db-credstore crate performs envelope encryption (AES-256-GCM KEK/DEK, key material via ring's SystemRandom). This is the most audit-exposed crypto surface and must be included in Phase 1. The ring imports (ring::aead, ring::rand) switch to aws-lc-rs equivalents under the fips feature flag.

EdDSA JWT decision -- Sandbox JWTs use EdDSA (sandbox_jwt.rs:117, validation at :177, :181-182), keyed by Ed25519 (bootstrap/jwt.rs:42 PKCS_ED25519). Ed25519 is approved only under FIPS 186-5 with a validated module. Decision needed: pin to ES256 in FIPS mode, or confirm coverage in the target aws-lc-fips certificate. Wire-format change either way.

Transitive dependency updates -- Several workspace dependencies pull in rustls and/or ring. Each needs feature flags updated for the FIPS build:

• reqwest: switch from rustls-tls-native-roots to using the globally-installed CryptoProvider
• sqlx with tls-rustls-ring-native-roots: feature name explicitly pins ring, needs updating
• kube with rustls-tls: needs feature flag update
• hyper-rustls: switch from ring to aws-lc-rs feature
• tokio-tungstenite: switch from rustls-tls-native-roots to aws-lc-rs backend

rustls 0.21 leg -- rustls 0.21.12 exists in the dependency graph via hyper-rustls 0.24.2 (AWS SDK path). The 0.21 leg predates pluggable CryptoProvider, so the workspace feature flag won't reach it. Either unify onto rustls 0.23 or account for a second unvalidated ring copy.

Phase 1 testing -- No test today would catch a crypto backend swap or a regression in algorithm restriction. Phase 1 needs negotiated-suite assertions, not just build-flag changes:

• Integration tests that verify the TLS cipher suites actually negotiated in FIPS mode are restricted to the approved set
• Tests that the ring::default_provider() sites are replaced by aws_lc_rs::default_provider() when the fips feature is active
• Negative tests confirming that non-FIPS algorithms (ChaCha20-Poly1305, X25519, Ed25519 host keys) are rejected during negotiation in FIPS mode

Go SDK -- GOFIPS140=v1.0.0 build flag for go 1.24+/1.26 toolchain is the cheapest FIPS win for the Go SDK. The Python SDK's BoringSSL gap is documented but out of scope for this issue.

UBI glibc supervisor image -- The supervisor currently uses an Alpine/musl static build (Dockerfile.supervisor). A musl static build cannot dynamically link a system OpenSSL FIPS provider. A UBI-based FIPS image variant needs a separate glibc build. The feat(build): add glibc-static supervisor libc variant commit is a step toward this, but a full UBI glibc variant is needed for the system-OpenSSL FIPS path.

Phase 2 (deferred): SSH transport FIPS validation

Phase 1 restricts SSH to FIPS-approved algorithms but the underlying implementations (ed25519-dalek, p256, aes from RustCrypto) remain non-validated modules. This is a known gap. The SSH transport only operates within the cluster's mTLS boundary (gateway-to-sandbox), providing defense-in-depth rather than being the primary trust boundary.

The NSSH1 nonce/HMAC handshake has been removed. SSH authentication is now unconditional (auth_none and auth_publickey both return Auth::Accept), with trust established via the unix socket (SO_PEERCRED) or the abstract socket in sidecar topology. This strengthens the Phase 2 deferral argument: the SSH layer carries no authentication of its own and operates entirely inside the mTLS boundary, making it even more clearly defense-in-depth.

If strict auditors require validated modules for the SSH layer, Phase 2 options include:

• Upstream russh support for aws-lc-rs as its crypto backend
• Replacing the embedded russh server with an OpenSSH subprocess (significant architecture change given the deep integration at ssh.rs -- 1700+ lines of process spawning, PTY management, channel handling, SFTP subsystem)

Note: ssh_tunnel.rs was deleted (#1029). russh is now 0.62 (was 0.57 in the original investigation).

Scope boundaries:

• The fips feature is off by default -- current behavior is preserved
• Phase 1 achieves FIPS-validated crypto for all TLS operations (the external-facing attack surface), credential-at-rest encryption, and FIPS-approved algorithms for SSH
• Phase 1 includes the EdDSA JWT decision (pin to ES256 or confirm aws-lc-rs coverage)
• Phase 1 includes negotiated-suite test assertions
• Phase 1 explicitly documents the SSH validation gap
• Phase 2 is deferred to actual audit requirements

Alternatives Considered

1. System OpenSSL for everything -- Replace rustls with the openssl crate and russh with libssh2 or OpenSSH subprocess. True FIPS validation for all operations via RHEL 9's OpenSSL 3.x (CMVP fix(vm): bump libkrun to v1.19.6 #4282). Rejected for Phase 1: massive rewrite, loses rustls memory safety guarantees, adds system library dependency, and significantly complicates cross-platform builds.

2. Partial compliance with documented exceptions -- FIPS for TLS only, document SSH as internal-only transport. This is essentially what Phase 1 achieves, but framed as the complete solution rather than a stepping stone. May not satisfy strict auditors.

3. No FIPS support -- Require FIPS-mode clusters to use custom crypto policy exceptions for OpenShell pods. Not viable for enterprise adoption in regulated environments.

4. gVisor RuntimeClass -- gVisor provides its own syscall interception and could theoretically handle crypto at the runtime level. Not applicable -- gVisor intercepts syscalls, not userspace crypto library calls.

Agent Investigation

Investigation performed with a coding agent pointed at the repo. Skills loaded: create-spike, generate-sandbox-policy. The agent traced every crypto dependency, configuration point, and algorithm choice across the workspace.

Crypto dependency map

The TLS and SSH subsystems use different crypto backends -- a critical finding for the migration path:

TLS path (all connections):
  rustls 0.23.38 -> ring 0.17
  rcgen 0.13 -> ring 0.17
  rustls-webpki -> ring 0.17
  quinn-proto -> ring 0.17

SSH path (sandbox transport):
  russh 0.62 -> aws-lc-rs
  russh 0.62 -> ed25519-dalek -> curve25519-dalek
  russh 0.62 -> aes, cbc, ctr (RustCrypto symmetric)
  russh 0.62 -> p256, p384, p521
  russh 0.62 -> libcrux-ml-kem (post-quantum)

Credential encryption path:
  openshell-driver-db-credstore -> ring 0.17 (aead, rand)
  openshell-driver-db-credstore -> sha2 (RustCrypto, for key ID)

Code references

Non-FIPS algorithm inventory

Existing FIPS awareness: Zero. The only mention of "FIPS" in the codebase is in an OCSF schema JSON referencing NIST FIPS 199 (information classification standard, unrelated to crypto).

Dead dependency: openshell-server/Cargo.toml:102 declares hmac = "0.12" with no remaining .rs usage in the server crate -- leftover from the removed NSSH1 handshake. Should be cleaned up regardless of FIPS work.

SSH RNG note: rand::rng() (userspace ChaCha12, OS-seeded) generates the SSH host key at ssh.rs:51. FIPS posture wants a validated DRBG for key material.

SHA-1 note: openshell-supervisor-network/src/l7/rest.rs:2274 uses SHA-1 for RFC 6455 Sec-WebSocket-Accept. This is protocol-mandated, not a security function. Permissible but will trip scanners; needs a documented exemption.

Test-only RSA note: rsa 0.9 at openshell-server/Cargo.toml:131 is a dev-dependency for test RS256 keys. Not in the shipped dependency graph.

Feature flag patterns: The codebase already uses workspace-level feature propagation (bundled-z3, dev-settings) and platform-conditional compilation via #[cfg(target_os = "linux")]. A #[cfg(feature = "fips")] pattern would be consistent.

Risks & open questions:

1. aws-lc-rs FIPS build requires CMake + Go, adding build toolchain complexity
2. russh's internal crypto (ed25519-dalek, p256, RustCrypto AES) is not FIPS-validated regardless of algorithm selection -- Phase 1 documents this gap
3. Does russh have upstream plans for an aws-lc-rs or FIPS backend?
4. Cross-compilation from macOS to linux/amd64 for FIPS container builds may require remote builds
5. SSH host key change from Ed25519 to ECDSA-P256 changes fingerprint -- sandboxes are ephemeral, so should not cause persistent trust issues
6. The current tree has aws-lc-rs but no aws-lc-fips-sys -- the FIPS variant is a distinct sys crate. Verify that the target aws-lc-fips-sys build matches CMVP feat(services): use stable hash-based service hostnames to relax name limits #4631
7. Transitive deps (reqwest, sqlx, tokio-tungstenite, hyper-rustls, kube) each need verification with aws-lc-rs provider
8. Single fips feature flag vs separate fips-tls/fips-ssh for phased rollout?
9. The musl supervisor image blocks the system-OpenSSL FIPS path; a UBI glibc variant is needed for production FIPS compliance
10. The rustls 0.21 leg (via hyper-rustls 0.24.2, AWS SDK path) predates pluggable CryptoProvider and will carry an unvalidated ring copy even after the workspace feature flag switch
11. EdDSA JWT wire-format change: pinning to ES256 in FIPS mode changes the JWT signing algorithm, requiring coordinated rollout between gateway and sandbox components

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request