Inference route API keys stored in plain object store #21

Activity

1. pimlock

   added this to the march-mvp milestone on Feb 27, 2026
2. pimlock

   added

   topic:securitySecurity issuesSecurity issues

   on Feb 27, 2026
3. johntmyers commented on Mar 5, 2026

   

   johntmyers

   on Mar 5, 2026

   Collaborator

   More actions

   🔒 security-review-agent

   Security Review

   Determination: Legitimate concern

   Summary

   Inference route API keys are stored as plaintext within protobuf-encoded blobs in the SQLite/Postgres object store with no field-level encryption. All API responses (create, update, list, get_sandbox_inference_bundle) return the full api_key without redaction, meaning any caller with gRPC access can read all stored inference provider keys.

   Severity Assessment

   • Impact: High — Compromise of the database (file theft, backup exposure, SQL injection in a future code path) or unauthorized gRPC access exposes all third-party inference API keys (e.g., NVIDIA, OpenAI, Anthropic keys). These keys grant direct access to paid inference endpoints.
   • Exploitability: Medium — Requires either database-level access (file system access to SQLite DB, database credentials for Postgres, or access to backups) or authenticated gRPC access to the ListInferenceRoutes / GetSandboxInferenceBundle endpoints. No special privileges beyond basic API access are needed to call these endpoints.
   • Affected components:
     • crates/navigator-server/src/inference.rs — create_inference_route (line 100), update_inference_route (line 141), list_inference_routes (lines 163-189), resolve_sandbox_inference_bundle (lines 274-278), list_sandbox_routes (line 322)
     • crates/navigator-server/src/persistence/mod.rs — put_message (lines 259-270) stores plaintext protobuf blobs
     • crates/navigator-core/src/proto/navigator.inference.v1.rs — InferenceRouteSpec.api_key (line 14) is a plain string field in the protobuf schema

   Attack Scenario

   1. Attacker gains read access to the SQLite database file (e.g., via a path traversal vulnerability, container escape, backup exposure, or compromised host).
   2. Attacker decodes any row in the objects table where object_type = 'inference_route' using the known protobuf schema.
   3. All inference API keys are extracted in plaintext, granting the attacker direct access to upstream inference providers (NVIDIA NIM, OpenAI, Anthropic, etc.) — potentially incurring costs or exfiltrating data.

   Alternative scenario: An attacker with any gRPC client access calls ListInferenceRoutes and receives all API keys in the response, no database access needed.

   Remediation Plan

   1. Encrypt at rest: Introduce field-level encryption for api_key before storage. Use an envelope encryption scheme (e.g., AES-256-GCM with a key derived from a server-managed secret or external KMS). Decrypt only when the key is needed for outbound inference requests.
   2. Redact on read: Strip or mask api_key in all gRPC responses (InferenceRouteResponse, ListInferenceRoutesResponse). Only the GetSandboxInferenceBundle internal path (used by the router to make outbound calls) should receive the decrypted key, and ideally over an internal-only channel.
   3. Restrict access: Add authorization checks on inference route management endpoints so that only admin-level callers can create/update routes, and no caller can read back raw keys.
   4. Audit logging: Log access to inference route secrets for forensic traceability.

   Additional Notes

   • The navigator-router crate already redacts api_key in its Debug impl for ResolvedRoute (config.rs:42), showing awareness of the sensitivity — but this protection does not extend to the persistence layer or gRPC API responses.
   • Provider credentials in grpc.rs have a similar pattern (stored in the same object store), which may warrant a separate but related review.
4. johntmyers

   added

   state:review-readyReady for human reviewReady for human review

   on Mar 5, 2026
5. drew

   assigned

   pimlock

   on Mar 5, 2026
6. pimlock commented on Mar 6, 2026

   

   pimlock

   on Mar 6, 2026

   CollaboratorAuthor

   More actions

   Resolved by #146.

   InferenceRouteSpec (which contained the api_key field) has been deleted. The persisted InferenceRoute now contains only a ClusterInferenceConfig with provider_name and model_id — no secrets. API keys live exclusively on the Provider records and are resolved at runtime when the server builds the inference bundle. The stored route object no longer contains any sensitive material.
7. pimlock

   closed this as completedin #146on Mar 6, 2026
8. factory-octavian

   added a commit that references this issue on Mar 31, 2026

   fix: openshell naming migration and launchable start up issues (NVIDI…

   ...

   9286f7d
9. maxamillion

   added a commit that references this issue on Apr 7, 2026

   Merge pull request NVIDIA#21 from LobsterTrap/merge/align-midstream-w…

   ...

   9a4c43a