Repository navigation
Simplify inference routing: introduce inference.local and remove implicit catch-all #133
Description
Summary
Simplify the inference routing system by removing the implicit catch-all mechanism and replacing it with an explicit inference.local hostname addressable inside every sandbox. Inference configuration moves from per-route CRUD to cluster-level config backed by the existing provider system.
Context
The current inference routing has two paths:
- Direct allow — Network policy explicitly allows traffic to a specific endpoint (e.g.,
api.anthropic.com). Works for any endpoint, not inference-specific. - Implicit catch-all — Requests that aren't directly allowed but are detected as inference calls get silently routed through the privacy router to a configured backend.
The catch-all is confusing. A typo in a policy (e.g., api.entropics.com instead of api.anthropic.com) silently reroutes inference to the local model instead of failing visibly. As John put it: "explicit policies for allowances and then we have this implicit secret inference catch-all which breaks the mental model."
Decisions
- Remove the implicit catch-all — No more
inspect_for_inferenceOPA action. If a request isn't explicitly allowed, it's denied. - Introduce
inference.local— An always-addressable hostname inside every sandbox that routes through our inference router. No credentials needed from the agent's perspective. inference.localdefaults to managed NVIDIA inference — If no local model is deployed (e.g., on Brev/CPU), the router points to managed NVIDIA endpoints. When a local model is available, it switches over.- Direct allow unchanged — Explicit network policy allows (e.g., Claude → Anthropic) continue as-is. The router is for "your custom agent" inference.
- Single model override — Router rewrites the model name from client to whatever is configured. Client-specified model is ignored.
- Cluster-level inference config — How
inference.localroutes is configured at the cluster level, not per-sandbox. Config is: provider name + model name. - Providers as the credential mechanism — Instead of routes carrying API keys, use the existing provider system for secure credential injection. New providers:
openai,anthropic,nvidia(all API-key-only for now). Related: Inference route API keys stored in plain object store #21, Inference route API keys exposed via ListInferenceRoutes #20. - Credential injection at supervisor level — Still planned independently of router changes.
Router Flow
- Request from agent hits
inference.local - Router detects the inference API format (OpenAI, Anthropic, etc.)
- Router fetches cluster inference config via gRPC (cached, periodically refreshed) → gets provider name + model name
- Router fetches provider credentials (API key) via the provider system
- Router makes the upstream request with the correct API key and model override
- API format translation (e.g., OpenAI ↔ Anthropic) is out of scope — handled in feat(router): add inference API translation between protocols #90
User Flow
# 1. Create a provider with credentials
nemoclaw provider create --name nvidia_build --type nvidia --from-existing
# 2. Configure cluster-level inference
nemoclaw cluster inference set --provider nvidia_build --model llama-3.1-8b
# 3. Inside any sandbox, agent hits inference.local — just works
curl http://inference.local/v1/chat/completions \
-d '{"model": "anything", "messages": [...]}'
# model is overwritten to llama-3.1-8b, routed to nvidia_build with injected API keyImplementation
Remove
- Remove
nemoclaw inference create/update/delete/listCLI commands - Remove inference route gRPC RPCs (
CreateInferenceRoute,UpdateInferenceRoute,DeleteInferenceRoute,ListInferenceRoutes,GetInferenceRoute) - Remove
InferenceRoute/InferenceRouteSpecdata model from proto (or deprecate) - Remove
inspect_for_inferenceOPA action and the implicit catch-all code path in the sandbox proxy - Remove
routing_hintconcept and route-level API key storage
Add
- Add
inference.localDNS/hostname resolution inside the sandbox (resolve to the router) - Add cluster-level inference configuration (proto fields, storage, gRPC endpoint)
- Add
nemoclaw cluster inference set/getCLI commands (provider name + model name) - Create
openai,anthropic, andnvidiaproviders- Note:
nvidiaone already exists.
- Note:
- Update the router to read cluster config (cached + refreshed) and fetch provider credentials
- Default route: managed NVIDIA inference when no local model is present
- Drop a skill/instructions in the sandbox telling agents about
inference.local
Update
- Update policy files (
dev-sandbox-policy.yaml,policy-local.yaml,policy-frontier.yaml, etc.) to remove inference catch-all rules - Update OPA rego rules — simplify to allow/deny (no tri-state)
- Update architecture docs (
architecture/security-policy.md, etc.)
GTC Priorities
- Primary: Awesome Brev cloud experience with managed endpoints (most users won't have Spark hardware)
- Secondary: In-cluster local model on Spark (aim for it, don't let it block)
Related Issues
- Inference route API keys stored in plain object store #21 — Provider configuration
- Inference route API keys exposed via ListInferenceRoutes #20 — Provider secrets
- feat(router): add inference API translation between protocols #90 — API format translation (OpenAI ↔ Anthropic)
Activity
pimlock commentedon Mar 5, 2026
Implementation Plan
Issue type:
featComplexity: High
Confidence: Medium-High — requirements are explicit; work still spans proto/server/sandbox/CLI/tests.
Summary
Keep the existing
InferenceRouteentity, gateway route CRUD RPCs, and bundle delivery RPC, while re-homing operator UX tonemoclaw cluster inference set/get. Runtime routing becomes explicit throughinference.local; the bundle effectively resolves to one managed route item. As part of this simplification, supportGET /v1/models(and model-discovery flow) for OpenAI/Anthropic-compatible providers.Scope
proto/inference.proto: retain route CRUD + bundle RPCs; evolve schema for managed singleton behavior (droprouting_hint, add version/generation metadata).crates/navigator-server/src/inference.rs: keep service/entity; enforce singleton managed route semantics and monotonic version updates.crates/navigator-sandbox/src/grpc_client.rs+crates/navigator-sandbox/src/lib.rs: continue bundle fetch path; consume single-item bundle and cache revision/version.crates/navigator-sandbox/src/proxy.rs: explicitinference.localhandling for CONNECT and absolute-form HTTP proxy requests; no DNS/hosts injection support.crates/navigator-sandbox/src/l7/inference.rs(or replacement path matcher in proxy): add support forGET /v1/modelsrouting in explicit inference flow.crates/navigator-sandbox/src/opa.rs+crates/navigator-sandbox/data/sandbox-policy.rego: simplify to allow/deny only and remove implicit catch-all interception path.crates/navigator-router/src/backend.rs(+crates/navigator-router/src/mock.rstests): preserve provider-backed auth and forced model override for generation calls; allow model-list passthrough semantics forGET /v1/models.crates/navigator-cli/src/main.rs+crates/navigator-cli/src/run.rs: deprecate/remove standalonenemoclaw inference ...; exposenemoclaw cluster inference set/getonly.getreturns provider + model only.crates/navigator-providers/src/lib.rs(+ provider plugins): supportopenaiandanthropic(API-key providers) with existingnvidia.python/navigator/sandbox.py+python/navigator/__init__.py: adjust SDK surface away from standalone route CRUD exposure.e2e/python/test_inference_routing.py+e2e/python/conftest.py: assertinference.localflow, singleton bundle behavior, andv1/modelssupport.architecture/docs + policy/example files: update implementation docs (internal-only note about managedInferenceRoutebacking).Implementation Steps
InferenceRoute/CRUD/bundle RPCs while reshaping fields for singleton cluster config semantics (including version metadata and removal ofrouting_hint).inference.local) and monotonic version increments on mutation.openaiandanthropicacross provider registry/CLI flows.cluster inference set/get; remove/deprecate standalone inference command UX.inference.localproxy routing (CONNECT + absolute-form HTTP); do not add DNS alias/hosts injection for non-proxy-aware clients.GET /v1/modelssupport in the inference routing path so model discovery works for OpenAI/Anthropic-compatible providers.Test Plan
inference.localhandling for CONNECT + non-CONNECT proxy HTTP, andGET /v1/modelspass-through.inference.local;GET /v1/modelssucceeds for supported providers; missing config returns actionable error; explicit direct endpoint allow behavior remains unchanged.Risks & Open Questions
InferenceRoutevsInferenceRouteSpecvs bundle response).openai/anthropic.cluster inference set --validateprobe in a follow-up (best-effort connectivity/model check without making config writes brittle).Documentation Impact
architecture/inference-routing.md,architecture/security-policy.md,architecture/sandbox.md,architecture/README.md.policy-*.yaml,deploy/docker/sandbox/dev-sandbox-policy.yaml,examples/inference/*.inference.local,cluster inference,v1/modelssupport), but keep managed-InferenceRoutebacking internal.Revision 4 — added
v1/modelssupport requirement, clarified simple set-time validation stance, kept singleton managed InferenceRoute internalsRevision 3 — keep InferenceRoute + CRUD/bundle internals; deprecate standalone CLI UX; switch to singleton managed route semantics with no routing_hint and single-item bundle
Revision 2 — incorporated maintainer clarifications: hard compatibility break accepted; no automatic provider fallback;
inference.localhandled at proxy layer; added exploration result for versioned server-side cluster inference config entityRevision 1 — initial plan
pimlock commentedon Mar 5, 2026
Captured implementation decisions from planning Q&A for future reference:
Q1) Backward compatibility for removing inference route CRUD RPCs/SDK
Decision: Hard break is acceptable now. No migration window is required.
Operational note: Notify internal channel about the breaking API/SDK change.
Q2) Fallback behavior when local model is unavailable
Decision: No automatic fallback based on local model presence/absence.
Behavior: Routing follows the provider configured at cluster level.
Failure mode: If no provider/config is set, return a clear actionable HTTP error with setup guidance (and docs link).
Q3) How should
inference.localbe made resolvable?Decision: Handle this at the proxy layer only; no DNS/hosts injection needed.
Rationale: In proxy mode, requests are routed via
HTTP_PROXY/HTTPS_PROXYand the proxy can matchinference.localdirectly from CONNECT/absolute-form request targets.Non-goal: Supporting non-proxy-aware clients via DNS aliasing is out of scope.
Q4) Should cluster metadata be versioned? How is metadata represented today?
Current state: Existing
ClusterMetadatais local CLI/bootstrap metadata stored in user config (~/.config/nemoclaw/...) and is not server-side cluster runtime config.Decision: Add a new server-side cluster inference config entity with explicit versioning (monotonic version increment on updates), instead of reusing bootstrap metadata.
Q5) Documentation coordination
Decision: Keep architecture/docs updates in lockstep with implementation and align with the parallel user-docs workstream.
pimlock commentedon Mar 5, 2026
Follow-up decision capture (supersedes part of prior Q&A note):
InferenceRouteas the underlying gateway entity.nemoclaw cluster inference set/get; standalonenemoclaw inference ...is deprecated/removed as the UX surface.routing_hintis dropped from the new model; routing is explicit viainference.local.pimlock commentedon Mar 5, 2026
Post-implementation audit: leftover dead code inventory, router/provider architecture design, and phased implementation plan.
https://gist.github.com/pimlock/fca000f2caa8ad803cc20e59a83a74d9
Key design decisions (from discussion):
ResolvedRouteviaAuthHeaderenumInferenceProviderProfileinnavigator-coreconsolidates provider-specific inference knowledge (base URL, protocols, auth style, credential keys)ProviderPlugin(credential discovery) andInferenceProviderProfile(API protocol) are separate concernspimlock commentedon Mar 5, 2026
Additional planning decisions captured:
v1/modelssupportGET /v1/modelsin the explicitinference.localrouting flow.Set-time validation strategy
johntmyers commentedon Mar 6, 2026
We will want to make sure that updates to the NemoClaw CLI skill and Policy Generation skill are handled as well.
fix(server): add field-level size limits to sandbox and provider crea…
pimlock commentedon Mar 6, 2026
Two items deferred to follow-up: #148 (default NVIDIA fallback + sandbox agent instructions for
inference.local).fix(server): add field-level size limits to sandbox and provider crea…
refactor(inference): simplify routing — introduce inference.local, re…
fix(server): add field-level size limits to sandbox and provider crea…
refactor(inference): simplify routing — introduce inference.local, re…