Repository navigation
feat(router): add inference API translation between protocols #90
Description
Problem
Today the router requires an exact protocol match between the client SDK and the backend route. If an agent uses the OpenAI SDK (POST /v1/chat/completions), only routes with openai_chat_completions in their protocols list are eligible. Similarly, an Anthropic SDK client can only hit anthropic_messages routes.
This means:
- Route operators must provision backends matching every SDK their agents might use
- Agents are constrained to SDKs that match available backends
- Adding a new backend provider requires all agents to adopt its SDK, or the operator must run parallel routes
Proposed Solution
Add an API translation layer in the router so that a request detected as one protocol (e.g., openai_chat_completions) can be transparently translated and forwarded to a route speaking a different protocol (e.g., anthropic_messages), and the response translated back.
Key design considerations
- Route selection: When no exact-match route exists for the detected protocol, the router should attempt translation to a compatible protocol that does have a route
- Request translation: Convert request body schema between protocols (e.g., OpenAI
messagesarray → Anthropicmessagesformat,max_tokenssemantics, tool/function calling schemas) - Response translation: Convert response body back to the client's expected schema (e.g., Anthropic response → OpenAI
ChatCompletionshape) - Streaming: Handle SSE streaming translation (
data: [DONE]vs Anthropic event types) - Feature parity: Some features don't map 1:1 (e.g., OpenAI
logprobs, Anthropicthinking). Define a strategy for unsupported features (error, best-effort, drop silently) - Header translation: Already partially exists (auth header rewriting per protocol). Extend to cover other protocol-specific headers (e.g.,
anthropic-version)
Scope of initial translation pairs
At minimum:
openai_chat_completions↔anthropic_messages
Nice to have:
openai_responses→ other protocolsopenai_completions→ chat-based protocols
Where this fits
The translation should live in navigator-router, likely as a new module between detect_inference_pattern() (in navigator-sandbox) and proxy_with_candidates(). The router already rewrites headers and model IDs in backend.rs — translation extends this to full schema transformation.
Acceptance Criteria
- A client using the OpenAI SDK can be routed to an Anthropic-compatible backend (and vice versa) without the client changing SDKs
- Streaming responses are translated correctly
- Non-translatable features produce clear errors rather than silent data loss
- Existing exact-match routing is unaffected (no performance regression for same-protocol routes)
- Translation pairs are extensible for future protocols
Activity
pimlock commentedon Mar 6, 2026
Exploration Notes: Architectural Approach
Intermediate Representation (IR) Pattern
Instead of building an N×M matrix of direct translators, use an IR pattern powered by the
genaicrate.Flow:
async-openaicrateasync-openaistructs →genai::chat::ChatRequest/ChatMessage(handles structural normalization, e.g., extracting system prompts for Anthropic)genaiclient, which handles provider-specific HTTP protocolsgenairesponse →async-openairesponse struct, serialize to JSON, return to clientDeterministic Routing (No Model-Name Guessing)
genaiby default infers the provider from the model string (e.g.,claude-3→ Anthropic). We bypass this since the same model name may exist across providers.Solution: Configure
genai::Clientwith a customModelMapperusing a routing key syntax:"{provider_name}|{model_name}"(e.g.,"anthropic|claude-3-5-sonnet-latest"). The mapper explicitly assigns theAdapterKind(protocol) and passes the raw model name to the provider.Custom Endpoints
For routing to custom infrastructure (e.g., local vLLM cluster with OpenAI-compatible API), use
genai'sServiceTargetResolveralongside theModelMapperto dynamically rewrite Base URL and Auth headers per-request.Open Implementation Items
async_openai::types::ChatCompletionRequestMessagevariants →genai::chat::ChatMessagestream: true, consumegenai'sChatStreamEvent, map toasync_openai::types::CreateChatCompletionStreamResponse, push to client via SSEcache_control). Decide between auto-injection, stripping, or lowest-common-denominator APIMerge pull request NVIDIA#90 from opendatahub-io/fix/konflux-rust-bui…