Website · Guide · Install · Quickstart · Proof · Certified selection
Firstpass is an open-source (Apache-2.0, Rust) proxy that sits between your agent and your LLM provider. Every request opens on the cheapest model in your ladder. Your own check (tests, a schema, a judge) runs on the real output. Pass, and it is served. Fail, and it escalates one rung and is checked again. Every decision leaves a hash-chained receipt anyone can re-derive.
Adopting it is one environment variable. Leaving is unsetting it.
Most model routing decides before the output exists: a classifier reads the prompt and guesses, or a gateway picks on price and availability. When the guess is wrong you find out in production, and nothing explains why.
The only moment you can answer "is this good enough to serve?" is after the answer exists. So Firstpass does not predict. It checks. The cheap model handles what it can prove it handled; the expensive model is spent only when a real check says the cheap answer is not enough. The cost saving is the consequence of that, not the mechanism.
- Route. The request opens on the cheapest rung. No per-prompt classifier picks the model.
- Prove. A gate checks the actual output: your unit tests, a JSON schema, an LLM judge, or self-consistency. A gate that errors abstains; it never passes.
- Escalate. Only on failure: one rung up, budget-capped, then gated again.
- The receipt. Each decision is written to a hash-chained trace that an outside party can re-verify with no proxy and no database in the loop.
What a receipt looks like (illustrative example, not a measured result)
firstpass export writes the log as JSONL; firstpass verify --file receipts.jsonl re-derives the hash chain from genesis. A single altered or reordered receipt breaks the chain at its index and exits non-zero. Downstream outcomes arrive via POST /v1/feedback onto a side table that never alters the sealed record.
All numbers below are from committed artifacts in docs/benchmarks/. Each one is tied to its dataset and ladder; none is a universal promise.
Gated escalation beats serving the cheap model alone. On 974 real MBPP coding tasks (haiku → opus ladder), gating lifted quality by +15.2 points over always-cheap (95% CI [+12.9, +17.5]) and made 0 of 974 tasks worse (artifact). Caveat: on this workload the gate runs asserts that are also shown in the prompt, so it is partly self-grading.
…at lower cost per success than always-top, if you name the ladder. 57% lower with an opus top rung, 12% lower with a sonnet top rung (opus · sonnet). Always-top still wins on raw success by 1 to 2 points (−2.3 on the opus ladder). A saving without its ladder means nothing.
A served-failure bound, earned on a real run. Pre-registered target: at most 10% wrong answers served at 95% confidence (α=0.10, δ=0.05). On 974 MBPP tasks with real test gates the run realized 7.7% served-failure (calibrated risk 5.5%) while serving 82% of requests from the cheap tier. The gate's false-accept rate on that run was 14.7%, stated openly. This is one run on one dataset: it shows the bound is computed from your outcomes, not that any deployment is guaranteed 10% (artifact, method).
Agent workloads: choose the model, not the request. On 500 real SWE-bench Verified agent runs per model (public leaderboard trajectories), no per-request cascade beat always-top. Choosing one model for the workload by evidence did: 374 vs 384 resolved against always-opus (difference not significant) at 71% lower cost. One recorded run per model, one scaffold (numbers).
Decision-rule replay. Replaying decision rules on recorded outcomes, the first-pass rule beat judge-escalate and conformal-cascade rules by +3.3 and +5.1 points (artifact).
Your savings depend on your workload, so every trace records the always-top counterfactual. You measure your number instead of trusting ours.
Reproduce it (each command labels itself and states what it costs)
cargo run -p firstpass-bench # simulation harness (free, self-labeled SIMULATION)
cargo run -p firstpass-bench -- --live # live benchmark (your key)
# the served-failure bound on real MBPP tasks (your key + Docker):
curl -sLO https://raw.githubusercontent.com/google-research/google-research/master/mbpp/mbpp.jsonl
FIRSTPASS_CODING_DATASET=./mbpp.jsonl \
cargo run --release -p firstpass-bench -- --coding-liveThe harness recomputes the bound from your run's gate and oracle outcomes with the same pre-registered α and δ. Provenance rules: docs/benchmarks/README.md.
A router that only ever says yes is not a router. Firstpass can test a cheaper model on your own traffic and refuse it. Add a [route.challenger]; an anytime-valid e-process, graded by your gates, decides certified (no worse than your margin) or halted (worse), with error at most alpha however often you look. It never switches models on its own: a halted challenger is stopped automatically, and adopting a certified one is a human config edit. (ADR 0014, evidence)
We ran it live on real Anthropic calls (about $29.70 total):
- MBPP, 974 pairs: halted. sonnet-5 95.4% vs haiku-4-5 84.6%, with haiku at 0.30× the cost per request. The router refused the cheaper model, halting at pair 137, and halted in 100 of 100 random orders.
- APPS, 290 contest problems: undecided. 65.2% vs 60.7%, with haiku at 0.09× the cost. Undecided in 194 of 200 random orders. That is the right answer: the gap is too close to the margin to call.
- The live run caught three real bugs in Firstpass itself, all fixed with tests: a challenger graded on "served" instead of "passed" had falsely certified haiku (#244); a halted challenger kept spending (#245); a 120 s timeout dropped the 33 longest problems and manufactured a halt (#246).
Paired mode (mode = "paired") re-runs the challenger off the hot path on the same request, under its own daily spend cap. In a replay of recorded paid calls it decided after about 76 to 127 requests, against 222 to 425 for live mode (about 3× fewer; replay).
Verification needs something to verify against. Firstpass is strongest where "correct" is checkable: code with tests, structured output against a schema, extraction. For open-ended prose the gates are a judge or self-consistency, which are weaker. If your output has no check, Firstpass is the wrong tool.
Related approaches and what is borrowed versus new (the verified cascade itself is prior art: FrugalGPT, AutoMix): docs/related-work.md.
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/dshakes/firstpass/releases/latest/download/firstpass-proxy-installer.sh | sh| pip / uvx | pip install firstpass · uvx --from firstpass firstpass-proxy |
| Homebrew | brew install dshakes/tap/firstpass-proxy |
| npm | npm i -g firstpass-proxy (not firstpass: that npm name is an unrelated CLI) |
| Docker | docker run -p 8080:8080 -e FIRSTPASS_BIND=0.0.0.0:8080 ghcr.io/dshakes/firstpass:latest (published on release tags) |
| Cargo | cargo install firstpass-proxy (crates.io; needs a Rust toolchain) |
| Binaries | macOS, Linux, Windows, checksummed — Releases |
Observe mode changes nothing:
firstpass-proxy # watches your traffic, touches nothing
export ANTHROPIC_BASE_URL="http://127.0.0.1:8080" # your agent now routes through firstpass
# use your agent normally; every call gets a receipt: what it would route, what you would saveConvinced by your own numbers? Switch on routing:
cp firstpass.example.toml firstpass.toml
FIRSTPASS_MODE=enforce FIRSTPASS_CONFIG=./firstpass.toml firstpass-proxyOr let Firstpass start the agent already pointed at it (firstpass launch claude; also codex, or openai -- <any OpenAI-compatible command>). It refuses to start if no proxy is listening, or if something that is not Firstpass holds the port.
Leaving is unset ANTHROPIC_BASE_URL.
Let an agent do it. firstpass onboard --apply detects your shell, whether the proxy is running and which agents you have, then does only what is missing. It is idempotent, firstpass onboard alone is a dry run, and firstpass offboard reverses it. firstpass kiosk puts one real request through the ladder and prints the receipt (keyless demo if no key is found). firstpass handshake prints one JSON document for callers with no terminal and never writes. GET /panel is a live receipts view served by the binary itself. Machine-readable setup: llms.txt, AGENTS.md, GET /v1/capabilities, firstpass mcp.
Point the agent at the proxy; nothing else about the agent changes.
| Agent | Route it through Firstpass | Endpoint it hits |
|---|---|---|
| Claude Code | export ANTHROPIC_BASE_URL="http://127.0.0.1:8080" or firstpass launch claude |
POST /v1/messages |
| Codex | firstpass launch codex (sets OPENAI_BASE_URL) |
POST /v1/responses |
| Gemini CLI | firstpass launch gemini (sets GOOGLE_GEMINI_BASE_URL) |
POST /v1beta/models/{m}:generateContent |
| aider | aider --openai-api-base=http://127.0.0.1:8080/v1 … |
POST /v1/chat/completions |
Observe (the default) forwards byte-identical, writes receipts, and relays the upstream's retry-after, request-id, anthropic-ratelimit-* and x-ratelimit-* headers so an agent backs off correctly on a 429. POST /v1/messages/count_tokens is passed through and writes no receipt (it is not a routing decision). Enforce adds gates, escalation and sessions:
- Tool turns are gated, not misread as empty. A reply that only calls a tool passes
non-emptyandjson-validwhen the request declared tools. The built-intool-args-validgate validates every tool call's arguments against the request'sinput_schema(Anthropic) /parameters(OpenAI) and fails on a tool the request never declared. - Sessions need no header. Turns are grouped by, in order:
x-firstpass-session, an agent session header (x-claude-code-session-id,session_id,conversation_id), Claude Code'smetadata.user_idsession,prompt_cache_key, then a salted hash of the conversation prefix plus tool names. Session promotion (a session that had to escalate starts there next turn) and per-session experiments therefore apply to Claude Code, Codex and aider without config. Derived ids are hashed; body-derived ids are computed only on enforce routes, so observe-only deployments never parse the body to derive a session. - Streaming fidelity.
thinking/redacted_thinkingblocks andtool_useare streamed, andstop_reasonis carried across providers. - Credentials stay scoped. A rung's configured
api_key_envalways wins and never falls back to the client's key; a Claude Code OAuth Bearer is accepted on the Anthropic path only.
A route for an agent (validated with firstpass doctor):
[[route]]
match = {} # every request; agent = "claude-code"/"codex"/"gemini-cli" is detected from User-Agent, others send x-firstpass-agent
mode = "enforce"
ladder = ["anthropic/claude-haiku-4-5", "anthropic/claude-sonnet-5"]
gates = ["non-empty", "json-valid", "tool-args-valid"]
[escalation]
max_rungs_per_request = 2
session_promotion = { after_failures = 2, window = "30m" }FIRSTPASS_MODE=enforce FIRSTPASS_CONFIG=./firstpass.toml firstpass-proxy. These gates check structure, not whether the code is right. For that, add your tests as a [[gate]] (Gates).
What the evidence says about granularity. On 500 real SWE-bench agent runs (full eval, $0, replay of public runs), switching models mid-session did not pay on either ladder tested. On haiku to opus: escalate-and-pin cost +$0.309 to +$0.316 per instance versus always-top at the same 384/500 resolved; per-session routing resolved 378/500 against 384/500; a run-level cascade (a proxy for per-call, which that data cannot identify) cost +$0.295 to +$0.340 per instance. Choosing the model per workload was far cheaper (374/500 at $0.219 per instance against 384/500 at $0.754), but non-inferiority at 1 point was not shown. So the practice is: route per workload or route (certified selection, above), keep a session sticky, and do not switch models mid-session. A per-repo pool selection arm scored better still, but it was a post-hoc arm and is exploratory; do not rely on it.
Compatibility, as measured (matrix; observe mode, fake upstream, firstpass 0.9.2): Claude Code and aider completed a turn; Codex is covered by a /v1/responses fixture request only (no standalone binary in the run); Gemini CLI is covered by a /v1beta/models/{m}:generateContent fixture request (no standalone binary in the run); cline and cursor-agent were skipped (interactive or cloud auth). Enforce mode with real CLIs is not in that matrix. More: agent guide.
Firstpass is a proxy in front of your provider calls. Your agent keeps its endpoint; the response is byte-identical to the served rung.
| Endpoint | For |
|---|---|
POST /v1/messages |
Anthropic Messages (Claude Code and anything Anthropic-native) |
POST /v1/chat/completions |
OpenAI Chat Completions and OpenAI-compatible clients |
POST /v1/responses |
OpenAI Responses, including tool calls |
POST /v1beta/models/{m}:generateContent |
Gemini CLI inbound (GOOGLE_GEMINI_BASE_URL) — observe and enforce |
GET /v1/models |
Model discovery for agent CLIs |
Vendors can be crossed mid-ladder: reasoning_effort is carried to Anthropic rungs as thinking, and tool calls survive the /v1/responses round trip.
| Effort | You get |
|---|---|
| None (observe mode) | Firstpass reports what it would route and save. Nothing changes. |
| One sentence (judge gate) | A second model grades each answer against your plain-English rubric. |
| One config line (consistency gate) | The model answers k times; agreement is the confidence (self-consistency, Wang et al. 2022). |
| Your existing tests | The strongest gate: generated code ships only if your suite passes. |
Built-in gates are non-empty, json-valid and tool-args-valid (agent tool calls); everything else is a [[gate]] block. Flaky gates auto-disable on an error budget. Your test runner becomes a gate via scripts/gates/run-tests.py (Gates).
anthropic and openai are built in. Eight more OpenAI-compatible platforms need no [[provider]] block: groq, deepseek, together, fireworks, mistral, openrouter, xai, cerebras. Any other OpenAI-compatible host (Azure, an aggregator, your own Ollama or vLLM) needs a [[provider]]; Gemini (dialect = "gemini"), AWS Bedrock (auth = "aws_sigv4") and Google Vertex (auth = "gcp_oauth") use the same shape. Every variant ships in firstpass.example.toml, guarded by a parse test.
[[route]]
mode = "enforce"
ladder = ["groq/llama-3.3-70b-versatile", "anthropic/claude-sonnet-5"]
gates = ["non-empty"]
[[price]] # required: endpoints ship built in, prices do not
model = "groq/llama-3.3-70b-versatile"
input_per_mtok = 0.59
output_per_mtok = 0.79Prices are deliberately not built in for these platforms: a stale price would write a wrong cost_usd into a tamper-evident receipt and mis-feed your [budget] caps, so the proxy refuses to start without a [[price]] for every rung. firstpass doctor fails loudly on a missing API key.
Verification status. The Anthropic path is live-verified end to end. The OpenAI-compatible, Gemini, Bedrock and Vertex adapters are implemented and offline-tested against recorded wire shapes, pending live verification (roadmap).
- A prompt too large for the cheap rung escalates instead of failing; the receipt records
context_overflowso capacity-forced escalations are not pooled with quality failures. - A session that had to escalate starts there next turn (
[escalation.session_promotion]), with a periodic downward probe so promotion is never a one-way ratchet. - Cached prompts are billed as cached. All three prompt-caching counters are priced, so receipts and
[budget]caps see the real cost. - A conversation that fits nowhere is condensed rather than refused (
[escalation.condense], off by default), only once every rung has overflowed.
The mechanism behind It says no, too:
[route.challenger]
ladder = ["openrouter/minimax-m2.5"] # what you'd like to switch to
percent = 10 # % of sessions served by it, randomized per sessionPost task outcomes to /v1/feedback; firstpass experiment (also GET /v1/experiments and the /panel view) reports certified, halted or undecided. The default margin is 5 points. Paired mode needs a real gate; a route whose only gate is non-empty is rejected. Halt state is per process and rebuilt from receipts at startup; for a global halt across replicas set redis_url under [route.challenger] (needs the redis-cache Cargo feature). Method and limits: ADR 0014.
The verified cache and session promotion are in-process by default, so behind a load balancer a retraction only reaches the replica that got the feedback. Share them through Redis (build with cargo install firstpass-proxy --features redis-cache):
[escalation.verified_cache]
ttl_secs = 900
redis_url = "redis://cache:6379/0"
[escalation.session_promotion]
after_failures = 2
window = "30m"
redis_url = "redis://cache:6379/0"Setting redis_url without the feature, or pointing it at an unreachable server, fails at startup rather than silently falling back.
x-firstpass-mode (or per route / env) selects a profile: cost, balanced, quality, latency, max. Same ladder, different serve threshold and escalation appetite.
- Learned start-rung bandit. UCB1 or Thompson sampling with discounted Beta posteriors (ADR 0007). It predicts where to start; the gate still decides what to serve.
- The bound. Split-conformal (Hoeffding UCB) or Learn-then-Test exact-binomial testing (
firstpass calibrate --method ltt), tracked under drift by adaptive conformal. Prometheus gauges:firstpass_serve_threshold,firstpass_realized_served_failure. - Off-policy evaluation.
firstpass opereplays your logged receipts against a candidate ladder with IPS / SNIPS / DR estimators and confidence intervals: rehearse a policy change before you ship it. - Elastic verification (validated research, phase-1 shipped, not default-on). A cheap k-sample probe decides how much proof to spend (ADR 0008). In a validation study (k=5, n=150) unanimous agreement was 99% oracle-correct and the probe alone decided 77% of traffic, with a traffic split of 12% / 23% / 65% (artifact).
Decision models such as TypeSafe's Jev answer one cheap closed-form question per query and can pick a model tier directly. Firstpass's position is narrower: predict the start, verify what is served. The same cheap signal can choose where the ladder opens, but the gate still runs on every attempt, so a wrong prediction costs money or latency, never an unverified answer.
- Simulation only: in Firstpass's own simulation harness (
cargo run -p firstpass-bench), a Jev-style router that serves unverified output shows a much higher served-failure rate than the gated arm. This is a simulation, not a live result (ADR 0013). [escalation.prior]is experimental and default-off. Replaying the prior on real recorded MBPP outcomes across three ladders, with OpenJev run locally as the prior source (not TypeSafe's hosted Jev), pooled $/success fell −4.6% with served-failure held (replay). The win is ladder-dependent and says nothing about hosted Jev's accuracy.- Blending a learned signal in was worse than the prior alone, so the prior alone stays the recommendation (replay).
- The
decisiongate is NOT-RECOMMENDED. Measured with local OpenJev against hidden-test labels on served MBPP answers, it missed the pre-registered bar; hosted Jev is unmeasured (study).
[escalation.prior]
provider = "typesafe" # the only accepted value today
base_url = "https://api.typesafe.ai" # default; point at a local OpenJev, e.g. http://127.0.0.1:8080
model = "jev-latest"
api_key_env = "TYPESAFE_API_KEY" # required even for a keyless local server (set any placeholder)
timeout_ms = 150 # any error fails open: no prior
strength = 10 # Beta pseudo-count weight of the prior
rungs = [
"rung 0 (claude-haiku) is the least capable tier that fully handles this request",
"rung 1 (claude-sonnet) is the least capable tier that fully handles this request",
]rungs needs exactly one entry per rung of every enforce route's ladder; a mismatch is rejected at parse. Run OpenJev locally (razorback16/openjev): OPENJEV_BACKEND=mlx python -m openjev on Apple Silicon, or docker compose up on NVIDIA. The decision gate is [[gate]] decision = { provider = "typesafe", model = "jev-latest", threshold = 0.6 }; transport errors or unparseable replies abstain, never pass.
Comparison on method, not marketing (RouteLLM, Not Diamond, Martian, FrugalGPT, AutoMix, CP-Router, and others): docs/related-work.md.
Configuration (12-factor, env-driven)
| Variable | Purpose | Default |
|---|---|---|
FIRSTPASS_MODE |
observe | enforce |
observe |
FIRSTPASS_BIND |
listen address | 127.0.0.1:8080 |
FIRSTPASS_CONFIG |
path to firstpass.toml (inert unless set) |
none |
FIRSTPASS_DB |
trace store path | firstpass.db |
FIRSTPASS_RECEIPTS |
best_effort | durable (durable spills receipts to disk under backpressure and drains on boot; the chain stays valid) |
best_effort |
Other endpoints: POST /v1/feedback · GET /v1/capabilities · GET /healthz · GET /metrics. route.mode = "enforce" is what enables enforcement per route.
Multi-tenant deployments add per-tenant auth (Argon2id), rate limits, gate-health scoping and AES-256-GCM key custody, all opt-in and default-off (ADR 0004).
Version 0.10.0, pre-1.0, developed in the open. Published on crates.io (firstpass-proxy, firstpass-core), PyPI, Homebrew, GHCR and GitHub Releases. Shipped: both wire dialects, all gate kinds, start-rung bandit, conformal and Learn-then-Test bounds, OPE, receipts with export and verify, certified model selection. Research or unfinished: elastic verification is not default-on, four provider dialects await live wire verification, and the hosted multi-tenant plane, a soak period and an external security audit are still open. GA is a checklist we publish (ADR 0003, GA handoff), not an adjective.
Website · Guide · How it works · The guarantee · SPEC · Example config · ADRs · Coding agents · Agent guide · llms.txt · License
Try cheap. Prove it. Escalate only on failure.
proof over prediction · receipts over adjectives
PRs here are gated by compass: agent review · security · cross-model audits · tests, then a human merges.
{ "trace_id": "0192f3a1-7c4e-7abc-9d21-4e8b1f0a2c33", "prev_hash": "9f2c…a1b7", // chains to the prior decision "attempts": [ { "rung": 0, "model": "anthropic/claude-haiku-4-5", "cost_usd": 0.0007, "gates": [{ "gate_id": "cargo-test", "verdict": "fail" }] }, // cheap tried first, gate caught it { "rung": 1, "model": "anthropic/claude-sonnet-5", "cost_usd": 0.0121, "gates": [{ "gate_id": "cargo-test", "verdict": "pass" }] } // escalated, proven, served ], "final": { "served_rung": 1, "total_cost_usd": 0.0128, "counterfactual_baseline_usd": 0.0630, "savings_usd": 0.0502 } }