Skip to content

feat(budget): per-client-key token budgets per model and window - #9

Merged
bntvllnt merged 27 commits into
mainfrom
feat/api-key-token-budgets
Nov 14, 2026
Merged

bntvllnt merged 27 commits into
mainfrom
feat/api-key-token-budgets

Conversation

@bntvllnt

@bntvllnt bntvllnt commented Nov 14, 2026 •

Copy link
Copy Markdown

What

Per-client-key token budgets: limit how many weighted tokens a labelled API key (access.api-key-labels) may spend per model pattern and per fixed window (day / week / month), e.g. cap claude-fable-* per team per week.

api-key-budgets:
  mode: observe            # observe = count + headers + "would reject" logs; enforce = 429
  timezone: UTC
  weights: {input: 1, output: 1, cache-write: 1, cache-read: 0.1}
  labels:
    team-x:
      - {models: "claude-fable-*", window: week, limit: 2000000}
      - {models: "*", window: day, limit: 20000000, mode: enforce}
  • Admission at the handler chokepoints, websocket turns, alpha search, Codex live, Muse relay, plus a second check after alias resolution in the conductor (an alias cannot reach a budgeted upstream model).
  • Charged from the usage pipeline with a ticket (same label + rules at admit and charge). Counters keyed by hashed label, never the raw key; persisted to the local KV store (batched flush every 2s, final flush at shutdown, preloaded on apply).
  • Enforce → 429 + Retry-After + rate_limit_error/budget_exhausted body. X-Budget-Remaining/Reset/Mode response headers (upstream cannot spoof them; exposed to CORS clients).
  • Hot reload: changing a limit or mode keeps the count; Home mode is a config error.
  • Docs: docs/api-key-budgets.md, example block in config.example.yaml.

Reliability (the feature must never take the API down)

  • No api-key-budgets block → one atomic nil check per request, 0 allocations, no flusher, no KV access, no headers.
  • Every engine entry point recovers and fails open; KV read/write errors fail open with a rate-limited warning; no I/O on the request path.
  • Header writes go through a writer wrapper applied by the goroutine that commits the response (fixes a fatal concurrent-map-write with keepalive goroutines found in review).
  • Shutdown waits are bounded; the KV store is not closed under a running flush.

Verification

  • go build, race build, go vet (only the known upstream host_test.go:1637 warning), gofmt clean.
  • go test ./...: only failures are pre-existing on pure upstream 905e70c (internal/discovery mDNS integration, internal/pluginhost Retry-After test).
  • -race on budget, handlers, config, usage, cliproxy and test/ packages: pass.
  • Dogfood (race binary, mock Claude upstream, SSE + non-stream keepalives on, 10 scenarios): observe → would-reject log; hot reload to enforce → 429 + Retry-After (stream and non-stream); unlabelled key untouched; alias → Fable blocked with 0 upstream calls; limit raise keeps count; restart keeps counters; timezone moves window; 400 req @ 50 concurrency all 200 with p50 1569 ms budgeted vs 1544 ms no-budget (1500 ms mock delay); 0 data races, 0 panics, 0 raw keys in logs.
  • Independent reviews: race/crash review (fixed), final review of middleware + shutdown + preload: nothing blocking.

Rollout

Deploy with no budgets (or mode: observe) first, calibrate 1–2 weeks from the window-close logs, then switch one rule at a time to enforce. Edit the config in place or via the management API: an atomic-rename replacement is not hot-reloaded (pre-existing watcher behaviour).

Follows #8 (merged).

AdmitBudget runs after model routing in the non-stream and stream paths, attaches the admission ticket, reports X-Budget-* headers and rejects with a 429 and Retry-After when an enforced budget is used up. X-Budget-* is reserved so upstreams cannot spoof it. Count-tokens is not budgeted. budget.SetDefault lets tests inject a clock.
After per-credential alias resolution the conductor checks the upstream models against the request's admission ticket. Exhausted pooled models are skipped; when none is left the request stops with the budget error, without MarkResult, cooldown or retry on another credential. Count-tokens is not checked.
…ponse writer

Admission can run on the stream bootstrap goroutine while the handler writes
keep-alives, and the non-stream keep-alive ticker writes from its own goroutine.
Setting headers there raced with those writes, and a concurrent map write is a
fatal error recover cannot catch. Keep-alive writers and the budget headers now
share a per-response lock; headers are skipped when a keep-alive holds it, as
the response is committing. Also corrects the stale comment: streams get headers.
With no api-key-budgets the engine still woke every 2s, locked its counters,
looked up the KV store and logged the memory-only notice. Start now spawns
nothing until a policy is installed, Apply starts or stops the flusher as
budgets appear or disappear, and Flush returns at once when there are no
counters. Wait is gone with the goroutine bookkeeping it served.
…etry failed loads

A new counter used to read its persisted total from the KV store on the
first request and on the usage dispatcher, so a stalled store could block
every request for a label. The read now runs in the background: admission
and charging use the in-memory value at once and the persisted total is
added when it arrives. Usage is not written to the store before that, so
nothing is double counted or lost. A failed read is retried by the next
flush until it succeeds; the final shutdown flush still writes everything.
… flush

Shutdown flushed the budget counters while the usage manager was still
running, so records published just before it were charged after the
flush and lost. Shutdown now stops the usage manager, waits for it to
deliver the queue, flushes the counters, then closes the KV store. Both
waits are bounded, so a stuck usage plugin or a stalled store cannot
hold shutdown; a store whose flush is still running is left open.

Add Manager.WaitStopped for the drain wait.
Nothing in production flushes per record or sets a flush interval, so
drop WithFlushInterval, the flushNow path, the flush timeout and the idle
sweep interval; the flush interval is now a constant. Ticket.Label had
no production caller; tests assert on admission instead. The benchmark
keeps the batched charge and admission measurements.
…t an upstream re-check

Cover the flusher lifecycle, asynchronous counter loading with retry, the
bounded shutdown drain and flush, that plugin-executor routes are admitted
on the requested model only, and that the first selected credential's
resolved models decide the alias rejection.
Admission can run on any goroutine, including ones that race a keep-alive
writer (images streams, Gemini interactions), so it must not touch the
response. AdmitBudget now only stores the X-Budget-* values in an atomic
per-request holder. BudgetHeadersMiddleware wraps the gin writer once per
budgeted request, and the writer sets the headers from the goroutine that
commits the response. Headers that arrive after the commit are dropped.

Without budgets, and for websocket upgrades, the writer is not wrapped.

This removes the per-response mutex, guardResponseWriter and writeGuarded
added earlier, restoring the keep-alive code to its original form.
stopUsageAndFlushBudgets derived its drain and flush limits from the
Shutdown context. When that context had already ended, the flush context was
born expired, the wait raced the flush goroutine and mostly lost, so the KV
store was left open and a spurious warning was logged, even without budgets.

Both waits are now limited by their own timeout (context.WithoutCancel), and
without a policy or counters the manager is only stopped, as before the
budget feature: no drain wait, no flush goroutine, the store closes.
@bntvllnt
bntvllnt changed the base branch from chore/sync-upstream-2026-11-14 to main November 14, 2026 22:07
@github-actions
github-actions Bot changed the base branch from main to dev November 14, 2026 22:07
@github-actions

Copy link
Copy Markdown

This pull request targeted main.

The base branch has been automatically changed to dev.

@github-actions

Copy link
Copy Markdown

This pull request targeted main.

The base branch has been automatically changed to dev.

@bntvllnt
bntvllnt changed the base branch from dev to main November 14, 2026 22:08
@bntvllnt
bntvllnt merged commit 3137e1b into main Nov 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant