Repository navigation
feat(budget): per-client-key token budgets per model and window - #9
Merged
Merged
Conversation
…ing and batched KV persistence
AdmitBudget runs after model routing in the non-stream and stream paths, attaches the admission ticket, reports X-Budget-* headers and rejects with a 429 and Retry-After when an enforced budget is used up. X-Budget-* is reserved so upstreams cannot spoof it. Count-tokens is not budgeted. budget.SetDefault lets tests inject a clock.
After per-credential alias resolution the conductor checks the upstream models against the request's admission ticket. Exhausted pooled models are skipped; when none is left the request stops with the budget error, without MarkResult, cooldown or retry on another credential. Count-tokens is not checked.
…ponse writer Admission can run on the stream bootstrap goroutine while the handler writes keep-alives, and the non-stream keep-alive ticker writes from its own goroutine. Setting headers there raced with those writes, and a concurrent map write is a fatal error recover cannot catch. Keep-alive writers and the budget headers now share a per-response lock; headers are skipped when a keep-alive holds it, as the response is committing. Also corrects the stale comment: streams get headers.
With no api-key-budgets the engine still woke every 2s, locked its counters, looked up the KV store and logged the memory-only notice. Start now spawns nothing until a policy is installed, Apply starts or stops the flusher as budgets appear or disappear, and Flush returns at once when there are no counters. Wait is gone with the goroutine bookkeeping it served.
…etry failed loads A new counter used to read its persisted total from the KV store on the first request and on the usage dispatcher, so a stalled store could block every request for a label. The read now runs in the background: admission and charging use the in-memory value at once and the persisted total is added when it arrives. Usage is not written to the store before that, so nothing is double counted or lost. A failed read is retried by the next flush until it succeeds; the final shutdown flush still writes everything.
… flush Shutdown flushed the budget counters while the usage manager was still running, so records published just before it were charged after the flush and lost. Shutdown now stops the usage manager, waits for it to deliver the queue, flushes the counters, then closes the KV store. Both waits are bounded, so a stuck usage plugin or a stalled store cannot hold shutdown; a store whose flush is still running is left open. Add Manager.WaitStopped for the drain wait.
Nothing in production flushes per record or sets a flush interval, so drop WithFlushInterval, the flushNow path, the flush timeout and the idle sweep interval; the flush interval is now a constant. Ticket.Label had no production caller; tests assert on admission instead. The benchmark keeps the batched charge and admission measurements.
…t an upstream re-check Cover the flusher lifecycle, asynchronous counter loading with retry, the bounded shutdown drain and flush, that plugin-executor routes are admitted on the requested model only, and that the first selected credential's resolved models decide the alias rejection.
Admission can run on any goroutine, including ones that race a keep-alive writer (images streams, Gemini interactions), so it must not touch the response. AdmitBudget now only stores the X-Budget-* values in an atomic per-request holder. BudgetHeadersMiddleware wraps the gin writer once per budgeted request, and the writer sets the headers from the goroutine that commits the response. Headers that arrive after the commit are dropped. Without budgets, and for websocket upgrades, the writer is not wrapped. This removes the per-response mutex, guardResponseWriter and writeGuarded added earlier, restoring the keep-alive code to its original form.
stopUsageAndFlushBudgets derived its drain and flush limits from the Shutdown context. When that context had already ended, the flush context was born expired, the wait raced the flush goroutine and mostly lost, so the KV store was left open and a spurious warning was logged, even without budgets. Both waits are now limited by their own timeout (context.WithoutCancel), and without a policy or counters the manager is only stopped, as before the budget feature: no drain wait, no flush goroutine, the store closes.
bntvllnt
changed the base branch from
chore/sync-upstream-2026-11-14
to
main
November 14, 2026 22:07
|
This pull request targeted The base branch has been automatically changed to |
|
This pull request targeted The base branch has been automatically changed to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Per-client-key token budgets: limit how many weighted tokens a labelled API key (
access.api-key-labels) may spend per model pattern and per fixed window (day / week / month), e.g. capclaude-fable-*per team per week.429+Retry-After+rate_limit_error/budget_exhaustedbody.X-Budget-Remaining/Reset/Moderesponse headers (upstream cannot spoof them; exposed to CORS clients).docs/api-key-budgets.md, example block inconfig.example.yaml.Reliability (the feature must never take the API down)
api-key-budgetsblock → one atomic nil check per request, 0 allocations, no flusher, no KV access, no headers.Verification
go build, race build,go vet(only the known upstreamhost_test.go:1637warning),gofmtclean.go test ./...: only failures are pre-existing on pure upstream 905e70c (internal/discoverymDNS integration,internal/pluginhostRetry-After test).-raceon budget, handlers, config, usage, cliproxy andtest/packages: pass.Rollout
Deploy with no budgets (or
mode: observe) first, calibrate 1–2 weeks from the window-close logs, then switch one rule at a time toenforce. Edit the config in place or via the management API: an atomic-rename replacement is not hot-reloaded (pre-existing watcher behaviour).Follows #8 (merged).