[aitianshi]

Closes #14

What

nullius standard run --max-usd USD stops a run at a spending cap. Every relay of the job (the units' own, in this process or in their containers, and the benchmark's) books each call in one ledger before sending it (app/nullius/standard/ledger.py). What was spent is each reply's real cost; only a call in flight is an estimate, since its cost is known when its reply comes back: the call's characters over three as input tokens, and as output the largest reply the run has had from that model so far, which the ledger keeps, or 4,096 tokens before any reply, never more than the limit the call names or the model's own, at the user's prices. A call is sent only while what was spent, what the calls in flight have booked and its own booking stay within the cap. The first that does not fit is refused by the relay itself (HTTP 402, a body and a header that name the relay, recorded refused), and so is every call after it, so a run stops at one point. A reply turns the booking into what the call cost; a provider's error gives it back; a reply with no token count, or cut off, keeps the whole booking as unsettled.

A unit the cap cut off ends as SpendingCapError, an outcome of its own: spending_cap (Spending cap), outside the agent's rate like the infrastructure outcomes, never a miss, in a bronze of its own in both themes. Nothing failed, so it is no longer read as an API error. On the live feed it is budget, an outcome the board already knew and nothing produced. The judge is not offered it and no cause suggests it: only the relay's refusal makes it. The parse's cache schema moves to 13, so a job cached with the earlier reading is read again. Units not begun are not run. result.json carries budget: max_usd, spent_usd, unsettled_usd when there is some, stopped, cut_off and not_run. A capped run is refused before it starts when a model it calls has no price.

Not in this change: the cap in the builder and on a host, shared over a run's jobs and shards, and the job page showing the budget (#45, in #48); a cap per unit (#47).

Built on #39 (the user's price list), which is on main now; main is merged into this branch, so the diff is the cap alone.

Touches

backend · templates and static · docs and site · the standard

Before and after

The job page of a run its cap stopped, at 1400 px in the dark theme; the images are not attached here.

Before	After
The unit cut off is "1 API error", in the blue of a provider's failure	"1 Spending cap", in bronze, still marked infra and outside the adjusted rate

Verified

• New app/tests/test_standard_ledger.py (10 tests): a call is sent while spent, in flight and its booking fit; the first that does not fit closes the ledger for every later one; an unknown cost keeps its booking and an error gives it back; a ledger that cannot be read lets no call out; 60 calls at once never book past the cap; four processes booking in one ledger lose nothing; what a booking is (the named limit, the model's own, the default, cache-write input, embeddings, an encoded file not counted).
• test_standard_relay.py: a call that fits is sent and settled at its cost; one that does not is answered 402 by the relay, marked, recorded refused and never reaches the provider, and neither does a smaller one after it; a provider's 429 gives the booking back; a reply without tokens keeps it; a model without a price is not sent; a relay without a cap books nothing; the relay as a process of its own books in the ledger it is given.
• test_standard_run.py: eight units under a cap the fourth call does not fit: three run, one is cut off (SpendingCapError, the outcome spending_cap), four are not run, result.json says so and no call went out past the cap; an ample cap changes nothing but the result; no cap, no ledger; a cap without a price for each model, or with a bad amount, is refused before anything is begun.
• test_classify.py: the cut-off unit has its own outcome, outside the rate, for the classifier and the live board (budget). test_outcome_styles.py, test_field_guide.py, test_judge.py and test_reconcile.py hold the new outcome to a colour in both themes, a row in the field guide and a place in the judge's table. test_standard_container.py: the relay's container is given the ledger and its folder alone.
• Backend suite python3 app/tests/parallel.py (1657 tests) and route suite (356 tests), OK, after main was merged in and again with the outcome. docs-drift script: 0 findings.
• The bronze was measured with the repository's palette helper against the eleven other outcomes: 34.8 from its nearest in the dark themes and 34.5 in light (the closest pairs before were 23.3 and 20.6), contrast 4.6 or more on every dark surface and 5.2 or more on light. Looked at on the job page in both themes.
• By hand, in a temporary profile against a stand-in provider with fixed usage (GSM8K, gpt-4o-mini at list price, 0.00024 USD a call, booked 0.0099):
  • ten units one at a time under --max-usd 0.011: five ran, the sixth was cut off, four were not run, 0.0012 spent, as computed by hand;
  • twelve units four at a time with slow replies under 0.03: three calls in flight filled the cap, the fourth was refused, eight units were not run;
  • --backend docker, on the merged code: the containers' relays share the ledger; three ran, one was cut off, two were not run;
  • an ample cap, and the benchmark's own script (Original) with --price: the run ended and the result carries what was spent;
  • a cap of 0, and a model with no price: refused before anything ran.
• With a real provider (OpenRouter, a model at 0.10 and 0.20 USD a million tokens, GSM8K's Original setup), the provider's own usage figure for the key read before and after each run:
  • five problems under --max-usd 2: all ran, each call's cost taken from the provider's reply; the ledger counted 0.0008868 USD and the provider billed 0.0008868;
  • 30 problems one at a time under a cap of 0.03 (run from the builder, feat(builder): give a recipe a spending cap and show it on the job page #48): 20 calls sent, the 21st refused by the relay with 0.0038 spent and 0.0262 booked for it, 9 problems not run; ledger 0.0037699, billed 0.0037699;
  • 24 problems 8 at a time under 2: the provider itself refused three calls with its own 402 (credits against calls in flight). They read as API error, not as the cap, and their bookings were given back; ledger 0.0044629, billed 0.0044629.
• The booking rule was changed after these runs showed its flaw (db813ca): a call was first booked at the model's whole output limit, 0.026 USD for calls that cost 0.0002, so 50 problems 15 at a time under 0.30 USD sent 11 calls and stopped with 0.002 spent. With the rule above, on the same provider and model:
  • 50 problems 15 at a time under 0.30 USD: all 50 ran; the first 15 calls were booked 0.00084 each, the later ones 0.00016 to 0.00032 as replies were seen; 0.0089511 spent;
  • 50 problems 5 at a time under 0.007 USD: 34 calls sent, the 35th refused with 0.0060 spent, 15 not run; ledger 0.0059737;
  • the two ledgers add up to 0.0149248 USD and the provider billed 0.0149248.
• Not verified: Docker Desktop, where the lock is shared between this machine's process and a VM's containers, and a real remote machine (a host was a container standing in, feat(builder): give a recipe a spending cap and show it on the job page #48).

Risk

• The relay now gives one answer of its own. It does so only for a run given --max-usd; without it no ledger exists and no call is booked.
• A booking is an estimate at list prices, not a bill. A call that costs more than was booked takes the spend past the cap by the difference; the next call is then refused.
• The calls in flight take room: with several units at a time and a cap close to one call's booking, the run stops early with little spent. The refusal says what was spent and what was booked.
• A twelfth outcome: a page or a script that lists outcomes by name does not know it. The classifier's order, the styles, the field guide and the judge's table are held to it by tests; under red-green colour blindness the bronze sits near Agent error, as other pairs of this palette already do, and the label carries it.
• A booking follows the replies the run has had, so a reply longer than any before it costs more than was booked: the spend can pass the cap by that difference for each call in flight. Before any reply a call is booked 4,096 output tokens: a cap below that for each unit at a time stops in the first wave (0.013 USD for 15 at a time on this model).
• The lock is flock on a file beside the ledger, as an install is locked. On Docker Desktop a benchmark's own relay (an option's embeddings) runs on the machine while the units' relays run in the VM; whether the lock holds between the two was not tested.
• A capped run needs a price under the name it sends. The public list had this model under its OpenRouter name; a model it lists only under its maker's name needs --price or an own price.

Undone by a revert; a job written with budget in its result stays readable.

[github-actions]

🔴 PR judge · 6 of 8 yes · #46 ↔ #14

Merge risk	🔴 MAJOR	the judge's reading of the diff; app/nullius/runs/classify.py is in the path of every run
Priority	🔴 EXTREME	the deepest path is app/nullius/standard/cli.py

Question	Verdict
Changes do what the issue asks · 2 of 2 criteria	✅ yes
Issue and pull request describe the same change	✅ yes
Description matches the diff, no more, no less	✅ yes
Diff stays in scope	✅ yes
Tests cover the change, or Verified says how	✅ yes
Docs follow the change	✅ yes
Risk named in the description	⚠️ partly
Before and after shown · 0 images · the diff touches what is visible	❌ no

Acceptance criteria, 2 of 2 met

#	Criterion	Verdict
1	A run can be given a spending cap, and no call goes out once the spend plus the calls in flight would pass it.	✅ met (cut)
2	A capped run's result says the cap stopped it, what was spent, and which units did not run.	✅ met (cut)

Checks the script runs itself, 4 of 4 pass

No machine attribution trailer in the commits or the description	✓
New code under app/, docs/, .github/ or .claude/	✓
No Flask import under the backend package	✓
No key-shaped string in the diff	✓

_{some of the diff was cut before the judge read it}

[aitianshi]

A review of the cap's three pull requests together found defects in this one's code. They are fixed in #49 (856e6f4), which also cuts the cap's documentation down (28383e4). The three are best merged together, in order.