[aitianshi]

Closes #45

What

The builder's Limits step takes a spending cap in USD for the whole run of a Nullius recipe. Its jobs that call a model (each dataset with each participant, on each machine) take equal shares of it, rounded down, as --max-usd on each job's command: the amount typed is the most the run spends, on this machine or on a host. A control that calls no model takes no share and is offered no cap. A Harbor recipe is not offered one: its calls do not pass the relay. A model without a price in the user's list is named at Limits before Run, since a capped run is refused without one.

The job page says the cap in its run totals. A run the cap stopped says so in a neutral notice, with what was spent, and lists the units never begun; the units cut off are its Spending cap trials. The live board is told the cap at job.start and says what was spent of it. Both show the cap as it was given ($0.395), not rounded to the cent as a cost is. merge adds the caps and spending of a launch's shards into one budget, when every shard had a cap. A job now carries budget, so the parse's cache schema moves to 14 and cached jobs are read again.

Two things differ from the issue as first written, and the issue is updated. The cap is one amount shared by the run's jobs, where the issue gave each job the whole cap: one number then holds for the whole run. A cap per unit is left to #47, because it changes how a unit counts and that is to be decided first.

Stacked on #46.

Touches

backend · routes · templates and static · docs and site · the standard

Before and after

Taken at 1400 px in the dark theme; the images are not attached here.

Before	After
Limits: number of problems, concurrent problems, attempts	The same, and Spending cap (USD)
Job page of a run the cap stopped: 21 of 30 planned trials, no word of a cap	A notice "Stopped at its spending cap of $0.03, with $0.0038 spent: 1 cut off, 9 not run", Spending cap $0.03 under Cost, and "9 not run" opening the list of units

Verified

• test_standard_recipe.py, SpendingCap: one job takes the whole cap and no cap adds nothing; two agents on two machines take a quarter each; three shards of 1 USD take 0.333333 each, never more than the cap in all; a control that calls no model takes no share; an amount that is no amount, or too small to share, names the Limits step; a missing price is said at Limits, and only a recipe with a cap asks.
• test_standard_shard.py: the merged job carries the shards' caps as one, and none when a shard had no cap; a recorded cap is read leniently, an amount missing is not zero. test_standard_run.py: the live feed carries the cap at job.start and the cut-off unit ends there as budget.
• app/ui_tests/test_capped_job.py: the job page of a stopped run (notice, cap row, the units not run, the Spending cap outcome), of a run within its cap, and of one without; the builder keeps a saved recipe's cap and a Harbor recipe takes none.
• Backend suite (1664 tests) and route suite (360 tests), OK. docs-drift script: 0 findings.
• By hand with a real provider, from the builder's Review step in a temporary profile (OpenRouter, a model at 0.10 and 0.20 USD a million tokens, GSM8K's Original setup). After each run the provider's own usage figure for the key was read and compared:
  • 24 problems, 8 at a time, under 2 USD: Review said "Spending cap $2.00"; the run ended within it and the job page shows the cap; the ledger counted 0.0044629 USD and the provider billed 0.0044629. The provider itself refused three calls with its own 402 (credits against calls in flight): they read as API error, not as the cap, and their bookings were given back.
  • 30 problems, one at a time, under 0.03 USD: 20 calls were sent, the 21st was refused by the relay (0.0038 spent, 0.0262 booked for the call), 9 problems were not run. The live board said "$0.0038 spent of the $0.03 cap" and the job page the notice above. Ledger 0.0037699, billed 0.0037699.
  • the same over this machine and a host (a container with an SSH server standing in for it), 0.06 USD in all: Review said "$0.06 in all · $0.03 for each of 2 jobs"; each shard stopped at its 0.03, with 0.0039 and 0.0038 spent, one problem cut off and eight not run. The host kept its own ledger and priced from this machine's list. merge gave one job with budget 0.06, 0.007686 spent, stopped, 2 cut off and 16 not run, and its page says so. Billed for both: 0.0076864.
• Several at a time, same provider and model, on a second key (50 problems each, from the builder). These were made while a call was booked at the model's whole output limit, 0.0262 USD; they are why feat(standard): stop a run at a spending cap #46 now books a call at what the run's replies have been. After that change, 15 at a time under 0.30 USD ran all 50 problems for 0.0090 USD, and 5 at a time under 0.007 USD stopped after 34 calls with 0.0060 spent, the job page saying "Stopped at its spending cap of $0.007, with $0.0060 spent: 1 cut off, 15 not run". Before it:
  • 15 at a time under 0.30 USD: 11 bookings fit, so 11 calls were sent and answered, 4 problems were cut off and 35 not run, as computed beforehand; 0.0020 spent;
  • 15 at a time under 0.395 USD, just above 15 bookings: 25 calls sent, the 26th refused, 24 not run (21 to 29 expected: the point depends on how many calls are in flight at that instant); 0.0047 spent;
  • 5 at a time under 0.135 USD: 26 calls sent, the 27th refused, 23 not run; 0.0049 spent;
  • the three ledgers add up to 0.0116098 USD and the provider billed 0.0116098.
  • 15 at a time under 0.05 USD (first key): one booking fits: 1 call sent, 14 cut off, 35 not run.
  • On the first key, whose account had about a cent of credit left, the provider itself refused most calls made in parallel with its own 402 (46 of 50 at 15 at a time under 0.50 USD). They read as API error, never as the cap, their bookings were given back and the cap did not stop that run.
  • Against a stand-in provider with fixed usage, 15 at a time under 0.65 USD with bookings of 0.04096: 40 calls sent, the 41st refused, 9 not run (32 to 44 expected).
• Browser: the Limits step in the dark and light themes at 1400, 760 and 390 px (no clipped text, no sideways scroll, no page error); the job pages above in both themes.
• Not verified: a capped recipe on the docker backend from the builder (the cap on docker was run from the command line in feat(standard): stop a run at a spending cap #46), a real remote machine, and Docker Desktop.

Risk

• Equal shares: a job that needs more than its share stops while another's goes unspent. Review says each job's share before Run.
• The price check at Limits names the models the recipe knows (the run's or a setup's, the judge's, the simulated user's, an option's embeddings). The run still checks every model it calls, an agent's side models included, and refuses on its own.
• Every cached job is parsed once more after the schema moves.
• A stopped run's notice is neutral, not a warning: nothing failed. A reader who expects a failure to be red may miss it; the outcome bar carries the same fact.

Undone by a revert: a recipe saved with max_usd then reads as one without a cap, and a job's budget stays in its result.json, unread.

[github-actions]

🔴 PR judge · 5 of 8 yes · #48 ↔ #45

Merge risk	🔴 MAJOR	the judge's reading of the diff; app/nullius/runs/parse.py is in the path of every run
Priority	🔴 EXTREME	the deepest path is app/nullius/standard/merge.py

Question	Verdict
Changes do what the issue asks · 4 of 4 criteria	✅ yes
Issue and pull request describe the same change	⚠️ partly
Description matches the diff, no more, no less	✅ yes
Diff stays in scope	✅ yes
Tests cover the change, or Verified says how	✅ yes
Docs follow the change	✅ yes
Risk named in the description	⚠️ partly
Before and after shown · 0 images · the diff touches what is visible	❌ no

Acceptance criteria, 4 of 4 met

#	Criterion	Verdict
1	A Nullius recipe can be given a spending cap in the builder, and the run it starts, here or on a host, stops at it as t…	✅ met (cut)
2	A Harbor recipe is not offered the cap.	✅ met (cut)
3	A run of several jobs (participants, datasets, shards) shares the cap in equal parts, and the merged job's result carri…	✅ met (cut)
4	The job page of a capped run shows the cap, what was spent, and, when the cap stopped it, the units cut off and those n…	✅ met (cut)

Checks the script runs itself, 4 of 4 pass

No machine attribution trailer in the commits or the description	✓
New code under app/, docs/, .github/ or .claude/	✓
No Flask import under the backend package	✓
No key-shaped string in the diff	✓

_{some of the diff was cut before the judge read it}

[aitianshi]

A review of the cap's three pull requests together found defects in this one's code. They are fixed in #49 (856e6f4), which also cuts the cap's documentation down (28383e4). The three are best merged together, in order.