A run keeps its pace into a rate limit and never runs a rate-limited unit again #27

Priority

extreme

Components

• app/nullius (backend)
• app/web (routes)
• templates and static files
• docs and the site
• catalogue (benchmarks, agents, capabilities)
• the standard (ports, kits, relay, runs)
• CI, workflows and skills

Today

A run of the standard starts its units at a fixed concurrency (ThreadPoolExecutor(self.concurrency) in standard/run.py) and keeps it whatever the provider answers. The relay passes a 429 to the caller unchanged, by design ("It never answers for the provider"), and a unit whose last model call was a 429 ends as RateLimitError, counted as infrastructure and left out of the rate. Nothing then happens to it: the unit is not run again, the units after it start at the same pace into the same limit, and the only remedy is the one the page gives, "On a shared API key, rate limits cause most failures: run few trials at once."

Wanted

A run that loses fewer units to a rate limit, without the relay answering for the provider. This touches how a run schedules its units, so the shape is the owner's to choose. A proposal, in two parts that can land separately:

• Slow down. When a unit ends rate-limited, no new unit starts for a while (a pause that doubles while they keep ending that way and resets on a unit that ends otherwise), and the concurrency in use is said in the progress lines.
• Run it again. A unit that ended rate-limited is run once more after the others, one at a time. Its first attempt stays in the job as evidence, and the result counts the second.

Open: where a second attempt's trial is kept so the pages and the live board read it as one unit, and whether Review should also warn when the chosen concurrency is above the limits a provider's headers state.

Acceptance

• The owner has chosen which parts to build.
• After a unit ends rate-limited, the next unit does not start at once (a test against an endpoint that answers 429).
• A unit that ended rate-limited is run again once, and the job's result counts its second attempt.
• The first attempt's record is still in the job.
• The relay still passes every 429 to the caller and records it.

Dependencies and related

none