Skip to content

Void Trials on provider capacity and overload errors - #1374

Merged
JacobStephens2 merged 3 commits into
mainfrom
issue-1362
Oct 26, 2026
Merged

JacobStephens2 merged 3 commits into
mainfrom
issue-1362

Conversation

@JacobStephens2

Copy link
Copy Markdown
Owner

Closes #1362.

Summary

Provider capacity and overload errors now classify as rate limits. A Trial that hits one is void with rate_limit and pauses the Experiment, which resumes under its saved Bar, instead of counting as a failed review.

 ErrorKind::from_text(text)
   usage_limit | usage limit | quota | spend limit | hit your limit  → UsageLimit
-  rate_limit | rate limit | 429                                     → RateLimit
+  rate_limit | rate limit | 429 | at capacity | overloaded          → RateLimit
+  the whole word 503 or 529                                         → RateLimit
   anything else                                                     → Other

Trials, Ticket runs and Scoring all classify through from_text. docs/guide.md, site/docs and site/how-it-works name the new void cause.

Evidence

  • Before: in the Run the A/A Experiment on the host #1267 A/A Experiment, four Trials ended as "outcome": "failed", "void_cause": null, "cause": "codex exited 1: Selected model is at capacity. Please try a different model." and counted as disagreements.
  • After:
    • tests/trial.rs::a_codex_capacity_error_is_a_rate_limited_void_trial: a fake Codex exits 1 with that exact message, and the Trial ends void with rate_limit and the same cause.
    • event_tests::provider_capacity_and_overload_errors_are_rate_limits, for Codex and Claude events:
      • capacity, OVERLOADED, overloaded_error, HTTP 503, HTTP 529, API Error: 503 and Error code: 529 → rate_limit;
      • quota exceeded stays usage_limit;
      • invalid JSON, src/check.rs:1503 and worker529 stay other.
    • retained_retryable_pause_resumes_with_the_same_budget_and_new_score now covers RateLimit as well as UsageLimit.

The first Ticket run wrote this fix and then failed on the capacity error itself. Its Continuation's reproduction step tested a branch that already held the fix, so it handed back needs-info (#1373). The Day shift reviewed the diff and opened this PR.

Merge Danger

Door: two-way

Blast Radius: classification

An unrelated error that contains "at capacity", "overloaded", or the whole word 503 or 529 would pause an Experiment instead of failing a Trial, and a Ticket run would report it as a limit. Both outcomes are visible and resumable.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LSAzXiA6BD9neiPgVQRMKD

…apacity. Please try a different model.)

2026-10-26T08:07:22Z, host thirdshift. Uncommitted work at the time of failure is included in this commit.
@JacobStephens2 JacobStephens2 added the test:macos On a PR into main: run the full 4-shard macOS suite instead of the one-job macOS gate (#893) label Oct 26, 2026
@JacobStephens2
JacobStephens2 merged commit 7f9adb3 into main Oct 26, 2026
21 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1362 branch October 26, 2026 14:11
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:macos On a PR into main: run the full 4-shard macOS suite instead of the one-job macOS gate (#893)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A model-at-capacity error fails a Trial instead of voiding it

1 participant