Skip to content

Pause Experiments at Harness plan-limit warnings - #1132

Merged
JacobStephens2 merged 3 commits into
issue-977from
issue-1008
Oct 23, 2026
Merged

JacobStephens2 merged 3 commits into
issue-977from
issue-1008

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

Pause Experiments at live Claude warnings/rejections and Codex quota reports at 85% in either window, including Trials from older Arm builds. Ordinary Runs continue.

plan-limit warning
  → mark a cut-short Trial void
  → stop both workers from starting Trials or retries
  → pause the Experiment
rerun the same Experiment
  → retain finished Trials and resume remaining work

Reconciles the latest issue-977 Graders and Verdicts while preserving limit-pause attempts and their fresh-retry budget.

Closes #1008

Evidence

  • Before: a_live_plan_limit_cuts_short_a_trial_and_keeps_its_evidence returned succeeded after a warning; the delayed-stop regression started four Trials instead of two.
    After: both regressions pass, alongside native Codex report detection, pause/resume and ordinary Run coverage.
  • Before: the Spec review's exact reproducer, built against baseline b8b9ec9, completed all six Trials despite the Baseline warning.
    After: the retained a_baseline_built_before_this_change_pauses_at_its_live_warning test passes with that actual baseline binary and with its fixed version-1 adapter for CI.
  • Before: repository-integrity grading replaced a warning's pause cause, so rerunning with --void-retries 0 exhausted the retry budget.
    After: a_plan_limit_pause_keeps_its_retry_budget_even_if_the_repository_changed passes: the warning remains a resumable pause and rerunning starts a fresh Trial.
  • Local validation: cargo test --test experiments --test trial --test fresh_run passes (56 tests; three subprocess entrypoints ignored). The actual pinned older-baseline regression also passes. cargo clippy --all-targets -- -D warnings, cargo fmt --check, and all 43 site tests selected by scripts/affected-tests issue-977 pass.
  • CI passed for exact HEAD 969d133, including the full Linux Rust suite, site and Console checks. Release builds and smoke gate passed.
  • Timed local selector: NEXTEST_TEST_THREADS=2 NEXTEST_STATUS_LEVEL=pass /usr/bin/time -p scripts/affected-tests issue-977 took 574.95 seconds, including compilation and site checks. After 43 site tests and 134 Rust tests passed, the redundant Rust run was deliberately interrupted when exact-head CI passed, as docs/testing.md directs. This timing is not a completed full-suite benchmark; CI supplies the full-suite result.

Test seams

  • Harness Interpretation's live decoding and completion interface.
  • Version-1 Trial CLI against fake Harness processes.
  • Experiment CLI, saved results and two-worker pause/resume behavior, including older Arms.
  • Ordinary Run CLI and Delivery after quota warnings.

Review coverage

Standards and Spec each reviewed all 27 changed files against issue-977, including generated site/docs/index.html. Fresh independent reviews after upstream reconciliation found no further findings. No changed file was left unread; no generated output was excluded.

The Standards P3 was addressed by separating pause policy from quota decoding. The Spec P1 was reproduced, fixed in Experiment supervision, and retained as a regression test.

Unaddressed findings

  • Standards: None. The one initial finding was addressed.
  • Spec: None. The one initial finding was verified and addressed.

Merge Danger

Door: two-way

No Trial request/result version change or data migration.

Blast Radius: Experiments

Trials now stop early at reported limits. An already-running peer may finish; the same Experiment resumes on rerun.

Built with codex · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 marked this pull request as ready for review October 23, 2026 05:56
@JacobStephens2
JacobStephens2 merged commit 8916839 into issue-977 Oct 23, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1008 branch October 23, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant