Skip to content

Record plan-limit stops as paused Commands with reset details - #1659

Merged
JacobStephens2 merged 5 commits into
mainfrom
issue-1619
Oct 30, 2026
Merged

JacobStephens2 merged 5 commits into
mainfrom
issue-1619

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Summary

Plan-limit stops now keep their own Command outcome and the facts needed to resume.

 Claude/Codex stopping event
- boolean limit flag → failed Experiment / sampled Scoring run
+ stopping window facts → paused Command + saved resume command
+ Needs you: Paused, reset time and copyable resume command
+ resumed work starts → earlier pause leaves Needs you

Finished Trials and Scores remain saved. Ordinary failures and older failed records keep their existing outcomes. Pauses neither send failure email nor count as failed outcomes. Automatic resume remains out of scope (#1653).

Closes #1619

Evidence

  • Before: cargo test --test experiments plan_limit_warnings_pause_both_workers_and_rerunning_resumes_void_trials -- --exact --nocapture failed: the Experiment state was paused, but the Command outcome was failed.
    After: the same regression passes, including both Harnesses, null reset fallback and resuming Void Trials.

  • Before: cargo test --test experiments model_graders::plan_limit_in_grading_keeps_trials_and_shows_a_resumable_paused_command -- --exact found missing limit facts after completed Trials reached the Grader.
    After: the regression covers Claude's 86% seven-day stop, RFC 3339 reset, show --json, Dismiss/Undo, and removal from Needs you while resumed grading is still running. Finished Trials are not repeated.

  • Scoring regression covers Claude warnings/rejections and Codex native reports, retaining completed Scores on resume. Decoder checks preserve each stopping window's own reset and explicit unknown fields.

  • node --test --test-name-pattern='Paused rows and Command details' console/tests/inbox.test.cjs passes: both locations copy the exact saved command to the clipboard. This exposed and fixed compact-row visibility and event delegation.

  • CI exposed copy feedback disappearing during a live Inbox refresh. A forced-refresh regression reproduced it; the existing data-copy-status preservation keeps feedback visible. cargo test --test console urls_in_inbox_and_peek_text_are_links_and_other_text_stays_escaped -- --exact also passes after limiting the new cause markup to paused rows.

  • Spec review's complete standalone parser reproduction failed before its correction (None instead of the known matching reset) and passed unchanged afterward. Its case is retained in src/harness/interpretation_tests.rs.

  • cargo clippy --all-targets -- -D warnings passed locally and in CI.

  • scripts/affected-tests main passed its local site checks (91.35 s) and all 126 Console browser checks (139.83 s). Its duplicate local Rust run was stopped after successful CI on the same final head, following docs/testing.md's CI reuse policy.

  • Full Linux CI on 2d704eee0019ca5a66edde53700bfa6f6b5f2e14: 3,610 Rust tests passed, 75 skipped, across 79 binaries. Nextest test runtime: 1,145.120 s, excluding compilation. The macOS gate, site and Console CI jobs also passed.

Test seams

  • CLI Commands with fake Harnesses: Experiment/Scoring execution, retained results, show, status, and Dismissal.
  • Harness stream interpretation: stopping-window facts and ordinary-session behavior.
  • Console HTTP and browser: paused rows, Command details, and clipboard actions.

Review coverage

Standards: 0 findings. Spec: 1 finding, fixed and verified. Both axes read all 41 changed paths through 42cc24f9, including site/docs/index.html, and reviewed the Spec correction. No generated-output exclusions. The final CI repair hunks in src/console/row.html and console/tests/inbox.test.cjs at 2d704eee were not reread by either axis after their one follow-up; both earlier versions were read. Final reports and counts are saved in .thirdshift-review-JxoMUH/ for the Session trace.

Unaddressed findings

Standards

None.

Spec

None. The matching-window reset finding was reproduced with its exact command, fixed, and retained as a regression.

Security

None.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Additive Command fields; older records remain readable without migration.

Blast Radius: outcomes

Command readers now distinguish plan-limit pauses across the Inbox, Console and status. Paused Scoring returns a nonzero exit code instead of reporting a completed sample; rerunning continues pending work.

Built with codex codex-cli 0.164.0 · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 6239668 into main Oct 30, 2026
@JacobStephens2
JacobStephens2 deleted the issue-1619 branch October 30, 2026 17:49
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A paused Experiment's Command is recorded as failed

1 participant