[JacobStephens2]

Summary

Before an Experiment runs its first Trial, it now prints what its remaining Trials will cost and, on a terminal, asks before going ahead. There is no cost ceiling in the Bar.

 experiment::run
   Stored::open                       # new, or resume
   if completed → print saved results
+  cost::confirm(experiment, --yes)
+    workers::remaining              # same queue the workers take
+    history: other Experiments' completed, non-void Trials
+             (every attempt's minutes), per suite and Harness
+    print count per suite and Arm, worst case, estimates
+    stdin is a terminal and no --yes → "Run these Trials? [y/N]"
+    declined → error; a fresh Experiment's record is discarded
   build Arms, run Trials, Graders, Verdict

thirdshift: Experiment 1b1a…: 2 Trials to run          (resume: "2 of 6 Trials remain")
thirdshift: spec-review, Baseline arm (codex): 1 Trial, no earlier Trials to estimate from
thirdshift: spec-review, Candidate arm (claude): 1 Trial, estimate 10 session-minutes
thirdshift: Worst case: 60 session-minutes (2 Trials × the Bar's 30-minute timeout)
Run these Trials? [y/N]

The total Estimate: line appears only when every suite and Arm has history. scripts/release.sh now passes --yes, so --with-experiment still asks only its own fail/inconclusive question. docs/guide.md (Experiments section, both synopses), the three hand-written site pages and docs/code-navigation.md describe the question and --yes. site/docs/index.html is regenerated.

Evidence

• Before: cargo test --test experiments cost:: → 5 failed. The Run never showed Run these Trials? [y/N] and ran its Trials straight away.
  After: cargo test --test experiments cost:: → 8 passed. These tests drive the CLI through the fake Harnesses, on a pseudo-terminal where it matters:
  • declining shows the count and worst case, starts no Trial and leaves no Experiment listed
  • y runs both Trials; --yes on a terminal, and a piped stdin, skip the question
  • the estimate is per suite and Harness, from earlier Experiments only: a void Trial of 100 minutes is left out; a codex Arm with no history shows no estimate
  • a resume counts only its remaining Trials (2 of 6 Trials remain)
  • a retried Trial counts every attempt (review finding; it failed first, with estimate 5/10 instead of 10/20)
• tests/release/experiments.rs now expects ["experiment", "--release", <commit>, "--json", "--yes"]. Those tests run only on Linux, so CI covers them. On macOS, cargo test --test release passed.
• Local validation on the merged head (f17a0605): npm test --prefix site passed, and cargo test -- --test-threads=3 passed (3162 passed, 0 failed). Under full default parallelism, four timing-bound unit tests in untouched modules failed: github::execution_tests ×2, session::private::tests, evals::experiment::execution::tests. That is the host-load case docs/testing.md describes (Experiment tests with 1s Trial timeouts fail under heavy local load on macOS #1472). Each passed when rerun alone.

Merge Danger

Door: two-way

A new prompt and flag. Reverting removes them, and saved Experiments are unchanged.

Blast Radius: Experiments

Anyone who runs thirdshift experiment from a terminal in a script without --yes now gets a question. Piped stdin behaves as before. A declined new Experiment's folder is removed, but only when this same command created it, before any Arm build or Trial.

Review

Reviewed with thirdshift-code-review against main. Standards raised 13 findings and Spec raised 5. The review read every changed file except site/docs/index.html, which is generated: npm run docs:check passed, and its source docs/guide.md was reviewed. The fixes made after the review also changed src/evals/experiment/store.rs (the new Stored::discard), and the review did not read that hunk.

Addressed: Standards 1 (history reads through the Files seam), 4 and Spec 1 (every attempt is counted; both tests kept), Standards 5 in part (Description::setup(arm)), 7, 8, 9, 12, 13; Spec 2 (docs now say the release Baseline probe comes first), 3 (a declined fresh Experiment is discarded; test kept), 4 (--release … [--yes] synopsis).

Unaddressed findings

Standards

• 2. confirm creates its terminal dependencies; there are no inline tests. Declined. The question's interface is the CLI, and tests/experiments/cost.rs drives it on a real pseudo-terminal through the fake Harnesses, as the issue asks ("Tests drive the CLI through the fake Harnesses"). Plan needs a stored Experiment. An Outside-style seam would add an abstraction with one caller.
• 3. history duplicates store::list. Declined. store::list (src/evals/experiment/store.rs) propagates the first unreadable Experiment's error and loads each one through Stored::load, which refreshes legacy Verdicts. history must skip unreadable Experiments, because one only narrows the estimate (cost.rs, history doc comment), and must read metadata only (Stored::retained).
• 5 (rest). Other Arm switches in execution.rs, store.rs and grading.rs; a Display label. Not changed. Those sites are outside this change, and cost.rs now uses the new Description::setup(arm).
• 6. (suite, harness) tuple key. Declined: a private type alias in one module, used in one lookup. A key type would add more than it names.
• 10. The question is gated on stdin only, unlike Setup's stdin-and-stderr check. Declined on the issue's line "It asks to confirm only when stdin is a terminal." The finding's command (cargo run --quiet -- experiment HEAD 2>/tmp/cost.log from an interactive terminal) needs an interactive terminal, so this headless session could not run it.
• 11. The cost lines are not kept in the Command record. The reviewer offered this as a note, not a fix. Confirming before logs::begin is what keeps a declined Experiment out of the Inbox.

Spec

• 5. A total Estimate: line, and yes accepted as well as y. The reviewer judged neither worth undoing. Both are documented in docs/guide.md.

Closes #1482

Built with claude 2.1.312 · default model · default effort

Security review outcome

No unaddressed introduced findings.