[JacobStephens2]

Summary

thirdshift experiment compares two Arms on private Eval suites, records the Bar before Trials start, and resumes interrupted work. thirdshift experiments [id] --json reads saved results.

experiment(candidate)
  fix Arms + Bar → build Arms → prepare Eval tasks
  alternate Trials on two workers → one result per Trial
  Stop / resume → keep finished Trials

Command records, Command logs and Activity logs use thirdshift's repository. Trial artifacts stay under ~/.thirdshift/evals/experiments/. The guide and site pages describe the workflow. Sandbox isolation is #1014; this ticket reports Trial outcomes before the separate grading/Verdict work.

Closes #1003

Evidence

• Before: cargo test --test experiments experiment_records_both_arms_the_bar_logs_stages_and_each_trial -- --exact failed because experiment was parsed as an Issue URL.
  After: The same test passes, checking Arms, Bar, stages, logs, Inbox and saved Trial results.
• Focused integration tests pass for Stop/resume, immutable Bar, two-worker interleaving, timeout cleanup, fresh Void Trial retries, and fixed Harness paths with distinct per-Arm flags.
• The Spec review's exact distinct_suite_task_pairs_keep_distinct_trial_results test failed with two saved results for four Trials, then passed after distinct suite/task identities received distinct paths. The regression is retained. A further retained regression verifies Stop during a retry keeps the original Void Trial budget.
• Before upstream reconciliation, /usr/bin/time -p env CARGO_BUILD_JOBS=2 scripts/affected-tests 85dd246df97ecd5c042cc0e4494ba9c1d2cd9bda passed on 4eb682b: all 43 site browser tests and all 2,421 Rust tests passed; Nextest skipped 58 existing tests. Wall time was 1,402.95s including compilation and site checks; Rust execution took 1,207.731s after 2m09s compilation. CI on that head also passed.
• After merging the advanced issue-977 base, 2b318f3 passes cargo check --all-targets, cargo fmt --check, cargo clippy --all-targets -- -D warnings, and cargo test --test experiments --test evals --test eval_arms --test command_surface (48 passed, 2 ignored). npm test --prefix site passes all 43 browser tests. The merge preserves the upstream setup-token credentials and private replay importer; imported tasks leave optional Trial inputs unset until configured for an Experiment.
• Final-head CI passes on 2b318f3, including the full Rust tests, formatting, Clippy, and site and Console browser checks. Linux and macOS release builds also pass.

Merge Danger

Door: two-way

Adds Experiment commands and private versioned artifacts; no migration of existing Command records.

Blast Radius: Experiments

Unaddressed findings

• Standards — statistical Bar settings (P3, optional Data Clumps): Kept the two public Bar schemas. The Experiment has a per-suite Trial-count map, timeout and Void rules; the Verdict has one Trial count and a prediction budget. src/evals/experiment.rs:60 takes the standing statistical defaults directly from verdict::Bar::default, so those defaults already have one source. This is the implementation's design decision; no Spec or Day shift decision is deferred. The state/result enums and the typed return from Arm preparation address the other two suggestions.
• Spec: None. The Trial identity finding was reproduced and fixed, with its test retained.

Review coverage

Independent Standards and Spec reviews initially compared against issue-977 at 85dd246df97ecd5c042cc0e4494ba9c1d2cd9bda, with a follow-up on 4eb682b. After upstream reconciliation, fresh independent reviewers compared the final head 2b318f3 against issue-977 at df4bb1ba2ae62173f159a6e396e96e232c96013b. Both axes read all 29 current changed files, including the generated manual output, and found no new findings. Generated-output exclusions: none. Unread changed files: none under either axis.

Test seams

• The experiment and experiments CLI, with real Git repositories, a private fixture suite, cached Arm builds and fake Harnesses.
• The Command record, Command log, Activity log, Inbox, show and stop CLI surfaces.
• The Arm's versioned --trial process protocol, including timeout and interruption cleanup.

Built with codex · gpt-6.1-sol · xhigh