[JacobStephens2]

Summary

Experiments measure a Candidate against a fresh Baseline before a change reaches production. This gathers the work for Spec #977: private Eval suites and replay import, Arm builds and fixed Harness versions, resumable paired Trials, grading and human rechecks, release checks, and live Console pages.

change → dependency-selected, approved Eval tasks
       → Baseline / Candidate builds + fixed Harness paths
       → interleaved Trials (two workers, provider proxy, bwrap, memory caps)
       → mechanical checks → blind model Graders → human Score rechecks
       → seeded paired Bayesian Verdict + wrong findings + pass^3 + Arm usage
       → experiments CLI / live Console / release wait

The CLI prepares time-safe repositories, approves proposed Eval tasks without starting work, and imports the private replay material once. Setup stores the Claude Trial token in Credentials. Trials and Graders retain private Session evidence and join the Experiment's Trace; Void retries, drop-outs, plan-limit pauses and interrupted work survive resume.

The whole-Spec review also fixes missed native Codex quota reports for current and older Arms, missing production user rules/hooks, stale Inbox groups after rechecks, and releases shipping changes absent from their measured Candidate. Executable lookup and metadata-table regression setup are shared.

Closes #977

Evidence

Each actionable behavioral finding was reproduced before its fix, and the regressions are retained:

Check	Before	After
Recheck → Inbox	Verdict became pass; status stayed Needs you	Confirmed pass moves the latest invocation to Done
Production user files	Claude rules and OpenCode instructions were missing	Real bwrap sessions retain Claude/Grok/Muse instructions, rules and hooks, plus OpenCode instructions
Codex native quota	Trial continued and succeeded after a native warning	Current Trials become void; older Code review and Spec review Arms pause with retained usage-limit evidence
Release resume and merged PR	A tag included arrived-during-pause.txt, absent from the Candidate	Changed main content is refused before publishing; an already-merged mismatched tree cannot be tagged

Final validation:

• /usr/bin/time -p scripts/affected-tests main: passed; 2,579 Rust tests passed (58 skipped), all 44 site tests passed, and the release-generator check passed. Overall wall time was 1,513.75s including compilation; the Rust test run took 1,341.37s after 81s compilation.
• cargo clippy --all-targets -- -D warnings and cargo fmt --check: passed.
• npm test --prefix console: 69 passed, 1 pre-existing skipped test.
• python3 -m unittest discover -s scripts -p 'test_*.py': all 14 passed.

Testing seams

• Whole binary against fake Harnesses: Command/Activity records, private artifacts, CLI views, resume, rechecks and Inbox groups.
• Standalone seeded Verdict: paired accuracy/wrong-finding intervals, drop-outs, pass^3 and forecasts.
• Sandbox launcher: real bwrap visibility/network/process checks and memory scopes where available.
• Console routes paired with CLI JSON and browser tests, including live Experiment pages and proposal approval.
• Release script against fake cargo/GitHub: waits, overrides, rechecks, interruption and matching measured/released content.

Unaddressed findings

Standards

• S1: statistical Bar data clump (P3 judgment), declined. src/verdict.rs:3-6,22-31 deliberately gives the standalone judge a scalar Trial count and prediction budget; src/evals/experiment.rs:41-55 persists per-suite counts, timeouts and Void rules. The explicit projection at src/evals/experiment/grading.rs:254-263 adapts those representations. CLI validation and the judge's validation protect separate entry seams and have different underflow/prediction checks. The Standards reviewer accepted retaining these interfaces. No behavioral defect was claimed. S2 and S3 were addressed.

Spec

• R3: agy provider-side web search (P1), deferred to #1215. src/harness/agy.rs:157-170 and docs/research/eval-sandboxes.md:831 record that no grounding-off switch is known. The supplied regression, cargo test --bin thirdshift harness::agy::tests::trials_do_not_keep_the_provider_grounding_invocation -- --exact --nocapture, failed with identical production and Trial arguments. It confirms the gap; the provider proxy cannot disable server-side grounding. The existing open needs-triage issue already records the evidence and the Day shift's choice between refusing unsupported Trials and permitting grounding. That product decision is not this review's to make. The full regression and patch remain with the Session's review artifacts; the temporary failing test is outside the active suite. R1, R2, R4 and R5 were addressed.

Security

• Security review of the issue-977 change against main (merge base d05aa97) found no unaddressed introduced findings: no failing proof-of-concept was produced, so no fix was committed here.

Review coverage

Both axes read all 121 changed files, including generated output. No changed file was left unread; no generated-output exclusions were taken. Final reports and each axis's complete files-read list are retained in .thirdshift-review-ARHrhS/standards.md and .thirdshift-review-ARHrhS/spec.md beside the Session trace.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

The code can be reverted; private Eval artifacts remain on the host. Real-login refresh, provider allowlists and production memory measurements remain Day shift checks, as recorded in the sandbox research.

Blast Radius: Factory

Session launching, Experiment scheduling/grading, Console views and release publication. Releases now refuse content changes that were absent from their fixed Candidate; a fresh release version measures current main.

Built with codex · gpt-6.1-sol · xhigh

Tickets

• The Verdict: a paired Bayesian interval and its rules #999 done
• The Eval task format, suites and time-safe repositories #1000 done
• The Trial flag: one session through a Run's code, one JSON line #1001 done
• Arm builds from commits, cached, with Harness versions fixed #1002 done
• thirdshift experiment: the Command, the Bar and running Trials #1003 done
• Code Graders and the Verdict for an Experiment #1004 done
• Model Graders with blind packets #1005 done
• Your rechecks of borderline and decisive Scores #1006 done
• Void Trials #1007 done
• Stopping at a plan-limit warning #1008 done
• Live checks for the Trial sandbox #1009 done
• A credentials section for Claude's setup token #1010 done
• Turn on lingering and save Claude's setup token #1011 done
• The Trial sandbox: bwrap, its own processes, fresh homes and logins #1012 done
• The Trial sandbox's network: an allowlisting proxy #1013 done
• Trials and Graders run in the sandbox #1014 done
• A memory cap for each Trial #1015 done
• The code-review suite and its Code review session #1016 done
• An import for #429's material #1017 done
• Run the import of #429's material on the thirdshift host #1018 done
• Prototype the Experiment pages #1019 done
• The Experiment pages in the Console #1020 landed with Add live Experiment pages to the Console #1270
• Approve a proposed Eval task #1021 landed with Approve proposed Eval tasks from the CLI and Console #1273
• Releases wait for their Experiment #1022 done
• Trials' Session traces in the Experiment's Trace, and each Arm's tokens and cost #1023 done
• Spec-review Trials have no Trial inputs or PR service #1251 landed with Complete Spec-review Trial inputs before Experiment builds #1272