[JacobStephens2]

Summary

After the Trials, each pair (Baseline Trial k with Candidate Trial k) gets base-tests and cross-tests, as evidence beside the Scores:

 experiment::run
   workers::run              # Trials, each with its code Grader Score
+  pairs::run                # per finished pair without retained evidence
+    for each Arm
+      new tests = files its final commit adds/changes since base matching new_tests.paths
+      base-test  = overlay them on base,              run new_tests.command per file
+      cross-test = overlay them on the other Arm's final commit, same commands
+      (#1484 overlay, offline in the code-Grader sandbox, Bar's timeout per run)
+    write pairs/<pair-id>.json   # never a Score
   model_grader::run

A task opts in through its manifest. The Spec doesn't say how an Arm's new tests are found or run, so this adds:

[process_grading.new_tests]
paths = ["tests/*.rs"]                                       # * within one path component
command = ["cargo", "test", "--offline", "--test", "{stem}"] # once per file; {path}, {stem}

The Experiment view's result format goes from 8 to 9, adding pairs. Each PairSide has test_files, fails_on_base, passes_on_other, disagreement, base/cross checks and error.

• Only checks that reached an exit status decide those booleans. A timeout or an unpreparable snapshot leaves them null.
• The text view prints a line per disagreement, per new test that passes on the base, per unfinished run and per error.
• The Console's Experiment page gains a Pair evidence table.
• Docs updated: docs/eval-tasks.md, docs/experiment-results.md, the guide, the glossary (Base-test, Cross-test), code navigation and the three hand-written site pages.

Evidence

Execution evidence

• Before: cargo test --test experiments pairs:: on the unchanged code, with the new tests added:

  thirdshift: invalid manifest …/task.toml: TOML parse error at line 30, column 18
  30 | [process_grading.new_tests]
  unknown field `new_tests`, expected one of `real_fix`, `test_files`, `test_fakes`, `fail_to_pass`, `pass_to_pass`
  …
  test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 165 filtered out
  

  After: the same command:

  test pairs::a_missing_other_arm_keeps_the_base_test_and_names_the_error ... ok
  test pairs::an_arm_without_new_tests_has_nothing_to_run ... ok
  test pairs::each_arms_new_tests_run_on_the_base_and_the_other_arms_commit ... ok
  test pairs::new_tests_run_offline ... ok
  test pairs::tasks_without_new_tests_have_no_pair_evidence ... ok
  test pairs::an_unpreparable_snapshot_decides_nothing ... ok
  test pairs::a_timed_out_base_run_is_not_reported_as_failing_on_the_base ... ok
  test pairs::new_tests_share_the_bar_timeout ... ok
  test result: ok. 8 passed; 0 failed
  

  How the acceptance criteria are covered:
  • JSON per Arm: each_arms_new_tests_run_on_the_base_and_the_other_arms_commit checks that the JSON shows, per Arm, whether its new tests fail on the base and pass on the other Arm's commit.
  • Disagreement, no Score change: the same test makes the Candidate's test fail on the Baseline's commit. That shows as disagreement: true while both Scores still pass.
  • No network: new_tests_run_offline runs under bwrap, which was present and not skipped.
  • Bar's timeout: new_tests_share_the_bar_timeout.
• Other suites:
  • cargo test --test experiments: 169 passed.
  • tests/console/experiments.rs: Pair evidence section and OpenAPI format 9.
  • npm test --prefix site and npm test --prefix console: pass.
  • npm run docs:check --prefix site: pass.
  • cargo clippy --all-targets: clean.
• scripts/affected-tests refs/remotes/origin/issue-1872 (full suite):
  • Every test binary passes except one test in an unrelated area, tests/process.rs findings::merged_fix_is_rechecked_after_a_later_fix_fails (Security fix publishing). It fails in this worktree only.
  • The same test passes in clean worktrees of both this branch's HEAD and issue-1872, so local worktree state causes it, not this change.
  • The binaries after tests/process.rs were then run with --no-fail-fast: all 51 pass.

Visual evidence

Before/After captures of the Console Experiment page with one retained pair. They are at 1440×900 and 390×844, made with Playwright's Chromium against the issue-1872 build and this branch.

They could not be published: this session's permission settings blocked uploading them to a secret Gist. They stay local at /tmp/before-desktop.2boKEZ.png / /tmp/after-desktop.26EK0P.png (desktop) and /tmp/before-mobile.J4IfoE.png / /tmp/after-mobile.5fKWVh.png (mobile).

• Before: no Pair evidence section.
• After: a Pair evidence table after Process Trial outcomes. It has columns for Pair, Arm, New tests, On the base (Fails/Passes) and On the other Arm (Passes / Fails: disagreement / Did not finish / Unavailable).

Merge Danger

Door: two-way

• The result format moves from 8 to 9 and gains one new optional manifest table.
• Nothing changes for Eval tasks without new_tests.
• Saved Experiments read unchanged and show pairs: [].

Blast Radius: Experiments

• Readers that pin format: 8 must accept 9; the OpenAPI const and docs moved together.
• Pair evidence runs after the Trials, before Model Graders. Each run adds up to the Bar's timeout per pair side.

Review

Two-axis review against refs/remotes/origin/issue-1872.

Standards raised 15 findings.

• Fixed: 1 (contract null rules, with a working version of its test), 2, 3, 4, 6, 8 (the Run → GraderRun part), 10, 11, 12 (Ran → CheckRun, exited) and 15.
• Finding 1's test as written didn't reach the Baseline's repository (its relative paths missed), so it failed on setup, not on the finding. a_missing_other_arm_keeps_the_base_test_and_names_the_error replaces it, and its run does set error.
• Finding 2's test failed as described and is kept, renamed an_unpreparable_snapshot_decides_nothing.

Spec raised 5 findings.

• Fixed: finding 5, whose test a_timed_out_base_run_is_not_reported_as_failing_on_the_base failed as described and is kept, and finding 7 (the test name).

Unread changed files: none. site/docs/index.html is generated output, excluded by both reviewers after npm run docs:check --prefix site passed, and not read.

Unaddressed findings

Standards

• 5. Data Clumps (suite, task, trial). Pair is the format-9 contract record (docs/experiment-results.md:426), and Identity also carries an Arm. pairs::run now builds the Identitys once (src/evals/experiment/pairs.rs:65). No further change.
• 7. disagreement duplicates passes_on_other. The acceptance criterion asks that "A cross-test failure is shown as a disagreement". The contract states the field is true exactly when passes_on_other is false (docs/experiment-results.md:77, src/evals/experiment/pairs.rs:274).
• 8 (glossary part). No "Pair evidence" glossary entry. The concept is defined by the new Base-test and Cross-test entries (GLOSSARY.md:412, GLOSSARY.md:416). "Pair evidence" is only a section heading.
• 9. Repeated Baseline/Candidate switches. This follows the established pattern, e.g. src/evals/experiment/store.rs:169.
• 13. execute as Middle Man over execute_runs. execute keeps the API for its caller at src/evals/experiment/grading.rs:79, so Scores' Check shape and callers stay untouched.
• 14. Six-parameter list. This predates the diff (src/sandbox/code_grader.rs:85).

Spec

• 1. Evidence in every panel Grader's packet. This is parent-Spec scope, owned by open Grade each pair of Trials with a Grader panel, reported beside the Verdict #1882 (its packet "carries the Ticket, both diffs, the evidence and the rubric"). The blind page is Add the blind pairwise review page to the Console #1885.
• 2. No blind pair page. The reviewer accepted the Console section, text view and JSON for this ticket. The blind page is Add the blind pairwise review page to the Console #1885.
• 6. Timeout per run rather than per pair side. This is the documented design: "Each run shares the Bar's timeout with preparing its snapshot" (docs/eval-tasks.md:139). The reviewer read it as intended.

Closes #1881

Built with claude 2.1.314 · opus · default effort

Security review outcome

No unaddressed introduced findings.