[JacobStephens2]

Summary

Each pair of Trials with retained evidence gets a blind page in the Console, with a CLI twin. People's calls are reported beside the Verdict.

GET  /experiments/<id>/pairs/<pair>               experiments <id> --pair <pair>
       Ticket · numbered acceptance checklist
       Change A | Change B   (code before tests, then evidence; sides drawn from the Bar's seed)
       no Arm, build, Harness, Model or model call
POST .../merge        a=ready|not-ready b=…        --merge <a> <b>          sealed until the preference
POST .../preference   preference=a|b|tie|both-bad  --prefer <p> [--rubric <criterion>=<p>]...
       → revealed: Arm and Setup per side, the Grader panel's calls, every reader's calls

 pairs::run
   Evidence::side
+    diff(own commit vs base)        # kept: Trial repositories are removed after completion
 reading::Observation
   pairs
+  people = pairwise::people(reviews/<pair>/<n>.json, pairs, seed)
+           # Arm-terms reviews, `blind`, people–panel agreement
 Experiment page
   Verdict
+  People's calls (links to blind pairs, calls table, agreement)

• Records: append-only files reviews/<pair-id>/<n>.json, attributed with the shared Actor (Console signed-in reader, or the Unix user on the CLI).
• Merge calls: a second merge call before the preference is refused. A review begun after the reader's own reveal is kept, but marked blind: false and left out of agreement.
• Result format 10 adds PairSide.diff, optional Pair.panel, people, and the documented PairPage that --pair --json emits.

Closes #1885

Evidence

Execution evidence

• Before: with the merge-call guard missing, cargo test --test console pairwise::sealed_merge_calls_cannot_be_replaced_or_remade_after_the_reveal (the Spec reviewer's test) failed:

  panicked at tests/console/pairwise.rs:453:5:
  assertion `left == right` failed
    left: 200
   right: 422
  test result: FAILED. 0 passed; 1 failed
  

  After: cargo test --test console pairwise → test result: ok. 5 passed; 0 failed
• cargo test --test experiments pairs::each_arms_new_tests_run_on_the_base_and_the_other_arms_commit → ok. It covers the real diff capture, plus CLI calls checked against the documented PairPage and Experiment contract.
• scripts/affected-tests origin/issue-1872 on f3aa5f7:
  • npm test --prefix site: PASS
  • npm test --prefix console: PASS, 171 tests
  • cargo test: PASS, 4082 passed, 0 failed, 866s
• The first affected run's Console browser tests failed only because console/node_modules was not installed (Cannot find module 'playwright'). They pass after npm ci --prefix console.

Visual evidence

Blind pair page, 1440×900, before any call. Before: the base Console has no such route and serves an empty 404.

Blind pair page, 390×844, before any call. The two changes stack on narrow screens.

Experiment page, 1440×900, after one CLI review. People's calls appear beside the Verdict.

Review

Reviewed with thirdshift-code-review against refs/remotes/origin/issue-1872: Standards found 10 findings, Spec found 7.

Addressed:

• P1: sealed merge calls are now refused before the preference.
• P2: a review begun after the reveal is marked not blind and kept out of agreement.
• P7/S3: the drift check now reaches people, panel and PairPage; --pair JSON and the record files are documented.
• S4: one list of preference values and one next-step command.
• S5: the changed-path listing is shared.
• S6: the Ticket input is read by ISSUE_FILE and must be a file.
• S7: one module path.
• S8: one reveal line for the CLI and the page.
• S9: the pages test reads the format number from the contract.
• S10: the workflow page describes the blind page.

Review coverage: both reviewers read every changed file except site/docs/index.html. That file is generated from docs/guide.md; it was excluded with npm run docs:check --prefix site passing, and was left unread by both axes. The review covered commits 934eb93 and bdfabac. The fix commit f3aa5f7 was not re-reviewed.

Unaddressed findings

Standards

• S1: the people setting is hard-coded as "beside" (ADR 0028). ADR 0028's people setting (off, beside or counted, with sample and overlap) is the scope of Count the panel or people in the Verdict, with seeded samples for several readers #1886: "the panel and the people are each off, beside or counted. The people's setting takes a sample size per reader, an overlap and a list of readers." This page reports beside, as Add the blind pairwise review page to the Console #1885 asks ("their calls are reported beside the Verdict").
• S2: Pair.panel is never written (speculative). Add the blind pairwise review page to the Console #1885's acceptance criteria require the reveal of "the panel's calls" and "people–panel agreement". Neither can be shown or tested without a read side for the panel's result. Grade each pair of Trials with a Grader panel, reported beside the Verdict #1882, which is open, produces it. The question of whether this contract shape suits Grade each pair of Trials with a Grader panel, reported beside the Verdict #1882 is filed as Align the people's blind-page rubric and panel result shape with the Grader panel (#1882) #1947.

Spec

• P3: the reader's rubric doesn't match the panel's per-side rubric. The rubric's criteria and shape belong to the panel's design, which Grade each pair of Trials with a Grader panel, reported beside the Verdict #1882 and the Spec settle. The research explicitly left them out of scope (docs/research/eval-grader-panel.md:14, "The rubric's criteria"). Filed as Align the people's blind-page rubric and panel result shape with the Grader panel (#1882) #1947.
• P4: no panel data in production. This is blocked on open Grade each pair of Trials with a Grader panel, reported beside the Verdict #1882, which writes the panel's calls. The read side is tested with retained panel calls in tests/console/pairwise.rs.
• P5: blinding holds only on the pair page. The acceptance criterion covers "The page's HTML", and tests/console/pairwise.rs::the_blind_page_names_no_arm_and_shows_the_ticket_checklist_diffs_and_evidence passes on it. A per-reader queue as the reader's entry point is Count the panel or people in the Verdict, with seeded samples for several readers #1886's "each reader's queue in the Console lists their sampled pairs".
• P6: PanelCalls is fixed in format 10 ahead of Grade each pair of Trials with a Grader panel, reported beside the Verdict #1882. This is the same decision as P3/S2. Filed as Align the people's blind-page rubric and panel result shape with the Grader panel (#1882) #1947.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

This is a new page, new routes and new private files under each Experiment. Result format 10 is additive. Older retained pairs read with an empty diff and no panel.

Blast Radius: Experiments

Only Experiment JSON consumers see the format bump. The Console's other pages are untouched apart from the new section on the Experiment page.

🤖 Generated with Claude Code

Built with claude 2.1.314 · opus · default effort