[JacobStephens2]

Closes #1882

Summary

The Bar can name a Grader panel. Each seat reads every finished pair of process Trials blind, once in each presentation order. Every seat's calls and the panel's vote are reported beside the Verdict, which they don't change.

experiment
  workers::run            Trials
  pairs::run              base-tests / cross-tests
  model_grader::run       Scores (review suites)
+ panel::run              for each pair × seat × order 1, 2:
+                           packet: Ticket, criteria, rubric, per side outcome + blinded diff + Pair evidence
+                           A/B shuffled per pair (Bar seed), swapped in order 2
+                           fresh sandboxed session → calls.json → validated Call by Arm
+                           saved per seat/order, so a resume asks only for missing calls
  refresh_verdict         unchanged

seat call   = orders agree ? that call : tie                     (merge: ready / not_ready / tie)
voters      = seats − arm-family seats, if --panel-arm-family-rule on (default) and another family sits
ballots     = per family (default; each family takes its own majority first) | per Model
panel call  = strict majority of ballots, else tie;  tied merge → not_ready

• Flags: --panel-seat harness=<h>,model=<m>[,effort=<e>] (repeatable), --panel-vote family|model and --panel-arm-family-rule on|off. They are part of the Bar, so changing them after the first Trial is refused. The Console's structured start options accept the same three.
• Families: a seat's family is its provider: the Harness's own, or the provider an OpenCode Model names.
• Result format 10: adds bar.panel and panel: [PanelPair], and the text view adds Panel … lines. docs/experiment-results.md, docs/eval-tasks.md (packet contract), the guide, the glossary's Bar entry, code navigation, the new Panel Grader prompt and the three hand-written site pages are all updated.
• Shared session: the model Grader and the panel now run through one private Grader session helper.
• Blinded diffs: added diff lines that report blinding would drop as identifying, such as Model: …, are removed from panel packets.

Evidence

Execution evidence

• Before (refs/remotes/origin/issue-1872 built in a temporary worktree): thirdshift experiment HEAD --panel-seat harness=claude,model=judge-one --json

  thirdshift: 08:33:09 unknown Experiment flag: --panel-seat
  exit=2
  

  After: cargo test --test experiments panel

  test panel::panel_settings_need_a_seat_and_named_values ... ok
  test panel::identifying_lines_an_arm_commits_stay_out_of_the_packet ... ok
  test panel::seats_from_an_arms_family_are_reported_but_do_not_vote ... ok
  test panel::with_the_rule_off_two_same_family_seats_are_one_vote_per_family ... ok
  test panel::a_three_seat_panel_reports_each_graders_calls_in_both_orders_and_its_majority ... ok
  test result: ok. 5 passed; 0 failed
  

• Acceptance criteria map to these fixture tests, which use fake Claude and Codex Graders:
  • The three-seat panel reports both orders and the per-pair majority.
  • A Grader that always prefers side A is a tie.
  • Two Anthropic seats are two votes per Model and one per family.
  • Anthropic seats are reported with votes: false while the Arms run on Claude, and vote with the rule off.
  • The Verdict's decisions and every Score match the same Experiment run without a panel.
• Unit tests: cargo test --bin thirdshift panel::tests covers a family split before the panel votes, OpenCode families and seat parsing. console::start::experiment covers the panel options.
• Final validation: scripts/affected-tests origin/issue-1872 on 4adf30b passed: site tests, Console tests, cargo build --locked and the full cargo test (795 s). Generated output was checked with cargo test --bin thirdshift prompts_page (14 passed) and npm run docs:check --prefix site (exit 0).

Visual evidence

None: no rendered UI changes. The Console Experiment page is unchanged; the blind pairwise page is #1885.

Merge Danger

Door: two-way

With no --panel-seat, the Bar has no panel and saved Experiments read as before. Format 10 only adds fields.

Blast Radius: Experiments

Format 10 changes format in the Experiment JSON and the OpenAPI constant. Only Experiments whose Bar names a panel run panel sessions. A panel Grader that fails or answers invalidly pauses the Experiment, as an unavailable model Grader already does, and the resume asks only for the missing calls.

Review

Reports: Standards found 8 findings and Spec found 6.

Addressed:

• S1: the prompt says "two Trials".
• S3: the diff deadline is now the Bar's timeout, and the null rule is reworded.
• S4: the shared Grader session; majority takes iterators; one flag string per Console array.
• S5: the preference is a serde enum.
• S6: Ballot, SeatCalls, shown_as_a and first_order; Seated was removed.
• P1: packet diffs are blinded. New test: identifying_lines_an_arm_commits_stay_out_of_the_packet.
• P5 (partly): the panel now runs after the model Grader.
• P6: per-Model ballots key by Model, as documented.

Review coverage:

• Changed after the review and not read by it: src/trial.rs, src/trial/reports.rs (blind_diff) and src/evals/experiment/model_grader.rs (the shared session). The fix commit also re-edited files the reviewers had read: panel.rs, panel/tests.rs, prompt.rs, experiment.rs, console/start/experiment.rs, the docs and tests/experiments/panel.rs.
• Generated output excluded from the Spec reviewer's reading: prompts/README.md, prompts/panel-grader.md, site/prompts/index.html and site/docs/index.html. The Standards reviewer read all of them. Regeneration was checked as listed under Evidence.

Unaddressed findings

Standards

• S2. The glossary has no entries for seat, family or presentation order. Declined here: docs/agents/domain.md sends new terms to /domain-modeling. "Grader panel" is already defined, and the docs define a seat's family where it is used.
• S5 (part). Family and the seat's Harness stay strings. The seat Harness follows the existing Setup.harness: String + Harness::named(..).expect(..) pattern (src/evals/experiment.rs Setup::choice). A family is shown and grouped, never matched against a closed set.
• S7. The format literal has no shared constant. That predates this change (format 9 was edited the same way in a6fc914); this change only follows the existing pattern.
• S8. arm_families is mild Feature Envy. It has one caller. The model Grader's exact-Model rule is a different rule, so there is no second user to move the walk onto Description for.
• S4 (part). panel::prepare and model_grader::prepare both reuse an Arm's pinned Harness. Kept separate: the panel also reuses earlier seats' pins, and the model Grader picks its Harness from User config.

Spec

• P2. Each packet side carries its outcome. Kept as part of "the evidence": a Trial that stopped to ask or failed has no pull request to merge, which the merge call needs. docs/eval-tasks.md documents it.
• P3. Small additions (Console options, Process Step families, format 10). No change needed. The Console options keep the CLI and API at parity (ADR 0026, "each Console serves an API that can do what the CLI can"). Format 10 is required by docs/experiment-results.md versioning.
• P4. A hosted Model's family is its host. Declined: Spec: blind pairwise grading of process Trials by a Grader panel and people #1872's Implementation Decisions say "Model families come from each seat's provider", and that is what the code does. Run: cargo test --bin thirdshift a_hosted_model_takes_its_makers_family failed with left: "fireworks-ai", right: "moonshot", as designed. Whether family should mean model lineage (docs/research/eval-grader-panel.md Q5.c) is a Spec decision, filed as Decide whether a Grader panel seat's family is its host or its Model's maker #1949.
• P5 (part). A panel failure pauses the Experiment. Running the reviewer's test, cargo test --test experiments panel::a_failing_panel_grader_does_not_withhold_the_verdict, gave invalid panel Grader response; see private Grader artifacts with left: Some(1), right: Some(0): the invalid response paused the Experiment. That is the intended behaviour: Spec: blind pairwise grading of process Trials by a Grader panel and people #1872 says the panel "pauses grading as an unavailable Grader does now", and a resume asks only for the missing calls. The panel now runs after the model Grader, so the pause holds back no model Score.

Security

None. The review checked the panel Grader sessions (same private session and bounded response read as the model Grader), the Harness pinned before Trials run, git diff in Trial repositories (a local probe found that repository config such as diff.external, textconv, core.fsmonitor and core.pager runs nothing under these flags), and the Console's panel_* options (keys are checked against the schema and values re-parsed).

Built with claude 2.1.314 · opus · default effort

Security review outcome

No unaddressed introduced findings.