[JacobStephens2]

Summary

Blind pairwise grading of each Baseline–Candidate pair of process Trials (Baseline Trial k with Candidate Trial k), by a Grader panel, by people, or both. The Bar fixes who decides before the first Trial (ADR 0028).

thirdshift experiment <candidate>
  cost::confirm             Arms' estimate, then per panel seat: sessions × frozen price      (#1884)
  workers::run              Trials, each with its code Grader Score
  pairs::run                per finished pair: base-tests + cross-tests, offline in the       (#1881)
                              code-Grader sandbox; each side's diff kept → pairs/<pair>.json
  model_grader::run         review Scores; Grader chosen by the family rule
  panel::run                pair × seat × order 1,2: blind packet → calls.json               (#1882)
                              stops at --panel-spending-limit (per run) and pauses           (#1884)
  refresh_verdict
    deciders::decide        who decides each judged pair under ADR 0028                       (#1886)
    grading::assess         Trial passes = code Grader passes && decider calls it ready
                              (counted modes only; statistics unchanged)

Console /experiments/<id>/pairs/<pair>   ⇄  thirdshift experiments <id> --pair <pair>      (#1885)
  Ticket · checklist · Change A | Change B (code before tests, evidence) — no Arm, no model call
  POST merge (sealed per side) → POST preference [+ rubric] → reveal Arms, panel, other readers
  append-only reviews/<pair>/<n>.json, attributed to the reader

How the Verdict is decided for each pair:

panel call  = per seat: orders agree ? call : tie
              ballots per family (default) | per Model; arm-family seats sit out while another family votes
              strict majority, else tie; tied merge → not ready
pair given to readers?  → people's call (ready only if every reader says so); not yet called → waiting_on_people
else panel counted?     → the panel's call stands in
else (people-only sample) → the pair is left out; each sampled pair of a process suite is one unit
both counted and people–panel agreement on the sample < --panel-agreement → inconclusive (says why)

• Bar flags: --panel-seat harness=…,model=…[,effort=…], --panel-vote family|model, --panel-arm-family-rule on|off, --panel off|beside|counted, --panel-families, --panel-agreement, --panel-spending-limit, --people off|beside|counted, --people-reader, --people-sample and --people-overlap. The Console's start options take the same flags.
• Providers (Allow Fireworks and Baseten through opencode, with a key per provider #1883):
  • The sandbox proxy allows api.fireworks.ai and inference.baseten.co.
  • OpenCode takes a key per provider under [opencode.keys], falling back to api_key.
• Result format 10: adds pairs, panel, panel_spend, people, bar.panel, bar.people, deciders, PairPage and the waiting_on_people state.
• Docs: the guide, docs/experiment-results.md, docs/eval-tasks.md, docs/trial.md, the glossary, code navigation, the Panel Grader prompt and the three hand-written site pages.

This review's fixes (760fcbba)

 grading::evaluate
-  unit = sample_only ? 1 : trials                         # every suite
+  unit = sample_only && suite has process tasks ? 1 : trials
 pairwise::page
   sides drawn from the seed
+  patch  = blind_diff(patch)                              # as the panel's packet
+  error  = "the evidence could not run"                   # git errors name trials/<hash>--baseline--…
 model_grader::prepare
-  first configured Model whose name differs from both Arms'
+  first configured Model from a family neither Arm runs on (panel::arm_families)
 panel::Sides::read
-  fs::read_to_string(task.directory.join(source))
+  pairwise::ticket(task)                                  # stays inside the task folder

The review also fixed two pieces of text. panel.rs and docs/experiment-results.md no longer say the panel is "never in" the Verdict. The guide's panel examples now seat Kimi through the Kimi Code Harness rather than OpenCode, as the Spec's Q6.4 says.

Evidence

Execution evidence

All three behaviour fixes started from the reviewer's failing test.

Spec P4: a sample-only Verdict resets every suite's statistics.

• Test: cargo test --bin thirdshift deciders::tests::a_sample_only_verdict_leaves_other_suites_statistics_alone
• Before:

  assertion `left == right` failed: {… "review":{… "bar":{… "trials_per_task":1}, … "retained_tasks":12 …}
    left: Number(1)
   right: 3
  

• After: test result: ok. 10 passed; 0 failed (all of deciders::tests).

Spec P1: the blind page shows identifying lines and error text.

• Test: cargo test --test console pairwise::identifying_lines_and_paths_stay_off_the_blind_page
• Before: the test panicked at tests/console/pairwise.rs:613. The page contained opus-fixture and unavailable: other Arm: Eval git failed: fatal: /h/trials/abc--baseline--0/attempt-1/repository.
• After: cargo test --test console pairwise → test result: ok. 8 passed; 0 failed.

Spec P6: the single model Grader excludes by exact Model name.

• Test: cargo test --test experiments model_graders::the_model_grader_comes_from_another_family_than_either_arm
• Before:

  assertion `left == right` failed: {"effort":null,"harness":{"name":"claude",…},"model":"judge-claude"}
    left: String("judge-claude")
   right: "judge-gpt"
  

• After: cargo test --test experiments model_graders → test result: ok. 14 passed; 0 failed.

Final validation: scripts/affected-tests refs/remotes/origin/main.

• Passed: npm test --prefix site, cargo build --locked and npm test --prefix console.
• cargo test had two failures, both in tests/process.rs findings::*, which this branch doesn't touch. They failed under the full run's load. Re-run alone, cargo test --test process findings:: → test result: ok. 18 passed; 0 failed.
• Regeneration checks: cargo test --bin thirdshift prompts_page → 14 passed. npm run docs:check --prefix site → exit 0.

Visual evidence

These captures were published by the Ticket PRs and are reused here. This review's fixes change only which diff lines and error text the blind page prints, not its layout.

Blind pair page, 1440×900, before any call (#1885). Before: the route doesn't exist.

Blind pair page, 390×844 (#1885).

Experiment page, People's calls with counted readers' queues, 1440×900 (#1886).

The Pair evidence table (#1881) has no published capture: that session couldn't upload one.

Seams under test

• thirdshift experiment candidate --json through the Experiment fixture, with fake Harness scripts as panel Graders (tests/experiments/{pairs,panel,model_graders}.rs).
• Console HTTP responses for the pair page, its POSTs and the Experiment page (tests/console/{pairwise,experiments}.rs).
• deciders::decide with grading::assess, for Verdicts under each Bar setting (src/evals/experiment/deciders/tests.rs).

Review coverage

Reviewed with thirdshift-code-review against refs/remotes/origin/main. Standards raised 12 findings and Spec raised 11. Every changed file was read by both axes, apart from generated output:

• prompts/panel-grader.md is excluded from both axes. Its source is src/prompt.rs, and cargo test --bin thirdshift prompts_page passes.
• site/docs/index.html is excluded from both axes. Its source is docs/guide.md, and npm run docs:check --prefix site passes.

Unaddressed findings

Standards

• S1. Credentials::read_from is pub(crate) for a test. Declined, as on Allow Fireworks and Baseten through OpenCode, with a key per provider #1942: the sandbox tests build OpenCodeKeys through the real Credentials parser (src/sandbox.rs:1993), not through a constructor that bypasses it.
• S3. Test deadline. The reviewer judged it compliant. The doc comment in tests/experiments/pairs.rs names what the 4s covers. No change.
• S4. Three Call types, two Preference types and two rubrics (Mysterious Name). The rubric and result shapes are open in Align the people's blind-page rubric and panel result shape with the Grader panel (#1882) #1947. Renaming before that decision would churn the same types twice.
• S5. Repeated pair walks (Duplicated Code). Declined, as on Estimate and cap the Grader panel's spend (#1884) #1951: panel::remaining must count unfinished pairs before any Trial, while panel::run skips them (src/evals/experiment/panel.rs, remaining/run).
• S6, the part about the git diff argument vector. The two call sites differ in purpose. pairs::diff splits the diff per file and keeps it; panel::Sides::read blinds the whole patch. The Ticket-read part of S6 is fixed.
• S7. Three seeded draws. Declined. Each draw answers a different question: the panel's first order, the reader's sample and the reader's sides. Keeping them independent leaves a reader's side A uncorrelated with the panel's first order. Merging them would change no result.
• S8. Repeated decided.calls match in grading::evaluate. Declined, as on Count the panel or people in the Verdict, with seeded samples for several readers #1953: only evaluate knows dropped_as_void (src/evals/experiment/grading.rs, evaluate).
• S9. WaitingOnPeople edits across five files. This follows the existing WaitingOnLeaks pattern (src/console/peek.rs, src/logs/reading/inbox.rs). Gathering them is a refactor outside this Spec.
• S10. Tuples and travelling parameters. These are judgement calls on private helpers with one or two callers. No change.
• S11. src/resend_key.rs owns OpenCode keys. That file is the Credentials module, which already holds every provider's key. Its name predates this Spec.
• S12. Small duplications in pairwise. These are judgement calls. pairwise::root() follows the six existing copies of the Experiments root, and gathering them all is outside this Spec.

Spec

• P2. A reader's queue sits on the Experiment page beside Arm-attributed Pair evidence. The reviewer's test a_readers_queue_is_not_shown_beside_arm_attributed_evidence failed as described: the page contains <td>Baseline</td><td><code>tests/left_test.py</code></td>. Moving the queue alone doesn't close the leak, because the Experiment page shows Arms, Trials and Scores to every reader. Whether to hide that evidence while a reader has pairs to read, to add a per-reader queue page, or to accept that the Experiment page isn't blind is for the Spec and the Day shift to decide → Keep a reader's queue of blind pairs away from Arm-attributed pair evidence #1956.
• P3. The people's rubric doesn't match the panel's. The reviewer's test readers_fill_in_the_panels_rubric failed (["correctness", "tests", "scope"] against ["criteria", "tests", "scope", "code"]). This is already open as a Spec decision → Align the people's blind-page rubric and panel result shape with the Grader panel (#1882) #1947.
• P5. A seat's family is its host. The reviewer's test kimi_is_one_family_whichever_harness_runs_it failed (left: "fireworks-ai", right: "moonshot"). That matches Spec: blind pairwise grading of process Trials by a Grader panel and people #1872's "Model families come from each seat's provider". Whether family should mean the Model's maker is open → Decide whether a Grader panel seat's family is its host or its Model's maker #1949.
• Minor: the per-run spending limit. Open → Should the Grader panel's spending limit span the whole Experiment rather than each run? #1952.
• Minor: integration coverage of panel-counted, stand-in and sample-only. Those Verdicts are tested at the deciders::decide + grading::assess seam (src/evals/experiment/deciders/tests.rs). Counted people is tested end to end through the Experiment fixture (tests/experiments/panel.rs), which exercises the same plumbing.
• waiting_on_people as scope creep. Declined, as on Count the panel or people in the Verdict, with seeded samples for several readers #1953. People's calls arrive after the run, so a counted Verdict must wait for them, as waiting_on_leaks waits for leak decisions (src/evals/experiment/store.rs, refresh_verdict).
• Sample-only statistics unit (P4's judgement part). Each sampled pair is one unit. This is open → Decide the statistical unit for a Verdict on the people's sample alone #1954.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Everything is opt-in through the Bar. Without --panel-seat or people flags, an Experiment runs as before, apart from base-tests and cross-tests for tasks that name process_grading.new_tests. The model Grader's choice is the exception: an Experiment can now pick a different Grader Model, when a configured Model shares a family with an Arm. Result format 10 is additive.

Blast Radius: Experiments

Consumers of Experiment JSON see format 10. The Console gains the pair page and the Experiment page gains its People's calls and Pair evidence sections. The sandbox proxy allows two more provider hosts, for OpenCode sessions only.

Closes #1872

Tickets

• Run base-tests and cross-tests for each pair of Trials #1881 landed with Run base-tests and cross-tests for each pair of Trials #1943
• Grade each pair of Trials with a Grader panel, reported beside the Verdict #1882 landed with Grade each pair of process Trials with a Grader panel, beside the Verdict #1948
• Allow Fireworks and Baseten through opencode, with a key per provider #1883 landed with Allow Fireworks and Baseten through OpenCode, with a key per provider #1942
• Estimate and cap the Grader panel's spend #1884 landed with Estimate and cap the Grader panel's spend (#1884) #1951
• Add the blind pairwise review page to the Console #1885 landed with Add the blind pairwise review page to the Console #1946
• Count the panel or people in the Verdict, with seeded samples for several readers #1886 landed with Count the panel or people in the Verdict, with seeded samples for several readers #1953

Built with claude 2.1.314 · opus · default effort