[JacobStephens2]

Closes #1886

Summary

The Bar now records who decides a Verdict, as ADR 0028 describes. The Grader panel and the people are each off, beside or counted. The people's setting also names the readers and fixes a seeded sample per reader with an overlap.

thirdshift experiment <candidate> --panel-seat … --panel counted \
  --people counted --people-reader ann@x --people-reader bo@x \
  --people-sample 7 --people-overlap 4 [--panel-families 2] [--panel-agreement 0.8]

The merge call that counts for each judged process pair (deciders::decide):

pair given to readers?      → people's call: ready per Arm only if every reader calls it ready
                              (their latest blind call); a reader yet to call → Waiting
else panel counted?         → the panel's merge call (stands in)
else (people-only sample)   → the pair is left out; each sampled pair is one unit of the statistics

How each pair's call feeds the Verdict (grading::assess):

 Trial passes = code Grader passes
+            && (counted mode) whoever decides its pair calls it ready
 Verdict
+  waiting_on_people   while a pair given to a reader has no blind call
+  inconclusive (why)  both counted, and people–panel agreement on the sample < the Bar's level

The rule for a panel standing in for people: with both counted, a voting panel that spans fewer families than --panel-families is refused before the Experiment is created.

Where the change shows:

• Queues appear on the Console's People's calls section, in the text view, and under people.queues.
• People–people agreement is now reported beside people–panel agreement.
• The Verdict carries a deciders report, which the Console and the CLI show as a line.
• A reader's call recalculates a completed Experiment's Verdict. Rechecks, leak decisions and these calls now share one publish step, which also releases the Trial repositories.

File layout:

src/evals/experiment/
├── deciders.rs      # new: roles, PeopleSetting, seeded queues, start checks, decide
├── panel.rs         # role, families, agreement; voting families; stand-in check
├── grading.rs       # counted calls, sample-only units, agreement gate, waiting_on_people
├── pairwise.rs      # people–people agreement; refuse calls when people off; recalc
└── store.rs         # refresh_completed, publish_decided (shared with rechecks/leaks)

Evidence

Execution evidence

• Before: cargo test --test experiments a_counted_peoples_call_that_settles_the_verdict_releases_trial_repositories was run on a212c9f, before the review fix (Standards finding 1).

  thread 'panel::a_counted_peoples_call_that_settles_the_verdict_releases_trial_repositories' panicked at tests/experiments/panel.rs:567:5:
  still retained: [".../trials/856c…--baseline--0/attempt-JNZv2Z/repository", ".../trials/856c…--candidate--0/attempt-DJEokd/repository"]
  test result: FAILED. 0 passed; 1 failed
  

  After: the same command on 418933b:

  test panel::a_counted_peoples_call_that_settles_the_verdict_releases_trial_repositories ... ok
  test result: ok. 1 passed; 0 failed
  

• The acceptance criteria are covered by new tests. --people, --panel counted and their flags do not exist on the base, so these tests have no Before run.
  • cargo test --bin thirdshift deciders::tests: 9 passed. They cover:
    • the four combinations on fixture data;
    • two readers sharing exactly the overlap, where a split call is not ready;
    • a stand-in with low agreement giving inconclusive (people–panel agreement on the sample is 4/8 merge calls, below the Bar's 0.8, …);
    • waiting on readers.
  • cargo test --test experiments -- panel::counted panel::a_stand_in: 2 passed. These cover the family refusal at start, and a counted people's CLI call deciding a real Experiment.
  • Console: cargo test --test console -- pairwise::each_readers_queue pairwise::with_people_off experiments::a_verdict_on_the_peoples_sample.
• scripts/affected-tests origin/issue-1872 on 418933b:

  PASS npm test --prefix site (37.23s)
  PASS cargo build --locked (0.10s)
  PASS npm test --prefix console (79.48s)
  PASS cargo test (938.00s)
  

  An earlier run failed one Console browser case, inbox.test.cjs "Landing, Ready and Stopping rows…". It is a theme-colour check unrelated to this diff. The file then passed alone (ℹ pass 80 ℹ fail 0), and the full rerun above passed.

Visual evidence

Both captures are of the Console Experiment page, People's calls section, at 1440×900. The fixture is a paused process Experiment with three pairs. The base build reads the same fixture without the fields it can't hold: the panel's role, families and agreement, and bar.people.

After: people are counted, with readers ann and bo, a sample of 2 and an overlap of 1. Each reader's queue shares exactly one pair.

Merge Danger

Door: two-way

Format 10 of the Experiment results is unreleased on this Spec branch, so this extends it in place. It adds bar.people, bar.panel.role/families/agreement, verdict.deciders, people.queues, people.people_agreement and the waiting_on_people Verdict.

Blast Radius: Experiments

Without the new flags, a Bar keeps today's behaviour: no panel, and people beside with no named readers. Experiments saved without the fields read as before. The Inbox and Peek now treat waiting_on_people as needing you. The release scripts don't list it, because a release Bar never counts people.

Review

Both axes reviewed a212c9f against refs/remotes/origin/issue-1872. Standards raised 10 findings and Spec raised 8.

Every changed file in that commit was read by both reviewers, except one generated file. site/docs/index.html was not read; it is generated from docs/guide.md, which was reviewed, and npm run docs:check --prefix site passed.

The review-fix commit 418933b changed four files the review never saw:

• src/evals/experiment/leaks.rs
• src/evals/experiment/rechecks.rs
• tests/console/experiments.rs
• tests/experiments/panel.rs (the added test)

Addressed:

• Standards 1: the retention bug, fixed through the shared Stored::publish_decided.
• Standards 2: the Verdict's gate count is now sample_agreement.
• Standards 5: the setting is renamed to PeopleSetting.
• Standards 7: the render's role switch, doc comment and capitalisation are fixed.
• Standards 8: the BTreeSet import.
• Standards 9: the module doc is updated.
• Spec 1: sample-only Verdicts now say "pairs".

Unaddressed findings

Standards

• 3: Feature Envy / Repeated Switches. The deciders counts are filled in grading::evaluate. evaluate is the only place that knows whether a task is Void-dropped (dropped_as_void, grading.rs). Waiting or unread pairs in Void-dropped tasks must not hold or count toward the Verdict, so the counts have to be made there.
• 4: Data Clumps. The (baseline, candidate) bools are not moved to panel::Merges. Merges holds three-valued Merge, including Tie, but a decided call is two-valued, since a tie is already resolved to not ready. Reusing Merges would bring back a state that cannot occur.
• 6: Shotgun Surgery / Primitive Obsession. WaitingOnPeople follows the existing pattern for WaitingOnLeaks, which has the same string lists in peek.rs and inbox.rs. Gathering them is a refactor outside this Ticket. The release scripts can't see waiting_on_people, because scripts/release.sh passes no people flags.
• 10: Blocking lock in refresh_completed. It runs only after reading state == Completed. A run of a completed Experiment takes the lock only to print and return (experiment.rs: "Experiment … is already completed"), and rechecks and leak decisions hold it briefly. With try_lock, a call landing during a recheck would leave the Verdict stale at waiting_on_people.

Spec

• 2: Each sampled pair is one statistical unit, though Spec: blind pairwise grading of process Trials by a Grader panel and people #1872 says "The statistics are unchanged". With three Trials per task, a sample drawn by pair leaves tasks partly read. The unchanged per-task statistics (verdict::validate needs the Bar's Trials in both Arms) can't take a partial task, so this PR keeps the method and makes the pair the unit. Whether that is acceptable, or whether the sample should be drawn by task instead, is a Spec decision for the Day shift → Decide the statistical unit for a Verdict on the people's sample alone #1954
• 3: A new waiting_on_people state and recalculation. People's calls arrive after the run ends. Without a waiting state, a counted Verdict would be computed before the calls it rests on. This follows the existing waiting_on_leaks and waiting_on_rechecks (store.rs refresh_verdict).
• 4: The 0.8 default for --panel-agreement. The Bar must hold some level, and Spec: blind pairwise grading of process Trials by a Grader panel and people #1872 names none, so I chose 0.8. It can be overridden per Experiment, but the default is the Day shift's call → Decide the default people–panel agreement level for a panel standing in for people #1955
• 5: Refusals when people are off, and when a task lacks new_tests.
  • Under ADR 0028, "off" means the people don't grade, so a call there would be recorded for nothing.
  • Pairs without new_tests have no pair page (pairs::run skips them). A reader given such a pair could never call it, and the Verdict would wait forever. Refusing at start is the explicit alternative.
• 6: The family re-check in panel::prepare comes after the Arms are built. The check at start (panel::check_stand_in in experiment::run, before Stored::open) covers both Arms' Harness and Model. Only Process Steps on another family are known later, and that re-check still refuses before the first Trial, which is what ADR 0028 requires.
• 7: Agreement appears in the Verdict line only when both are counted. Both agreements are always reported under people: in the Console, the text view and JSON. The Verdict line shows the sample agreement where it decides the Verdict.
• 8: Arm families can count toward the stand-in rule. This follows ADR 0028's wording, "the voting panel must span a set number of families", together with the arm-family rule that's already in place. ADR 0028 governs over grill Q5.1(a).

Built with claude 2.1.314 · opus · default effort

Security review outcome

No unaddressed introduced findings.