[JacobStephens2]

Summary

The Grader panel's cost is estimated per seat before the Experiment, reported apart from the Arms, and capped by an optional Bar setting, --panel-spending-limit <usd>.

 experiment
   cost::confirm
     Trial groups, worst case, estimate
+    Grader panel seat N (harness/model): S sessions (P pairs × 2 orders), estimate $X at $Y per <basis>
+    Grader panel estimate: $T for priced seats, apart from the Arms
+    Grader panel spending limit: $L per run, then grading pauses
   workers, pairs, model Grader
   panel::run
     for each owed call (pair, seat, order)
+      if this run's priced panel spend >= limit: pause
       Grader session -> save call
+      price the new session(s) from the frozen table (unpriceable -> pause; call kept)
 experiments <id> [--json]
   accounting (Arms only, unchanged)
+  panel_spend {sessions, cost_usd, seats[]}   # every retained panel session, retries included

• Estimate: each seat's remaining sessions, priced from the Experiment's frozen table. The price per session is the seat's measured mean in this Experiment once it has priced sessions, and otherwise the research's assumed 400k-input (80% cache reads), 15k-output session. A seat whose Model has no frozen price is listed as such.
• Spending limit: counts the spend of each run of the command and is checked before each panel session. Reaching it pauses the Experiment, as an unavailable Grader does. A resume gets the limit afresh and never regrades a saved call. Whether the limit should instead span the whole Experiment is filed as Should the Grader panel's spending limit span the whole Experiment rather than each run? #1952 for the Day shift (see below).
• A resume with only panel calls left asks Run the remaining Grader panel sessions? [y/N] on a terminal.
• Docs: bar.panel.spending_limit, panel_spend, PanelSpend and PanelSeatSpend are added within the unreleased format 10 of docs/experiment-results.md. The guide, the glossary's Bar entry, code-navigation and the three hand-written site pages are updated, and the manual is regenerated.

Evidence

Execution evidence

• Before (base origin/issue-1872, with this branch's tests/experiments/panel.rs): cargo test --test experiments panel::

  thread 'panel::a_low_spending_limit_pauses_grading_and_a_resume_continues_from_completed_calls' panicked at tests/experiments/panel.rs:449:5:
  assertion `left == right` failed: thirdshift: unknown Experiment flag: --panel-spending-limit
  thread 'panel::the_estimate_lists_each_panel_seats_expected_cost' panicked at tests/experiments/panel.rs:433:9:
  ...
  test result: FAILED. 5 passed; 5 failed; 0 ignored; 0 measured; 173 filtered out
  

  After (9ea1047c), the same command:

  test panel::the_estimate_lists_each_panel_seats_expected_cost ... ok
  test panel::a_low_spending_limit_pauses_grading_and_a_resume_continues_from_completed_calls ... ok
  test panel::an_unpriceable_panel_session_under_a_limit_keeps_its_completed_call ... ok
  test panel::a_resume_with_only_panel_calls_left_asks_about_the_panel_alone ... ok
  test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 173 filtered out
  

• Review fix: an_unpriceable_panel_session_under_a_limit_keeps_its_completed_call on 72fddea9 failed with assertion left == right failed: a completed call was graded again / left: 3 / right: 2. On 9ea1047c it passes, because the call is saved before its session is priced.
• Validation on 9ea1047c:
  • cargo test --test experiments: 182 passed, 0 failed.
  • cargo test --bin thirdshift: 1710 passed, 0 failed.
  • cargo test --test console: 274 passed, 0 failed.
  • npm test --prefix site: 68 passed, 0 failed.
  • npm run docs:check --prefix site: passed.
  • cargo clippy --all-targets: clean.
• Validation on 72fddea9: scripts/affected-tests origin/issue-1872 selected the site, Console and full Rust suite checks. The site and Console checks passed. The full cargo test --no-fail-fast run, which outlasted the script's foreground window, passed with 0 failures.

Visual evidence

These are text-only additions to existing paragraphs of the hand-written pages, with no CSS or layout change. Each pair is at 1440×900 desktop, scrolled to the Grader panel paragraph. No mobile capture was taken, because responsive layout is unaffected.

Home (site/index.html):

How it works:

Architecture:

Review coverage

Both reviewers read every changed file except site/docs/index.html. That file is generated from docs/guide.md by site/generate-docs.mjs, and both reviewers confirmed the exclusion: they reviewed the guide's diff and re-ran npm run docs:check --prefix site, which passed. It is intentionally unread. No changed file was left unread otherwise.

The fixes for Standards 3–6 and 8 and Spec 1 (in 9ea1047c) were not re-reviewed.

Unaddressed findings

Fixed in 9ea1047c:

• Standards 1 and Spec 2 (an unpriceable session was graded again on every resume).
• Standards 3 (shared options::positive), 4 and 5 (SeatEstimate, SessionPrice, SessionTokens, with the basis rendered from the constant), 6 (SeatSpend::mean_usd; the dropped spend error is now commented as the same narrowing as unreadable Arm history) and 8 (cost::counted).
• Spec 1 (the panel-only question).

Standards

• 2. Duplicated loop shape between panel::remaining and panel::run (judgement). Declined. The two loops apply different rules:

  • run owes a call only once both Trials of the pair have finished (src/evals/experiment/panel.rs:617, .any(|identity| !trial_results::finished(...))), and it reads each pair's sides once, lazily.
  • remaining (src/evals/experiment/panel.rs:566) must count unfinished pairs too, because the estimate runs before any Trial.

  A shared iterator would have to carry both rules.

• 7. Files seam versus Native and Path::exists (judgement). Declined.

  • The Files seam (src/evals/experiment/trial_results/files.rs:13) serves the read-only Experiment views, which is where panel::spend is used from reading.rs.
  • panel::run launches sessions and writes calls on the live directory, as model_grader::run does (src/evals/experiment/model_grader.rs:113).
  • remaining feeds the live pre-run estimate.

Spec

• 3. The limit applies per run, so repeated resumes can spend without bound. Declined, because this is a decision for the Spec and the Day shift. Filed as Should the Grader panel's spending limit span the whole Experiment rather than each run? #1952 (needs-triage). The issue's wording is "resuming continues from the completed calls". The Bar can't change after the first Trial (src/evals/experiment/store.rs, "Bar cannot change after the first Trial"), so a whole-Experiment limit could never be lifted by a resume. The per-run reading is documented in the guide and the results format.
• 4. A spending-limit pause logs the Command outcome as failed. Declined; the reviewer also did not raise it as a defect. The issue asks for the pause "as an unavailable Grader does now", and an unavailable panel or model Grader session today also ends as failed: only session::limits::Reached is logged as paused (src/evals/experiment.rs:495, otherwise "failed" at :505). The Experiment's own state is paused, with the reason in pause_reason.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

New optional Bar field and view fields within the unreleased format 10. Existing Experiments deserialize unchanged because spending_limit defaults to absent.

Blast Radius: small

Only Experiments that name a Grader panel see new estimate lines, panel_spend or the limit. The Arms' accounting, the Verdict and the cost ceiling are untouched.

Closes #1884

Built with claude 2.1.314 · opus · default effort