Deepen the Grader panel's retained-call lifecycle #1968

What to build

Give Grader panel cost estimates, execution and resume, and reports one owner for retained calls. They should agree on which seat and presentation orders are complete, so an Experiment resumes without paying for a completed call again and reports a pair only when every seat has read it in both orders.

Report an unreadable retained call before launching more panel sessions, including when an earlier order is still missing. Today estimates and execution treat a file's presence as completion, while reporting decodes it and stops examining a pair at its first missing order. A malformed later call can therefore stay hidden while missing earlier calls are paid for.

Acceptance criteria

• One deep Grader panel module owns expected call identities, retained-call capture, pending work, pair completeness, and durable call publication. Cost estimation, execution/resume and reporting consume its interface; their independent task × Trial × seat × order completion traversals are removed.
• Every process Eval task is included, whether judged or measured, and non-process Eval tasks are excluded. Task order, Trial index, seat order and presentation orders 1 then 2 remain stable. With no panel configured, no panel call files are read or written.
• Estimates count every missing order, including pairs whose Trials have not finished, and count each pending pair once per seat. Execution launches missing calls only after both Trials satisfy the existing terminal predicate. A report is produced only with both orders for every seat, using the existing tally unchanged.
• Capture examines the complete expected identity set rather than stopping at a missing order. Only a not-found read means pending. Other read errors and saved-call decoding errors propagate with pair, seat and order context, including a malformed later order behind an earlier missing one. Unexpected files outside the expected identity set are ignored as before.
• Fresh responses retain their existing response validation and Arm mapping. Publication uses the existing atomic write, marks a call complete in memory only after it succeeds, and does not replace a completed call. A failed publication leaves that work pending. A subsequent pricing failure preserves the successfully published call for resume.
• Saved call filenames and JSON, Grader workspace names, Experiment result format 10, fixed seat executables, seeded presentation, voting and family rules, the current per-run spending limit, and separate panel/Arm accounting remain compatible. Loading saved calls keeps the current structural decoding rules rather than adding new rubric or evidence requirements.
• Shared-interface tests cover absent, partial, multi-seat and complete histories, process-task membership, errors hidden behind earlier missing orders, stable captured projections, successful publication and failed publication. Command regressions verify that corrupt retained calls prevent further Grader sessions and that valid partial histories resume with matching estimates and reports.
• Existing panel voting, order-flip, family, counted-people, low-limit pause/resume, unpriceable-session retention and panel-only confirmation regressions continue to pass. Update the domain-to-code index and document the earlier corrupt-call diagnostic where resume behavior is described, regenerating derived documentation when needed.

Blocked by

None (can start immediately).

Decisions

The design tree was settled in three rounds: scope and compatibility → ownership and storage seam → completion, scheduling and publication → caller integration, verification and documentation. Every branch below is settled; no human decision or release prerequisite remains.

Round 1: constraints and ownership

1. Keep the scope inside the Grader panel. Deepen retained-call ownership, including pending work and publication, rather than changing pair evidence, people's reviews, the rubric or Verdict statistics. This is the repeated responsibility shared by three active callers, and it avoids importing the unresolved policy decisions in Align the people's blind-page rubric and panel result shape with the Grader panel (#1882) #1947, Decide whether a Grader panel seat's family is its host or its Model's maker #1949, Should the Grader panel's spending limit span the whole Experiment rather than each run? #1952, Decide the statistical unit for a Verdict on the people's sample alone #1954, Decide the default people–panel agreement level for a panel standing in for people #1955 and Keep a reader's queue of blind pairs away from Arm-attributed pair evidence #1956.
2. Use the fixed Bar and task list as the identity source. A call is identified by process Eval task, Trial index, configured seat index and presentation order. Enumerate all planned process pairs, not only judged tasks or pairs with Base-tests and Cross-tests. This preserves existing coverage and supports panel grading without optional pair evidence.
3. Keep one private retained-call module within the Grader panel. Its interface supplies per-seat remaining counts, ordered executable work, complete pair inputs for reporting, and publication of a validated fresh call. It owns paths and completion facts; callers receive the task/seat/order information needed to build packets without reconstructing storage or completion rules. This supplies depth beyond a shallow iterator or path helper.
4. Reuse the existing filesystem seam. Native and Observed are already production and test adapters for retained Experiment reads. Capture uses that interface, while publication uses real temporary-directory storage and the existing atomic writer. There is no new filesystem trait, alternate store, persisted queue or migration to maintain.

Round 2: call state and execution

5. Capture all expected saved calls once per operation. Missing calls remain pending; successfully decoded calls are complete. Inspect later seats and orders even when earlier calls are absent, and give decoding/read failures identity context. In particular, do not convert permission errors into pending work or silently regrade malformed files. This makes completion facts agree before more paid work starts.
6. Keep saved-call decoding compatible. Decode the retained Call shape as today; do not rerun fresh-response semantic validation on historical calls. Fresh responses still require both sides, valid labels, every rubric criterion and nonempty reasons/evidence before publication. These are different existing contracts, and tightening the saved format would turn a local refactor into a data migration.
7. Separate planned pending work from executable pending work inside the module. Estimates include calls for Trials still to run. Executable selection additionally checks both Trial identities with the existing terminal predicate, once per pair; reports require saved-call completeness without requiring retained Trial repositories. This preserves the three callers' legitimate differences while sharing identity and completion policy.
8. Retain execution order and lazy preparation. Execute pending items in task, Trial, seat and order sequence; keep one lazily prepared blind packet source per pair and load machine caps only when execution needs them. Keep the existing interruption checks, timeout, seed and swapped second order. A completed pair requires exactly two decoded orders for each configured seat before invoking the existing tally; no partial vote is published.
9. Make publication the completion transition. Accept only a missing call belonging to the captured history, create its parent directory, atomically publish the already validated Call, then update the owned history. Reject an attempt to publish an already completed call. A write failure returns an error without changing in-memory completion. This keeps execution, later estimates and reporting anchored to durable outcomes.
10. Keep publish-before-pricing and the current budget policy. Price new sessions only after their calls have been saved; preserve saved calls when pricing fails or a later session hits the per-run limit. Keep workspace/session discovery and spend calculation outside this refactor. This retains the existing guarantee that resume does not pay for completed calls twice without deciding Should the Grader panel's spending limit span the whole Experiment rather than each run? #1952's whole-Experiment budget question.
11. Use operation-local captures, with existing writer ownership. Cost confirmation, execution and each read get a fresh capture; projections from a capture do no further reads. Execution advances its own capture after successful publication. Reuse the Experiment's existing exclusive writer lock and atomic per-file writes. Do not claim an atomic snapshot across concurrently published files: a later read may see new calls, as today.

Round 3: integration and verification

12. Propagate retained-call failures through cost confirmation. Make the remaining-count, seat-estimate and cost-plan construction chain fallible rather than dropping call-history errors. Preserve the existing best-effort treatment of unreadable spend measurements. Cost confirmation already runs before Arm preparation and Trials, so both interactive confirmation and --yes must reject corrupt retained calls before paid work proceeds.
13. Test through the shared interface. Use temporary storage and the existing Observed adapter to exercise pending counts, executable work and complete report inputs from the same capture. Cover multiple seats and Trials, non-process exclusion, a malformed later order after an earlier missing one, an injected read error, and projections that do not reread captured paths. Verify successful and failed publication through the same interface with real temporary storage. These tests concentrate completion policy instead of copying three discovery test suites.
14. Retain command tests for integration and policy. Add a resume regression with an earlier missing order and a later malformed retained call, asserting no additional Grader invocation and no overwrite of retained calls. Add or extend a valid sparse multi-seat resume regression so remaining session estimates, actual resumed invocations and the final report agree. Preserve existing low-limit and pricing-failure regressions because they verify durable completion after side effects. Keep tally and family tests because they cover policy, not shallow discovery helpers.
15. Validate the affected behavior and update navigation. Run focused Grader panel tests and Experiment panel command regressions, then the repository's affected-test selector against main. Update the domain-to-code index to identify the new owner and tests, and correct resume documentation for early corrupt-call failures. No new domain term or ADR is needed: the Grader panel already names this module, ADR 0028's Bar semantics are preserved, and this private implementation choice is readily reversible.
16. Publish one standalone Ticket. The refactor, error propagation, regressions and documentation form one verifiable change that fits one fresh context window. There are no external blockers, release steps, new infrastructure requirements or reason to create a Spec with separate Tickets.

Architecture review

Strength: Strong · Dependencies: local-substitutable

Files: src/evals/experiment/panel.rs (remaining: 670–692; execution: 697–813; reporting: 951–996), src/evals/experiment/cost.rs, src/evals/experiment/panel/tests.rs, src/evals/experiment/trial_results/files.rs, tests/experiments/panel.rs, docs/code-navigation.md, docs/guide.md, docs/experiment-results.md

Before

flowchart LR
  E[Cost estimates] -. presence means complete .-> F[Retained calls]
  X[Execution and resume] -. presence means complete .-> F
  R[Panel reports] -. decode until first missing .-> F
  classDef leak stroke:#dc2626,stroke-width:2px;
  class E,X,R leak

After

flowchart LR
  E[Cost estimates] --> C[Grader panel retained-call module]
  X[Execution and resume] --> C
  R[Panel reports] --> C
  C -->|existing filesystem adapter| F[Retained calls]
  classDef deep stroke-width:3px;
  class C deep

Problem: Three callers repeat call identity and completion policy, so mere file presence can hide unreadable calls behind missing work.

Solution: Deepen the Grader panel module with one retained-call interface that owns capture, completion, executable work and durable publication at the existing storage seam.

Wins

• Depth: one interface owns completion
• Locality: one retained-call policy
• Leverage: estimates, execution and reports
• Existing filesystem adapter supplies tests
• Corrupt calls fail before spending

Deletion test: Removing the new module would spread identity, partial-history interpretation and publication ordering back across three callers; deleting their repeated traversals concentrates meaningful complexity rather than moving path construction.

Covered by: No matching open issue among the 169 titles and bodies reviewed. #1952 changes spending policy, #1949 changes family policy and #1947 changes rubric/result shape; none proposes this deepening.

Review basis: main at 6ce3c940fec989e1499fdb1f3ef524655b85d078; recent commit history concentrated the scan on Experiments. No ADR conflict. The worktree was unchanged; scripts/affected-tests main reported no affected tests during this review. Implementation tests above are acceptance work for this Ticket.