[JacobStephens2]

Summary

#429 replay manifests + cached original issues + grading index
  → code-review Eval tasks (3 Trials per Arm)
  → Spec-review Eval tasks (1 Trial per Arm)
rerun → preserve existing files, add only missing Eval tasks

Adds thirdshift evals import-replays, reading private host data from ~/.thirdshift/replays/ into ~/.thirdshift/evals/suites/. Catch criteria and real/wrong findings remain outside Trial inputs. The format docs describe the grading index to prepare on the host, since the replay script leaves grading-file structure unspecified. Only made-up fixtures were used; the private import was not run.

Closes #1017

Evidence

• Before: cargo test --test evals imports_replay_bugs_and_scored_specs_once_as_preparable_eval_tasks failed because the import action was missing; cargo test --test evals imported_suites_follow_production_review_sources failed on src/session.rs dependency selection.
  After: cargo test --test evals passed all 13 tests, including both regressions, repeat imports, partial-import recovery, retained local edits, invalid grades and preparation of both session kinds.
• cargo check --bin thirdshift and cargo clippy --all-targets -- -D warnings passed.
• After merging upstream Arm preparation, cargo fmt --check, cargo check --bin thirdshift and cargo test --test evals --test eval_arms passed (20 focused tests).
• npm test --prefix site passed all 43 tests, including generated-manual freshness and browser checks.
• CI passed for final head d9b3b44, including Rust, Console and site checks.
• /usr/bin/time -p scripts/affected-tests origin/issue-977 passed: all 43 site tests and 2,417 Rust tests passed, with 57 Rust tests skipped by the repository configuration. Total wall time was 1,753.78 s including site checks and compilation; Rust compilation took 1m 17s and Nextest execution took 1,565.729 s (17 slow tests).

Testing seams

• Whole binary CLI in a temporary home, with made-up Multi-family review: have another model family review a Run's work #429 manifests, cached issue bodies, grading files and real local Git repositories. Observe import JSON, evals list, private Eval task files and evals prepare output.

Review coverage

Independent thirdshift-code-review axes compare against issue-977, pinned at 7f88dd683d4f308f8aef82948893a935503f6fe3 (the remote branch; the local branch is stale in another worktree). After reconciling upstream #1002 in d9b3b44, each axis read all 12 paths in that pinned diff, including the two files inherited from upstream and the rendered HTML hunks. Standards: 0 findings; Spec: 0 findings. No generated-output exclusions and no unread changed files. The PR itself changes 10 files against its current base.

Reports are saved in this worktree at .thirdshift-review-TMr3wI/standards.md and .thirdshift-review-TMr3wI/spec.md.

Unaddressed findings

• Standards: None.
• Spec: None.

Merge Danger

Door: two-way

Blast Radius: Evals

The new command creates approved private Eval tasks. Existing valid Eval tasks and suite declarations are preserved. Incomplete source grading fails before either suite is published.

Built with codex · gpt-6.1-sol · xhigh