[JacobStephens2]

Summary

Add versioned private Eval task/suite manifests, dependency selection and throwaway repositories that expose only base history. Proposed Eval tasks are listed and refused preparation. Spec-review repositories get a pushable local bare origin.

~/.thirdshift/evals/suites/<suite>/
├── suite.toml             # dependencies and Trials per Arm
└── tasks/<task>/task.toml # commits, session, inputs, state and success criteria

evals prepare → detached base + staged reviewed patch + declared inputs
              → prune and verify every ref

The format is documented with invented data; no real Eval tasks are checked in. This adds preparation commands, not the Experiment runner or approval workflow.

Closes #1000

Evidence

• Before: the first CLI regression failed because evals was unsupported. Independent Spec reproductions also failed for diff.noprefix=true, diff.context=0, and a silently dropped submodule commit update.
  After: all nine cargo test --test evals tests pass. All three exact reviewer reproduction commands pass after fixes, with their Rust regressions retained. Tests exercise base-only history and absent future objects, a planted later ref, a real push to the bare origin, all five dependency categories, and proposal refusal.
• cargo check, cargo clippy --all-targets -- -D warnings, and git diff --check pass. Final validation also exposed the existing 65-line help limit; the compact entry fixes it, and cargo test --test command_surface passes (20 tests, one fixture entry point ignored).
• Final scripts/affected-tests issue-977 run passed all 43 site/browser checks. Its duplicate local Rust phase was stopped when CI run 39887910032 passed on exact head 860b3293aafbbd2c081d9ee67bc0c12e80cf6a7d, following the repository's CI-reuse policy. That run passed full cargo test, formatting, all-target clippy, Console tests and Site tests.

Test seams

• thirdshift evals list and touched: private manifest loading, proposal exclusion and dependency selection.
• thirdshift evals prepare: time-safe Git history, reviewed changes, task inputs and a pushable Spec-review origin.
• thirdshift evals verify: the history check preparation uses, including a planted later commit.

Review

Reviewed against issue-977 using independent Standards and Spec reviewers. Addressed the Standards fixture Message Chains judgment call and both Spec findings, including the follow-up context-setting reproduction. Every behavioral reproduction was run as written before accepting its finding. Reports retain initial/follow-up findings and each axis's final files-read list.

Both reviewers read all 12 changed files, including generated site/docs/index.html. The final compact-help correction in src/main.rs was added after their follow-ups and remains unread by those reviewers; its existing command-surface tests pass. No other changed content remains unread. Generated-output exclusions: none.

Unaddressed findings

Standards

None.

Spec

None.

Merge Danger

Door: two-way

Additive CLI and private format; preparation requires a new destination and leaves source repositories unchanged. Prepared repositories are disposable.

Blast Radius: Evals

Built with codex · gpt-6.1-sol · xhigh