Skip to content

Add Code review Trials and suite execution - #1135

Merged
JacobStephens2 merged 4 commits into
issue-977from
issue-1016
Oct 23, 2026
Merged

JacobStephens2 merged 4 commits into
issue-977from
issue-1016

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

Code review Trial
  prepared historical diff → production code-review skill → retained Standards + Spec reports
Code review Experiment
  fresh or legacy replay import → 3 Trials per Arm per Eval task → pass^3

Adds the review-only Session kind and Session prompt, uses the implement session's reviewer-independence rules, and regenerates the prompts page. Replay imports now supply the required Code review Trial inputs; reimport completes missing legacy inputs while preserving local criteria, comments and custom Trial inputs.

Closes #1016

Evidence

  • Before: a_code_review_trial_reviews_the_prepared_diff_and_keeps_both_reports failed with unknown code-review; the Experiment regression failed at Code-review Trials require #1016.
    After: both regressions pass, including mixed outcomes proving pass^3 is 1 for an Arm passing all three Trials and 0 for an Arm passing only two.
  • Before: the Spec reviewer's exact reproduction and full import-to-Experiment regression failed with Eval task needs a [trial] table.
    After: the retained regression imports a suite, runs all six Trials, keeps both reports, and reports pass^3. The legacy-import regression also passes and preserves local edits.
  • cargo test --test evals --test experiments --test trial: 56 passed, 2 subprocess entrypoints ignored after merging the updated Base branch. Both Code review regressions and the mechanical-score regression pass after the fixture cleanup.
  • cargo test --bin thirdshift prompts_page: 12 passed; regeneration and docs checks pass. Site browser checks: 43 passed.
  • cargo check --all-targets and cargo fmt --check pass. CI on the exact final head, d3bafe1, passed the full Rust suite, Clippy, formatting, site and Console checks.
  • scripts/affected-tests ac8d24494e52f42268e524f716be20e4f71af361 passed all 43 site checks. Its duplicate Rust run was stopped after exact-head CI passed, reusing that result as docs/testing.md directs.

Merge Danger

Door: two-way

Blast Radius: Evals

Enables Code review Trials and completes missing private replay-manifest inputs. Existing custom Trial tables and grading criteria are preserved.

Test seams

  • The whole binary's versioned --trial protocol against fake Harnesses: the Code review prompt, reviewer independence, and retained reports.
  • The whole binary's evals import-replays, experiment, and experiments --json interfaces against fake Harnesses: imported and prepared review inputs, preservation of local edits on reimport, three Trials per Arm, and pass^3.

Review coverage

Reviewed against the updated issue-977 head, pinned at ac8d24494e52f42268e524f716be20e4f71af361, after merging its Void Trial changes into this branch. Fresh independent reviewers inspected the reconciled diff and the final fixture cleanup.

  • Standards: 1 low-priority duplication finding, addressed by sharing the report-setup fixture. Follow-up confirmed resolution.
  • Spec: 1 P2 finding, reproduced as written, fixed, and verified with the retained regression. Follow-up confirmed resolution.
  • Both reviewers read all 21 changed files, including generated output. Generated-output exclusions: none. Unread changed files: none.

Unaddressed findings

  • Standards: None.
  • Spec: None.

Built with codex · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 04c2489 into issue-977 Oct 23, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1016 branch October 23, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant