Repository navigation
Multi-family review: have another model family review a Run's work #429
Description
Rewritten on 9 October 2026, after the third grilling session (rounds 8–10). The full record is in .scratch/grill-with-docs/429-multi-family-review.md (#618). Below are the decisions and the plan.
Decided
-
The term is Second opinion: an agent session that reviews another session's work, on its own Harness, Model and Effort set in the User config. It can review three things:
- a Run's change, after its implement session;
- a Spec branch, after its Spec review;
- an Architect plan, before thirdshift marks it ready.
It's usually another model family, but it never has to be. The term reaches
CONTEXT.mdwith the Ticket that builds it. -
Its job is to catch what slips through. Every factory PR has merged itself since about 27 September. The same-model review and CI are the only checks before main.
-
Where it could go, each one an opt-in User config setting, off by default:
- a Second opinion in every Run;
- a Second opinion after the Spec review;
- the Spec review itself on its own Harness, Model and Effort;
- a Second opinion on each Architect plan's Strong call, before the plan is marked ready.
-
Nothing gets built before a replay on past work supports it. Each piece has a bar fixed in advance, in the table below. A piece that misses its bar isn't built.
-
Evidence-backed review findings shipped as Evidence-backed review findings #486 (One review instruction for the prompts that run the review #487–thirdshift keeps each reviewer's report beside the Session log #490), in 0.10.1 on 7 October.
-
Codex's review sub-agents get asked to start fresh: Codex sessions that run the code review ask for fresh sub-agents #616. In 54 Codex sessions, no reviewer received the implementer's work. But the model chose
fork_turnsitself each time, and a Resume could hand its first turn to the reviewers.
What the record showed
- 16 bugs reached main on thirdshift after merging, between 25 September and 6 October. About 10 showed up only in live runs. About 6 are timing bugs: what happens when other Runs act at the same moment, or a pass repeats (A thirdshift-plugin-* temp dir can outlive its Run #24, Merge the Base branch again when it moves while CI runs #57, Repair loop keeps re-checking the same commit after a CI-fix Repair makes no change #102, Check a conflict Repair against the commit it merged, not the moved ref #175, Take in Foreign commits that land during a Repair before pushing #182, A Pickup run takes a Spec whose Tickets are all closed on every pass, and never reaches the next Ready issue #303).
- Two of those six were caught and then declined. Repair loop keeps re-checking the same commit after a CI-fix Repair makes no change #102 and A Pickup run takes a Spec whose Tickets are all closed on every pass, and never reaches the next Ready issue #303 were raised by the review, then declined as needing a decision that nobody made.
- Authors declined about 7 findings per PR, about 1,150 in all, and nobody read them, until Evidence-backed review findings #486 made each decline cite its evidence.
- The Architecture review almost always rates its own pick Strong. 209 reviews produced 202 plans and 4 ideas, so it said Strong 98% of the time, and nobody else checks a plan.
- A second look finds bugs. About half of the 30 Spec reviews fixed a behaviour bug the Tickets' reviews had passed.
- Codex's record since 6 October:
- 27 plans (Deepen Harness process execution around one owned lifetime #448–Deepen failed Worktree acquisition recovery with verified retry authority #530) and four Specs (Spec: four more Harnesses (agy, Grok Build, Muse Code, OpenCode) #432, Spec: Deepen Command execution for Git and GitHub #465, Evidence-backed review findings #486, Security runs and the Security review: thirdshift finds, reproduces and fixes vulnerabilities #427).
- Three bugs reached main from its PRs: CI red on issue-427: test-macos #559, Report reproduction failures separately from completed Security audits #603 and Security review: surface the session's reason when it deliberately ends incomplete #607.
- Its reviewer reports are kept since 7 October (thirdshift keeps each reviewer's report beside the Session log #490).
The replays
The replays are read-only review sessions over past work, run one at a time in throwaway worktrees. Results go in a research note in docs/research/, with the script beside it, in a PR, and a summary here.
- Skill: the bug replays use today's review skill. The Spec replay uses the skill as it was before Evidence-backed review findings #486 (479a1fb), to match the recorded Spec reviews.
- Effort: Codex runs at xhigh. Claude runs at max as the baseline and at high as the Second opinion being tried. A Claude fail in the reverse direction therefore leaves Claude at max untested.
| Replay | Arms | Bar to build |
|---|---|---|
| Six timing bugs (five PRs, reviewed as merged) | Claude in a fresh session (max); Codex (xhigh); Claude with a timing checklist (max), written by a fresh agent that hasn't seen the bugs; one arm each for agy, Grok Build, Muse Code and OpenCode, at their best settings | A Second opinion in a Run if the other family catches at least 2 bugs that the fresh same-model arm misses and the checklist arm doesn't. If the checklist arm catches as many, the checklist goes into the skill instead. Dropped if no arm catches more than 1. A new-Harness arm that passes must also pass the Spec replay. |
| Codex's three bugs, reverse (#559, #603, #607; four PRs, reviewed as merged) | Claude (high); Codex in a fresh session (xhigh) | Counts for a Second opinion in a Run if Claude catches at least 2 that fresh Codex misses. The kept reports also show whether the factory's own review raised each bug. |
| Plan check, forward | Codex (xhigh), and Claude in a fresh session (max), each check 20 plans that Claude wrote | Built if it stops at least 3, and the Day shift, grading without knowing which arm stopped them, would have stopped most of the ones it stops |
| Plan check, reverse | Claude (high), and Codex in a fresh session (xhigh), each check 20 of the 27 plans Codex wrote | At least 3 of the 20, graded the same way |
| Spec reviews | Codex reviews 30 Spec branches as they stood before their Spec reviews | A Second opinion after the Spec review if the bugs the recorded review missed include a real behaviour bug in at least 6 of the 30, and no more than half of a graded sample turns out wrong. The Spec review on its own Harness only if Codex alone finds real behaviour bugs in at least as many Specs as the recorded review did (about 15). |
- Sessions: 54 on Claude, 79 on Codex and 20 on the four new Harnesses.
- Schedule:
- The Claude sessions run first, from 9 October, alongside Weeding, since they share no limits with it.
- Weeding then pauses while the Codex and new-Harness sessions run.
- Security runs pause only during the Muse arm, since they run on Muse.
- Each stage stops at the first limit warning and picks up later.
- Grading: Claude grades the bugs against their fixes. The Day shift grades the plans the checks stop, in both directions, plus about 20 Spec findings.
- Later:
- the reverse Spec replay, once Codex has run enough Specs;
- a reverse timing-bug replay, once timing bugs surface in Codex's work.
After the replays
For each piece that passes, a short grilling settles the questions that wait on the numbers. Then /to-spec and /to-tickets file it. The questions:
- Who acts on a Second opinion's findings.
- What happens when the plan check disagrees.
- Whether standalone Runs and Tickets get separate switches.
Research
docs/research/multi-family-review.mddocs/research/model-strength-for-review.md
Activity
Merge pull request #495 from JacobStephens2/grilling-429-second-opinion
JacobStephens2 commentedon Oct 8, 2026
From #427's trial of Cloudflare's security-audit skill on 2026-10-07, folded in here because it bears on reviewer independence:
fork_turns: "none", according to Codex's rollout files. If that holds forthirdshift-code-review, its Standards and Spec sub-agents don't review with fresh context under Codex, the Harness the factory's User config uses by default. That undercuts the reason the skill uses sub-agents at all.agents.max_concurrent_threads_per_session.codex exec --jsonshows neither sub-agents nor their tokens. The tokens it reports were about half the total.To check first: read a recent Codex Run's rollout files for the code review's sub-agents, and see whether they were forked with the parent's turns. If they were, the Session prompt can ask Codex for
fork_turns: "none".Measurements: the Research results in
.scratch/grill-with-docs/427-security-work.md.JacobStephens2 commentedon Oct 10, 2026
On the 8 October comment about Codex sub-agents: checked on 9 October against Codex's rollout files.
fork_turnsitself:"none"23 times,"all"6 times, and left it out 25 times. Why"all"didn't carry the parent's turns is an inference: Codex seems to copy only completed turns, and a headless session's one turn is still running. A Resume has a completed first turn, so a review there could hand it to the reviewers. That case hasn't been seen.The thread cap and the hidden sub-agent tokens in
codex exec --jsondon't bear on the replays, which run one reviewer session at a time.Record #429's rewrite and #618 in the grilling log
Grilling log for #429: rounds 8 to 10, the revised replay plan (#618)
JacobStephens2 commentedon Oct 11, 2026
The replays have reported. The full write-up is in #657 (
docs/research/second-opinion-replays.md). There were 183 read-only sessions, graded against the bars fixed in Q6, Q29 and Q30.What this means for #429:
What the bug replays did show: fresh Claude at max, on today's review skill, caught 4 of the 6 timing bugs. The recorded in-session reviews raised 2, and both were declined. A fresh session and the new skill changed together, so that gain can't be split between them.
Per Q33, a short grilling comes next for what passed: who acts on a Second opinion's findings, and what happens when the plan check disagrees. Then
/to-spec.Merge pull request #657 from JacobStephens2/429-replays
JacobStephens2 commentedon Oct 17, 2026
Triage Notes
The replays are complete; do not repeat the decision about whether to build a Second opinion in ordinary Runs. Those arms failed, as did the checklist and replacing the Spec review's Harness. A Second opinion after the Spec review passed and remains the supported implementation candidate.
The later October 12 follow-up also found that Claude at max met the Codex-plan check bar exactly. Its correct stops were the same two plans fresh Codex stopped, so the result does not settle whether to build the plan check.
Next decision: finish the recorded Round 11 choices: default reviewer settings, who fixes findings, what the reviewer sees, placement and incomplete-review behavior, payoff evaluation, and whether to include the plan check. Then publish the implementation Spec/Tickets. Existing private replay findings remain private.
Merge pull request #1127 from JacobStephens2/issue-1017
JacobStephens2 commentedon Oct 26, 2026
From #877's grilling (
docs/grill-with-docs/877-composable-by-file.md, Q3.5): #877 will let a user write a Custom process, a named sequence of Steps, each an agent session running a named skill on its own Harness, Model and Effort. Steps can run in parallel.What this changes for #429:
thirdshift evals import-replaysfilled from that material. The body above keeps "replay" because it describes that finished work.#429's plan otherwise stands.