Trial: the Security review's two modes #549

Parent

#427

What to build

Before the Security review is built, measure its two candidate modes on the factory's own repositories, report-only, so that the Day shift can choose between them:

• Full audit: the skill's full audit, quick, scoped to the diff between a merged pull request's merge base and its head.
• Guidance mode: one session reviews the diff with the skill's attack classes and method.

How to run it.

• Use three recent merged pull requests, and run each mode on Claude Code (claude-opus-5-5, medium) and on Codex (gpt-6.1-sol, xhigh).
• Run in throwaway clones with no origin remote.
• Run at most two sessions at once, each under a timeout.
• This is roughly twelve sessions. Security runs and the Security review: thirdshift finds, reproduces and fixes vulnerabilities #427's trial of whole-repository audits put one at $14–20 at list price on Claude Code.

What to measure:

• cost, or tokens on Codex;
• wall time;
• sub-agents;
• refusals or changes of Model;
• whether the session asked anything;
• findings by verdict.

Where results go.

• The measurements go in a public research note among the repository's research notes.
• The findings go only in a private file on the machine (mode 600), with an empty grade line under each. Never put them in the pull request, an issue, or a comment.

Acceptance criteria

• The research note gives each mode's measurements on each Harness, for each pull request, and describes the method.
• No finding's title or detail appears in the pull request, the note, or any issue or comment. The note names only the private file's path.
• The private file lists every finding by mode and Harness, each with a blank grade line.
• Nothing is pushed to any repository but this one, and no advisory, issue or label is created.