[JacobStephens2]

Summary

Research for #427 (security work). Docs only.

 docs/research/
+├── security-agent-passes.md      # how precise LLM security audits are, the two projects #427 names, alternatives, GitHub's free features, disclosure, cost
+└── security-skill-candidates.md  # whether any upstream skill beats Cloudflare's security-audit-skill, and how it would serve each Run's review and a pass

Evidence

• Before: Security runs and the Security review: thirdshift finds, reproduces and fixes vulnerabilities #427 named two projects to study and asked how precise LLM security audits are, with no sources.
  After: every claim cites its source and grade, and the numbers the grilling leans on were checked against the originals:
  • Precision: Semgrep's 2025 study (Claude Code right on 14% of its findings, Codex on 18%) and Anthropic's disclosure ledger (92.7% of 6,123 externally reviewed findings valid).
  • Agent-written vulnerabilities: the SusVibes leaderboard (about 60% of functionally correct solutions insecure), and a replay in which a security-hardening skill cut reintroduction by 13.8 points (arXiv:2606.23130, Table 4).
  • Fix correctness: PatchBench (45.3% of tasks solved correctly) and the AIxCC SoK (37.7% of Claude Code's validated patches semantically wrong).
  • cloudflare/security-audit-skill, read in full at c1c8a8c: its sandbox rule, how to ask for a scoped diff run, its floor of seven sub-agents, and its finding schema against GitHub's advisory fields.
  • GitHub's REST docs: a draft advisory and a temporary private fork can be created through the API, but no CI runs in the fork, and merging it is a web-UI button.
• Local facts (this machine's sandbox limits, the seven repositories' security settings, costs from the Command logs) are marked as checked by test.

Merge Danger

Door: two-way

Blast Radius: none

Docs only: no code, prompts or skills change.

🤖 Generated with Claude Code