Repository navigation
[SCR-68] Measure the defect rate of each agent - #76
Conversation
Every merged pull request in this fleet ends with an Assisted-by trailer naming the agent and model that wrote it. Nothing read it. At fleet volume the first sign that one agent produces junk is a person noticing, months late, that a certain kind of change keeps coming back. Report, per agent, how often its merged work was reverted or hot-fixed inside a window after it landed. Show the merged count beside every rate: a zero percent rate from one merged PR is a sample size, not a quality signal, and a report that hides the denominator invites exactly that reading.
|
Warning Review limit reachedNext included review available in 21 minutes. View limit detailsLimit details: You’ve used all 2 included reviews currently available. Your 81 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Comment |
…h failures gh api --paginate emits one JSON array per page when the --jq filter itself wraps output in [...], so any repo with more than one page of merged PRs, commits, or PR files broke json.loads() and silently collapsed to "no data" / 0% rates. Also stop treating gh api failures (auth, rate limit, network) as empty results, since a working fallback that hides a real failure is worse than a loud one. Fix a cross-repo PR-number collision that silently dropped a defect PR from the TOTAL row when analyzing multiple repos with colliding PR numbers, and tighten the hotfix-title match so "Fixture ..." titles no longer count as a fix of an unrelated PR.
…po collision, argv limit - reverts-#N matching used bare substring search, so PR #1 was falsely flagged as reverted by a commit that actually reverts #10 (or any #1X). Require the number not be followed by another digit. - fetch_default_branch swallowed all gh failures and silently fell back to "main", contradicting the fix in fetch_merged_prs that this same PR makes for the same class of bug. - the per-repo branch cache file flattened "owner/repo" by replacing '/' with '_', so repos like "a_b/c" and "a/b_c" collided on one filename. Nest by the real path components instead. - prs_json was passed to python via argv, which can exceed the OS argument-size limit for a repo with many closed PRs; read it from a file.
There was a problem hiding this comment.
Summary
Verified the correctness-lens findings against the code and fixed four confirmed bugs (commit 994d07d):
- False revert match (Major) —
reverts #{num}/revert of #{num}used bare substring matching, so a commit reverting PR #10 would also flag PR #1 as reverted (its digits are a prefix of "10"). Added a negative digit lookahead so the match requires a full number. - Silent branch-fallback swallowing gh failures (Major) —
fetch_default_branchcaught allgh apierrors (auth, rate limit, network) and silently substituted"main". This directly contradicts the stated purpose of this PR's prior commit ("stop swallowing gh failures") —fetch_merged_prswas fixed butfetch_default_branchwasn't. Now propagates failures like its sibling function. - Repo-name collision in the branch cache file (Minor) — flattening
owner/repoby replacing/with_meant e.g.a_b/canda/b_cboth mapped tobranch_a_b_c, letting one repo's commits get queried against another repo's default branch in a multi-repo run. Fixed by nesting a real directory per repo instead of string-mangling. - argv size limit (Minor) — the per-repo PR JSON was passed to
python3viaargv, which can exceed the OS argument-size limit for a repo with many closed PRs (this file already avoids that pattern elsewhere, reading from files instead). Now written to a temp file and read from disk.
Discarded: "PRs merged less than --window-days ago are counted as non-defects before their window closes" — this is inherent to any windowed defect-rate measurement, not a diff-introduced bug with a concrete wrong-output trigger; it's a documented tradeoff (the script already warns readers to check the merged-count denominator), not something to silently "fix" by guessing at different semantics.
Note: I could not execute the test suite in this sandbox (script execution requires interactive approval here), so I verified the fixes by careful static tracing against agent-defect-rate.test.sh's existing cases rather than by running them. Worth a human re-run of ./agent-defect-rate.test.sh before merge.
🧑⚖️ Council
| Model | Lens | Result |
|---|---|---|
| GPT-5.6 (Codex) | correctness | confirmed (4 issues fixed, 1 rejected — see above) |
| Gemini 3 Pro | performance | no findings (provider error/timeout) |
| Kimi K3 | security | no findings (provider error/timeout) |
| Grok 4.5 | maintainability | no findings (timed out) |
| GPT-5.6 (scope) | scope | no findings |
…x, cross-repo issue collision - Filter PRs to the repo's default branch before counting them: revert commits are only ever fetched from that branch, so a PR merged elsewhere could never be flagged as a defect and just diluted the denominator. - Match GitHub's own "Revert" button commit subject, which appends " (#N)" for the revert PR's own number, not just an exact string. - Stop letting a cross-repo "Closes other/repo#100" collide with an unrelated same-numbered issue in a later same-repo "Fixes #100". Adds test coverage for all three.
There was a problem hiding this comment.
Council-synthesized review
Verified each council claim against the code and repo conventions (pr-standards.md's nine closing-keyword list and "Assisted-by anywhere in the body" rule). Four confirmed Major-severity correctness bugs, all in agent-defect-rate's core matching logic — each one causes the tool to misreport the exact metric it exists to produce (which agent produced a defect). One Minor issue rolled into this summary.
No fixes pushed this cycle — three automated fix cycles have already landed on this branch (150ce63, 994d07d, 738f875), so per the loop guard this pass is comment-only. None of the four findings below overlap with what those cycles already fixed (confirmed by diffing each prior commit), so they're newly surfaced, not persistent misses.
Rolled-in Minor
closing_issue_numbers(line 358) comparesref_repo != repocase-sensitively, so a self-reference likeCloses Owner/Repo#100when invoked asowner/repois wrongly treated as cross-repo and dropped. GitHub repo paths are case-insensitive; narrow trigger, low impact.
Rejected
- Gemini (performance) and Grok (maintainability) errored/timed out — no findings to evaluate.
- Kimi (security) and GPT-5.6 (scope) reported no findings — confirmed nothing further after reading the diff.
🧑⚖️ Council
| Model | Lens | Result |
|---|---|---|
| GPT-5.6 (Codex) | correctness | confirmed (4 of 4 claims) |
| Gemini 3 Pro | performance | no findings (errored: HTTP 429 / timeout) |
| Kimi K3 | security | no findings |
| Grok 4.5 | maintainability | no findings (timed out) |
| GPT-5.6 (scope) | scope | no findings |
| Synthesizer (Sonnet 5) | conventions + test coverage | confirmed 1 new finding (word-boundary gap in closing_issue_numbers), no test coverage for any of the 4 confirmed bugs |
| def parse_agent(body): | ||
| if not body: | ||
| return "unattributed" | ||
| m = re.search(r'Assisted-by:\s*(\S+)', body) |
There was a problem hiding this comment.
Major — misattributes defects to the wrong agent. re.search(r'Assisted-by:\s*(\S+)', body) returns the first match in the body, not the real trailer. pr-standards.md explicitly says the Assisted-by line is accepted "anywhere in the body" (not just at the end), so this isn't hypothetical: any PR body that quotes the trailer format before stating the real one (e.g. a PR editing pr-standards-templates/agents-block.md or pr-standards.md itself, both of which contain the literal string Assisted-by: <agent>:<model> as example text) will attribute the PR to whatever agent name appears in the quoted example instead of the actual author. Since this script's entire purpose is per-agent attribution, this silently corrupts the metric. Anchor to the last match (or a line that looks like a trailer, e.g. ^Assisted-by:) instead of the first search() hit.
| for target in candidates_fix_targets: | ||
| # Match Fixes #<target> case-insensitively, allow any closing keyword variant? | ||
| # Spec says Fixes #<pr number>; be permissive: fixes/fix. | ||
| if re.search(rf'(?i)fixes\s*#\s*{target}\b', body): |
There was a problem hiding this comment.
Major — undercounts defects. is_fix_body only matches the fixes keyword: re.search(rf'(?i)fixes\s*#\s*{target}\b', body). This is inconsistent with closing_issue_numbers (line 356) and with pr-standards.md, both of which treat fix/fixes/fixed (and close/closes/closed/resolve/resolves/resolved) as equivalent closing keywords. A later PR with a neutral title and a body like Fixed #30 or Fix #30 won't be detected as a hotfix for PR #30, silently lowering that agent's defect rate. Reuse the same keyword set/regex used in closing_issue_numbers.
| # bare "#123", which would otherwise let an unrelated same-numbered issue | ||
| # in this repo falsely match a later "Fixes #123". | ||
| nums = [] | ||
| for m in re.finditer(r'(?i)(?:close|closes|closed|fix|fixes|fixed|resolve|resolves|resolved)\s*:?\s*(?:([A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)\#|\#)(\d+)', body or ""): |
There was a problem hiding this comment.
Major — false positive closing-reference match. The keyword alternation (close|closes|closed|fix|fixes|fixed|resolve|resolves|resolved) has no left word-boundary, so it matches inside ordinary words that happen to end the same way: e.g. body text "shared prefix #10" or "an unclosed #10" gets parsed as a closing reference to issue #10 purely because prefix/unclosed end in fix/closed. That false linked-issue number then feeds is_hotfix_defect's candidates_fix_targets, so an unrelated later PR that legitimately says Fixes #10 about something else gets wrongly counted as fixing this PR. Add a \b (or (?<!\w)) before the keyword alternation.
| # Plain substring matching would let PR #1 match inside "reverts #10" | ||
| # (the digits of #10 start with "1"), so require the number not be | ||
| # followed by another digit. | ||
| if re.search(rf'reverts #{num}(?!\d)', low) or re.search(rf'revert of #{num}(?!\d)', low): |
There was a problem hiding this comment.
Major — false positive revert match. re.search(rf'reverts #{num}(?!\d)', low) has no left boundary, so a commit message like "Unreverts #11" (re-applying a previously reverted change — a real convention in some projects) contains "reverts #11" as a substring and is wrongly counted as evidence PR #11 was reverted, when the commit's intent is the opposite. The digit-suffix boundary was already fixed in 994d07d for this exact line; the missing boundary is on the left side of the same match. Add a boundary before reverts/revert of too (e.g. (?<!\w)reverts #{num}).
Closes #68
What
Adds
agent-defect-rate, a script that reports how often each agent's merged work came back — reverted, or hot-fixed within a window.Why
Every PR body in this fleet ends with
Assisted-by: <agent>:<model>. Nothing read it. Agents open most of the PRs, so the first sign that one of them produces junk is a person noticing, months late, that a certain kind of change keeps coming back. This is the number that says it earlier.The report shows the merged count beside every rate on purpose. A zero percent rate from one merged PR is a sample size, not a quality signal, and a table that hides the denominator invites exactly that misreading.
A PR counts as a defect when, inside the window after it merged, either a revert commit lands on the default branch or a later merged PR touches the same files and marks itself a fix.
How I verified
The test stubs
ghonPATH, so it runs offline and pins the window arithmetic in both directions:Against a real repo:
That run is also the first evidence the number is worth having. Twenty-five of twenty-eight merged PRs carry no trailer, so today the metric mostly measures how recently the disclosure rule landed. It gets useful as the unattributed row shrinks.
Proof: n/a — a terminal report with no user-visible surface. Its output is the two runs above.
Assisted-by: muse:meta-code, claude-personal-1:claude-opus-5