Skip to content

[SCR-68] Measure the defect rate of each agent - #76

Merged
itsryanarmani merged 4 commits into
mainfrom
scr-68-measure-agent-defect-rate
Aug 31, 2026
Merged

itsryanarmani merged 4 commits into
mainfrom
scr-68-measure-agent-defect-rate

Conversation

@itsryanarmani

Copy link
Copy Markdown
Owner

Closes #68

What

Adds agent-defect-rate, a script that reports how often each agent's merged work came back — reverted, or hot-fixed within a window.

Why

Every PR body in this fleet ends with Assisted-by: <agent>:<model>. Nothing read it. Agents open most of the PRs, so the first sign that one of them produces junk is a person noticing, months late, that a certain kind of change keeps coming back. This is the number that says it earlier.

The report shows the merged count beside every rate on purpose. A zero percent rate from one merged PR is a sample size, not a quality signal, and a table that hides the denominator invites exactly that misreading.

A PR counts as a defect when, inside the window after it merged, either a revert commit lands on the default branch or a later merged PR touches the same files and marks itself a fix.

How I verified

The test stubs gh on PATH, so it runs offline and pins the window arithmetic in both directions:

$ ./agent-defect-rate.test.sh
ok - PR with no trailer lands under unattributed
ok - PR with trailer is grouped under its agent
ok - revert inside window counts
ok - json output for revert inside window is correct
ok - revert outside window does not count
ok - Reverts #<pr> counts (case-insensitive)
ok - revert of #<pr> counts
ok - rate maths 50% for agent-a
ok - rate maths 0% for agent-b
ok - worst rate first
ok - TOTAL row correct (3 merged 1 defect 33.3%)
ok - hotfix touching same files counts
ok - hotfix outside window does not count
ok - --json prints valid JSON
---
14 passed, 0 failed

Against a real repo:

$ ./agent-defect-rate pooriaarab/vibecodereview --since 2026-08-01
defect rate since 2026-08-01 (window 7d) — pooriaarab/vibecodereview
AGENT                                  MERGED DEFECTS    RATE  DEFECT PRS
-------------------------------------- ------ ------- -------  ----------
unattributed                               25       4   16.0%  #1 #2 #4 #6
claude-personal-1:claude-opus-5             1       0    0.0%  -
claude-personal-2:claude-opus-5             1       0    0.0%  -
gemini-personal:gemini-3.1-pro-preview      1       0    0.0%  -
TOTAL                                      28       4   14.3%  #1 #2 #4 #6

That run is also the first evidence the number is worth having. Twenty-five of twenty-eight merged PRs carry no trailer, so today the metric mostly measures how recently the disclosure rule landed. It gets useful as the unattributed row shrinks.

Proof: n/a — a terminal report with no user-visible surface. Its output is the two runs above.

Assisted-by: muse:meta-code, claude-personal-1:claude-opus-5

Every merged pull request in this fleet ends with an Assisted-by
trailer naming the agent and model that wrote it. Nothing read it. At
fleet volume the first sign that one agent produces junk is a person
noticing, months late, that a certain kind of change keeps coming back.

Report, per agent, how often its merged work was reverted or hot-fixed
inside a window after it landed. Show the merged count beside every
rate: a zero percent rate from one merged PR is a sample size, not a
quality signal, and a report that hides the denominator invites exactly
that reading.
@coderabbitai

coderabbitai Bot commented Aug 30, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 21 minutes.

View limit details

Limit details: You’ve used all 2 included reviews currently available. Your 81 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7914b56e-4362-42be-b1d2-66408f0a7670

📥 Commits

Reviewing files that changed from the base of the PR and between 6d988a2 and 738f875.

📒 Files selected for processing (3)
  • README.md
  • agent-defect-rate
  • agent-defect-rate.test.sh

Comment @coderabbitai help to get the list of available commands.

vibecodereview[bot] added 2 commits August 30, 2026 00:55
…h failures

gh api --paginate emits one JSON array per page when the --jq filter itself
wraps output in [...], so any repo with more than one page of merged PRs,
commits, or PR files broke json.loads() and silently collapsed to "no data" /
0% rates. Also stop treating gh api failures (auth, rate limit, network) as
empty results, since a working fallback that hides a real failure is worse
than a loud one. Fix a cross-repo PR-number collision that silently dropped
a defect PR from the TOTAL row when analyzing multiple repos with colliding
PR numbers, and tighten the hotfix-title match so "Fixture ..." titles no
longer count as a fix of an unrelated PR.
…po collision, argv limit

- reverts-#N matching used bare substring search, so PR #1 was falsely
  flagged as reverted by a commit that actually reverts #10 (or any #1X).
  Require the number not be followed by another digit.
- fetch_default_branch swallowed all gh failures and silently fell back to
  "main", contradicting the fix in fetch_merged_prs that this same PR makes
  for the same class of bug.
- the per-repo branch cache file flattened "owner/repo" by replacing '/'
  with '_', so repos like "a_b/c" and "a/b_c" collided on one filename.
  Nest by the real path components instead.
- prs_json was passed to python via argv, which can exceed the OS
  argument-size limit for a repo with many closed PRs; read it from a file.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Verified the correctness-lens findings against the code and fixed four confirmed bugs (commit 994d07d):

  1. False revert match (Major) — reverts #{num} / revert of #{num} used bare substring matching, so a commit reverting PR #10 would also flag PR #1 as reverted (its digits are a prefix of "10"). Added a negative digit lookahead so the match requires a full number.
  2. Silent branch-fallback swallowing gh failures (Major) — fetch_default_branch caught all gh api errors (auth, rate limit, network) and silently substituted "main". This directly contradicts the stated purpose of this PR's prior commit ("stop swallowing gh failures") — fetch_merged_prs was fixed but fetch_default_branch wasn't. Now propagates failures like its sibling function.
  3. Repo-name collision in the branch cache file (Minor) — flattening owner/repo by replacing / with _ meant e.g. a_b/c and a/b_c both mapped to branch_a_b_c, letting one repo's commits get queried against another repo's default branch in a multi-repo run. Fixed by nesting a real directory per repo instead of string-mangling.
  4. argv size limit (Minor) — the per-repo PR JSON was passed to python3 via argv, which can exceed the OS argument-size limit for a repo with many closed PRs (this file already avoids that pattern elsewhere, reading from files instead). Now written to a temp file and read from disk.

Discarded: "PRs merged less than --window-days ago are counted as non-defects before their window closes" — this is inherent to any windowed defect-rate measurement, not a diff-introduced bug with a concrete wrong-output trigger; it's a documented tradeoff (the script already warns readers to check the merged-count denominator), not something to silently "fix" by guessing at different semantics.

Note: I could not execute the test suite in this sandbox (script execution requires interactive approval here), so I verified the fixes by careful static tracing against agent-defect-rate.test.sh's existing cases rather than by running them. Worth a human re-run of ./agent-defect-rate.test.sh before merge.

🧑‍⚖️ Council
Model Lens Result
GPT-5.6 (Codex) correctness confirmed (4 issues fixed, 1 rejected — see above)
Gemini 3 Pro performance no findings (provider error/timeout)
Kimi K3 security no findings (provider error/timeout)
Grok 4.5 maintainability no findings (timed out)
GPT-5.6 (scope) scope no findings

…x, cross-repo issue collision

- Filter PRs to the repo's default branch before counting them: revert
  commits are only ever fetched from that branch, so a PR merged
  elsewhere could never be flagged as a defect and just diluted the
  denominator.
- Match GitHub's own "Revert" button commit subject, which appends
  " (#N)" for the revert PR's own number, not just an exact string.
- Stop letting a cross-repo "Closes other/repo#100" collide with an
  unrelated same-numbered issue in a later same-repo "Fixes #100".

Adds test coverage for all three.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Council-synthesized review

Verified each council claim against the code and repo conventions (pr-standards.md's nine closing-keyword list and "Assisted-by anywhere in the body" rule). Four confirmed Major-severity correctness bugs, all in agent-defect-rate's core matching logic — each one causes the tool to misreport the exact metric it exists to produce (which agent produced a defect). One Minor issue rolled into this summary.

No fixes pushed this cycle — three automated fix cycles have already landed on this branch (150ce63, 994d07d, 738f875), so per the loop guard this pass is comment-only. None of the four findings below overlap with what those cycles already fixed (confirmed by diffing each prior commit), so they're newly surfaced, not persistent misses.

Rolled-in Minor

  • closing_issue_numbers (line 358) compares ref_repo != repo case-sensitively, so a self-reference like Closes Owner/Repo#100 when invoked as owner/repo is wrongly treated as cross-repo and dropped. GitHub repo paths are case-insensitive; narrow trigger, low impact.

Rejected

  • Gemini (performance) and Grok (maintainability) errored/timed out — no findings to evaluate.
  • Kimi (security) and GPT-5.6 (scope) reported no findings — confirmed nothing further after reading the diff.
🧑‍⚖️ Council
Model Lens Result
GPT-5.6 (Codex) correctness confirmed (4 of 4 claims)
Gemini 3 Pro performance no findings (errored: HTTP 429 / timeout)
Kimi K3 security no findings
Grok 4.5 maintainability no findings (timed out)
GPT-5.6 (scope) scope no findings
Synthesizer (Sonnet 5) conventions + test coverage confirmed 1 new finding (word-boundary gap in closing_issue_numbers), no test coverage for any of the 4 confirmed bugs

Comment thread agent-defect-rate
def parse_agent(body):
if not body:
return "unattributed"
m = re.search(r'Assisted-by:\s*(\S+)', body)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major — misattributes defects to the wrong agent. re.search(r'Assisted-by:\s*(\S+)', body) returns the first match in the body, not the real trailer. pr-standards.md explicitly says the Assisted-by line is accepted "anywhere in the body" (not just at the end), so this isn't hypothetical: any PR body that quotes the trailer format before stating the real one (e.g. a PR editing pr-standards-templates/agents-block.md or pr-standards.md itself, both of which contain the literal string Assisted-by: <agent>:<model> as example text) will attribute the PR to whatever agent name appears in the quoted example instead of the actual author. Since this script's entire purpose is per-agent attribution, this silently corrupts the metric. Anchor to the last match (or a line that looks like a trailer, e.g. ^Assisted-by:) instead of the first search() hit.

Comment thread agent-defect-rate
for target in candidates_fix_targets:
# Match Fixes #<target> case-insensitively, allow any closing keyword variant?
# Spec says Fixes #<pr number>; be permissive: fixes/fix.
if re.search(rf'(?i)fixes\s*#\s*{target}\b', body):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major — undercounts defects. is_fix_body only matches the fixes keyword: re.search(rf'(?i)fixes\s*#\s*{target}\b', body). This is inconsistent with closing_issue_numbers (line 356) and with pr-standards.md, both of which treat fix/fixes/fixed (and close/closes/closed/resolve/resolves/resolved) as equivalent closing keywords. A later PR with a neutral title and a body like Fixed #30 or Fix #30 won't be detected as a hotfix for PR #30, silently lowering that agent's defect rate. Reuse the same keyword set/regex used in closing_issue_numbers.

Comment thread agent-defect-rate
# bare "#123", which would otherwise let an unrelated same-numbered issue
# in this repo falsely match a later "Fixes #123".
nums = []
for m in re.finditer(r'(?i)(?:close|closes|closed|fix|fixes|fixed|resolve|resolves|resolved)\s*:?\s*(?:([A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)\#|\#)(\d+)', body or ""):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major — false positive closing-reference match. The keyword alternation (close|closes|closed|fix|fixes|fixed|resolve|resolves|resolved) has no left word-boundary, so it matches inside ordinary words that happen to end the same way: e.g. body text "shared prefix #10" or "an unclosed #10" gets parsed as a closing reference to issue #10 purely because prefix/unclosed end in fix/closed. That false linked-issue number then feeds is_hotfix_defect's candidates_fix_targets, so an unrelated later PR that legitimately says Fixes #10 about something else gets wrongly counted as fixing this PR. Add a \b (or (?<!\w)) before the keyword alternation.

Comment thread agent-defect-rate
# Plain substring matching would let PR #1 match inside "reverts #10"
# (the digits of #10 start with "1"), so require the number not be
# followed by another digit.
if re.search(rf'reverts #{num}(?!\d)', low) or re.search(rf'revert of #{num}(?!\d)', low):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major — false positive revert match. re.search(rf'reverts #{num}(?!\d)', low) has no left boundary, so a commit message like "Unreverts #11" (re-applying a previously reverted change — a real convention in some projects) contains "reverts #11" as a substring and is wrongly counted as evidence PR #11 was reverted, when the commit's intent is the opposite. The digit-suffix boundary was already fixed in 994d07d for this exact line; the missing boundary is on the left side of the same match. Add a boundary before reverts/revert of too (e.g. (?<!\w)reverts #{num}).

@itsryanarmani
itsryanarmani merged commit 8a3686c into main Aug 31, 2026
2 checks passed
@itsryanarmani
itsryanarmani deleted the scr-68-measure-agent-defect-rate branch August 31, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure the defect rate of each agent

1 participant