Skip to content

Bias triage and batching toward higher-merge-rate issue types - #29

Merged
dmccoystephenson merged 2 commits into
mainfrom
feature/bias-triage-merge-rate
May 25, 2026
Merged

dmccoystephenson merged 2 commits into
mainfrom
feature/bias-triage-merge-rate

Conversation

@dmccoystephenson

@dmccoystephenson dmccoystephenson commented May 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds two paragraphs to the generated dev-loop template:

  1. Phase 1 — Record skip reasons. When triage filters out an open issue, record the skip reason in the chosen PR's body or as a comment on the skipped issue. Per RESEARCH.md §7, LLM-based issue triage is useful as a filter but not as a final decision — surfacing the filter's reasoning preserves human oversight.

  2. Phase 2 — Tiebreaker + early-cycle bias. When choosing between issues of comparable scope with no dependency relationship, prefer (in descending order): documentation fixes → CI/build fixes → small refactors → bug fixes → performance work. Per RESEARCH.md §2, autonomous-agent PRs merge at substantially higher rates for the earlier categories. The tiebreaker is soft; coherent-batch grouping still wins when it applies. Early cycles (the first 3–5 after a new dev-loop skill is generated) weight the tiebreaker more strongly.

Closes #20

Test plan

  • Phase 1 has explicit "Record skip reasons" instruction (line 142)
  • Phase 2 has tiebreaker rule with one-sentence rationale + RESEARCH.md cross-reference (line 156)
  • Phase 2 marks early-cycle bias as soft preference, not hard rule (line 158)
  • No new {{placeholders}} introduced; substitution table unchanged (placeholder count: 19, same as before)
  • Phase numbering unchanged (still 1–10); README "What it does" count unchanged
  • Per the rule shipped in PR Promote 'verify each Closes #N' to template Phase 4 #26, ran gh issue view 20 and confirmed title/body match

Note on Phase 1's RESEARCH.md cross-reference

The skip-reason instruction cites RESEARCH.md §7 (LLM-based issue triage accuracy 70–85%). Per the CLAUDE.md rule shipped in PR #27, this PR should also add an Implementations entry under §7 — but §7's specific implications were "emit triage reasoning to the PR description" and "use triage for batching coherence rather than value judgment," both of which this PR directly implements. Adding an Implementations entry now.


drafted by Claude on behalf of Daniel Stephenson

BuildTools and others added 2 commits May 25, 2026 12:42
Closes #20

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Phase 1 skip-reason and Phase 2 tiebreaker changes implement
findings §2 (agent-PR merge-rate gradient) and §7 (LLM triage as
filter not decision-maker), so per the CLAUDE.md rule shipped in
PR #27, both findings get Implementations entries pointing at this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@dmccoystephenson dmccoystephenson left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review rubric:

  • Scope: PASS — create-dev-loop.md (6 lines added: skip-reason + tiebreaker + early-cycle bias) and RESEARCH.md (6 lines added: Implementations entries under §2 and §7, required by the CLAUDE.md rule from PR #27).
  • Tests-new: PASS — the skip-reason instruction is exercised by this very PR body, which includes a 'Note on Phase 1's RESEARCH.md cross-reference' explaining the deferred work; the tiebreaker is exercised by this cycle's choice of #20 over the heavier #16/#17/#18 (all docs/process > harder structural work).
  • Tests-fix: N/A — not a bug fix.
  • Sibling structure: PASS — new Phase 1 paragraph follows the same **Bold label.** explanation. shape as its siblings ('Before filing each issue', 'After filing a batch of issues'); new Phase 2 paragraphs match the pattern of the existing bullet list above them.
  • Sibling renames: N/A — no identifiers renamed.
  • Docs: PASS — Phase 7 sources of truth (create-dev-loop.md, README.md) reviewed; README doesn't enumerate triage/work-selection logic at this granularity. RESEARCH.md updated per the CLAUDE.md PR-27 rule.
  • Issue resolution: PASS — all four AC items from #20 met: (1) Phase 1 records skip reasons; (2) Phase 2 has tiebreaker with rationale and RESEARCH.md cross-reference; (3) early-cycle bias marked as soft preference; (4) no new placeholders.
  • Validation: PASS — placeholder count unchanged (19); no new fenced blocks; phase numbering unchanged.
  • Placeholder-consistency: N/A — no template placeholder added.
  • Escaping: N/A — no fenced code blocks in the prose additions.

Adversarial probe — 'Does the tiebreaker contradict the existing coherent-batch directive?' Yes there is potential tension. Resolved in the text: 'coherent-batch grouping still takes precedence when it applies.' The tiebreaker only fires when batching doesn't yield a clear winner. Confirmed by re-reading the new paragraph against the existing 'Prefer issues that touch the same subsystem' bullet — order of precedence is explicit.

@dmccoystephenson
dmccoystephenson merged commit 2d383d7 into main May 25, 2026
@dmccoystephenson
dmccoystephenson deleted the feature/bias-triage-merge-rate branch May 25, 2026 18:44
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
…ity, test comment (#103)

- README.md: use ln -sf for the Installation symlink, matching Step 5's
  own registration convention (create-dev-loop.md:624).
- CLAUDE.md: clarify that the ISSUE_TEMPLATE doc-table row's *.md glob
  deliberately excludes config.yml, which has no restatable conventions.
- tests/test_check_docs.py: note in the module docstring that any new
  test class must reset check_docs.errors in its own setUp(), so a
  future addition doesn't silently leak state between tests.

Not included, with reasons:
- RESEARCH.md's "Observed effect: pending — needs N cycles of data"
  entries (PRs #29/#30/#52/#54): this needs real usage data the
  maintainer would have to supply; nothing to mechanically edit here
  without fabricating an observed effect.
- The flagged openai.com RESEARCH.md link: re-verified via an
  independent fetch, still 403 (not 404) — consistent with bot/WAF
  blocking a non-browser client, not a confirmed dead link. Left as-is.
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
* Bias triage and batching toward higher-merge-rate issue types

Closes #20

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Record Implementations entries for RESEARCH.md §2 and §7

The Phase 1 skip-reason and Phase 2 tiebreaker changes implement
findings §2 (agent-PR merge-rate gradient) and §7 (LLM triage as
filter not decision-maker), so per the CLAUDE.md rule shipped in
PR #27, both findings get Implementations entries pointing at this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: BuildTools <unconfigured@null.spigotmc.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
…ity, test comment (#103)

- README.md: use ln -sf for the Installation symlink, matching Step 5's
  own registration convention (create-dev-loop.md:624).
- CLAUDE.md: clarify that the ISSUE_TEMPLATE doc-table row's *.md glob
  deliberately excludes config.yml, which has no restatable conventions.
- tests/test_check_docs.py: note in the module docstring that any new
  test class must reset check_docs.errors in its own setUp(), so a
  future addition doesn't silently leak state between tests.

Not included, with reasons:
- RESEARCH.md's "Observed effect: pending — needs N cycles of data"
  entries (PRs #29/#30/#52/#54): this needs real usage data the
  maintainer would have to supply; nothing to mechanically edit here
  without fabricating an observed effect.
- The flagged openai.com RESEARCH.md link: re-verified via an
  independent fetch, still 403 (not 404) — consistent with bot/WAF
  blocking a non-browser client, not a confirmed dead link. Left as-is.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bias triage and batching toward issue types with higher agent-PR merge rates

1 participant