Skip to content

Reframe self-review as rubric-based and CI-grounded - #15

Merged
dmccoystephenson merged 1 commit into
mainfrom
feature/rubric-based-self-review
May 25, 2026
Merged

dmccoystephenson merged 1 commit into
mainfrom
feature/rubric-based-self-review

Conversation

@dmccoystephenson

@dmccoystephenson dmccoystephenson commented May 25, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Replaces the free-form Phase 4 self-review with an objective yes/no rubric anchored on CI status.
  • Adds an adversarial framing instruction ("assume FAIL unless you have direct evidence of PASS") to counter LLM self-preference bias.
  • Caps intrinsic critique at one pass per PR. Phase 6 now distinguishes internal critique (capped) from external signals — reviewer comments and CI failures — which remain uncapped.
  • Adds a new SELF_REVIEW_RUBRIC placeholder so generated skills can append repo-specific objective rubric items.
  • Adds an edge-case entry for CI failures on the PR.

Motivation (per RESEARCH.md)

Per the new convention in CLAUDE.md, design changes cite the empirical basis:

  • §1 — Self-critique without an external signal is unreliable. Huang et al. ICLR 2024 (arXiv:2310.01798), Kamoi et al. TACL 2024 (arXiv:2406.01297), and arXiv:2604.06996 on self-preference bias all converge on: intrinsic self-correction is neutral-to-harmful without an external grounding signal. This change uses CI status as that signal and replaces the previous "review for logic errors, edge cases, anti-patterns…" framing with an objective rubric.
  • §5 — Reflection iterations plateau early. Self-Refine (arXiv:2303.17651) and Reflexion (arXiv:2303.11366) show gains concentrate in iterations 1–2; later iterations regress without external feedback. The one-pass cap is a direct application.

The previous flow already implicitly ran self-review once (linear phases, no loop back to Phase 4), so the cap is more of a documentation/intent change than a behavioral one — but making it explicit prevents future drift and gives the agent a principled reason to refuse re-critiquing.

Retrofit follow-up

Per CLAUDE.md's "Promoting a rule into the template" rule, this template change needs retrofit PRs in existing child skills so they don't silently lag:

  • dmccoystephenson/example-l-dev-loop
  • dmccoystephenson/example-q-dev-loop
  • dmccoystephenson/example-a-dev-loop

Not included in this PR — those are separate repos. Will follow up after this merges (or can be assigned).

Test plan

  • All {{placeholder}} tokens still have rows in the Step 4 substitution table (verified: SELF_REVIEW_RUBRIC added)
  • Conditional block syntax {{#if SELF_REVIEW_RUBRIC}}...{{/if}} matches the pattern used by other optional placeholders (CODE_PATTERNS, LINT_CMD, REVIEWER, CLAUDE_MD)
  • Generated skill compiles cleanly when /create-dev-loop is run against a real repo with and without repo-specific rubric items
  • The new Phase 4 reads as actionable instructions, not advice
  • Phase 6's "external vs. internal" distinction is clear without re-reading Phase 4

🤖 Generated with Claude Code


drafted by Claude on behalf of Daniel Stephenson

Replaces the free-form Phase 4 self-review with an objective yes/no
rubric anchored on the CI signal, adversarially framed, and explicitly
capped at one intrinsic-critique pass per PR. Phase 6 now distinguishes
internal critique (capped) from external signals (uncapped). Adds a
SELF_REVIEW_RUBRIC placeholder so generated skills can append
repo-specific rubric items, and an edge-case entry for CI failures on
the PR.

Motivation: empirical findings in RESEARCH.md sections 1 and 5. LLM
self-critique without an external grounding signal is neutral-to-harmful
(Huang ICLR 2024, Kamoi TACL 2024); rubric items are vulnerable to
self-preference bias unless phrased objectively (arXiv:2604.06996);
reflection iterations plateau by round 2.

Template-only change. Per CLAUDE.md's promotion rule, existing child
skills (dpm-dev-loop, herald-dev-loop, acsf-dev-loop) need retrofit
PRs in their respective repos as follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@dmccoystephenson
dmccoystephenson merged commit 9a8a0eb into main May 25, 2026
@dmccoystephenson
dmccoystephenson deleted the feature/rubric-based-self-review branch August 3, 2026 05:01
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
Replaces the free-form Phase 4 self-review with an objective yes/no
rubric anchored on the CI signal, adversarially framed, and explicitly
capped at one intrinsic-critique pass per PR. Phase 6 now distinguishes
internal critique (capped) from external signals (uncapped). Adds a
SELF_REVIEW_RUBRIC placeholder so generated skills can append
repo-specific rubric items, and an edge-case entry for CI failures on
the PR.

Motivation: empirical findings in RESEARCH.md sections 1 and 5. LLM
self-critique without an external grounding signal is neutral-to-harmful
(Huang ICLR 2024, Kamoi TACL 2024); rubric items are vulnerable to
self-preference bias unless phrased objectively (arXiv:2604.06996);
reflection iterations plateau by round 2.

Template-only change. Per CLAUDE.md's promotion rule, existing child
skills (a-private-repo-1, a-private-repo-2, acsf-dev-loop) need retrofit
PRs in their respective repos as follow-up.

Co-authored-by: BuildTools <unconfigured@null.spigotmc.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant