Skip to content

Structure Phase 9 self-audit prompts and label gap issues - #34

Merged
dmccoystephenson merged 1 commit into
mainfrom
feature/structured-self-audit
May 25, 2026
Merged

dmccoystephenson merged 1 commit into
mainfrom
feature/structured-self-audit

Conversation

@dmccoystephenson

@dmccoystephenson dmccoystephenson commented May 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Three changes to the generated dev-loop template:

  1. Phase 9 self-audit becomes a structured rubric. Replaces the free-form bullet list with a 6-item rubric (identity drift, instruction clarity, edge case coverage, phase friction, drift candidates, external-signal quality) — each scored PASS / FAIL / no-signal with a one-line justification. Mirrors the Phase 4 self-review structure shipped in PR Reframe self-review as rubric-based and CI-grounded #15. Grounded in RESEARCH.md §1 and §5 (structured rubrics outperform free-form for LLM critique).

  2. Filed gap issues now carry a label from a 5-label taxonomy: template-rule, repo-specific, edge-case, research-gap, process. Makes triage faster and surfaces patterns across child-skill repos.

  3. Step 6 creates the labels at repo-creation time so they exist when the first cycle's Phase 9 fires. Uses a single for loop over a colon-delimited list of name:color:description entries.

Closes #24

Test plan

Note on label taxonomy duplication

The taxonomy is listed in two places: Phase 9 (where the agent picks one) and Step 6 (where the agent creates the labels). They must stay in sync. A future PR could DRY this — e.g. a single placeholder {{GAP_LABEL_TAXONOMY}} — but that's overengineering for 5 entries. Listed in both spots verbatim; renaming-siblings-together rule applies if anyone changes the names.


drafted by Claude on behalf of Daniel Stephenson

Closes #24

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@dmccoystephenson dmccoystephenson left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review rubric:

  • Scope: PASS — 2 files, 44 lines; each change maps to one of #24's 4 AC items or the CLAUDE.md PR-27 Implementations rule.
  • Tests-new: PASS — the new self-audit rubric is exercised by this very cycle (I'll run it after merge); the new label taxonomy is exercised by the Step 6 loop syntax (verified shell expansion works with IFS=:).
  • Tests-fix: N/A — not a bug fix.
  • Sibling structure: PASS — Phase 9 rubric block matches the Phase 4 rubric structure (bold-item-name + colon + question; PASS/FAIL/no-signal scoring); new Step 6 bash block matches the existing Step 6 bash-block style (fenced bash, no surrounding prose interruption).
  • Sibling renames: PASS — the label-taxonomy names (template-rule, repo-specific, etc.) appear verbatim in Phase 9 AND Step 6; siblings renamed/added together.
  • Docs: PASS — Phase 7 sources (create-dev-loop.md, README.md) reviewed; README is high-level and doesn't enumerate Phase 9 internals (correctly out of scope). RESEARCH.md §1 and §5 get Implementations entries per PR-27 rule.
  • Issue resolution: PASS — all 4 AC items met: (1) structured checklist replaces bullet list; (2) substitution table unchanged; (3) gap-issue command takes --label; (4) labels created in Step 6 before first cycle runs.
  • Validation: PASS — placeholder count unchanged at 22; Step 6 bash block uses standard \``` plain fence (Step 6 is OUTSIDE the outer template, so no escape needed); Phase 9 fenced block uses leading-backslash escape (Phase 9 is INSIDE the outer template).
  • {{EXTERNAL_SIGNAL_LABEL}}: N/A — no CI; manual validation against diff served as anchor.
  • Placeholder-consistency: N/A — no template placeholder added.
  • Escaping: PASS — confirmed by reading the diff: Step 6 label-creation block uses plain triple-backticks (it's outside the embedded template), Phase 9 issue-create block uses \``bash` with leading backslash (inside template).

Adversarial probe — 'Does the new rubric's External-signal-quality item make sense for repos where the external anchor is just "run the test command locally"?' Considered: yes, the item asks 'did it catch the kinds of issues it's supposed to catch this cycle' — for any anchor (CI, manual validation, local tests), the agent can score it. No exclusion needed.

@dmccoystephenson
dmccoystephenson merged commit c348a32 into main May 25, 2026
@dmccoystephenson
dmccoystephenson deleted the feature/structured-self-audit branch May 25, 2026 18:58
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
Closes #24

Co-authored-by: BuildTools <unconfigured@null.spigotmc.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Structure Phase 9 self-audit prompts and label resulting gap issues

1 participant