Skip to content

Add pre-edit localization verification and pre-merge regression gate - #31

Merged
dmccoystephenson merged 1 commit into
mainfrom
feature/localization-regression-gates
May 25, 2026
Merged

dmccoystephenson merged 1 commit into
mainfrom
feature/localization-regression-gates

Conversation

@dmccoystephenson

@dmccoystephenson dmccoystephenson commented May 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Two structural additions to the generated dev-loop template, grounded in RESEARCH.md §3 (localization is the real bottleneck on uncontaminated benchmarks):

  1. Phase 3 — Localization verification. Before writing any code, list the target files and verify each with test -f + a grep for the named surface area from the issue. If the issue says the bug is in validatePermission and grep finds nothing, the localization is wrong — stop and re-triage.

  2. Phase 8 — Regression gate. For each Closes #N that describes a bug or incorrect behavior, require a new/modified test (or validation step, for projects whose external anchor is manual validation) in the diff. Hardens the Phase 4 Tests-fix rubric item into a merge-time gate.

Generalized cleanly with the {{EXTERNAL_SIGNAL_LABEL}} infrastructure shipped in PR #28: test-having repos check for a new test, no-CI repos check for a new validation step.

Closes #18

Test plan

Fixture-validation note

#18's test plan asks for "a fixture issue where the agent would naively localize to the wrong file" — same as #17's fixture ask, this requires fixture tooling that doesn't exist. The localization check is currently authority-based, not enforced. A pre-edit check could be wired into a hook in the future.


drafted by Claude on behalf of Daniel Stephenson

Closes #18

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@dmccoystephenson dmccoystephenson left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review rubric:

  • Scope: PASS — 2 files (create-dev-loop.md, RESEARCH.md); each line directly maps to an AC item or the CLAUDE.md PR-27 Implementations rule.
  • Tests-new: PASS — the localization step is exercised by this PR's own pre-edit verification (grep for Phase 3 — Implementation and Phase 8 — Merge headings before editing).
  • Tests-fix: N/A — not a bug fix.
  • Sibling structure: PASS — new Phase 3 localization block matches the 'Bold label. explanation + numbered list' pattern of other Phase 3 sub-blocks; new Phase 8 regression gate matches the bold-label structure of the do-not-auto-merge sibling below it.
  • Sibling renames: N/A — no identifiers renamed.
  • Docs: PASS — Phase 7 sources of truth (create-dev-loop.md, README.md) reviewed; README doesn't enumerate inner Phase steps at this granularity (correctly stays high-level). RESEARCH.md §3 gets an Implementations entry per the CLAUDE.md PR-27 rule.
  • Issue resolution: PASS-with-deferral — AC 1 (Phase 3 localization step) met; AC 2 (Phase 8 regression gate) met; AC 3 (fixture test) explicitly deferred in PR body, same reason as PR #30.
  • Validation: PASS — placeholder count unchanged; new content uses only inline backticks (no new fenced blocks to escape); phase numbering unchanged.
  • Placeholder-consistency: N/A — no new placeholders.
  • Escaping: N/A — no fenced code blocks added.

Adversarial probe — 'Does the regression gate work for test-less repos like create-dev-loop itself?' The gate prose says '(or, for projects whose external anchor is manual validation, a new validation step)'. Confirmed by re-reading the new text: the OR clause keeps the gate applicable to projects without a test suite — verifiable by checking whether the diff adds a new test-plan checkbox or updates the project's manual-validation guidance. The generalization works.

@dmccoystephenson
dmccoystephenson merged commit 70aa4e7 into main May 25, 2026
@dmccoystephenson
dmccoystephenson deleted the feature/localization-regression-gates branch May 25, 2026 18:50
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
…31)

Closes #18

Co-authored-by: BuildTools <unconfigured@null.spigotmc.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dmccoystephenson added a commit that referenced this pull request Aug 25, 2026
Finding 1's entry covers both defects: a revert that silently does nothing
returns Tests-fix to unanchored judgment while still reporting an anchor's
verdict, and a ladder rung whose interruption leaves the head undoing its own
fix damages the anchor it substitutes for. Finding 3's entry covers the
consequence for the regression gate it motivated in PR #31 — manufactured
evidence cannot distinguish a real fix from a coincidental patch.
dmccoystephenson added a commit that referenced this pull request Aug 27, 2026
…dder rung 1 (#136)

* Make the checkout form Phase 4's default Tests-fix revert and gate rung 1

The Tests-fix rubric item prescribed `git stash push` on a tree that Phase 3
has already committed and Phase 4 has already pushed, so the stash saved
nothing, the check ran against the unmodified tree, and the no-op read as a
false-negative in the test. The checkout form is now the default, stash is
demoted to the genuinely-uncommitted case, and the revert must be confirmed to
have changed the tree before either half is trusted.

The fallback ladder loses its "retry with checkout first" preamble, since the
retry is now the default, and rung 1 gains an entry condition: a headless or
timeout-bounded dispatch must not push a head that undoes its own fix, because
a killed session leaves nothing to restore it.

Closes #116
Closes #132

* Record PR #136 under RESEARCH.md findings 1 and 3

Finding 1's entry covers both defects: a revert that silently does nothing
returns Tests-fix to unanchored judgment while still reporting an anchor's
verdict, and a ladder rung whose interruption leaves the head undoing its own
fix damages the anchor it substitutes for. Finding 3's entry covers the
consequence for the regression gate it motivated in PR #31 — manufactured
evidence cannot distinguish a real fix from a coincidental patch.

* Keep rung 1 CI-specific and revert to the merge base, not the branch tip

Two self-review findings.

Substituting {{EXTERNAL_SIGNAL_LABEL}} into the ladder preamble and rung 1 was
wrong: rung 1 pushes a commit and reads a check result, so it is inherently
CI-specific and cannot be performed at all where the anchor is a manual
checklist. The literal wording is restored and a sentence now states that
unavailability outright.

The checkout form said `git checkout origin/<default> -- <src files>`, which
also pulls in anything that landed on the default branch since the branch was
cut. The merge base is named as the correct revert target.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add pre-edit localization verification and pre-merge regression test gate

1 participant