Repository navigation
Add pre-edit localization verification and pre-merge regression gate - #31
Merged
Merged
Conversation
Closes #18 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dmccoystephenson
left a comment
Member
Author
There was a problem hiding this comment.
Self-review rubric:
- Scope: PASS — 2 files (create-dev-loop.md, RESEARCH.md); each line directly maps to an AC item or the CLAUDE.md PR-27 Implementations rule.
- Tests-new: PASS — the localization step is exercised by this PR's own pre-edit verification (
grepforPhase 3 — ImplementationandPhase 8 — Mergeheadings before editing). - Tests-fix: N/A — not a bug fix.
- Sibling structure: PASS — new Phase 3 localization block matches the 'Bold label. explanation + numbered list' pattern of other Phase 3 sub-blocks; new Phase 8 regression gate matches the bold-label structure of the do-not-auto-merge sibling below it.
- Sibling renames: N/A — no identifiers renamed.
- Docs: PASS — Phase 7 sources of truth (
create-dev-loop.md,README.md) reviewed; README doesn't enumerate inner Phase steps at this granularity (correctly stays high-level). RESEARCH.md §3 gets an Implementations entry per the CLAUDE.md PR-27 rule. - Issue resolution: PASS-with-deferral — AC 1 (Phase 3 localization step) met; AC 2 (Phase 8 regression gate) met; AC 3 (fixture test) explicitly deferred in PR body, same reason as PR #30.
- Validation: PASS — placeholder count unchanged; new content uses only inline backticks (no new fenced blocks to escape); phase numbering unchanged.
- Placeholder-consistency: N/A — no new placeholders.
- Escaping: N/A — no fenced code blocks added.
Adversarial probe — 'Does the regression gate work for test-less repos like create-dev-loop itself?' The gate prose says '(or, for projects whose external anchor is manual validation, a new validation step)'. Confirmed by re-reading the new text: the OR clause keeps the gate applicable to projects without a test suite — verifiable by checking whether the diff adds a new test-plan checkbox or updates the project's manual-validation guidance. The generalization works.
6 of 7 tasks
6 of 7 tasks
dmccoystephenson
added a commit
that referenced
this pull request
Aug 25, 2026
Finding 1's entry covers both defects: a revert that silently does nothing returns Tests-fix to unanchored judgment while still reporting an anchor's verdict, and a ladder rung whose interruption leaves the head undoing its own fix damages the anchor it substitutes for. Finding 3's entry covers the consequence for the regression gate it motivated in PR #31 — manufactured evidence cannot distinguish a real fix from a coincidental patch.
10 of 11 tasks
dmccoystephenson
added a commit
that referenced
this pull request
Aug 27, 2026
…dder rung 1 (#136) * Make the checkout form Phase 4's default Tests-fix revert and gate rung 1 The Tests-fix rubric item prescribed `git stash push` on a tree that Phase 3 has already committed and Phase 4 has already pushed, so the stash saved nothing, the check ran against the unmodified tree, and the no-op read as a false-negative in the test. The checkout form is now the default, stash is demoted to the genuinely-uncommitted case, and the revert must be confirmed to have changed the tree before either half is trusted. The fallback ladder loses its "retry with checkout first" preamble, since the retry is now the default, and rung 1 gains an entry condition: a headless or timeout-bounded dispatch must not push a head that undoes its own fix, because a killed session leaves nothing to restore it. Closes #116 Closes #132 * Record PR #136 under RESEARCH.md findings 1 and 3 Finding 1's entry covers both defects: a revert that silently does nothing returns Tests-fix to unanchored judgment while still reporting an anchor's verdict, and a ladder rung whose interruption leaves the head undoing its own fix damages the anchor it substitutes for. Finding 3's entry covers the consequence for the regression gate it motivated in PR #31 — manufactured evidence cannot distinguish a real fix from a coincidental patch. * Keep rung 1 CI-specific and revert to the merge base, not the branch tip Two self-review findings. Substituting {{EXTERNAL_SIGNAL_LABEL}} into the ladder preamble and rung 1 was wrong: rung 1 pushes a commit and reads a check result, so it is inherently CI-specific and cannot be performed at all where the anchor is a manual checklist. The literal wording is restored and a sentence now states that unavailability outright. The checkout form said `git checkout origin/<default> -- <src files>`, which also pulls in anything that landed on the default branch since the branch was cut. The merge base is named as the correct revert target.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two structural additions to the generated dev-loop template, grounded in RESEARCH.md §3 (localization is the real bottleneck on uncontaminated benchmarks):
Phase 3 — Localization verification. Before writing any code, list the target files and verify each with
test -f+ agrepfor the named surface area from the issue. If the issue says the bug is invalidatePermissionand grep finds nothing, the localization is wrong — stop and re-triage.Phase 8 — Regression gate. For each
Closes #Nthat describes a bug or incorrect behavior, require a new/modified test (or validation step, for projects whose external anchor is manual validation) in the diff. Hardens the Phase 4 Tests-fix rubric item into a merge-time gate.Generalized cleanly with the
{{EXTERNAL_SIGNAL_LABEL}}infrastructure shipped in PR #28: test-having repos check for a new test, no-CI repos check for a new validation step.Closes #18
Test plan
{{placeholders}}(placeholder count unchanged); the regression gate reuses{{EXTERNAL_SIGNAL_LABEL}}indirectly via the prosecreate-dev-loop.mdexists, contains "Phase 3 — Implementation" and "Phase 8 — Merge" headingsgh issue view 18and confirmed title/body matchgit diff --name-only origin/main...HEADand confirmed no matches against do-not-auto-merge listFixture-validation note
#18's test plan asks for "a fixture issue where the agent would naively localize to the wrong file" — same as #17's fixture ask, this requires fixture tooling that doesn't exist. The localization check is currently authority-based, not enforced. A pre-edit check could be wired into a hook in the future.
drafted by Claude on behalf of Daniel Stephenson