Skip to content

Track implementation outcomes back into RESEARCH.md - #27

Merged
dmccoystephenson merged 1 commit into
mainfrom
feature/track-implementation-outcomes
May 25, 2026
Merged

dmccoystephenson merged 1 commit into
mainfrom
feature/track-implementation-outcomes

Conversation

@dmccoystephenson

@dmccoystephenson dmccoystephenson commented May 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Closes the feedback loop between research-grounded predictions and what we actually observe by:

  • Adding an Implementations convention to RESEARCH.md's "How to use this document" section: when a finding gets implemented, record the PR + ship date + observed-effect placeholder under the finding.
  • Adding a corresponding bullet to CLAUDE.md's "Grounding work in research" so the rule appears at the same point of friction (PR authoring) where forward citations are already required.
  • Retroactively populating Implementations entries for findings that have already been acted on:

The "observed effect" column starts at "pending" — replacing it with real numbers requires running multiple cycles, which only becomes routine once #24 (structured self-audit) lands. This PR establishes the data-recording habit so the column has somewhere to go.

Closes #25

Test plan

  • RESEARCH.md "How to use" section gains the Implementations convention with format example
  • CLAUDE.md "Grounding work in research" gains a matching bullet pointing at RESEARCH.md
  • Three retroactive entries placed under §1, §4, §5 — confirmed by grep -n "^\*\*Implementations\." RESEARCH.md returning lines 44/104/124 (one per finding)
  • No changes to create-dev-loop.md template; substitution table unchanged
  • PR Reframe self-review as rubric-based and CI-grounded #15 ship date 2026-05-25 confirmed via gh pr view 15 --json mergedAt
  • Per the rule shipped in PR Promote 'verify each Closes #N' to template Phase 4 #26, ran gh issue view 25 and confirmed issue title/body match this PR's work

Note on AC #4

Issue #25's fourth AC ("at least one updated outcome entry, i.e. not just 'pending'") cannot be honestly satisfied yet — none of the three implementations has been measured across enough cycles to replace "pending" with a real observation. Marking that AC explicitly as deferred rather than fabricating an outcome.


drafted by Claude on behalf of Daniel Stephenson

Closes #25

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@dmccoystephenson dmccoystephenson left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review rubric:

  • Scope: PASS — only CLAUDE.md (1 line) and RESEARCH.md (15 lines across 4 places) modified; all changes implement #25's AC #1–#3 directly.
  • Tests-new: PASS — the convention introduced is prose; the three retroactive Implementations entries demonstrate the format being applied, making the convention self-exemplifying.
  • Tests-fix: N/A — not a bug fix.
  • Sibling structure: PASS — new How-to-use bullet follows the same **When ...** — ... shape as its three neighbors; Implementations subsections all use the same **Implementations.**\n- PR #N (summary): shipped DATE. Observed effect: ... shape.
  • Sibling renames: N/A — no identifiers renamed.
  • Docs: PASS — Phase 7 sources of truth for this repo are create-dev-loop.md and README.md; neither changed because this PR adds a process rule to RESEARCH.md/CLAUDE.md (themselves Phase 7 sources for the project, both updated together to keep them in sync).
  • Issue resolution: PARTIAL but transparent — AC #1, #2, #3 met; AC #4 ('at least one updated outcome entry not just pending') explicitly deferred in the PR body since outcomes require multi-cycle data that doesn't yet exist. Calling this PASS-with-noted-deferral rather than FAIL because the AC is structurally impossible to satisfy in the same PR that introduces the tracking.
  • Validation: PASS — grep -n '^\*\*Implementations\.' confirms the three entries land under §1, §4, §5; gh pr view 15 --json mergedAt confirmed PR #15 ship date.
  • Placeholder-consistency: N/A — no template change.
  • Escaping: N/A — the fenced code block added to RESEARCH.md is not inside the create-dev-loop.md outer markdown template; it's free-standing in RESEARCH.md, so no triple-backtick escape needed.

Adversarial probe: 'Should Implementations also appear in the Adding-a-new-finding template at the bottom of RESEARCH.md?' Considered and rejected — that template is for new findings, which by definition have no implementations yet; adding a blank subsection would be noise. Leaving as-is is consistent with the issue's 'optional subsection' framing.

@dmccoystephenson
dmccoystephenson merged commit 85bdad5 into main May 25, 2026
@dmccoystephenson
dmccoystephenson deleted the feature/track-implementation-outcomes branch May 25, 2026 18:35
dmccoystephenson pushed a commit that referenced this pull request May 25, 2026
The Phase 1 skip-reason and Phase 2 tiebreaker changes implement
findings §2 (agent-PR merge-rate gradient) and §7 (LLM triage as
filter not decision-maker), so per the CLAUDE.md rule shipped in
PR #27, both findings get Implementations entries pointing at this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dmccoystephenson added a commit that referenced this pull request May 25, 2026
* Bias triage and batching toward higher-merge-rate issue types

Closes #20

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Record Implementations entries for RESEARCH.md §2 and §7

The Phase 1 skip-reason and Phase 2 tiebreaker changes implement
findings §2 (agent-PR merge-rate gradient) and §7 (LLM triage as
filter not decision-maker), so per the CLAUDE.md rule shipped in
PR #27, both findings get Implementations entries pointing at this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: BuildTools <unconfigured@null.spigotmc.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
Closes #25

Co-authored-by: BuildTools <unconfigured@null.spigotmc.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
* Bias triage and batching toward higher-merge-rate issue types

Closes #20

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Record Implementations entries for RESEARCH.md §2 and §7

The Phase 1 skip-reason and Phase 2 tiebreaker changes implement
findings §2 (agent-PR merge-rate gradient) and §7 (LLM triage as
filter not decision-maker), so per the CLAUDE.md rule shipped in
PR #27, both findings get Implementations entries pointing at this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: BuildTools <unconfigured@null.spigotmc.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Track implementation outcomes back into RESEARCH.md after each evidence-backed change ships

1 participant