Skip to content

Track implementation outcomes back into RESEARCH.md after each evidence-backed change ships #25

Closed

Description

@dmccoystephenson

Problem

The "How to use" section of RESEARCH.md requires citing findings forward when proposing changes, but it does not require recording outcomes backward when a change ships. This means:

  1. We can't tell from RESEARCH.md which findings have been implemented vs. which are still pending.
  2. When a research-backed change ships, we don't capture whether it worked in practice — we have the prediction (expected impact from the literature) but no record of the observed effect.
  3. Future contributors can't distinguish "this finding hasn't been acted on yet" from "this finding was acted on and it didn't help / was reverted."

Motivation

The whole point of grounding work in research is to learn from outcomes. Without an outcome log, we're consuming the literature but not contributing back to our own understanding. Over a few template-promotion cycles, this becomes the difference between empirically-informed and empirically-grounded design.

Connected to RESEARCH.md §8 — drift: without outcome tracking, we can't tell when a prior finding has been superseded by what we've actually observed in our own loops.

Confidence: low (this is a process improvement, not an empirical claim — but the lack of any back-pressure mechanism is visible).

Proposal

Extend each finding section in RESEARCH.md with an optional "Implementations" subsection:

**Implementations.**
- PR #15 (rubric-based self-review): shipped 2026-05-25. Observed effect: [pending — needs N cycles of data].
- PR #X: ...

When PR #15-style work ships, the same PR (or a follow-up) adds an entry under the relevant finding's "Implementations" subsection. Initially the observed-effect column reads "pending"; after a few cycles, the entry is updated based on observed behavior (rate of caught issues, false positives, etc.).

The CLAUDE.md "Grounding work in research" section gets a new bullet:

If your PR implements a finding from RESEARCH.md, also add an "Implementations" entry under that finding with the PR number and a placeholder "observed effect: pending".

Acceptance criteria

Open question

Where do "observed effects" actually get measured? Phase 9 self-audit is the natural place if #24 lands (structured self-audit), but until then this risks being aspirational.

Related


drafted by Claude on behalf of Daniel Stephenson

Activity

  1. added a commit that references this issue on May 25, 2026
    85bdad5
  2. added a commit that references this issue on Aug 3, 2026
    8e7c8ca
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions