Skip to content

docs(adr): accept ADR-0074 with live calibration evidence - #669

Closed
delete-cloud wants to merge 1 commit into
mainfrom
docs/adr-0074-accept
Closed

delete-cloud wants to merge 1 commit into
mainfrom
docs/adr-0074-accept

Conversation

@delete-cloud

@delete-cloud delete-cloud commented Jul 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

Flip ADR-0074 Proposed → Accepted: implementation merged (#668), and the live o6n acceptance is now complete.

  • Two-round calibration: round 1 (0.4/0.3, sre-infra refactor: use agentkit session result reducer #182) was miscalibrated on token-overlap-scale samples — the semantic noise band 0.433–0.493 passed the floor; round 2 (sre-infra feat: expose workspace publication artifact refs #183) raised recall_min_score to 0.5.
  • Final live values: recall_min_score=0.5, recall_min_overlap=0.3.
  • Regression proof (evidence in this PR): restic probe injects zero cross-topic noise and grounds from the sre KB corpus (Repo references, distances 0.742–0.781, answer cites volsync-restic-async-backup); relevant seeded-topic recall survives (overlap 0.5333, sole injected item).
  • New follow-ups recorded: rendered recall scores carry no scale label (caused the round-1 miscalibration; feeds D0), and D9: bge-m3 short-summary similarity bands are barely separable.

Test plan

  • Docs + evidence only; CI full suite gates the PR.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation
    • Updated the semantic recall ADR to accepted status and marked its acceptance criteria as complete with validation details and rollout notes.
    • Expanded dogfood evidence to include additional runs, timestamps, and topic probe results.
    • Added new JSONL evidence entries covering recent semantic-memory seed, probe, and run outcomes.

Two-round o6n calibration (floors 0.5/0.3 live via sre-infra #182/#183);
restic regression grounds from the KB corpus with zero cross-topic noise
while relevant seeded-topic recall survives. Evidence updated (28 records).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6b501f07-1b53-44e0-8100-ba9eb070e07e

📥 Commits

Reviewing files that changed from the base of the PR and between a4b2eee and 01f3289.

📒 Files selected for processing (3)
  • coding-agent/docs/adr/0074-semantic-recall-relevance-threshold.md
  • coding-agent/docs/dogfood/SEMANTIC_MEMORY_RUN_EVIDENCE.md
  • coding-agent/docs/dogfood/semantic_memory_dogfood_evidence.jsonl

📝 Walkthrough

Walkthrough

This PR updates documentation only: ADR-0074's status changes from Proposed to Accepted with completed acceptance criteria details, and the dogfood semantic memory evidence report and its JSONL data file are updated with new run metrics and evidence records reflecting recall floor calibration.

Changes

ADR acceptance and evidence documentation

Layer / File(s) Summary
ADR-0074 status and acceptance criteria
coding-agent/docs/adr/0074-semantic-recall-relevance-threshold.md
Status changed from Proposed to Accepted; acceptance criteria checklist marked complete with pytest validation notes, PR #668 merge confirmation, calibration summary, and regression evidence.
Dogfood evidence updates
coding-agent/docs/dogfood/SEMANTIC_MEMORY_RUN_EVIDENCE.md, coding-agent/docs/dogfood/semantic_memory_dogfood_evidence.jsonl
Summary metrics increased and Records table expanded with new seed/topic probe rows; JSONL file appends corresponding seed, topic probe (fail/pass), run-summary, and run entries.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Related issues: None specified

Related PRs: References PR #668 (implementation merge mentioned in ADR acceptance criteria)

Suggested labels: documentation

Suggested reviewers: None specified

🐰 Status flipped from proposed to done,
Checklists checked, the ADR won,
Dogfood records grow in rows,
Recall floors calibrated, as evidence shows,
A quiet hop through docs, review complete!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: accepting ADR-0074 and adding live calibration evidence.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/adr-0074-accept

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@delete-cloud
delete-cloud deleted the docs/adr-0074-accept branch July 3, 2026 08:40
@delete-cloud
delete-cloud restored the docs/adr-0074-accept branch July 5, 2026 04:34
@delete-cloud
delete-cloud deleted the docs/adr-0074-accept branch July 5, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant