[delete-cloud]

Summary

• add deterministic recall evaluation reports for no-recall, accepted-memory, topic-range, and combined recall variants
• add low-cardinality recall and memory Prometheus metrics
• tighten OTLP attribute value filtering for raw recall/memory strings

Verification

• uv run pytest tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py -v
• uv run pytest tests/coding_agent/test_recall_context.py tests/coding_agent/test_memory_review.py tests/coding_agent/evaluation/ -v
• uv run pytest tests/coding_agent/test_observability_platform_smoke.py tests/coding_agent/test_context_system_smoke.py -v
• uv run ruff format --check --preview docs/cross_topic_memory/GOAL_PROGRESS.md docs/adr/0046-cross-topic-memory-and-topic-range-search.md src/coding_agent/recall_evaluation.py src/coding_agent/observability.py tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py
• uv run ruff check src/coding_agent/recall_evaluation.py src/coding_agent/observability.py tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py
• git diff --check -- .

Local review: Boole found one OTLP trace value leak; fixed and re-reviewed with no blockers.

Summary by CodeRabbit

• New Features

  • Added recall evaluation reports that compare multiple retrieval modes.
  • Introduced new memory and topic recall metrics for improved system observability.

• Bug Fixes

  • Fixed sensitive data exposure in telemetry exports by filtering sensitive values before transmission.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6607ce77-7062-45b5-a171-3db8c2f0de8e

📥 Commits

Reviewing files that changed from the base of the PR and between 070ef08 and fc7650e.

📒 Files selected for processing (6)

• coding-agent/docs/adr/0046-cross-topic-memory-and-topic-range-search.md
• coding-agent/docs/cross_topic_memory/GOAL_PROGRESS.md
• coding-agent/src/coding_agent/observability.py
• coding-agent/src/coding_agent/recall_evaluation.py
• coding-agent/tests/coding_agent/test_observability.py
• coding-agent/tests/coding_agent/test_recall_evaluation.py

---

📝 Walkthrough

Walkthrough

This PR introduces a deterministic recall evaluation framework to assess memory and topic retrieval quality across four modes, and extends observability to record and safely expose metrics for these evaluations. The changes add the recall_evaluation.py module with dataclass contracts, update observability configuration and filtering to handle memory/recall dimensions, and provide comprehensive test coverage with documentation tracking.

Changes

Recall Evaluation and Observability Instrumentation

Layer / File(s)	Summary
Recall evaluation contracts and types coding-agent/src/coding_agent/recall_evaluation.py	RecallEvalVariant enum with four modes (no recall, accepted-memory, topic-range, combined). Frozen dataclasses define RecallEvalCase inputs with validation, RecallEvalVariantResult per-variant outputs, and RecallEvalReport for full results with to_dict() serialization.
Recall evaluation implementation coding-agent/src/coding_agent/recall_evaluation.py	evaluate_recall_variants() entry point runs all four variants; _evaluate_variant() helper executes topic recall planners, extracts candidate IDs from topic results and accepted memories, computes candidate_count and matched_expected via set intersection, and derives status (matched vs empty).
Observability metric configuration and allowlists coding-agent/src/coding_agent/observability.py	Expanded sensitive-value filtering to block token/key/password/private/log substrings, forbid memory_id label, extend allowed attribute keys with memory_* and recall_* dimensions, and define constants for recall source/status and memory kind/status enumerations.
Observability OTLP filtering and metric recording coding-agent/src/coding_agent/observability.py	Added _attribute_value_allowed() to filter OTLP exports rejecting attribute values containing sensitive tokens. PrometheusMetricsRecorder methods record topic recall run outcomes/candidate counts, memory candidate states, and memory review transitions with normalized dimensions.
Observability exposition and module-level recording functions coding-agent/src/coding_agent/observability.py	exposition_text() emits new Prometheus metrics (topic_recall_runs_total, topic_recall_candidates, memory_candidates_total, memory_reviews_total) with type/help declarations. Added internal property selectors and exception-safe module-level recording functions (record_topic_recall_metric, record_memory_candidate_metric, record_memory_review_metric).
Recall evaluation test suite coding-agent/tests/coding_agent/test_recall_evaluation.py	Tests validate variant ordering and per-variant candidate/matched counts, verify report.to_dict() omits raw summary/query text, confirm RecallEvalCase rejects forbidden raw-content markers. Helper _topic() builds deterministic test topic records.
Observability test suite for recall metrics coding-agent/tests/coding_agent/test_observability.py	Tests assert Prometheus emission of low-cardinality recall/memory label sets, normalization of unlisted/unsafe values to unknown, absence of high-cardinality identifiers and raw evidence strings, and OTLP filtering of raw evidence attribute values while preserving normalized status fields.
Documentation and progress tracking coding-agent/docs/adr/0046-cross-topic-memory-and-topic-range-search.md, coding-agent/docs/cross_topic_memory/GOAL_PROGRESS.md	ADR acceptance criteria mark tests as complete. GOAL_PROGRESS documents G142 phase: evaluation variants, new metrics, no-leak/cardinality/serialization regression tests, and OTLP raw-value leak fix via sensitive-string filtering.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 Recall now speaks in metrics clear,
With variants evaluated far and near—
From memory's archive to topic's range,
Each observation safe, no secrets strange,
Four modes of truth, filtered, secure!

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/cross-topic-memory-g142-eval-observability

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}