G142 recall eval and observability

coding-agent/docs/adr/0046-cross-topic-memory-and-topic-range-search.md

@@ -169,7 +169,8 @@ multi-agent task graph work are out of scope.
 - [x] `test_accepted_memory_context_pack_is_reference_only`
 - [x] `test_recall_plan_records_anchor_and_topic_recall_links`
 - [x] `test_recall_context_pack_contains_reference_evidence`
-- [ ] `test_recall_metrics_omit_high_cardinality_labels`
+- [x] `test_recall_eval_report_compares_all_recall_variants`
+- [x] `test_recall_metrics_omit_high_cardinality_labels`
 - [ ] `test_console_memory_recall_renders_safe_provenance`
 - [ ] `test_cross_topic_memory_e2e_smoke`
 - [ ] `uv run pytest tests/coding_agent/test_topic_layer_smoke.py -v`

coding-agent/docs/cross_topic_memory/GOAL_PROGRESS.md

@@ -279,3 +279,64 @@ phase.
   - Added disabled mode and no-leak query validation.
 - Remaining risks:
   - G142 still needs recall evaluation and low-cardinality recall/memory metrics.
+
+## G142_RECALL_EVAL_AND_OBSERVABILITY
+
+### Before
+
+- Goal id: G142_RECALL_EVAL_AND_OBSERVABILITY
+- Intended files:
+  - `docs/cross_topic_memory/GOAL_PROGRESS.md`
+  - `docs/adr/0046-cross-topic-memory-and-topic-range-search.md`
+  - `src/coding_agent/recall_evaluation.py`
+  - `src/coding_agent/observability.py`
+  - `tests/coding_agent/test_recall_evaluation.py`
+  - `tests/coding_agent/test_observability.py`
+- Verification commands:
+  - `uv run pytest tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py -v`
+  - `uv run pytest tests/coding_agent/test_recall_context.py tests/coding_agent/test_memory_review.py tests/coding_agent/evaluation/ -v`
+  - `uv run ruff format --check --preview docs/cross_topic_memory/GOAL_PROGRESS.md docs/adr/0046-cross-topic-memory-and-topic-range-search.md src/coding_agent/recall_evaluation.py src/coding_agent/observability.py tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py`
+  - `uv run ruff check src/coding_agent/recall_evaluation.py src/coding_agent/observability.py tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py`
+  - `git diff --check -- .`
+- Stop criteria:
+  - Eval helper compares no recall, accepted-memory recall, topic-range recall,
+    and combined topic+memory recall deterministically.
+  - Recall/memory metrics emit only low-cardinality labels.
+  - Metrics omit `memory_id`, `topic_id`, `task_id`, `run_id`, and
+    `session_id` labels.
+  - Recall evaluation reports and metrics avoid raw evidence/log content.
+
+### After
+
+- Changed files:
+  - `docs/cross_topic_memory/GOAL_PROGRESS.md`
+  - `docs/adr/0046-cross-topic-memory-and-topic-range-search.md`
+  - `src/coding_agent/recall_evaluation.py`
+  - `src/coding_agent/observability.py`
+  - `tests/coding_agent/test_recall_evaluation.py`
+  - `tests/coding_agent/test_observability.py`
+- Tests run:
+  - `uv run pytest tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py -q`
+  - `uv run pytest tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py -v`
+  - `uv run pytest tests/coding_agent/test_recall_context.py tests/coding_agent/test_memory_review.py tests/coding_agent/evaluation/ -v`
+  - `uv run pytest tests/coding_agent/test_observability_platform_smoke.py tests/coding_agent/test_context_system_smoke.py -v`
+  - `uv run ruff format --check --preview docs/cross_topic_memory/GOAL_PROGRESS.md docs/adr/0046-cross-topic-memory-and-topic-range-search.md src/coding_agent/recall_evaluation.py src/coding_agent/observability.py tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py`
+  - `uv run ruff check src/coding_agent/recall_evaluation.py src/coding_agent/observability.py tests/coding_agent/test_recall_evaluation.py tests/coding_agent/test_observability.py`
+  - `git diff --check -- .`
+- Results:
+  - Red check before implementation failed on missing
+    `coding_agent.recall_evaluation` and missing recall/memory metric wrappers,
+    confirming the target tests covered the new behavior.
+  - Added deterministic recall evaluation reports for no-recall,
+    accepted-memory, topic-range, and combined recall variants.
+  - Added recall and memory metrics:
+    `topic_recall_runs_total`, `topic_recall_candidates`,
+    `memory_candidates_total`, and `memory_reviews_total`.
+  - Added no-leak and cardinality tests for recall/memory metrics and recall
+    evaluation report serialization.
+  - Local review found an OTLP trace value leak for raw recall/memory attribute
+    values; fixed by filtering sensitive string values before OTLP
+    serialization and adding a regression test.
+- Remaining risks:
+  - G143 still needs Developer Console memory/recall rendering and safe review
+    surface coverage.

coding-agent/src/coding_agent/observability.py

@@ -36,14 +36,20 @@
 _SENSITIVE_PROMETHEUS_VALUE_PARTS = frozenset({
     "content",
     "env",
+    "key",
+    "log",
     "message",
     "output",
+    "password",
+    "private",
     "prompt",
+    "raw",
     "result",
     "secret",
     "stderr",
     "stdout",
     "text",
+    "token",
 })
 _FORBIDDEN_PROMETHEUS_LABELS = frozenset({
     "run_id",
@@ -61,6 +67,7 @@
     "interaction_id",
     "job_name",
     "launch_id",
+    "memory_id",
     "pod_name",
     "tool_call_id",
     "workflow_name",
@@ -79,10 +86,15 @@
     "executor_kind",
     "eval_status",
     "hitl_status",
+    "memory_kind",
+    "memory_review_status",
+    "memory_status",
     "model",
     "operation",
     "policy_decision",
     "provider",
+    "recall_source",
+    "recall_status",
     "risk_level",
     "schedule_kind",
     "schedule_status",
@@ -170,6 +182,17 @@
     }),
     "eval_status": frozenset({"failed", "passed", "skipped"}),
     "hitl_status": frozenset({"approved", "rejected", "requested", "timed_out"}),
+    "memory_kind": frozenset({
+        "command_memory",
+        "decision",
+        "fact",
+        "incident",
+        "procedure",
+        "project_convention",
+        "unknown",
+    }),
+    "memory_review_status": frozenset({"accepted", "archived", "rejected"}),
+    "memory_status": frozenset({"accepted", "archived", "candidate", "rejected"}),
     "operation": frozenset({
         "checkpoint_load",
         "checkpoint_save",
@@ -179,6 +202,20 @@
         "tape_load",
     }),
     "policy_decision": frozenset({"allow", "approval_required", "deny"}),
+    "recall_source": frozenset({
+        "accepted_memory",
+        "none",
+        "topic_and_memory",
+        "topic_range",
+        "unknown",
+    }),
+    "recall_status": frozenset({
+        "disabled",
+        "empty",
+        "failed",
+        "matched",
+        "unknown",
+    }),
     "risk_level": frozenset({"low", "medium", "high"}),
     "schedule_kind": frozenset({"interval", "manual", "once", "unknown"}),
     "schedule_status": frozenset({"active", "completed", "disabled", "paused"}),
@@ -261,6 +298,40 @@
     "unavailable",
     "unknown",
 })
+_TOPIC_RECALL_SOURCES: Final[frozenset[str]] = frozenset({
+    "accepted_memory",
+    "none",
+    "topic_and_memory",
+    "topic_range",
+    "unknown",
+})
+_TOPIC_RECALL_STATUSES: Final[frozenset[str]] = frozenset({
+    "disabled",
+    "empty",
+    "failed",
+    "matched",
+    "unknown",
+})
+_MEMORY_KINDS: Final[frozenset[str]] = frozenset({
+    "command_memory",
+    "decision",
+    "fact",
+    "incident",
+    "procedure",
+    "project_convention",
+    "unknown",
+})
+_MEMORY_STATUSES: Final[frozenset[str]] = frozenset({
+    "accepted",
+    "archived",
+    "candidate",
+    "rejected",
+})
+_MEMORY_REVIEW_STATUSES: Final[frozenset[str]] = frozenset({
+    "accepted",
+    "archived",
+    "rejected",
+})
 _PROMETHEUS_HISTOGRAM_BUCKETS = (5.0, 10.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 5000.0)
 _PROMETHEUS_HTTP_HISTOGRAM_BUCKETS = (
     1.0,
@@ -397,10 +468,19 @@ def _otlp_attributes(attributes: Mapping[str, Any]) -> list[dict[str, Any]]:
     return [
         {"key": key, "value": _otlp_value(value)}
         for key, value in attributes.items()
-        if _attribute_allowed(key)
+        if _attribute_allowed(key) and _attribute_value_allowed(value)
     ]
 
 
+def _attribute_value_allowed(value: Any) -> bool:
+    if not isinstance(value, str):
+        return True
+    normalized_tokens = {
+        token for token in re.split(r"[^a-zA-Z0-9]+", value.casefold()) if token
+    }
+    return not bool(normalized_tokens & _SENSITIVE_PROMETHEUS_VALUE_PARTS)
+
+
 def _resource_attributes(attributes: Mapping[str, Any]) -> list[dict[str, Any]]:
     resource: dict[str, Any] = {"service.name": "coding-agent"}
     session_id = attributes.get("session_id")
@@ -648,6 +728,39 @@ def record_executor_capability(
         with self._lock:
             self._set("executor_capability_status", labels, 1.0)
 
+    def record_topic_recall_run(
+        self,
+        *,
+        source: str,
+        status: str,
+        candidate_count: int,
+    ) -> None:
+        labels = {
+            "source": _prometheus_metric_part(source, _TOPIC_RECALL_SOURCES),
+            "status": _prometheus_metric_part(status, _TOPIC_RECALL_STATUSES),
+        }
+        histogram_labels = {"source": labels["source"]}
+        with self._lock:
+            self._inc("topic_recall_runs_total", labels)
+            self._observe(
+                "topic_recall_candidates",
+                histogram_labels,
+                max(0.0, float(candidate_count)),
+            )
+
+    def record_memory_candidate(self, *, kind: str, status: str) -> None:
+        labels = {
+            "kind": _prometheus_metric_part(kind, _MEMORY_KINDS),
+            "status": _prometheus_metric_part(status, _MEMORY_STATUSES),
+        }
+        with self._lock:
+            self._inc("memory_candidates_total", labels)
+
+    def record_memory_review(self, *, status: str) -> None:
+        labels = {"status": _prometheus_metric_part(status, _MEMORY_REVIEW_STATUSES)}
+        with self._lock:
+            self._inc("memory_reviews_total", labels)
+
     def exposition_text(self) -> str:
         lines: list[str] = []
         with self._lock:
@@ -838,6 +951,53 @@ def exposition_text(self) -> str:
                         executor_capabilities,
                     )
                 )
+            topic_recall_counters = self._topic_recall_counters
+            topic_recall_histograms = self._topic_recall_histograms
+            if topic_recall_counters:
+                lines.extend((
+                    "# HELP topic_recall_runs_total Topic recall runs by source and status.",
+                    "# TYPE topic_recall_runs_total counter",
+                ))
+                lines.extend(
+                    self._format_metric(
+                        "topic_recall_runs_total",
+                        topic_recall_counters,
+                    )
+                )
+            if topic_recall_histograms:
+                lines.extend((
+                    "# HELP topic_recall_candidates Recall candidates returned by source.",
+                    "# TYPE topic_recall_candidates histogram",
+                ))
+                lines.extend(
+                    self._format_histograms(
+                        histograms=topic_recall_histograms,
+                        metric_name="topic_recall_candidates",
+                        buckets=_PROMETHEUS_HISTOGRAM_BUCKETS,
+                    )
+                )
+            memory_counters = self._memory_counters
+            if memory_counters:
+                lines.extend((
+                    "# HELP memory_candidates_total Topic-derived memory candidates by kind and status.",
+                    "# TYPE memory_candidates_total counter",
+                ))
+                lines.extend(
+                    self._format_metric(
+                        "memory_candidates_total",
+                        memory_counters,
+                    )
+                )
+                lines.extend((
+                    "# HELP memory_reviews_total Memory review transitions by status.",
+                    "# TYPE memory_reviews_total counter",
+                ))
+                lines.extend(
+                    self._format_metric(
+                        "memory_reviews_total",
+                        memory_counters,
+                    )
+                )
         return "\n".join(lines) + ("\n" if lines else "")
 
     def reset(self) -> None:
@@ -972,6 +1132,36 @@ def _executor_capability_gauges(
             if key[0] == "executor_capability_status"
         }
 
+    @property
+    def _topic_recall_counters(
+        self,
+    ) -> dict[tuple[str, tuple[tuple[str, str], ...]], float]:
+        return {
+            key: value
+            for key, value in self._counters.items()
+            if key[0] == "topic_recall_runs_total"
+        }
+
+    @property
+    def _topic_recall_histograms(
+        self,
+    ) -> dict[tuple[str, tuple[tuple[str, str], ...]], list[float]]:
+        return {
+            key: value
+            for key, value in self._histograms.items()
+            if key[0] == "topic_recall_candidates"
+        }
+
+    @property
+    def _memory_counters(
+        self,
+    ) -> dict[tuple[str, tuple[tuple[str, str], ...]], float]:
+        return {
+            key: value
+            for key, value in self._counters.items()
+            if key[0] in {"memory_candidates_total", "memory_reviews_total"}
+        }
+
     def _inc(self, metric: str, labels: Mapping[str, str], amount: float = 1.0) -> None:
         key = (metric, tuple(sorted(labels.items())))
         self._counters[key] = self._counters.get(key, 0.0) + amount
@@ -1149,6 +1339,39 @@ def record_executor_capability_metric(
         return
 
 
+def record_topic_recall_metric(
+    *,
+    source: str,
+    status: str,
+    candidate_count: int,
+) -> None:
+    try:
+        _DEFAULT_PROMETHEUS_RECORDER.record_topic_recall_run(
+            source=source,
+            status=status,
+            candidate_count=candidate_count,
+        )
+    except Exception:
+        return
+
+
+def record_memory_candidate_metric(*, kind: str, status: str) -> None:
+    try:
+        _DEFAULT_PROMETHEUS_RECORDER.record_memory_candidate(
+            kind=kind,
+            status=status,
+        )
+    except Exception:
+        return
+
+
+def record_memory_review_metric(*, status: str) -> None:
+    try:
+        _DEFAULT_PROMETHEUS_RECORDER.record_memory_review(status=status)
+    except Exception:
+        return
+
+
 def reset_prometheus_metrics() -> None:
     _DEFAULT_PROMETHEUS_RECORDER.reset()