Track per-step knowledge retrieval, rule application, and decision evidence #442

Problem

We need to inspect which specific knowledge-base entries were encountered at each step and how they relate to verified decisions. Today, routing and execution traces provide pieces of this information, but no unified report connects:

step → retrieved/read knowledge → applicable rule → source/test evidence → decision or finding.

Without these links, we cannot reliably tell whether routing selected the right knowledge, whether a rule was actually checked, which rules contributed to a finding, or where knowledge coverage was missing.

Existing building blocks

Based on inspection of the local implementation:

• src/infermatrix_copilot/direct_routing.py: Direct returns knowledge_routes with owner, path, reason, quick_map, and quick_map_status, plus related_knowledge, scope validation, and diagnostics.knowledge_snapshot. This is a routing response; durable capture depends on the consuming host.
• src/infermatrix_copilot/engine/agent_runtime/runner.py: Strict writes review_knowledge_context with the step and related page paths, and agent_dispatch with injected skill names. These events do not provide a uniform rule-to-decision mapping.
• src/infermatrix_copilot/run_trace.py: append-only per-run JSONL events already provide an execution timeline.
• src/infermatrix_copilot/trace_store.py: trace/1 supports run_id, step, rule_ids, model/tool payload references, decisions, and outcomes. Schema support alone does not guarantee complete instrumentation across Direct and Strict.

Requested behavior

Provide a durable, queryable knowledge-use record for Direct and Strict, with a human-readable per-run summary. Reuse existing tracing infrastructure where practical.

Distinguish these states explicitly:

1. Retrieved: routing/search selected or returned an entry.
2. Read/provided: the specific content or excerpt was delivered to the agent; record partial/truncated content honestly. This is delivery evidence, not proof of model attention.
3. Applied: the agent explicitly linked a rule to a check or decision, with a short rationale and observable evidence.

Record at least:

• Run/request ID, step ID, event ID and ordering, repository, and execution mode.
• Knowledge snapshot/version or content hash; page path, section/rule ID where available, and excerpt range or content reference.
• Selection mechanism and reason: intent signal, changed-file scope, search query, mandatory guide, or briefing injection.
• Disposition such as applicable, not applicable, or unresolved, with a concise reason.
• Links to source evidence at the frozen head SHA, validation commands/results, and related findings or decisions.

Do not infer “applied” from retrieval or delivery alone. Preserve unresolved or unavailable evidence as an explicit gap. Record concise decision justifications and evidence references; private model chain-of-thought is not required.

Example summary

Illustrative format, not an assertion about an existing run:

Acceptance criteria

• Direct and Strict expose the same core knowledge-use semantics, documenting host responsibilities and unavailable instrumentation.
• A run/request can be exported as ordered machine-readable events and a readable step-by-step summary.
• The report separates retrieved entries, delivered excerpts, and explicitly applied rules; omitted rule IDs remain unknown rather than guessed.
• Applied-rule records link to checks/decisions and source/test evidence, or explicitly identify missing verification.
• Records identify the knowledge snapshot/content version and reviewed source head so later KB changes do not silently change their meaning.
• Routing fallbacks, no-hit cases, truncation, failed reads, and interrupted/partial runs remain visible.
• Tests cover retrieval without application, partial delivery, an applied rule with evidence, and snapshot/head attribution.
• Existing redaction and repository isolation are preserved; traces and reports stay outside the product knowledge tree.