Repository navigation
Description
Goal
Implement persistent, agent-invisible MCP interaction observability for the Standards Engine so real agent/tool interactions can be measured, replayed or analyzed where evidence permits, and used to improve agent-facing contracts. Preserve the existing qualification observer v3 rather than replacing or mischaracterizing it.
This is a Coding-Standards implementation first. Tuldok will later be an adopting consumer, not a second telemetry design in this PR.
Existing code and reference points
- Production MCP dispatch/catalog:
tools/standards_engine/standards_engine/mcp.py,mcp_catalog.py,runtime_identity.py,contract_discovery.py. - Observer v3 is currently part of qualification, not continuous production telemetry:
tools/standards_engine/tests/codex_discovery_client.py,replay_discovery_qualification.py,INPUT-DISCOVERY-QUALIFICATION.md, anddocs/plans/input-contract-discovery/observer-reconciliation.md. - Preserve canonical interface contracts, on-demand schema discovery, stable catalog/run identities and replay semantics. Current source README identifies interface 46; verify source HEAD/runtime identity before changing generated contracts.
- Apply the repository's Core Standards and Standards Router to the task; record relevant decisions, test and review evidence.
Required capability
- Instrument the existing MCP interaction boundary without advertising observer tools to agents or changing business-domain operation outcomes. Capture one bounded, correlated record per observable request/response pair, covering initialize/catalog discovery and tool calls as applicable. Represent incomplete calls and unknown outcomes explicitly; do not manufacture completions.
- Persist locally in SQLite with declared schema version, migration/recovery/retention rules and consistent transaction ownership. Design for restart-safe readback and bounded resource usage; telemetry failure must not silently change an already committed domain result. Specify degraded/fail-open behavior and make telemetry health observable to the operator.
- Capture correlation/chronology, tool operation name, safe input shape or redacted fields, structured error/rejection class, validation/discovery attempts, duration and request/response byte counts. Capture interface version, catalog/schema/implementation digests and deployment identity sufficient to distinguish stale-client effects. Track cross-call continuations where the protocol supplies a legitimate identity.
- Privacy and authority: do not store credentials, tokens, raw standards bodies, complete repository content, full prompt/model transcripts, arbitrary tool output, or sensitive argument fields. Use an explicit allowlist/redaction/digest/size policy for useful input diagnostics; recording input structure is not permission to retain secrets. Separate operator read/export from the agent tool catalog. Do not assume a same-user local filesystem is a security boundary.
- Build operator-side inspection/reporting to answer: which operations are heavily used, which fields/schema variants repeatedly fail, how many repairs/discovery calls precede success, where time/serialized bytes accumulate, and what changed between catalog revisions. Use accurate denominators; bytes are not tokens/billing; correlation does not establish model intent.
- Preserve independent event evidence for replay/qualification. Reuse observer-v3 ideas and, where feasible, common pure assessment helpers, but do not turn the test harness into a production recorder or add a second MCP protocol/authorization workflow. Historical qualification artifacts/verdicts must not be rewritten.
- Keep deployment opt-in or explicitly operator-configured until privacy/retention expectations are decided. No automatic prompt, standard, or schema rewrites based solely on traces.
Milestones / acceptance
- A. Baseline and design: inspect current interface 46 source, observer v3, current persistence and diagnostics owners; write a bounded design/implementation plan with schema, event flow, sanitization, migration, retention, configuration and failure semantics. Identify existing tests to reuse.
- B. Narrow vertical slice: capture catalog identity plus validated tool call begin/end/typed failure to SQLite through a single authoritative MCP boundary, with agent-visible output and the normal tool catalog unchanged. No extra tools, hidden model calls or new retry behavior.
- C. Operator feedback: offer a bounded read-only CLI/report or similar operator-only inspection for repair loops, error distribution, latency and byte accounting.
- D. Qualification: instrument realistic cold-stdio sessions and process replacement; test exact source/catalog attribution, failed validation, tool exceptions, interrupted calls, storage unavailable/locked/full, privacy redaction, bounded concurrency, migration/retention and source replay. Compare a baseline to the candidate with recorded evidence, including call counts and bytes; keep the observer v3 tests green.
- E. Delivery: branch + PR with focused commits, tests, docs, operational configuration and limitations. Independent review before merge, following Coding-Standards workflow. No changes to Tuldok in this PR.
Out of scope
Changing normative standards, granting agents review/publication authority, automatic self-modifying tool contracts, cross-machine telemetry, generic surveillance of Codex, model token/billing measurements without an actual provider surface, and a new MCP proxy unless shown necessary.
Follow-on
After the Coding-Standards observer is qualified, apply the pattern to MrScripty/Tuldok, whose existing agent registry and MCP bridge already expose draft/graph/release/run workflows. Use observed agent failures to improve schema discovery, typed feedback and training/dataset workflow ergonomics, not to bypass human review or exact execution approvals.
Activity