Skip to content

Implement persistent, agent-invisible MCP interaction observability #3

Open

Description

@MrScripty

Goal

Implement persistent, agent-invisible MCP interaction observability for the Standards Engine so real agent/tool interactions can be measured, replayed or analyzed where evidence permits, and used to improve agent-facing contracts. Preserve the existing qualification observer v3 rather than replacing or mischaracterizing it.

This is a Coding-Standards implementation first. Tuldok will later be an adopting consumer, not a second telemetry design in this PR.

Existing code and reference points

  • Production MCP dispatch/catalog: tools/standards_engine/standards_engine/mcp.py, mcp_catalog.py, runtime_identity.py, contract_discovery.py.
  • Observer v3 is currently part of qualification, not continuous production telemetry: tools/standards_engine/tests/codex_discovery_client.py, replay_discovery_qualification.py, INPUT-DISCOVERY-QUALIFICATION.md, and docs/plans/input-contract-discovery/observer-reconciliation.md.
  • Preserve canonical interface contracts, on-demand schema discovery, stable catalog/run identities and replay semantics. Current source README identifies interface 46; verify source HEAD/runtime identity before changing generated contracts.
  • Apply the repository's Core Standards and Standards Router to the task; record relevant decisions, test and review evidence.

Required capability

  1. Instrument the existing MCP interaction boundary without advertising observer tools to agents or changing business-domain operation outcomes. Capture one bounded, correlated record per observable request/response pair, covering initialize/catalog discovery and tool calls as applicable. Represent incomplete calls and unknown outcomes explicitly; do not manufacture completions.
  2. Persist locally in SQLite with declared schema version, migration/recovery/retention rules and consistent transaction ownership. Design for restart-safe readback and bounded resource usage; telemetry failure must not silently change an already committed domain result. Specify degraded/fail-open behavior and make telemetry health observable to the operator.
  3. Capture correlation/chronology, tool operation name, safe input shape or redacted fields, structured error/rejection class, validation/discovery attempts, duration and request/response byte counts. Capture interface version, catalog/schema/implementation digests and deployment identity sufficient to distinguish stale-client effects. Track cross-call continuations where the protocol supplies a legitimate identity.
  4. Privacy and authority: do not store credentials, tokens, raw standards bodies, complete repository content, full prompt/model transcripts, arbitrary tool output, or sensitive argument fields. Use an explicit allowlist/redaction/digest/size policy for useful input diagnostics; recording input structure is not permission to retain secrets. Separate operator read/export from the agent tool catalog. Do not assume a same-user local filesystem is a security boundary.
  5. Build operator-side inspection/reporting to answer: which operations are heavily used, which fields/schema variants repeatedly fail, how many repairs/discovery calls precede success, where time/serialized bytes accumulate, and what changed between catalog revisions. Use accurate denominators; bytes are not tokens/billing; correlation does not establish model intent.
  6. Preserve independent event evidence for replay/qualification. Reuse observer-v3 ideas and, where feasible, common pure assessment helpers, but do not turn the test harness into a production recorder or add a second MCP protocol/authorization workflow. Historical qualification artifacts/verdicts must not be rewritten.
  7. Keep deployment opt-in or explicitly operator-configured until privacy/retention expectations are decided. No automatic prompt, standard, or schema rewrites based solely on traces.

Milestones / acceptance

  • A. Baseline and design: inspect current interface 46 source, observer v3, current persistence and diagnostics owners; write a bounded design/implementation plan with schema, event flow, sanitization, migration, retention, configuration and failure semantics. Identify existing tests to reuse.
  • B. Narrow vertical slice: capture catalog identity plus validated tool call begin/end/typed failure to SQLite through a single authoritative MCP boundary, with agent-visible output and the normal tool catalog unchanged. No extra tools, hidden model calls or new retry behavior.
  • C. Operator feedback: offer a bounded read-only CLI/report or similar operator-only inspection for repair loops, error distribution, latency and byte accounting.
  • D. Qualification: instrument realistic cold-stdio sessions and process replacement; test exact source/catalog attribution, failed validation, tool exceptions, interrupted calls, storage unavailable/locked/full, privacy redaction, bounded concurrency, migration/retention and source replay. Compare a baseline to the candidate with recorded evidence, including call counts and bytes; keep the observer v3 tests green.
  • E. Delivery: branch + PR with focused commits, tests, docs, operational configuration and limitations. Independent review before merge, following Coding-Standards workflow. No changes to Tuldok in this PR.

Out of scope

Changing normative standards, granting agents review/publication authority, automatic self-modifying tool contracts, cross-machine telemetry, generic surveillance of Codex, model token/billing measurements without an actual provider surface, and a new MCP proxy unless shown necessary.

Follow-on

After the Coding-Standards observer is qualified, apply the pattern to MrScripty/Tuldok, whose existing agent registry and MCP bridge already expose draft/graph/release/run workflows. Use observed agent failures to improve schema discovery, typed feedback and training/dataset workflow ergonomics, not to bypass human review or exact execution approvals.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions