Skip to content

feat(engine): record MCP observations by default and never drop them - #8

Merged
MrScripty merged 3 commits into
mainfrom
feat/observer-always-on
Nov 12, 2026
Merged

MrScripty merged 3 commits into
mainfrom
feat/observer-always-on

Conversation

@MrScripty

Copy link
Copy Markdown
Owner

Summary

The recorder exists to show how agents use the tool interface so the interface can be improved. As built it could not: it was off unless two flags were passed, so the application purpose was never recorded; it gave up for the life of the process after a 20 ms lock wait, so servers started together recorded nothing; and it stored argument shapes rather than values. This reworks it to five requirements.

  • Always on. Every server process records to ${XDG_DATA_HOME:-~/.local/share}/standards-engine/observations/interactions-v2.sqlite3 with no flags (directory 0700, file 0600). --no-observe or STANDARDS_ENGINE_OBSERVE=off turns it off.
  • Invisible. Tool results are unchanged, a healthy recorder writes nothing to stderr, and a recording failure never changes or fails a call.
  • No dropped records. A busy database queues the row in a bounded in-memory backlog and delivers it at the next call or at close. The database uses write-ahead mode.
  • Seven-day window. Unchanged. A full file now evicts its oldest records instead of refusing new ones.
  • Useful values. See below.

What is recorded

  • Navigation calls: targets, options and route facts as sent, including fact names and values outside the vocabulary, each string cut at 200 characters.
  • Authoring calls: identifiers and enum words only. Never bodies, titles, summaries, rationales, intents or evidence text.
  • Results: kind, status, code, and the selected or returned module IDs. Never content text.
  • A client thread id only as a keyed hash, with a main/subagent marker, and the client name and version. No other request metadata is read.

Fields worth a deliberate decision: free-text route facts, consumer IDs that are repository paths, and client name and version.

Operational notes

  • Recording starts when server processes restart. Registrations need no change.
  • While a server runs there are three files (database, -wal, -shm), and the database must be on a local disk. After a power loss the last few records can be missing.
  • Schema v2 uses a new file. A v1 file is refused and left untouched.
  • Every test and harness launch passes --no-observe; a guard test fails if a new launch site forgets. In-process MCPServer construction records only when given a configuration.
  • Not recordable: a process killed while rows are queued loses them, and a database unreachable for a process's whole life is reported only on stderr.

Verification

  • The full CI sequence passed locally on this exact tree: 652 engine tests, the twelve other suites, structural verifier 73/73, with the default observation location redirected and left empty.
  • 8 and then 16 processes started at the same instant on a new database kept every record (176 of 176, 352 of 352, none lost). The previous code kept 42 of 176 in the 8-process run.
  • A real server process with no flags created the database at the default path with the expected modes; a raw thread id and a workspace path sent in request metadata were not stored.
  • A privacy test sends sentinel strings in authoring bodies, titles, rationales and metadata and asserts none appears in any database file.

Stack

Third of five. This branch contains the commits from #6 and #7; review the last commit.

🤖 Generated with Claude Code

MrScripty and others added 3 commits November 12, 2026 07:20
…iliation

Six engine tests have failed on main since 22 October, when two changes
landed without the tests that depend on them being updated:

- Interface 48 (ad56443). Four publication tests asserted the literal
  "interface 47;" in the server instructions, and two contract examples
  still carried interface_version 47. The assertion now reads the number
  from the interface the server under test serves, so it cannot go stale
  again, and the examples say 48.
- Publication checkout reconciliation (8c5a4ad). The create-and-retire
  apply test still asserted that apply leaves the working tree untouched.
  It now asserts the documented behaviour: HEAD, index and worktree present
  the published candidate and no other local difference exists.

No production code changes. The suite-input manifest is regenerated; it was
already stale on main because 5bce311 added files without refreshing it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The MCP server opens the snapshot store for each tool call, and every open
of an existing store ran PRAGMA integrity_check and foreign_key_check over
the whole file. On a 282 MB store that is about 0.6 s added to every
route, read and related call, growing with the file.

The full audit now runs at most once per process for each store file,
identified by device and inode. It still runs on the first open in every
process, throughout a version 1 migration, and for a file put in place
under a different inode. Application id, schema version and exact schema
checks still run on every open. Measured on a copy of that store, five
consecutive opens in one process go from about 0.6 s each to 0.6 s for the
first and under 1 ms for the rest.

Known limits, documented in the snapshots README: a one-shot CLI call is a
new process and still audits; a store deleted and recreated in place can
reuse its inode on ext4 and is then not audited again by a server that is
already running.

Tests cover one audit for two opens, a replaced file, a corrupt
replacement, the checks that stay on every open, a fresh process rejecting
a corrupted file, migration keeping all of its audits, and the server
auditing once across several tool calls.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The recorder exists to show how agents use the tool interface so the
interface can be improved. As built it could not do that: it was off
unless two flags were passed, so the application purpose was never
recorded; it gave up for the life of the process after a 20 ms lock wait,
so servers started together recorded nothing; and it stored argument
shapes rather than values, so it could not say which standard was read,
which facts were sent or which rejection came back.

Always on. Every server process records to
${XDG_DATA_HOME:-~/.local/share}/standards-engine/observations/
interactions-v2.sqlite3 with no flags. --no-observe or
STANDARDS_ENGINE_OBSERVE=off turns it off. In-process MCPServer
construction records only when given a configuration, and every test and
harness launch passes --no-observe; a guard test fails if a new launch
site forgets.

Invisible. Tool results are unchanged, a healthy recorder writes nothing
to stderr, and a recording failure never changes or fails a call.

No dropped records. A busy database queues the row in a bounded in-memory
backlog and delivers it at the next call or at close. The database uses
write-ahead mode. Eight processes started at the same instant on a new
database now keep all 176 starts and terminals; the previous code kept 42.
Loss that does occur (backlog overflow) is counted in the database.

Seven-day window. Unchanged. A full file now evicts its oldest records
instead of refusing new ones. Schema v2 uses a new file; a v1 file is
refused and left untouched.

Useful values. Navigation calls keep targets, options and route facts as
sent, bounded to 200 characters per string. Authoring calls keep only
identifiers and enum words, never bodies, titles, rationales or evidence.
Results keep kind, status, code and the selected or returned module IDs.
A client thread id is stored only as a keyed hash, with a main or
subagent marker. No other request metadata is read.

The report gains a readable text summary (rejection rates, latency and
size percentiles, refused fact names and values, most-read targets,
repeat reads per thread) and a --since-days filter.

MCP-OBSERVABILITY.md is rewritten for the new contract.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 47985069-d426-4351-a76c-41d73754f7df
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@MrScripty
MrScripty merged commit 764c674 into main Nov 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant