Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions evaluation/standards-effectiveness/generated/suite-inputs.json
Original file line number Diff line number Diff line change
Expand Up @@ -7347,7 +7347,7 @@
]
},
{
"digest": "sha256:8d8859b36d951bbf7a747da81d597df19b07b6653f1e36acc48b4b131b90c0a8",
"digest": "sha256:c85b4e2d525e7d644261658b19e35b815e86f6386754d327f56de5bc8be52caa",
"path": "tools/standards_engine/README.md",
"state": "present",
"uses": [
Expand Down Expand Up @@ -7405,7 +7405,7 @@
]
},
{
"digest": "sha256:489fea805dbfd6f84bcb28374b467da192d56243100c0a58d568d9acf6180e08",
"digest": "sha256:146231132e8fbff19b5d917c1268bb75a0195cff4478d0e44df6bf04b8ba6aa2",
"path": "tools/standards_engine/contracts/examples/a1-examples.json",
"state": "present",
"uses": [
Expand Down Expand Up @@ -7875,7 +7875,7 @@
]
},
{
"digest": "sha256:7f51e403f03f5c57be8b1ca5736a79bcf139c87dd9eb4703ce3f7e59f1ba3d01",
"digest": "sha256:30eec6a6c472938048821b5669d17e054aa7cdee637647961b4012a380245e62",
"path": "tools/standards_engine/tests/test_analysis.py",
"state": "present",
"uses": [
Expand Down Expand Up @@ -8344,7 +8344,7 @@
]
},
{
"digest": "sha256:828be47647e4332067893b79a508e85ed65ff1d524c41daf8f09fa41f6b7d2a2",
"digest": "sha256:e6f3adf5543fad7cbd24b31b68b21d08029e542e11d80d2899d30b2ced682452",
"path": "tools/standards_snapshots/README.md",
"state": "present",
"uses": [
Expand Down Expand Up @@ -8441,7 +8441,7 @@
]
},
{
"digest": "sha256:bda178630b8226af0de5c42a7da6290ace2ed1745e3d2a99057deb0bae6ac617",
"digest": "sha256:6853039f09ef90b9d4ba9f54e4df7050b13b0a1941764af95250b44f7237ecae",
"path": "tools/standards_snapshots/standards_snapshots/store.py",
"state": "present",
"uses": [
Expand Down Expand Up @@ -14116,7 +14116,7 @@
"path": "evaluation/standards-effectiveness/suite-registry.toml"
},
"repository_index": {
"digest": "sha256:25f5d13e9650e10c5bcc198f50ea81aa00f90799ca3f9ba2a5a433a658262229",
"digest": "sha256:96dd89639c5b1a45d6c6692e969cea93338efdd3e5604fb2c2d30e6cfe254fd7",
"uses": [
{
"check": "python-package-contract",
Expand Down
16 changes: 11 additions & 5 deletions tools/standards_engine/PURPOSE-SEPARATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,14 @@ two-entry and 32 MiB accounted-retention limit shared by snapshot and draft
projection entries. Snapshot keys contain the exact verified
captured paths, bytes and source revision, plus the installed compiler function.
Python dictionary equality distinguishes hash collisions. Each new operation
opens and validates its store, checks snapshot lifecycle and verifies complete
content identity before lookup. Missing, quarantined, purged, replaced or corrupt
storage therefore remains observable. Equal verified inputs may share pure
opens its store, checks its application, version and exact schema, checks snapshot
lifecycle and verifies complete content identity before lookup. Missing,
quarantined, purged or replaced storage, and content that no longer matches its
identity, therefore remain observable. The store's complete SQLite
integrity/foreign-key audit is not repeated by every operation: a process runs it
on its first open of a store file, as described under
[Snapshot admission](../standards_snapshots/README.md#admission-and-maintenance-work).
Equal verified inputs may share pure
compilation even when obtained from distinct store roots; store-specific handles,
access checks and result envelopes remain fresh. No store facts enter the cache.

Expand Down Expand Up @@ -185,8 +190,9 @@ Review, candidate verification, application and recovery retain their existing
fresh replay paths. New snapshot capture retains both independent compiler passes.
Main advancement affects new captures, never the base of an existing draft. Old
explicit revisions retain historical meaning and can have live stale status.
Corruption, quarantine, purge or missing/replaced stores reject before projection
reuse. Server restart reconstructs cold, and close releases both entry kinds.
Stored content that fails validation, quarantine, purge or missing/replaced stores
reject before projection reuse. Server restart reconstructs cold and repeats the
store's complete SQLite audit; close releases both entry kinds.

### Incremental successor construction

Expand Down
4 changes: 4 additions & 0 deletions tools/standards_engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,10 @@ The process-owned compilation cache also retains exact captured-content identity
proofs. Identity and compilation for one capture share one LRU entry and the existing
byte budget. The Snapshot codec still defines identity, and every read reloads actual
bytes, verifies file integrity and checks the stored identity and current lifecycle.
These per-read checks are separate from the store's complete SQLite
integrity/foreign-key audit, which a process runs on its first open of a store file
and not on each store-backed call; see
[Snapshot admission](../standards_snapshots/README.md#admission-and-maintenance-work).

Focused authoring operations reuse canonical revision decoding through
`ProposalMaterials`; the single record is released on success or exceptional exit.
Expand Down
4 changes: 2 additions & 2 deletions tools/standards_engine/contracts/examples/a1-examples.json
Original file line number Diff line number Diff line change
Expand Up @@ -1274,7 +1274,7 @@
"value": {
"kind": "input-contract-result",
"purpose": "application",
"interface_version": 47,
"interface_version": 48,
"catalog_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"operation": "runtime_info",
"root": "RuntimeInfoCall",
Expand Down Expand Up @@ -1420,7 +1420,7 @@
"value": {
"kind": "output-contract-result",
"purpose": "authoring",
"interface_version": 47,
"interface_version": 48,
"catalog_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"operation": "review",
"dialect": "https://json-schema.org/draft/2020-12/schema",
Expand Down
42 changes: 40 additions & 2 deletions tools/standards_engine/tests/test_analysis.py
Original file line number Diff line number Diff line change
Expand Up @@ -687,8 +687,39 @@ def apply_ready(readiness_id: str) -> ProposalApplication:
}
)
)
# Apply advances this unchanged main checkout to its publication.
self.assertEqual(applied.checkout.status, "current")
return engine._authoring.read_application(applied.application.id)

def assert_checkout_presents(
application: ProposalApplication,
present: tuple[str, ...],
absent: tuple[str, ...] = (),
) -> None:
# Interface 47: HEAD, index and worktree all present the exact
# published candidate, and no other local difference appears.
self.assertEqual(
git_output(repository, ("rev-parse", "HEAD")).strip(),
application.candidate.oid.encode("ascii"),
)
for path in present:
self.assertEqual(
(repository / path).read_bytes(),
engine._repository.read_file(
application.candidate, RepositoryPath.parse(path)
),
path,
)
for path in absent:
self.assertFalse((repository / path).exists(), path)
self.assertEqual(
git_output(
repository,
("status", "--porcelain=v1", "--untracked-files=all"),
),
b"",
)

def assert_commit(
application: ProposalApplication,
revision: ProposalRevision,
Expand Down Expand Up @@ -798,7 +829,10 @@ def assert_commit(
impact_path: "100644",
},
)
self.assertFalse((repository / standard_path).exists())
assert_checkout_presents(
create_application,
(standard_path, policy_path, impact_path),
)

facade.close()
engine = StandardsEngine.open_repository(
Expand Down Expand Up @@ -933,7 +967,11 @@ def assert_commit(
)
self.assertIn(b"[[tombstone]]", retired_policy)
self.assertNotIn(b"[[policy_unit]]", retired_policy)
self.assertFalse((repository / standard_path).exists())
assert_checkout_presents(
retire_application,
(policy_path,),
absent=(standard_path, impact_path),
)

facade.close()
engine = StandardsEngine.open_repository(
Expand Down
13 changes: 12 additions & 1 deletion tools/standards_engine/tests/test_consumer_publication.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,23 @@
import subprocess
import sys
import tempfile
import tomllib
import unittest

from tools.standards_engine.standards_engine.tools import INTERFACE_CONTRACT
from tools.standards_engine.tests.test_agent_workflow import ROOT, decisions, evidence, prepare_repository
from tools.standards_engine.tests import test_proposal_consumers as fixtures
from tools.standards_engine.tests.test_proposal_consumers import registration


def installed_interface_version(root):
# The server announces the interface declared by the repository it serves.
# Read that declaration instead of repeating a number that each interface
# edition would otherwise leave stale here.
with (root / INTERFACE_CONTRACT).open('rb') as source:
return tomllib.load(source)['interface_schema_version']


class ConsumerPublicationTest(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='consumer-publication-')
Expand All @@ -43,7 +53,8 @@ def rpc(self, method, params, purpose='authoring'):
self.assertEqual(process.returncode,0,process.stderr)
self.assertEqual(process.stderr,'')
responses={row['id']:row for row in map(json.loads,process.stdout.splitlines())}
self.assertIn('interface 47;',responses[1]['result']['instructions'])
self.assertIn(f'interface {installed_interface_version(self.root)};',
responses[1]['result']['instructions'])
return responses[2]

def call(self,name,arguments,purpose='authoring',error=False):
Expand Down
116 changes: 91 additions & 25 deletions tools/standards_engine/tests/test_storage_lifecycle_transport.py
Original file line number Diff line number Diff line change
@@ -1,25 +1,36 @@
"""Cold MCP reads coexist with a reserved writer only when no purge is due.

One server process audits its store once, on its first call; a newly started
process audits again and still rejects a corrupt store.

The fixture owns its database, evidence and processes. Publication, authorization
and interpreter timing are not simulated; no production store is accessed.
"""
from __future__ import annotations

import json
import os
import shutil
import sqlite3
import subprocess
import sys
import time
import unittest
from contextlib import closing
from pathlib import Path
from unittest import mock

from tools.standards_engine.standards_engine.mcp import MCPServer
from tools.standards_engine.tests import test_agent_interaction_transport as transport
from tools.standards_engine.tests.test_mcp import initialize, request
from tools.standards_engine.tests.test_route_content import known_assertions
from tools.standards_snapshots.standards_snapshots import (
AggregateChild, AggregateRecord, AggregateRoot, ChildHandle,
SnapshotError, SnapshotId, SnapshotModule,
)
from tools.standards_snapshots.standards_snapshots import store as snapshot_store

AUDIT_STATEMENTS = ('PRAGMA integrity_check', 'PRAGMA foreign_key_check')


class StorageLifecycleTransportTest(unittest.TestCase):
Expand All @@ -39,6 +50,85 @@ def call(self, purpose, operation, arguments, *, rejected=False):
self, purpose, operation, arguments, rejected=rejected, delivery='on-demand',
)

def cold_open_failure(self):
"""Run one cold authoring read whose store open fails; return its private stderr."""
# Open-time failure retains the existing MCP private-stderr
# boundary (no structured result), not an invented tool outcome.
messages = [
{'jsonrpc': '2.0', 'id': 1, 'method': 'initialize', 'params': {
'protocolVersion': '2025-11-25', 'capabilities': {},
'clientInfo': {'name': 'storage-open-fixture', 'version': '1'}}},
{'jsonrpc': '2.0', 'method': 'notifications/initialized'},
{'jsonrpc': '2.0', 'id': 2, 'method': 'tools/call', 'params': {
'name': 'read', 'arguments': {'snapshot': self.snapshot, 'target': 'core'}}},
]
process = subprocess.run(
[sys.executable, '-P', '-m', 'tools.standards_engine.standards_engine.mcp',
'--repo-root', str(self.root), '--purpose', 'authoring',
'--output-schemas', 'on-demand'],
env={**os.environ, 'PYTHONPATH': str(transport.fixture.ROOT)},
input=''.join(json.dumps(message) + '\n' for message in messages),
text=True, capture_output=True, timeout=90,
)
self.assertEqual(process.returncode, 0, process.stderr)
failed = next(json.loads(line)['result'] for line in process.stdout.splitlines()
if json.loads(line).get('id') == 2)
self.assertTrue(failed['isError'])
self.assertNotIn('structuredContent', failed)
self.assertIn('Engine invocation failed', failed['content'][0]['text'])
return process.stderr

def test_one_server_process_audits_its_store_once_across_calls(self):
# The fixture built this store inside the test process. Discard that
# record so the server below starts as a newly launched process does.
with snapshot_store._AUDITED_FILES_LOCK:
snapshot_store._AUDITED_FILES.clear()
opens = []
connect = sqlite3.connect

def traced_connect(database, *arguments, **options):
connection = connect(database, *arguments, **options)
if Path(database).resolve() == self.path.resolve():
opens.append([])
connection.set_trace_callback(opens[-1].append)
return connection

server = MCPServer(self.root, purpose='application')
self.addCleanup(server.close)
initialize(server)
arguments = {'snapshot': self.snapshot, 'target': 'core'}
with mock.patch.object(snapshot_store.sqlite3, 'connect', traced_connect):
results = [
server.dispatch(request('tools/call', {'name': 'read', 'arguments': arguments},
identifier))['result']
for identifier in (2, 3, 4)
]
for result in results:
self.assertFalse(result['isError'], result)
self.assertEqual(result['structuredContent']['content'],
transport.fixture.PurposeTransportTest.expected)
# Every call still opens and closes its own store connection.
self.assertEqual(len(opens), 3)
self.assertEqual([[statement for statement in statements if statement in AUDIT_STATEMENTS]
for statements in opens], [list(AUDIT_STATEMENTS), [], []])
for statements in opens:
self.assertIn('PRAGMA application_id', statements)
self.assertIn('PRAGMA user_version', statements)
self.assertTrue(any('FROM sqlite_schema' in statement for statement in statements))

def test_cold_server_process_rejects_a_corrupt_store_on_its_first_call(self):
retained = self.path.with_name('retained-valid-store.sqlite3')
shutil.copyfile(self.path, retained)
try:
with closing(sqlite3.connect(self.path, isolation_level=None)) as corruptor:
corruptor.execute(
"INSERT INTO content_files VALUES ('missing', 'a', X'', 0, 'digest')")
self.assertIn('SNAPSHOT_STORE.FOREIGN_KEY_FAILURE', self.cold_open_failure())
finally:
os.replace(retained, self.path)
recovered = self.call('authoring', 'read', {'snapshot': self.snapshot, 'target': 'core'})
self.assertEqual(recovered['content'], transport.fixture.PurposeTransportTest.expected)

def test_cold_agent_observations_match_while_another_process_holds_writer(self):
cases = [
('read', {'snapshot': self.snapshot, 'target': 'core'}),
Expand Down Expand Up @@ -82,31 +172,7 @@ def test_cold_due_maintenance_keeps_busy_then_expires_all_dependents_atomically(
with closing(sqlite3.connect(self.path, isolation_level=None)) as writer:
writer.execute('BEGIN IMMEDIATE')
try:
# Open-time failure retains the existing MCP private-stderr
# boundary (no structured result), not an invented tool outcome.
messages = [
{'jsonrpc': '2.0', 'id': 1, 'method': 'initialize', 'params': {
'protocolVersion': '2025-11-25', 'capabilities': {},
'clientInfo': {'name': 'storage-busy-fixture', 'version': '1'}}},
{'jsonrpc': '2.0', 'method': 'notifications/initialized'},
{'jsonrpc': '2.0', 'id': 2, 'method': 'tools/call', 'params': {
'name': 'read', 'arguments': {'snapshot': self.snapshot, 'target': 'core'}}},
]
process = subprocess.run(
[sys.executable, '-P', '-m', 'tools.standards_engine.standards_engine.mcp',
'--repo-root', str(self.root), '--purpose', 'authoring',
'--output-schemas', 'on-demand'],
env={**os.environ, 'PYTHONPATH': str(transport.fixture.ROOT)},
input=''.join(json.dumps(message) + '\n' for message in messages),
text=True, capture_output=True, timeout=90,
)
self.assertEqual(process.returncode, 0, process.stderr)
self.assertIn('SNAPSHOT_STORE.BUSY', process.stderr)
failed = next(json.loads(line)['result'] for line in process.stdout.splitlines()
if json.loads(line).get('id') == 2)
self.assertTrue(failed['isError'])
self.assertNotIn('structuredContent', failed)
self.assertIn('Engine invocation failed', failed['content'][0]['text'])
self.assertIn('SNAPSHOT_STORE.BUSY', self.cold_open_failure())
self.assertIsNotNone(writer.execute(
'SELECT 1 FROM snapshot_roots WHERE snapshot_id = ?', (str(expired),)
).fetchone())
Expand Down
Loading
Loading