Skip to content

Research: remote MCP catalog, tool-contract drift and optional MCP Trust Gate (kill-first) #157

Open

Description

@PhysShell

2026-11-15 PRIORITY UPDATE: PARKED / NOT AN ACTIVE 007 ROADMAP ITEM. The owner no longer considers 007 an active product direction. Do not implement the MCP Trust Gate or start this research on behalf of 007. The actionable, narrower AI-Assistent simulation investigation now lives in AI-Assistent-skeleton #30 and its research page. Keep this issue only as historical prior art, subject to a new explicit GO if 007 is ever resumed.

Classification and scope

P2 RESEARCH / KILL-FIRST / NO IMPLEMENTATION AUTHORIZED. Evaluate remote MCP server discovery, tool-contract drift, and capability enforcement for agents. Do not connect the catalog to a privileged agent, enable arbitrary third-party endpoints, or introduce a new mandatory proxy/gate until the gap is demonstrated.

Source (checked against repository README and CONTRIBUTING on 2026-11-15): punkpeye/awesome-remote-mcp-servers. This is a discovery catalog, not an independently security-audited allowlist.

  • Remote MCP: provider-operated HTTPS endpoint, typically Streamable HTTP (or legacy SSE) rather than a locally spawned stdio process. Authentication varies.
  • Catalog contribution CI requires a public endpoint to answer MCP initialize and a Glama connector badge. This verifies neither safe tool behavior nor trustworthy outputs.
  • CONTRIBUTING allows accelerated handling of automated-agent PRs marked 🤖🤖🤖; membership is not a security endorsement.
  • 🔓 means no authentication, not necessarily zero cost or read-only.

Relationship to existing work (avoid duplication)

  • 007 #74 owns generic change admission, capability policy, immutable evidence and fail-closed reduction.
  • 007 #156 owns untrusted repository intake and repo-controlled execution surfaces. This issue is about remote MCP endpoint/tool capabilities and tool-schema drift; it does not duplicate repository/package scanning.
  • 007 #53 covers containment/process boundaries. Any enforcement must reuse a tested existing boundary where possible.
  • Assurance-as-Code concept is a consumer of verifiable policy/evidence, not another competing execution runtime.

Candidate landscape: explore, do not integrate by default

These are provider/catalog claims, not verified operational endorsements; exact endpoints, authorization, cost, limits, privacy and quality must be independently tested.

Candidate Listed remote endpoint Candidate use / owner Check first
Firedrill https://mcp.firedrill.run/mcp Agent scenario/synthetic-tool simulations; 007 / Assurance-as-Code Exportable reproducible scenarios, independent oracle, isolation
CodeRifts https://app.coderifts.com/mcp Approved API-contract changes and decision receipts; 007 / requirements-to-code Actual authorization semantics; can approval be bypassed?
MCP Checkup https://mcpcheckup.com/mcp External MCP endpoint/tool-contract checks Probe scope, tool-schema diff, attestation verifiability and threat model
Papers by Ouroboros https://papers-mcp.vercel.app/mcp Research papers for Own.NET / CGC / Assurance-as-Code Citation coverage, freshness, relevance, reproducibility vs existing search
DDMarketer https://www.ddmarketer.com/api/mcp Demand Radar complaint-driven lead discovery Source provenance, sampling bias, unverifiable 0–100 scores, zero-LLM baseline
Specpack https://prompt-generator-website.com/mcp Requirements Workbench: spec / AGENTS.md / CLAUDE.md generator as prior art Determinism, semantic validation, compatibility vs StrictDoc/FRET and typed baseline

Hypothesis: optional MCP Trust Gate

Only if built-in client/network restrictions and existing tools cannot meet the need, explore a provider-neutral pre-call admission layer:

agent intent + task permissions
  -> endpoint identity, transport, auth and scopes
  -> discovered tool inventory + canonical schema snapshot/diff
  -> per-tool/action policy (read/write/destructive, data classification, egress)
  -> allow / ask-human / deny / unknown
  -> authorized call and evidence receipt (when allowed)

Minimum contract to evaluate:

  1. Explicit endpoint allowlist and endpoint identity, including redirect/DNS/TLS trust boundaries; do not confuse URL text with authenticated identity.
  2. Snapshot and diff tools/list names, input schemas, descriptions, annotations, transport and security-relevant metadata. Hash only a documented canonical representation; treat descriptions/annotations as untrusted, not authority.
  3. Per-task tool allowlist, least-privilege OAuth/API-key handling, scopes and outbound data labels; block exfiltration of source, internal DB records, credentials and unrelated project data by default.
  4. Re-evaluate relevant policy at invocation time (TOCTOU: inventory may change after inspection). Mutation/destructive tools require explicit authorization.
  5. Auditable but redacted receipts: endpoint identity/version (if attested), tool-schema fingerprint, policy ID/revision, decision, caller, sanitized inputs/outputs or hashes, result/coverage.
  6. UNKNOWN/DEFER/BLOCK for changed/unaudited schema, unverifiable identity, denied egress, or unavailable enforcement. Never treat a successful MCP initialize as SAFE.
  7. Separate discovery/monitoring from sandbox/runtime enforcement. Avoid falsely claiming policy enforces network restrictions if agent can bypass the gate through direct HTTP.

Bounded evaluation (no production data)

  • Read actual 007 run/tool/sandbox interfaces at a pinned SHA; list already-enforced capabilities versus design-only claims.
  • Compare native MCP client allowlists/permissions, existing proxies, API gateway policies and MCP Checkup before designing any custom component. Record overlap and maintenance costs.
  • Freeze 6–8 synthetic fixtures: read-only stable tool, added destructive tool, changed input schema, misleading tool description/prompt injection, endpoint redirects or identity change, revoked/expired token, untrusted data egress, incomplete discovery.
  • Predetermine oracle and expected ALLOW / HUMAN_APPROVAL / BLOCK / UNKNOWN for every fixture; include negative controls.
  • Compare baseline against optional gate on true prevented policy violations, false blocks, TOCTOU cases, p95 overhead, operator effort, privacy risk and coverage honesty.
  • Separately evaluate each of six catalog candidates by real value over existing tools; do not infer adoption from listing.
  • Produce a small portable McpToolPolicyDecision / evidence sample and a decision REUSE / ADAPT / PARK / KILL. No production integration without an owner-approved follow-up.

Stop conditions: native permissions already cover the cases; no reproducible incremental violation detection; unacceptable bypass surface; requires sending internal data to third parties; or upkeep dwarfs demonstrated benefit.

Non-goals: a new MCP standard, general MCP marketplace, mass endpoint scanning, trusting agent-made assertions as evidence, bypassing OAuth, changing existing 007 #74/156 acceptance semantics, or writing implementation code in this research issue.

Research page / follow-up clarification (2026-11-15)

  • Detailed overview and all six provider candidates: Remote MCP Servers — catalog, agent simulations, MCP Trust Gate.
  • Firedrill endpoint verified: both the catalog and the provider's current agent quickstart list https://mcp.firedrill.run/mcp. Recheck vendor documentation when connecting; earlier discrepancy note was corrected.
  • Firedrill is not a code sandbox. Per the vendor overview, the tested agent/model calls run in the user's runner; the platform supplies synthetic users, stateful tools, scenarios and score reports. A local process retains the user's permissions. Compare with Ancha and straightforward local mock tools as baselines.
  • Candidate agent-simulation fixture: tool returns green CI despite a required security job missing; deterministic oracle must reject merge despite apparent success. Separate injection, changed-tool-schema, revoked-token and user-approval scenarios. Check tool calls, ordering and state effects, not only final prose.

Activity

  1. PhysShell commented on Nov 21, 2026

    @PhysShell
    OwnerAuthor

    MCP Events / Reactive Agents is a separate research seam (21.11.2026)

    Clare Liguori talks + technical notes · ext-events WG repo.

    The modelcontextprotocol/ext-events repository is explicitly an experimental incubation space, with the Events extension marked Beta; it is not evidence of a stable MCP Events core feature or guarantee of secure callbacks. Event-triggered wakeups (subscribe → wait → wake → resume) are a separate concern from this issue's remote-MCP tool discovery, schema drift and trust gate.

    Do not add event support to the MCP Trust Gate. If this seam is ever revisited, desk-audit a pinned beta spec for event identity, ordering, duplicate/replay, authorization and resumption; compare with #42/#43 and keep delivery failures as UNKNOWN or fail-closed. A repeated event cannot repeat an external mutation without reconciliation/idempotency.

    Respect the 15 Nov 2026 decision: 007 is PARKED; no active work, PoC or new MCP integration is authorized. This comment is a source pointer only, not a scope expansion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions