Skip to content

Add the detector as a local web app and MCP server - #7

Merged
ichen27 merged 1 commit into
mainfrom
feat/detector-app-and-mcp
Nov 15, 2026
Merged

ichen27 merged 1 commit into
mainfrom
feat/detector-app-and-mcp

Conversation

@ichen27

@ichen27 ichen27 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Problem

The detectors only ran as batch scripts over the evaluation sets. There was no way to score a new text.

What this adds

  • detecteval.serve: a web page, a JSON API (POST /api/detect) and an MCP endpoint (/mcp, tool detect_ai_text) in one small process.
  • detecteval.worker: the model process. It starts on the first request and is stopped after five idle minutes, so the service costs about 50 MB while idle.
  • detecteval.detector and detecteval.calibration: passages of up to 500 words, and bands derived from how often human reference texts score as high. A result never states a verdict.
  • results/calibration.json: reference scores built from the committed long-set scores. A private human reference is read from outside the repository when present.
  • launcher.json: the manifest for this project's own deployment.

The classifier is the main signal and Binoculars an optional second opinion. Clef is left out because it flagged 42% of real pre-ChatGPT student writing.

Validation

  • bash ci/run.sh: ruff clean, 38 tests pass (10 new).
  • Ran the service locally and through the launcher deployment: health, page, a human text (band none), an AI text (band strong), a two-passage text with Binoculars, a too-short text, and an MCP tools/call.
  • A request to /mcp with a foreign Host header is refused (421).

Risks

  • The worker takes 10 to 30 seconds to start on the first request.
  • Binoculars needs about 3 GB and is skipped when free memory is under 35%.
  • The live deployment runs from the canonical checkout; AGENTS.md says how to handle that.

A small Starlette service scores text with the trained classifier, and
Binoculars on request, in a worker process that starts on demand and
stops when idle. Results give bands and the reference rates behind them,
never a verdict. The MCP endpoint exposes detect_ai_text on loopback.
Includes the launcher manifest used for this project's own deployment.
@ichen27
ichen27 merged commit 2db96f7 into main Nov 15, 2026
1 check passed
@ichen27
ichen27 deleted the feat/detector-app-and-mcp branch November 15, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant