Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 26 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,29 @@
# ai-text-detection-eval
# Proofmark

Repository: https://github.com/ichen27/ai-text-detection-eval (private).
Canonical checkout: `/Volumes/SamsungSSD1/code/ai-text-detection-eval` on the Mac mini SSD.
Repository: https://github.com/ichen27/proofmark (public; it was named ai-text-detection-eval until
2026-11-16, and the old address redirects).
Canonical checkout: `/Volumes/SamsungSSD1/code/ai-text-detection-eval` on the Mac mini SSD. The directory
and the Python package (`detecteval`) keep their original names.

## Shipping decisions

Recorded 2026-11-16.

- **Launcher:** yes. Deployed to the app launcher as `proofmark`.
- **Repository visibility:** public, MIT licence for the code.
- **Name:** brand name, Proofmark, on GitHub and the launcher.
- **Announcement:** X only. Post link: not posted yet.

The repository is public, so nothing private may be committed: no texts, titles, ids or per-document
scores from private writing, and no results for documents whose authorship is uncertain (use
`private_eval report --omit-group`).

## What this is

A small, reproducible experiment on detecting AI-written text. It compiles labelled text sets and
compares three kinds of detector on them: decision models asked a typed question with no training
(Cloudflare Clef), statistical detectors that need no training (Binoculars, Fast-DetectGPT), and a
classifier fine-tuned on examples. It is an evaluation harness, not a detector product.
A local AI-text detector (web app and MCP server) and the reproducible evaluation behind it. The
evaluation compiles labelled text sets and compares three kinds of detector on them: decision models
asked a typed question with no training (Cloudflare Clef), statistical detectors that need no training
(Binoculars, Fast-DetectGPT), and a classifier fine-tuned on examples. The app uses the classifier.

## Layout

Expand Down Expand Up @@ -55,13 +70,13 @@ classifier fine-tuned on examples. It is an evaluation harness, not a detector p

## The deployed detector

The service is deployed on the Mac mini through the app launcher as `ai-detector`
(https://app-ai-detector.chenagent.com, owner sign-in) from [launcher.json](launcher.json), and is
registered in the MCP hub as the lazy HTTP backend `ai-detector` (tools prefixed `ai_detector__`).
The service is deployed on the Mac mini through the app launcher as `proofmark`
(https://app-proofmark.chenagent.com, owner sign-in) from [launcher.json](launcher.json), and is
registered in the MCP hub as the eager HTTP backend `proofmark` (tools prefixed `proofmark__`).

- It runs from this canonical checkout with its own environment at
`/Volumes/SamsungSSD1/tools/ai-text-detection/venv`, so `ci/run.sh` and `uv sync` here do not disturb
it. Code changes take effect on restart: `node src/cli.mjs restart ai-detector` in the launcher checkout.
it. Code changes take effect on restart: `node src/cli.mjs restart proofmark` in the launcher checkout.
Keep the checkout on `main` when not actively working, or develop service changes in a worktree.
- After changing dependencies, redeploy with the launcher's `deploy` command so the build step runs.
- The service must stay light: models load in the worker on demand and the worker is stopped when idle.
Expand Down
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Ivan Chen

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
33 changes: 22 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,25 @@
# ai-text-detection-eval
# Proofmark

How well can AI-written text be told from human writing, and what improves it?
An AI-likelihood score for writing, with measured error rates. It is evidence, not proof.

This repository is a small, reproducible experiment. It compiles labelled texts and compares three kinds
of detector on them:
Proofmark is two things in one repository:

- **A detector you can run.** A local web app and MCP server give a text an AI likelihood from 1% to 99%
and say how often real human writing scores the same. Nothing leaves your machine. See
[Use the detector](#use-the-detector-web-app-and-mcp-server).
- **The evaluation behind it.** A reproducible comparison of three kinds of detector on labelled texts,
including one student's private pre-ChatGPT writing. It is the reason the app uses a trained classifier
and does not ask a general model.

The three kinds of detector compared:

1. asking a general-purpose *decision model* (Cloudflare's Clef, on Workers AI) with no training;
2. statistical detectors that need no training (Binoculars, Fast-DetectGPT);
3. a classifier trained on examples, with and without examples from current models.

It is an evaluation harness, not a detector. Nothing here should be used to accuse anyone of using AI.
Status: a working research project, not a product. The classifier is a small model trained in an hour,
and the private-writing test covers one writer. Nothing here should be used on its own to accuse anyone
of using AI.

## Results in brief (15 November 2026)

Expand Down Expand Up @@ -203,8 +213,8 @@ Prerequisites: Python 3.11+, [uv](https://docs.astral.sh/uv/). Calling the Clef
text also needs a Cloudflare account with Workers AI.

```sh
git clone https://github.com/ichen27/ai-text-detection-eval.git
cd ai-text-detection-eval
git clone https://github.com/ichen27/proofmark.git
cd proofmark
uv sync
bash ci/run.sh # lint and unit tests; needs no credentials
uv run python -m detecteval.report # first study: rebuild report and figures from committed scores
Expand Down Expand Up @@ -345,7 +355,7 @@ Binoculars and the classifier run locally: about 30 minutes and 60 minutes on an
| Reports | [results/report.md](results/report.md), [results/comparison.md](results/comparison.md), [results/private_writing.md](results/private_writing.md), with `.json` twins | Markdown, JSON | `report`, `compare`, `private_eval report` |
| Figures | [results/figures/](results/figures/) | PNG | `report`, `compare` |
| Reference scores for the detector app | [results/calibration.json](results/calibration.json) | JSON | `calibration` |
| Trained classifiers (2 × 306 MB) | [release v0.3.0](https://github.com/ichen27/ai-text-detection-eval/releases/tag/v0.3.0) | Hugging Face model folders, gzipped tar | Extract into `RAW_DATA_DIR/models/`, then `python -m detecteval.classify raid mirrors`; or retrain with `train_classifier` |
| Trained classifiers (2 × 306 MB) | [release v0.3.0](https://github.com/ichen27/proofmark/releases/tag/v0.3.0) | Hugging Face model folders, gzipped tar | Extract into `RAW_DATA_DIR/models/`, then `python -m detecteval.classify raid mirrors`; or retrain with `train_classifier` |

SHA-256 of the release files: `classifier-raid.tar.gz` `81ae99d8aa214fe596a5799cac1b5a19c8f67da5989284e9639a3431d535a9f5`,
`classifier-mirrors.tar.gz` `84b41fcb72116e3a19c03e13c670aeccfc1a08f0184dead8b409ab02825b9d0e`.
Expand Down Expand Up @@ -377,8 +387,9 @@ a dataset, prompts or the seed, regenerate scores and reports in the same pull r

## Licence and attribution

No licence has been chosen for the code yet, so default copyright applies. The evaluation sets include
short excerpts from RAID and from Liang et al. (2023), both MIT-licensed, and Stack Exchange posts under
CC BY-SA 4.0, which stay under that licence. Sources, attributions, the papers whose methods are
The code is released under the [MIT licence](LICENSE). The data keeps its own terms: the evaluation sets
include short excerpts from RAID and from Liang et al. (2023), both MIT-licensed, and Stack Exchange posts
under CC BY-SA 4.0, which stay under that licence. The released classifier weights are fine-tuned from
`distilbert/distilroberta-base` (Apache-2.0). Sources, attributions, the papers whose methods are
implemented here and the models downloaded at run time are listed in
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
Binary file modified docs/screenshots/detector-app.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 4 additions & 4 deletions launcher.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"id": "ai-detector",
"name": "AI Text Detector",
"description": "Scores how AI-like a text is and how often human writing scores the same. Runs locally.",
"icon": "AI",
"id": "proofmark",
"name": "Proofmark",
"description": "An AI-likelihood score for writing, with how often human writing scores the same. Runs locally.",
"icon": "PM",
"color": "#d9e4f2",
"tags": ["tools", "writing"],
"favorite": false,
Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
[project]
name = "detecteval"
version = "0.1.0"
description = "Zero-shot evaluation of decision models as AI-generated-text detectors"
description = "Proofmark: AI-likelihood scores for writing with measured error rates, and the evaluation behind them"
license = "MIT"
readme = "README.md"
requires-python = ">=3.11"
dependencies = ["numpy>=1.26", "matplotlib>=3.8", "duckdb>=1.1"]
Expand Down
4 changes: 2 additions & 2 deletions src/detecteval/serve.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""The detector as a small web app and an MCP server, in one process.
"""Proofmark: the detector as a small web app and an MCP server, in one process.

python -m detecteval.serve # needs the "ml" and "serve" dependency groups

Expand Down Expand Up @@ -216,7 +216,7 @@ async def api_detect(request: Request) -> JSONResponse:


mcp = MCPServer(
"AI text detector",
"Proofmark",
instructions="Scores how AI-like a text is, with measured error rates. A result is evidence, not a verdict: "
"never present it as proof that someone used AI.")

Expand Down
7 changes: 4 additions & 3 deletions src/detecteval/static/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>AI text detector</title>
<title>Proofmark</title>
<style>
:root {
--paper: #eef1f4; --surface: #ffffff; --ink: #16202b; --muted: #5b6673; --rule: #d5dbe2;
Expand Down Expand Up @@ -102,9 +102,10 @@
</head>
<body>
<main>
<h1>AI text detector</h1>
<h1>Proofmark</h1>
<p class="lede">Paste writing to get an AI likelihood from 1% to 99%, and how often real human writing
scores the same. Everything runs on this machine; the text is not sent anywhere.</p>
scores the same. It is evidence, not proof. Everything runs on this machine; the text is not sent
anywhere.</p>

<form id="form">
<label for="text">Text to score</label>
Expand Down
Loading