diff --git a/AGENTS.md b/AGENTS.md index 63c11ba..ac72d60 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,14 +1,29 @@ -# ai-text-detection-eval +# Proofmark -Repository: https://github.com/ichen27/ai-text-detection-eval (private). -Canonical checkout: `/Volumes/SamsungSSD1/code/ai-text-detection-eval` on the Mac mini SSD. +Repository: https://github.com/ichen27/proofmark (public; it was named ai-text-detection-eval until +2026-11-16, and the old address redirects). +Canonical checkout: `/Volumes/SamsungSSD1/code/ai-text-detection-eval` on the Mac mini SSD. The directory +and the Python package (`detecteval`) keep their original names. + +## Shipping decisions + +Recorded 2026-11-16. + +- **Launcher:** yes. Deployed to the app launcher as `proofmark`. +- **Repository visibility:** public, MIT licence for the code. +- **Name:** brand name, Proofmark, on GitHub and the launcher. +- **Announcement:** X only. Post link: not posted yet. + +The repository is public, so nothing private may be committed: no texts, titles, ids or per-document +scores from private writing, and no results for documents whose authorship is uncertain (use +`private_eval report --omit-group`). ## What this is -A small, reproducible experiment on detecting AI-written text. It compiles labelled text sets and -compares three kinds of detector on them: decision models asked a typed question with no training -(Cloudflare Clef), statistical detectors that need no training (Binoculars, Fast-DetectGPT), and a -classifier fine-tuned on examples. It is an evaluation harness, not a detector product. +A local AI-text detector (web app and MCP server) and the reproducible evaluation behind it. The +evaluation compiles labelled text sets and compares three kinds of detector on them: decision models +asked a typed question with no training (Cloudflare Clef), statistical detectors that need no training +(Binoculars, Fast-DetectGPT), and a classifier fine-tuned on examples. The app uses the classifier. ## Layout @@ -55,13 +70,13 @@ classifier fine-tuned on examples. It is an evaluation harness, not a detector p ## The deployed detector -The service is deployed on the Mac mini through the app launcher as `ai-detector` -(https://app-ai-detector.chenagent.com, owner sign-in) from [launcher.json](launcher.json), and is -registered in the MCP hub as the lazy HTTP backend `ai-detector` (tools prefixed `ai_detector__`). +The service is deployed on the Mac mini through the app launcher as `proofmark` +(https://app-proofmark.chenagent.com, owner sign-in) from [launcher.json](launcher.json), and is +registered in the MCP hub as the eager HTTP backend `proofmark` (tools prefixed `proofmark__`). - It runs from this canonical checkout with its own environment at `/Volumes/SamsungSSD1/tools/ai-text-detection/venv`, so `ci/run.sh` and `uv sync` here do not disturb - it. Code changes take effect on restart: `node src/cli.mjs restart ai-detector` in the launcher checkout. + it. Code changes take effect on restart: `node src/cli.mjs restart proofmark` in the launcher checkout. Keep the checkout on `main` when not actively working, or develop service changes in a worktree. - After changing dependencies, redeploy with the launcher's `deploy` command so the build step runs. - The service must stay light: models load in the worker on demand and the worker is stopped when idle. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..56f61d2 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Ivan Chen + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index c29e400..813c393 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,25 @@ -# ai-text-detection-eval +# Proofmark -How well can AI-written text be told from human writing, and what improves it? +An AI-likelihood score for writing, with measured error rates. It is evidence, not proof. -This repository is a small, reproducible experiment. It compiles labelled texts and compares three kinds -of detector on them: +Proofmark is two things in one repository: + +- **A detector you can run.** A local web app and MCP server give a text an AI likelihood from 1% to 99% + and say how often real human writing scores the same. Nothing leaves your machine. See + [Use the detector](#use-the-detector-web-app-and-mcp-server). +- **The evaluation behind it.** A reproducible comparison of three kinds of detector on labelled texts, + including one student's private pre-ChatGPT writing. It is the reason the app uses a trained classifier + and does not ask a general model. + +The three kinds of detector compared: 1. asking a general-purpose *decision model* (Cloudflare's Clef, on Workers AI) with no training; 2. statistical detectors that need no training (Binoculars, Fast-DetectGPT); 3. a classifier trained on examples, with and without examples from current models. -It is an evaluation harness, not a detector. Nothing here should be used to accuse anyone of using AI. +Status: a working research project, not a product. The classifier is a small model trained in an hour, +and the private-writing test covers one writer. Nothing here should be used on its own to accuse anyone +of using AI. ## Results in brief (15 November 2026) @@ -203,8 +213,8 @@ Prerequisites: Python 3.11+, [uv](https://docs.astral.sh/uv/). Calling the Clef text also needs a Cloudflare account with Workers AI. ```sh -git clone https://github.com/ichen27/ai-text-detection-eval.git -cd ai-text-detection-eval +git clone https://github.com/ichen27/proofmark.git +cd proofmark uv sync bash ci/run.sh # lint and unit tests; needs no credentials uv run python -m detecteval.report # first study: rebuild report and figures from committed scores @@ -345,7 +355,7 @@ Binoculars and the classifier run locally: about 30 minutes and 60 minutes on an | Reports | [results/report.md](results/report.md), [results/comparison.md](results/comparison.md), [results/private_writing.md](results/private_writing.md), with `.json` twins | Markdown, JSON | `report`, `compare`, `private_eval report` | | Figures | [results/figures/](results/figures/) | PNG | `report`, `compare` | | Reference scores for the detector app | [results/calibration.json](results/calibration.json) | JSON | `calibration` | -| Trained classifiers (2 × 306 MB) | [release v0.3.0](https://github.com/ichen27/ai-text-detection-eval/releases/tag/v0.3.0) | Hugging Face model folders, gzipped tar | Extract into `RAW_DATA_DIR/models/`, then `python -m detecteval.classify raid mirrors`; or retrain with `train_classifier` | +| Trained classifiers (2 × 306 MB) | [release v0.3.0](https://github.com/ichen27/proofmark/releases/tag/v0.3.0) | Hugging Face model folders, gzipped tar | Extract into `RAW_DATA_DIR/models/`, then `python -m detecteval.classify raid mirrors`; or retrain with `train_classifier` | SHA-256 of the release files: `classifier-raid.tar.gz` `81ae99d8aa214fe596a5799cac1b5a19c8f67da5989284e9639a3431d535a9f5`, `classifier-mirrors.tar.gz` `84b41fcb72116e3a19c03e13c670aeccfc1a08f0184dead8b409ab02825b9d0e`. @@ -377,8 +387,9 @@ a dataset, prompts or the seed, regenerate scores and reports in the same pull r ## Licence and attribution -No licence has been chosen for the code yet, so default copyright applies. The evaluation sets include -short excerpts from RAID and from Liang et al. (2023), both MIT-licensed, and Stack Exchange posts under -CC BY-SA 4.0, which stay under that licence. Sources, attributions, the papers whose methods are +The code is released under the [MIT licence](LICENSE). The data keeps its own terms: the evaluation sets +include short excerpts from RAID and from Liang et al. (2023), both MIT-licensed, and Stack Exchange posts +under CC BY-SA 4.0, which stay under that licence. The released classifier weights are fine-tuned from +`distilbert/distilroberta-base` (Apache-2.0). Sources, attributions, the papers whose methods are implemented here and the models downloaded at run time are listed in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). diff --git a/docs/screenshots/detector-app.png b/docs/screenshots/detector-app.png index 424fd6d..d9bc1db 100644 Binary files a/docs/screenshots/detector-app.png and b/docs/screenshots/detector-app.png differ diff --git a/launcher.json b/launcher.json index 44e4801..e5e46f2 100644 --- a/launcher.json +++ b/launcher.json @@ -1,8 +1,8 @@ { - "id": "ai-detector", - "name": "AI Text Detector", - "description": "Scores how AI-like a text is and how often human writing scores the same. Runs locally.", - "icon": "AI", + "id": "proofmark", + "name": "Proofmark", + "description": "An AI-likelihood score for writing, with how often human writing scores the same. Runs locally.", + "icon": "PM", "color": "#d9e4f2", "tags": ["tools", "writing"], "favorite": false, diff --git a/pyproject.toml b/pyproject.toml index 18ea49b..829920f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,8 @@ [project] name = "detecteval" version = "0.1.0" -description = "Zero-shot evaluation of decision models as AI-generated-text detectors" +description = "Proofmark: AI-likelihood scores for writing with measured error rates, and the evaluation behind them" +license = "MIT" readme = "README.md" requires-python = ">=3.11" dependencies = ["numpy>=1.26", "matplotlib>=3.8", "duckdb>=1.1"] diff --git a/src/detecteval/serve.py b/src/detecteval/serve.py index 8a98ec9..9e8575e 100644 --- a/src/detecteval/serve.py +++ b/src/detecteval/serve.py @@ -1,4 +1,4 @@ -"""The detector as a small web app and an MCP server, in one process. +"""Proofmark: the detector as a small web app and an MCP server, in one process. python -m detecteval.serve # needs the "ml" and "serve" dependency groups @@ -216,7 +216,7 @@ async def api_detect(request: Request) -> JSONResponse: mcp = MCPServer( - "AI text detector", + "Proofmark", instructions="Scores how AI-like a text is, with measured error rates. A result is evidence, not a verdict: " "never present it as proof that someone used AI.") diff --git a/src/detecteval/static/index.html b/src/detecteval/static/index.html index 86f0d9f..3ceddfe 100644 --- a/src/detecteval/static/index.html +++ b/src/detecteval/static/index.html @@ -3,7 +3,7 @@ -AI text detector +Proofmark