Repository navigation
Compare Clef with Binoculars and a trained classifier on longer, held-out texts - #2
Merged
Merged
Conversation
… request limits Shared pieces for the second study: texts cut to fixed word lengths, a calibration/test split for thresholds, a small logistic regression for combining detectors, Wilson intervals, and a per-item time limit in hub batches so one slow request cannot lose the rest. Adds an optional ml dependency group for the local detectors.
…th rungs Documents of 300 words or more, kept up to 500, so each can be scored at 75, 150, 300 and 500 words. Human forum posts published after the Clef models were released stand in for writing the models cannot have seen, with 2019 posts from the same sites as a control. Every text is assigned to a calibration or a test half.
… detectors Binoculars and Fast-DetectGPT run on a small local model pair. The classifier is fine-tuned on RAID, optionally with mirror texts written by four current models; four other current models and all student essays are kept out of training. compare reports every detector and their combinations on the test half only.
Binoculars works through each text in slices, the classifier pads batches to fixed lengths and checkpoints, and both cap Apple-GPU memory so a job fails cleanly instead of exhausting the machine. The classifier uses a smaller encoder, and the mirrors recipe is now a short update of the RAID-trained model. compare restricts every head-to-head table to texts all detectors have scored and adds a local-detectors view.
830 documents of 300 to 500 words with a calibration/test split, 79 fresh 2026-model texts and 264 mirror texts. Generation stopped when the MCP gateway went down; the remaining texts are documented as outstanding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first study asked three Clef decision models whether short texts were AI-written. It left four gaps: texts were at most 150 words, the flagging bar was set on the same texts it was scored on, every detector was the same kind (a model being asked), and nothing was trained.
What this adds
data/long_set.jsonl): 830 documents of 300 to 500 words from seven RAID domains, student essays and Stack Exchange posts, each scored cut to 75, 150, 300 and 500 words. Every text belongs to a calibration or a test half.compare: every detector and their combinations, with thresholds and combination weights set on the calibration half and all numbers measured on the test half.Results (test half)
Short texts, including 2026-model text, bar set for 5% of humans:
Same documents at 75 / 150 / 300 words (human and 2023-era AI text): Clef 49% / 59% / 81%; all three combined 92% / 95% / 98%.
The trained classifier flags 9 of 47 non-native TOEFL essays against 0 of 48 native essays; Clef flags 0 and 1.
Full tables are in
results/comparison.md.Not finished
The MCP gateway used to reach Workers AI went down during the run, so the paid steps stopped part-way:
Each step resumes where it stopped. The README lists them under "What is not finished".
Validation
bash ci/run.sh: ruff clean, 27 tests pass.python -m detecteval.reportreproduces the first study's report with no diff.python -m detecteval.comparerebuildsresults/comparison.mdand the figures from committed scores.Risks