Skip to content

Compare Clef with Binoculars and a trained classifier on longer, held-out texts - #2

Merged
ichen27 merged 7 commits into
mainfrom
feat/longer-texts-detectors-classifier
Nov 13, 2026
Merged

ichen27 merged 7 commits into
mainfrom
feat/longer-texts-detectors-classifier

Conversation

@ichen27

@ichen27 ichen27 commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Problem

The first study asked three Clef decision models whether short texts were AI-written. It left four gaps: texts were at most 150 words, the flagging bar was set on the same texts it was scored on, every detector was the same kind (a model being asked), and nothing was trained.

What this adds

  • A long evaluation set (data/long_set.jsonl): 830 documents of 300 to 500 words from seven RAID domains, student essays and Stack Exchange posts, each scored cut to 75, 150, 300 and 500 words. Every text belongs to a calibration or a test half.
  • Human posts published after the Clef models were released, with posts from the same sites in 2019 as a control.
  • Binoculars and Fast-DetectGPT on a local 0.5B model pair.
  • A classifier (DistilRoBERTa) trained on RAID, then updated with "mirror" texts from four current models. Four other current models, the evaluation forum sites and all student essays are kept out of training.
  • compare: every detector and their combinations, with thresholds and combination weights set on the calibration half and all numbers measured on the test half.

Results (test half)

Short texts, including 2026-model text, bar set for 5% of humans:

Detector AUROC AI caught Humans flagged
Clef 27B, asked 0.89 65% 5%
Binoculars (0.5B pair) 0.81 45% 2%
Classifier, RAID only 0.88 61% 5%
Classifier, RAID + 264 mirrors 0.97 86% 6%
All three combined 0.97 83% 6%

Same documents at 75 / 150 / 300 words (human and 2023-era AI text): Clef 49% / 59% / 81%; all three combined 92% / 95% / 98%.

The trained classifier flags 9 of 47 non-native TOEFL essays against 0 of 48 native essays; Clef flags 0 and 1.

Full tables are in results/comparison.md.

Not finished

The MCP gateway used to reach Workers AI went down during the run, so the paid steps stopped part-way:

  • long 2026-model texts: 79 of 160 generated, no humanized or polished texts;
  • Clef has not scored the long 2026-model texts, so it has no 300-word result on current models and no 500-word result;
  • Clef-flash is partly scored and left out of the comparison; Clef-omni is not started;
  • training mirrors: 264 of 1,000.

Each step resumes where it stopped. The README lists them under "What is not finished".

Validation

  • bash ci/run.sh: ruff clean, 27 tests pass.
  • python -m detecteval.report reproduces the first study's report with no diff.
  • python -m detecteval.compare rebuilds results/comparison.md and the figures from committed scores.

Risks

  • Test halves are small (about 170 to 260 human and 110 to 180 AI texts); per-group rows smaller.
  • Recent forum posts include AI-assisted writing, so they are not a clean "unseen human" sample.
  • The local detectors need 3 to 5 GB of free memory each and must be run one at a time.

… request limits

Shared pieces for the second study: texts cut to fixed word lengths, a
calibration/test split for thresholds, a small logistic regression for
combining detectors, Wilson intervals, and a per-item time limit in hub
batches so one slow request cannot lose the rest. Adds an optional ml
dependency group for the local detectors.
…th rungs

Documents of 300 words or more, kept up to 500, so each can be scored at
75, 150, 300 and 500 words. Human forum posts published after the Clef
models were released stand in for writing the models cannot have seen,
with 2019 posts from the same sites as a control. Every text is assigned
to a calibration or a test half.
… detectors

Binoculars and Fast-DetectGPT run on a small local model pair. The
classifier is fine-tuned on RAID, optionally with mirror texts written by
four current models; four other current models and all student essays are
kept out of training. compare reports every detector and their
combinations on the test half only.
Binoculars works through each text in slices, the classifier pads batches
to fixed lengths and checkpoints, and both cap Apple-GPU memory so a job
fails cleanly instead of exhausting the machine. The classifier uses a
smaller encoder, and the mirrors recipe is now a short update of the
RAID-trained model. compare restricts every head-to-head table to texts
all detectors have scored and adds a local-detectors view.
830 documents of 300 to 500 words with a calibration/test split, 79 fresh
2026-model texts and 264 mirror texts. Generation stopped when the MCP
gateway went down; the remaining texts are documented as outstanding.
@ichen27
ichen27 merged commit 630606e into main Nov 13, 2026
1 check passed
@ichen27
ichen27 deleted the feat/longer-texts-detectors-classifier branch November 13, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant