Skip to content

Complete the second study: Clef on current-model text, full mirrors, 500 words - #4

Merged
ichen27 merged 4 commits into
mainfrom
feat/finish-second-study
Nov 15, 2026
Merged

ichen27 merged 4 commits into
mainfrom
feat/finish-second-study

Conversation

@ichen27

@ichen27 ichen27 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Problem

The second study stopped part-way when the MCP gateway went down: half the long 2026-model texts were missing, Clef had not scored any of them, there were no humanized or polished long texts, and the classifier update used 264 of 1,000 planned mirrors.

What this adds

  • The remaining long texts: 148 from 2026 models (now including forum questions and student essays), 47 humanized rewrites and 56 human texts polished by a current model. The long set is 1,003 documents.
  • Clef 27B scores for all of them at 75, 150, 300 and 500 words; Clef-flash and Clef-omni at 150 and 300.
  • 943 training mirrors and a classifier update retrained on them.
  • A simpler comparison report now that every detector has scored every document.

Results (test half, bar set for 5% of humans)

Detector Short texts 300 words
Clef 27B, asked 65% 76%
Binoculars (0.5B pair) 45% 73%
Classifier, RAID only 61% 69%
Classifier, RAID + mirrors 92% 91%
All three combined 95% 95%
  • From 300 to 500 words Clef goes from 63% to 74% on the documents that long.
  • A one-prompt humanizer drops Clef from 77% to 42% at 300 words; the updated classifier still flags all 19.
  • The updated classifier flags 4 of 47 non-native essays and 1 of 48 native ones (RAID-only: 8 and 0; Clef: 0 and 1).
  • Human text polished by a model: the classifier flags 24 of 27, Clef 6 of 27.

Full tables are in results/comparison.md.

Validation

  • bash ci/run.sh: ruff clean, 27 tests pass.
  • python -m detecteval.report reproduces the first study's report with no diff.
  • python -m detecteval.compare rebuilds results/comparison.md and the figures from committed scores.

Risks and gaps

  • GLM 5.3 failed many requests, so it wrote 10 of its 20 long texts and 193 of its 250 mirrors.
  • Test halves are small; group rows rest on 19 to 110 texts.
  • Human writing that has never been online is still untested.
  • The README links release v0.3.0 for the retrained classifier, which is published after this merges.

Once every detector has scored every document, the comparison drops the
local-only tables, includes Clef in the 300-to-500-word comparison and
reports seen and unseen generators on the long set. Rewrites now carry
the id of their source text so paired changes can be measured.
1,003 documents: adds the remaining 2026-model texts, 47 humanized
rewrites and 56 polished human texts. 943 training mirrors.
@ichen27
ichen27 merged commit 158ef7c into main Nov 15, 2026
1 check passed
@ichen27
ichen27 deleted the feat/finish-second-study branch November 15, 2026 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant