Skip to content

Add results on private, pre-ChatGPT writing - #6

Merged
ichen27 merged 3 commits into
mainfrom
feat/private-writing-results
Nov 15, 2026
Merged

ichen27 merged 3 commits into
mainfrom
feat/private-writing-results

Conversation

@ichen27

@ichen27 ichen27 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Problem

Every false-positive figure so far came from public human text, which may be in the models' training data and is not the kind of writing detectors are used on.

What this adds

  • Aggregate results from running every detector on one writer's private Google Docs, with the flagging bar left where the public data put it. Texts, titles and per-document scores are not in the repository.
  • private_eval now reports documents as well as passages, includes the local-only combination, can keep a group off the cloud step (--skip-group), and shows what each detector catches when the bar is set on the certainly-human private group.

Results (300-word passages, bar set for 5% of public human texts)

Detector Public humans Private, dated before 30 Nov 2022
Clef 27B, asked 5% 42% (21 of 50)
Binoculars (0.5B pair) 4% 4% (2 of 50)
Classifier, RAID + mirrors 3% 8% (4 of 50)
Binoculars + classifier 6% 22% (11 of 50)
All three combined 5% 44% (22 of 50)

With the bar re-set on the private passages, the classifier still catches 85% of AI text at 300 words; Clef catches 65%.

Validation

  • bash ci/run.sh: ruff clean, 28 tests pass.
  • results/private_writing.md and .json were checked to contain no document ids, titles or text.

Risks

  • One writer, mostly high-school essays; passages cut from one document are not independent.
  • The re-set bar is chosen and judged on the same private passages, so that table is indicative.
  • A personal journal was scored by the local detectors only; it was not sent to Clef.

… private human text

Adds the local-only combination, counts of documents with a flagged
passage, a table of what each detector catches when the bar is set on the
certainly-human private group, and an option to keep a group's texts off
the cloud step.
Counts and rates only; texts, titles and per-document scores stay outside
the repository.
@ichen27
ichen27 merged commit c586c52 into main Nov 15, 2026
1 check passed
@ichen27
ichen27 deleted the feat/private-writing-results branch November 15, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant