Skip to content

Add a test on private writing that keeps the texts out of the repository - #5

Merged
ichen27 merged 1 commit into
mainfrom
feat/private-writing-test
Nov 15, 2026
Merged

ichen27 merged 1 commit into
mainfrom
feat/private-writing-test

Conversation

@ichen27

@ichen27 ichen27 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Problem

Every human text in the study is public, so it may be in a model's training data, and recent public text turned out to include AI-assisted writing. Private documents written before language models were common are the clean test of false accusations.

What this adds

  • detecteval.private_eval with three steps: clef, local and report. It reads RAW_DATA_DIR/private/manifest.jsonl, scores each document at 75, 150, 300 and 500 words, and reports how many each detector and combination flags.
  • The flagging bar and combination weights are the ones already set on the long set's calibration half; the private documents play no part in choosing them.
  • Texts, ids and per-document scores stay under RAW_DATA_DIR/private. The report holds counts and rates only.
  • binoculars.run, classify.run and score_long.score now accept any list of texts.

Validation

  • bash ci/run.sh: ruff clean, 28 tests pass (one new test for the private-text loader).
  • Ran all three steps end to end on ten public stand-in texts; the stand-ins and their output were then deleted.

Not included

No private documents have been scored yet; this PR adds the tooling only.

…ut of the repository

private_eval scores documents listed in RAW_DATA_DIR/private with Clef,
Binoculars and both classifiers, then reports how many each detector flags
using the bar and combination weights already set on the long set's
calibration half. The detectors' entry points now accept any list of texts.
@ichen27
ichen27 merged commit 073077b into main Nov 15, 2026
1 check passed
@ichen27
ichen27 deleted the feat/private-writing-test branch November 15, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant