Repository navigation
Complete the second study: Clef on current-model text, full mirrors, 500 words - #4
Merged
Merged
Conversation
Once every detector has scored every document, the comparison drops the local-only tables, includes Clef in the 300-to-500-word comparison and reports seen and unseen generators on the long set. Rewrites now carry the id of their source text so paired changes can be measured.
1,003 documents: adds the remaining 2026-model texts, 47 humanized rewrites and 56 polished human texts. 943 training mirrors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The second study stopped part-way when the MCP gateway went down: half the long 2026-model texts were missing, Clef had not scored any of them, there were no humanized or polished long texts, and the classifier update used 264 of 1,000 planned mirrors.
What this adds
Results (test half, bar set for 5% of humans)
Full tables are in
results/comparison.md.Validation
bash ci/run.sh: ruff clean, 27 tests pass.python -m detecteval.reportreproduces the first study's report with no diff.python -m detecteval.comparerebuildsresults/comparison.mdand the figures from committed scores.Risks and gaps