Skip to content

Give each detector result an AI likelihood percentage - #11

Merged
ichen27 merged 1 commit into
mainfrom
feat/detector-percentage
Nov 15, 2026
Merged

ichen27 merged 1 commit into
mainfrom
feat/detector-percentage

Conversation

@ichen27

@ichen27 ichen27 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Problem

Labels and odds were still hard to read at a glance; a single number was asked for.

Change

  • calibration.probability_curve: an isotonic fit of "is AI" against score with human and AI text weighted equally, so the output is the chance a passage is AI-written at even prior odds. A private human reference, when present, shares the human weight equally with the public one.
  • Each passage gets ai_likelihood_percent (1 to 99). Labels follow it: under 30% Human-like, 30 to 60% Slightly AI-like, 60 to 85% AI-like, 85% and up Strongly AI-like. The whole text gets the average weighted by passage length.
  • The page leads with the number and a bar; the MCP result carries the same fields and the tool description states the caveat.

Is the number honest?

Fitted on the calibration half and checked on the test half (classifier, 300 words): passages given 85% or more were AI-written 99% of the time, 60 to 85% were 72%, 30 to 60% were 39%, under 30% were 3%. With the private reference included, 4% of the writer's pre-ChatGPT passages score 85% or more.

Validation

Lint clean, 52 tests pass (curve monotonicity and class balance, likelihood bounds, labels, length weighting). Rendered one- and three-passage results in headless Chrome; README screenshot updated.

Risks

  • The percentage assumes even prior odds. Where most text is human-written the real chance is lower; the page, the limits list and the MCP tool description say so.
  • A single number invites over-reading. It never reaches 0% or 100%, and the false-alarm odds stay next to it.

A never-decreasing curve fitted to the reference scores turns a detector
score into the chance a passage is AI-written at even odds, from 1% to
99%. Labels now follow that number, the whole text gets a length-weighted
average, and the page and MCP results lead with it. A private human
reference, when present, counts as much as the public one in the curve.
@ichen27
ichen27 merged commit 6f60956 into main Nov 15, 2026
1 check passed
@ichen27
ichen27 deleted the feat/detector-percentage branch November 15, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant