Why AI-Text Detectors Disagree About Who Cheated
A measurement study of AI-text detectors. The popular claim is that they all read one underlying signal. That is partly a statistical trick of the setup. Measure it correctly and the detectors disagree, in ways that decide which students get falsely accused.
Why you should care
Picture a student who writes an essay, turns it in, and a week later sits across from a teacher who says a detector flagged it as AI. No chatbot was used. The student now has to prove a negative. This happens, and it happens more to some students than others.
So it matters a lot whether these detectors measure something real and shared, or whether each one has its own quirks and its own set of victims. A recent line of research argues for the shared story. Detectors come in very different shapes. Some measure how "surprised" a base model (a plain language model, before any special training) is by the words, since a model finds its own writing unsurprising and human writing more surprising. Some are trained classifiers (a model trained on labeled AI-versus-human examples to sort new text). Some compare two models against each other. Several papers argue that underneath, they all track a single hidden axis, one underlying thing, such as a writing style left behind by a model's training. If that were true, the detectors would be interchangeable, and a wrong call from one would be a wrong call from all of them. We set out to check that one-axis claim. The standard way to check it gives the wrong answer, for a reason built into the setup.
The trap in the obvious way to measure
Here is the natural way to ask "do these detectors measure one thing." Grab a pile of documents, some written by people and some by AI. Score every document with every detector. Now you have a table of numbers, one column per detector. If the columns all go up and down together, the detectors agree. If they move on their own, they measure different things. The standard one-number summary of "how many separate directions are in a table like this" is the participation ratio: a value near 1 means the columns move as one, and a value near the number of detectors means they move independently.
The trap is mixing people and AI in the same pile. Every good detector scores AI text high and human text low, because that is the whole job. So on a mixed pile every detector rises on the AI documents and falls on the human ones, and they all look nearly identical, no matter what else each one actually measures. That is exactly what we saw: on the mixed pile, the first shared direction explains 83% of the variation and the participation ratio is 1.42. That reads as "basically one signal." But that 1.42 is created by the pile having two groups in it. It is not evidence that the detectors share a mechanism. You would get the same "one signal" answer even if the detectors were reading totally different things.
The fix is to look at each group by itself: score the human documents alone, and the AI documents alone, so there is no human-versus-AI split to fake the agreement. Do that and the first shared direction drops from 83% down to 58 to 64%. Try the toggle below to see the difference.
Measuring it the honest way
Even after that fix, a participation ratio above 1 is not yet proof of more than one signal. Picture one true shared signal, plus a bit of random noise unique to each detector. That also pushes the ratio above 1. We checked this by building fake data out of exactly one hidden signal (a factor, in the statistics term). The participation ratio came back at 3.35, far above 1, even though the truth was one factor. So the ratio by itself cannot settle anything.
The test that can settle it is called parallel analysis (line up each candidate signal against the biggest signal you would get from pure random noise, and keep only the ones that clearly beat noise). On that same one-signal fake data, parallel analysis returns the correct count of one factor.
Then we made the group of detectors as different from each other as we could, so nobody could say the result came from cherry-picking. The final group has nine detectors built on six different base models: surprise-based scores from two unrelated language models, three detectors that read a document's "typicality" (how ordinary the writing looks to a base model), and RADAR, a detector trained specifically to spot AI text. We reran the whole measurement on five random shuffles of the data, checking the human group and the AI group each time. In nine of those ten checks, parallel analysis found two real signals, not one. The participation ratio settled at 3.00, with a 95% confidence range of 2.95 to 3.06 (rerun the shuffle many times and almost all answers land in that window). Making the detectors more varied pushed the count up, not down, which is the opposite of what a cherry-picked group would do. So the honest verdict is that AI-text detection uses at least two signals: more than one shared axis, and fewer than nine independent ones, a small handful of genuinely different directions.
Why two signals is not a technicality
Two signals means detectors do not share one blind spot. The clearest place to see it is the writing of people who learned English as a second language, which researchers call ESL (English as a Second Language) writing.
Two published papers looked like they flatly disagreed. One found that deployed detectors call ESL essays "AI" far too often. Another found the opposite for a typicality detector: ESL essays look unusual to a base model, so they land on the human side and get under-flagged. Both are true, and length is the main axis we can point to (a middle length of 104 words versus 429 words in the two studies). To test it we set each detector's cutoff to flag 5% of native-English writers, then measured how often it flags ESL writers. On the long essays, the typicality detectors under-flag ESL writers. Truncate those same essays to about 100 words, the length students actually hand in, and one detector, DeBERTa, flips to over-flagging 9.4% of them, while another, ELECTRA, barely budges. Which detector over-flags an ESL writer depends on the detector and on how long the essay is.
And the harm is not hypothetical. One popular commercial detector, run at its own default setting, calls 51.6% of 91 real ESL exam essays "AI," which matches an earlier report of about 61% on the same set. (This is a different detector at its shipped cutoff, so it is a picture of real-world harm, not a number you can line up directly against the 9.4% above.) These are confident mistakes: 25 of the 91 essays were rated at an average 89% chance of being AI, and every single one was written by a person. A school that buys one detector is choosing a particular, length-dependent list of who gets falsely accused.
The part that did not work
The most useful thing to report is one of our own findings that failed when we tested it properly. We ran an experiment asking whether a text is easier to detect because of a model's training recipe (the steps used to fine-tune it) or because of its base-model typicality. Across six random seeds (a seed is the starting point for the randomness, so a different seed is a fresh independent try), typicality is the stronger predictor in both prompting styles (0.456 versus 0.161 in one, 0.361 versus 0.193 in the other). A single seed had made it look as though the winner flips depending on how you prompt the model, which would have been a striking result. The flip was that one seed, and it vanished across the other five. We report the corrected result.
That near-miss was one of four different ways our own analysis handed us a confident but wrong answer before we caught it. Each one passed the usual sanity checks and was caught only by a targeted test. Mixing the two groups would have "confirmed" one signal, for the mechanical reason from earlier. Averaging perplexity (a measure of surprise that has a long tail, where a few extreme values dominate) let a handful of broken generations swing a reported average between 288 and 54,983 across otherwise identical runs. Using a base model's own score as both the detector and the thing it was measured against made a correlation come out at exactly 1.0, which is just a number compared with itself, not a discovery. And the single-seed flip. We boiled these down to a short checklist any study like this can run: score each group on its own, average surprise in log space, keep the detector separate from what you compare it against, and rerun across seeds.
Where this does not settle things
This is a measurement on one family of AI writing and two collections of human writing. The exact numbers belong to that setup. The claim that detection uses more than one signal is the sturdy part, because it comes from the factor test on a varied set of detectors and holds up across seeds. The exact count and the exact participation ratio would shift with a different set of detectors. We established the factor count on one AI generator. The clearest next step is to confirm it on text from an unrelated generator. And the long-versus-short comparison uses two essay collections that differ in topic and style as well as length, so length is a strong driver here but not a perfectly isolated one.
Takeaways
- The popular "detectors all read one axis" result is partly a trick of measuring on a mixed human-and-AI pile, where the group label forces the detectors to look identical.
- Measured within each group, on a varied nine-detector set, an honest factor test finds at least two real signals, steady across five seeds.
- Because detectors do not share one signal, they do not share one blind spot. The same short ESL essay is "AI" to one detector and "human" to another, and a deployed detector falsely flags about half of real ESL exam essays.
- One run is not a result. One of our own findings reversed under more seeds, which is why the four-item checklist above is the part of this work worth carrying to your own project.