Our Benchmark Caught Us Twice

A team that built a benchmark tool for AI-text detectors found that it caught two of their own wrong conclusions within a week, including a claim that a simple detector performed below chance based on a 12-document smoke test, which was later corrected when a 6-million-document corpus showed a score of 0.766. The tool enforces rules such as refusing to answer on short text, treating scores as non-probabilities, and reporting 'n/a' when sample sizes are too small to measure false-positive rates.

Our Benchmark Caught Us Twice We built a tool that measures AI-text detectors. Not another detector — a referee. It runs several detection methods against the same documents, attacks those documents the way somebody trying to evade detection actually would, and reports where each method fails. Within a week it had caught two wrong conclusions. Both were ours. That is not a confession dressed up as a lesson. It is the thing we were hoping the tool would do, working earlier and more bluntly than expected. But the second mistake is worth describing carefully, because the wrong version was a much better story than the right one, and we came close to telling it. Why measure detectors instead of building one Almost every AI detector leads with a single number, usually AUROC, usually above 0.95. That number is close to useless for the decision people actually make with these tools, which is “do I accuse this person?” AUROC averages performance over every possible threshold, including thresholds nobody would deploy. A detector can post an excellent AUROC and still be catastrophically wrong at the one setting that matters. Recent work found detectors assigning near-certain “machine” labels to 60% of human-written documents once the topic drifted away from their training data — while their headline numbers still looked fine. So our tool leads with a different question: how much machine text do you catch if you accept wrongly accusing one person in a hundred? And then the harder one: one in a thousand? Three rules fell out of that, and all three are enforced in code rather than recommended in comments: Refusing to answer is a result. Short text carries too little signal for any published method. A detector that answers anyway is guessing with a confident face. Ours declines, and the refusal is recorded and reported alongside the accuracy — because a tool that only answers the easy documents will otherwise look excellent. A score is not a probability. Our detectors return raw numbers, and the probability field stays empty until someone has actually calibrated the method against labelled data. Turning a raw score into a confident-looking percentage is the most common way this whole category misleads people. In our code it is a type error. The measurements refuse too. You cannot observe a one-in-a-thousand false-positive rate with fewer than a thousand human documents — the smallest rate a sample of n can even express is 1/ n . So when the sample is too small, our tool prints n/a instead of a number. This turns out to matter more than we expected. The first mistake: twelve documents The tool ships with a tiny fixture — twelve documents, six written between 1813 and 1859 and six generated by a model — so that new users can run it immediately. It is labelled, loudly, as a smoke test that supports no conclusions. We then drew a conclusion from it. Our simplest detector is a deliberately weak one: twelve surface statistics like sentence-length variation and punctuation habits. No model, no training. It exists as a control. On the twelve-document fixture it scored below chance — worse than a coin flip. We wrote that down in the methodology file, along with our reasoning: the hand-assigned directions of those twelve features must be wrong on balance. Then we wired up a real corpus — six million documents across eleven text generators — and ran it again on five thousand. It scored 0.766. Comfortably above chance. The feature directions were fine all along. Twelve documents simply were not enough to tell, and the confident diagnosis we wrote from them was wrong. We left the correction in the file rather than quietly editing it out. The warnings about small samples were already there; we had written them ourselves, and then walked straight past them. That seemed worth recording. The second mistake: the headline we nearly published This one is better. Along with the detectors, the tool ships attacks — the cheap tricks somebody would actually use. The cheapest is called zero width : insert invisible characters between words. It costs seconds. Copy the result into a document and it looks completely unchanged, because it is unchanged to a reader. Only the underlying bytes move. We ran three detectors across three thousand documents, clean and attacked. The result was striking: | detector | clean | with invisible characters | |---|---|---| | twelve-feature baseline | 0.756 | 0.744 | | Binoculars | 0.781 | 0.618 | | Fast-DetectGPT | 0.787 | 0.548 | The best detector fell to 0.548. That is a coin flip. Meanwhile our deliberately weak baseline barely moved. The ranking inverted. On clean text the sophisticated methods clearly won. Under an attack that costs nothing, the crude one won — because it never looks at tokens in the first place, so scrambling the tokenisation does not touch it. That is a genuinely good story. It has a table, a reversal, and a villain you can demonstrate in ten seconds. We had the post half-drafted in our heads. Then we ran the control we had nearly skipped. The control we nearly skipped Both of those attacks are undone by Unicode normalisation — a standard cleanup pass that strips invisible characters and folds lookalike letters back to plain ASCII. Our tool already had the function. We had simply not been applying it before scoring. So we added a “defended” mode: apply the cleanup to every document, then measure again. | detector | attack | clean | attacked | defended | recovered | |---|---|---|---|---|---| | Binoculars | invisible chars | 49.7% | 14.7% | 49.7% | 100% | | Fast-DetectGPT | invisible chars | 51.3% | 18.1% | 51.3% | 100% | | Fast-DetectGPT | synonym swaps | 51.3% | 50.0% | 50.0% | 0% | catch rate at a one-in-a-hundred false-positive rate Full recovery. Both detectors restored exactly, and the ranking flipped back to normal. The collapse was never a property of the detectors. It was a property of not running a cleanup pass . The honest claim is not “invisible characters break the best detectors.” It is “invisible characters break the best detectors in any deployment that forgets to normalise its input, and the repair is one function call.” That is a duller sentence. It is also the true one. Note the third row, the one at 0%. Swapping synonyms is not a Unicode trick, so a Unicode cleanup does nothing about it — exactly as it should. That row is what makes the other two believable. A defence that appeared to repair everything would mean our harness was broken, not that we had found a great defence. What actually caught us Neither mistake was caught by being careful. Both were caught by machinery we had built for other reasons. The first was caught because we eventually pointed the tool at a real corpus, and because the tool refuses to print numbers a small sample cannot support — so the twelve-document result had n/a scattered across it, which made it obviously untrustworthy the moment we looked again. The second was caught by a control slice. And when we added it, we nearly left one cell out: we reasoned that a Unicode cleanup obviously cannot matter for a synonym-swap attack, so why spend an hour of GPU time confirming it? Then we checked, and found the cleanup alters 61 of 1,500 documents on that slice anyway — because real text is full of ligatures, curly quotes and lookalike characters that normalisation rewrites whether or not anybody attacked them. Which produced a third finding we were not looking for: the defence is not free. It edits legitimate documents too. Under the invisible-character attack it restores 1,439 of 1,500 documents to their original form — not all of them. There was a fourth catch, on the way in. Before publishing any numbers we checked our implementation of Binoculars against the authors’ own code, line by line, and found we had it wrong in two places: we were taking a perplexity measurement from the wrong one of the two models, and applying an alignment shift the original does not apply. The resulting scores were off by 4.6% to 12.4% — larger than the gap between the reference implementation’s own two recommended thresholds. Both versions produced completely plausible-looking numbers. Only comparison caught it. The point We are not going to tell you AI-text detection works. It mostly does not, at the confidence people want from it, and the failures fall hardest on people writing in a second language. But the useful lesson from a week of this is narrower and more portable: the exciting version of a finding is the one most likely to be wrong, and the thing that catches it is usually a control you were tempted to skip. Every mistake above was caught by a rule that felt like bureaucratic overhead when we wrote it. Refuse to print numbers the sample cannot support. Measure the defended case, not just the attacked one. Check your implementation against the original before you quote it. None of those are clever. They are just the parts that were still standing when the exciting story turned out to be wrong. The code, the corpus wiring, and both corrections — left in rather than edited out — are at sv-pro/ai-detector-bench https://github.com/sv-pro/ai-detector-bench .