{"slug": "our-benchmark-caught-us-twice", "title": "Our Benchmark Caught Us Twice", "summary": "A team that built a benchmark tool for AI-text detectors found that it caught two of their own wrong conclusions within a week, including a claim that a simple detector performed below chance based on a 12-document smoke test, which was later corrected when a 6-million-document corpus showed a score of 0.766. The tool enforces rules such as refusing to answer on short text, treating scores as non-probabilities, and reporting 'n/a' when sample sizes are too small to measure false-positive rates.", "body_md": "# Our Benchmark Caught Us Twice\n\nWe built a tool that measures AI-text detectors. Not another detector — a referee. It runs several detection methods against the same documents, attacks those documents the way somebody trying to evade detection actually would, and reports where each method fails.\n\nWithin a week it had caught two wrong conclusions. Both were ours.\n\nThat is not a confession dressed up as a lesson. It is the thing we were hoping the tool would do, working earlier and more bluntly than expected. But the second mistake is worth describing carefully, because the wrong version was a much better story than the right one, and we came close to telling it.\n\n## Why measure detectors instead of building one\n\nAlmost every AI detector leads with a single number, usually AUROC, usually above 0.95. That\nnumber is close to useless for the decision people actually make with these tools, which is\n*“do I accuse this person?”*\n\nAUROC averages performance over every possible threshold, including thresholds nobody would\ndeploy. A detector can post an excellent AUROC and still be catastrophically wrong at the one\nsetting that matters. Recent work found detectors assigning near-certain “machine” labels to\n**60% of human-written documents** once the topic drifted away from their training data —\nwhile their headline numbers still looked fine.\n\nSo our tool leads with a different question: *how much machine text do you catch if you\naccept wrongly accusing one person in a hundred?* And then the harder one: one in a thousand?\n\nThree rules fell out of that, and all three are enforced in code rather than recommended in comments:\n\n**Refusing to answer is a result.** Short text carries too little signal for any published\nmethod. A detector that answers anyway is guessing with a confident face. Ours declines, and\nthe refusal is recorded and reported alongside the accuracy — because a tool that only\nanswers the easy documents will otherwise look excellent.\n\n**A score is not a probability.** Our detectors return raw numbers, and the probability field\nstays empty until someone has actually calibrated the method against labelled data. Turning\na raw score into a confident-looking percentage is the most common way this whole category\nmisleads people. In our code it is a type error.\n\n**The measurements refuse too.** You cannot observe a one-in-a-thousand false-positive rate\nwith fewer than a thousand human documents — the smallest rate a sample of *n* can even\nexpress is 1/*n*. So when the sample is too small, our tool prints `n/a`\n\ninstead of a number.\nThis turns out to matter more than we expected.\n\n## The first mistake: twelve documents\n\nThe tool ships with a tiny fixture — twelve documents, six written between 1813 and 1859 and six generated by a model — so that new users can run it immediately. It is labelled, loudly, as a smoke test that supports no conclusions.\n\nWe then drew a conclusion from it.\n\nOur simplest detector is a deliberately weak one: twelve surface statistics like sentence-length\nvariation and punctuation habits. No model, no training. It exists as a control. On the\ntwelve-document fixture it scored *below chance* — worse than a coin flip. We wrote that down\nin the methodology file, along with our reasoning: the hand-assigned directions of those twelve\nfeatures must be wrong on balance.\n\nThen we wired up a real corpus — six million documents across eleven text generators — and ran it again on five thousand. It scored 0.766. Comfortably above chance. The feature directions were fine all along. Twelve documents simply were not enough to tell, and the confident diagnosis we wrote from them was wrong.\n\nWe left the correction in the file rather than quietly editing it out. The warnings about small samples were already there; we had written them ourselves, and then walked straight past them. That seemed worth recording.\n\n## The second mistake: the headline we nearly published\n\nThis one is better.\n\nAlong with the detectors, the tool ships attacks — the cheap tricks somebody would actually\nuse. The cheapest is called `zero_width`\n\n: insert invisible characters between words. It costs\nseconds. Copy the result into a document and it looks completely unchanged, because it *is*\nunchanged to a reader. Only the underlying bytes move.\n\nWe ran three detectors across three thousand documents, clean and attacked. The result was striking:\n\n| detector | clean | with invisible characters |\n|---|---|---|\n| twelve-feature baseline | 0.756 | 0.744 |\n| Binoculars | 0.781 | 0.618 |\n| Fast-DetectGPT | 0.787 |\n0.548 |\n\nThe best detector fell to 0.548. That is a coin flip. Meanwhile our deliberately weak baseline barely moved.\n\n**The ranking inverted.** On clean text the sophisticated methods clearly won. Under an attack\nthat costs nothing, the crude one won — because it never looks at tokens in the first place,\nso scrambling the tokenisation does not touch it.\n\nThat is a genuinely good story. It has a table, a reversal, and a villain you can demonstrate in ten seconds. We had the post half-drafted in our heads.\n\nThen we ran the control we had nearly skipped.\n\n## The control we nearly skipped\n\nBoth of those attacks are undone by *Unicode normalisation* — a standard cleanup pass that\nstrips invisible characters and folds lookalike letters back to plain ASCII. Our tool already\nhad the function. We had simply not been applying it before scoring.\n\nSo we added a “defended” mode: apply the cleanup to every document, then measure again.\n\n| detector | attack | clean | attacked | defended | recovered |\n|---|---|---|---|---|---|\n| Binoculars | invisible chars | 49.7% | 14.7% | 49.7% |\n100% |\n| Fast-DetectGPT | invisible chars | 51.3% | 18.1% | 51.3% |\n100% |\n| Fast-DetectGPT | synonym swaps | 51.3% | 50.0% | 50.0% | 0% |\n\n*(catch rate at a one-in-a-hundred false-positive rate)*\n\nFull recovery. Both detectors restored exactly, and the ranking flipped back to normal.\n\nThe collapse was never a property of the detectors. It was a property of *not running a\ncleanup pass*. The honest claim is not “invisible characters break the best detectors.” It is\n“invisible characters break the best detectors in any deployment that forgets to normalise its\ninput, and the repair is one function call.”\n\nThat is a duller sentence. It is also the true one.\n\nNote the third row, the one at 0%. Swapping synonyms is not a Unicode trick, so a Unicode\ncleanup does nothing about it — exactly as it should. That row is what makes the other two\nbelievable. A defence that appeared to repair *everything* would mean our harness was broken,\nnot that we had found a great defence.\n\n## What actually caught us\n\nNeither mistake was caught by being careful. Both were caught by machinery we had built for other reasons.\n\nThe first was caught because we eventually pointed the tool at a real corpus, and because the\ntool refuses to print numbers a small sample cannot support — so the twelve-document result\nhad `n/a`\n\nscattered across it, which made it obviously untrustworthy the moment we looked\nagain.\n\nThe second was caught by a control slice. And when we added it, we nearly left one cell out: we reasoned that a Unicode cleanup obviously cannot matter for a synonym-swap attack, so why spend an hour of GPU time confirming it? Then we checked, and found the cleanup alters 61 of 1,500 documents on that slice anyway — because real text is full of ligatures, curly quotes and lookalike characters that normalisation rewrites whether or not anybody attacked them.\n\nWhich produced a third finding we were not looking for: **the defence is not free.** It edits\nlegitimate documents too. Under the invisible-character attack it restores 1,439 of 1,500\ndocuments to their original form — not all of them.\n\nThere was a fourth catch, on the way in. Before publishing any numbers we checked our implementation of Binoculars against the authors’ own code, line by line, and found we had it wrong in two places: we were taking a perplexity measurement from the wrong one of the two models, and applying an alignment shift the original does not apply. The resulting scores were off by 4.6% to 12.4% — larger than the gap between the reference implementation’s own two recommended thresholds. Both versions produced completely plausible-looking numbers. Only comparison caught it.\n\n## The point\n\nWe are not going to tell you AI-text detection works. It mostly does not, at the confidence people want from it, and the failures fall hardest on people writing in a second language.\n\nBut the useful lesson from a week of this is narrower and more portable: **the exciting version\nof a finding is the one most likely to be wrong, and the thing that catches it is usually a\ncontrol you were tempted to skip.**\n\nEvery mistake above was caught by a rule that felt like bureaucratic overhead when we wrote it. Refuse to print numbers the sample cannot support. Measure the defended case, not just the attacked one. Check your implementation against the original before you quote it. None of those are clever. They are just the parts that were still standing when the exciting story turned out to be wrong.\n\nThe code, the corpus wiring, and both corrections — left in rather than edited out — are at\n[sv-pro/ai-detector-bench](https://github.com/sv-pro/ai-detector-bench).", "url": "https://wpnews.pro/news/our-benchmark-caught-us-twice", "canonical_source": "https://ai2rules.dev/blog/our-benchmark-caught-us-twice/", "published_at": "2026-08-05 00:00:00+00:00", "updated_at": "2026-08-15 07:42:49.541513+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-research"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/our-benchmark-caught-us-twice", "markdown": "https://wpnews.pro/news/our-benchmark-caught-us-twice.md", "text": "https://wpnews.pro/news/our-benchmark-caught-us-twice.txt", "jsonld": "https://wpnews.pro/news/our-benchmark-caught-us-twice.jsonld"}}