Why Detecting AI-Generated Text Is Harder Than You Think (And What I Built Anyway)

A developer built an AI-text detector that reports a calibrated confidence score and the specific statistical signals that fired, rather than a single accuracy claim, after finding that text lacks the robust watermarking available to image models. The tool combines perplexity, sentence burstiness and known-watermark probes, with verdicts calibrated by text length, and is available at detectaiwatermarks.com.

Every "AI detector" landing page promises 99% accuracy. Then you paste in a paragraph you actually wrote yourself and it flags you as ChatGPT. I kept seeing this in the wild — students wrongly accused, editors discarding human copy, and a pile of tools that were really just guessing. So I went down the rabbit hole of how AI-content detection actually works, built a tool to test the claims, and learned that the honest answer is far more interesting than the marketing. Image models can embed an invisible statistical watermark the SynthID-style approach , and that's a real, checkable signal. Text is different. A model generates tokens probabilistically; there's no natural place to hide a bit string that survives copy-paste. Researchers have proposed watermarking the logit distribution green/red token lists , but it breaks under: If a tool claims 100% accuracy on short text, it's lying. Anyone who's actually benchmarked one knows it. In practice detectors lean on a few weaker, statistical signals: None of these is a watermark. They're probabilities , and they fail on edited, mixed, or short content. That gap between "statistical likelihood" and "provenance" is where most products quietly pretend to be something they aren't. I got tired of the gap, so I built a detector that reports honestly — a confidence score plus an explanation of which signals fired, rather than a single fake certainty. You can try it here: https://detectaiwatermarks.com https://detectaiwatermarks.com The technical choices I'd highlight: python A naive "is this AI?" check fails. A useful one returns evidence. signals = { "perplexity": score perplexity text , "burstiness": sentence variance text , "watermark scan": probe known watermark text , often None, that's fine } Verdict is calibrated per-length: a tweet = an essay. verdict = calibrate signals, length=len text