# Why Detecting AI-Generated Text Is Harder Than You Think (And What I Built Anyway)

> Source: <https://dev.to/chenjiayan/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway-44lp>
> Published: 2026-09-24 02:16:42+00:00

Every "AI detector" landing page promises 99% accuracy. Then you paste in a

paragraph you actually wrote yourself and it flags *you* as ChatGPT. I kept

seeing this in the wild — students wrongly accused, editors discarding human

copy, and a pile of tools that were really just guessing.

So I went down the rabbit hole of how AI-content detection actually works,

built a tool to test the claims, and learned that the honest answer is far more

interesting than the marketing.

Image models can embed an invisible statistical watermark (the SynthID-style

approach), and that's a real, checkable signal. **Text is different.** A model

generates tokens probabilistically; there's no natural place to hide a bit

string that survives copy-paste. Researchers have proposed watermarking the

*logit* distribution (green/red token lists), but it breaks under:

If a tool claims 100% accuracy on short text, it's lying. Anyone who's actually

benchmarked one knows it.

In practice detectors lean on a few weaker, statistical signals:

None of these is a watermark. They're *probabilities*, and they fail on edited,

mixed, or short content. That gap between "statistical likelihood" and

"provenance" is where most products quietly pretend to be something they aren't.

I got tired of the gap, so I built a detector that reports honestly — a

confidence score plus an explanation of *which* signals fired, rather than a

single fake certainty. You can try it here: [https://detectaiwatermarks.com](https://detectaiwatermarks.com)

The technical choices I'd highlight:

```
python
# A naive "is this AI?" check fails. A useful one returns evidence.
signals = {
    "perplexity": score_perplexity(text),
    "burstiness": sentence_variance(text),
    "watermark_scan": probe_known_watermark(text),  # often None, that's fine
}
# Verdict is calibrated per-length: a tweet != an essay.
verdict = calibrate(signals, length=len(text))
```