{"slug": "why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway", "title": "Why Detecting AI-Generated Text Is Harder Than You Think (And What I Built Anyway)", "summary": "A developer built an AI-text detector that reports a calibrated confidence score and the specific statistical signals that fired, rather than a single accuracy claim, after finding that text lacks the robust watermarking available to image models. The tool combines perplexity, sentence burstiness and known-watermark probes, with verdicts calibrated by text length, and is available at detectaiwatermarks.com.", "body_md": "Every \"AI detector\" landing page promises 99% accuracy. Then you paste in a\n\nparagraph you actually wrote yourself and it flags *you* as ChatGPT. I kept\n\nseeing this in the wild — students wrongly accused, editors discarding human\n\ncopy, and a pile of tools that were really just guessing.\n\nSo I went down the rabbit hole of how AI-content detection actually works,\n\nbuilt a tool to test the claims, and learned that the honest answer is far more\n\ninteresting than the marketing.\n\nImage models can embed an invisible statistical watermark (the SynthID-style\n\napproach), and that's a real, checkable signal. **Text is different.** A model\n\ngenerates tokens probabilistically; there's no natural place to hide a bit\n\nstring that survives copy-paste. Researchers have proposed watermarking the\n\n*logit* distribution (green/red token lists), but it breaks under:\n\nIf a tool claims 100% accuracy on short text, it's lying. Anyone who's actually\n\nbenchmarked one knows it.\n\nIn practice detectors lean on a few weaker, statistical signals:\n\nNone of these is a watermark. They're *probabilities*, and they fail on edited,\n\nmixed, or short content. That gap between \"statistical likelihood\" and\n\n\"provenance\" is where most products quietly pretend to be something they aren't.\n\nI got tired of the gap, so I built a detector that reports honestly — a\n\nconfidence score plus an explanation of *which* signals fired, rather than a\n\nsingle fake certainty. You can try it here: [https://detectaiwatermarks.com](https://detectaiwatermarks.com)\n\nThe technical choices I'd highlight:\n\n```\npython\n# A naive \"is this AI?\" check fails. A useful one returns evidence.\nsignals = {\n    \"perplexity\": score_perplexity(text),\n    \"burstiness\": sentence_variance(text),\n    \"watermark_scan\": probe_known_watermark(text),  # often None, that's fine\n}\n# Verdict is calibrated per-length: a tweet != an essay.\nverdict = calibrate(signals, length=len(text))\n```\n\n", "url": "https://wpnews.pro/news/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway", "canonical_source": "https://dev.to/chenjiayan/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway-44lp", "published_at": "2026-09-24 02:16:42+00:00", "updated_at": "2026-09-24 02:59:34.004420+00:00", "lang": "en", "topics": ["ai-safety", "ai-tools", "large-language-models", "natural-language-processing"], "entities": ["ChatGPT", "SynthID", "detectaiwatermarks.com"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway", "markdown": "https://wpnews.pro/news/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway.md", "text": "https://wpnews.pro/news/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway.txt", "jsonld": "https://wpnews.pro/news/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway.jsonld"}}