AI Detection Scores: Why the Percentages are Lying to You

AI detection scores are misleading because they measure perplexity and burstiness, not actual AI authorship, according to a technical writer who found their own hand-written documentation flagged as 85% AI. The false positive rate is extremely high for academic or technical prose, and the only way to validate scores is through side-by-side comparison with a known human-written baseline of the same genre.

AI Detection Scores: Why the Percentages are Lying to You The core issue is that these tools aren't "detecting" AI; they are measuring perplexity and burstiness. When a detector spits out a high percentage, it's essentially saying the text is highly predictable based on its training data. This creates a massive false positive problem for non-native English speakers or technical writers who use a standardized, formal style. I ran a test with a few paragraphs of my own highly structured technical documentation—written entirely by hand—and one tool flagged it as 85% AI. The "evidence" was simply that my sentence length was too consistent. If you're trying to build a reliable AI workflow or implement a content policy, relying on these scores is a recipe for disaster. Here is the reality of those numbers: Perplexity: High predictability = High AI score. Burstiness: Low variance in sentence structure = High AI score. False Positive Rate: Extremely high for academic or technical prose. The only way to actually validate these scores is to run a side-by-side comparison with a known human-written baseline of the same genre. If the "human" baseline also scores 40% AI, then the detector's 90% score on your target text is actually only a 50% delta, which is far from definitive proof. Stop treating these percentages as a "truth meter" and start treating them as a "predictability index." Next RAG for Business: A Practical Implementation Guide → /en/threads/3492/