I got burned by a client who swore my original writeup was "AI-generated." It wasn't. But the accusation cost me a week of back-and-forth, so I did what any annoyed engineer does: I ran the detectors against each other until I understood what they actually measure.
This is not a scientific paper. It's 40-ish samples, half human (mine), half generated, pushed through every free detector I could find without an account wall.
I used three buckets of text:
Every sample went through each detector twice, a day apart, because I wanted to see score stability. Some tools wobble 20+ points between runs on identical text. That alone tells you how much to trust a single number.
None of them reliably separated my casual writing from the model's casual writing. The tools that flagged human text as AI did it most aggressively on formal, low-personality prose. The more your writing sounds like documentation, the more likely you are to get falsely flagged.
The detectors basically measure surprise. Words and structures that are statistically common get read as "machine." Which means plain, competent writing is increasingly indistinguishable from generated text, and the only real defense is voice: specifics, opinions, errors of emphasis that no model would pick.
They're decent at catching wholesale generation. If a student submits a fully generated essay, most tools light up. It's the gray middle (edited AI drafts, AI-assisted outlines, human text that happens to be boring) where everything falls apart.
I keep a working list of which detectors drift and which hold steady, and I update it when tools change models. That list lives on my AI tools review page if you want the current state without re-running this yourself.
If you write for a living, the detectors are a fact of life now. Learn their failure modes instead of fearing the number. Notes: this writeup is part of a loose series I keep across projects. The money-angle version of these experiments lives at Extra Hustles, and I track the FIRE math of "is writing even worth it hourly" over at Firenomics. Gear that survives my abuse gets written up at Durable Picks, and the health experiments go to Hackedself. Code for the test harness sits in my repos.