This week I packaged a small analyzer I had been testing as a Chrome extension: Copy Tell. It checks product descriptions and posts on Gumroad and Substack for the kind of stock phrasing an unedited AI writing tool tends to produce, and shows a small notice if enough of it shows up.
No ML model. Pure lexical and stylometric scoring: stock phrase density, symmetric list framing used as filler, sentence length uniformity, a few other signals. Everything runs in the page's own JavaScript context. Nothing is sent anywhere.
I tested it against a 100 sample labeled set: 50 AI generated product descriptions, 50 real human written descriptions pulled from published npm package READMEs.
Results:
tp 39, fp 1, tn 49, fn 11
precision 0.975, recall 0.78, f1 0.867, accuracy 0.88
Precision is strong. When it calls something likely AI, it is right 97.5% of the time. Recall is the honest weak point: it misses 22% of the AI samples outright. The 11 misses share a pattern, all written in a plainer, more literal register (a garden hose description, an area rug, a smart thermostat) with none of the stock phrasing the analyzer is built to catch. That is a real limitation of a rule based approach, not a bug I can patch away without adding actual language understanding, which is a different, heavier tool than this one.
One false positive too: axe-core's real README, flagged mainly for a single phrase a human author also happened to use. A fair miss, not cherry picked.
The test set itself has a gap worth stating plainly: it is AI copy versus real technical documentation, not real marketplace listing copy. Sellers tend to lean on superlatives more than a README does, so real world precision on actual Gumroad or Substack listings is probably lower than the number above. Not yet measured against real seller copy specifically. That validation is the obvious next step.
Extension is live and free: https://chromewebstore.google.com/detail/copy-tell-ai-description/ehpkbnkcpbfkagcfmbofldfnpnmopcbi
If you run it against your own listing and it gets a call wrong in either direction, I would like to know.