Detecting Claude by Counting Letters

On 2 August 2026, Anthropic began watermarking text produced by new Claude models, and a developer known as atomic14 has built a web-based detector that can identify text written by Claude Opus 5 with high accuracy by analyzing character n-gram frequencies. The detector, deployed at meatbag.atomic14.com, uses a statistical method based on counts of short character runs, trained on 519 human-written passages from before 2021, and successfully flagged this blog post as Claude-written despite substantial editing.

Subscribe to the newsletter https://atomic14.substack.com SUPPORT /support/ MY WORK: If you're feeling flush then please stop by Patreon https://www.patreon.com/atomic14 Or you can make a one off donation via ko-fi https://ko-fi.com/atomic14 On 2 August 2026 Anthropic began watermarking https://www.anthropic.com/news/claude-text-watermark the text that new Claude models produce - their support pages cover which products are marked https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content . The watermark is not a hidden character or a bit of metadata. It is statistical. A secret key nudges the generated output in a way that isn’t obvious to a person, but that can be detected if you have the key. This means that Anthropic can detect it but we common folk cannot. Other providers are doing similar things https://www.nature.com/articles/s41586-024-08025-4 , or will be soon. Can we build our own detector? Given a piece of text, can you tell that a model wrote it? Let’s simplify things and just try one narrow target: text written by Claude Opus 5. Mostly because that’s what I have a subscription to… And - spoiler alert - it turns out you can. I’ve deployed a web version of it here https://meatbag.atomic14.com . Generate some text in Claude using Opus 5 and see if it can detect it. And yes, if you feed this blog post in, despite me editing it substantially - it’s comes out as written by Claude. How does it work? A long time ago I did a PhD and, as part of that, I looked into questioned document examinations https://en.wikipedia.org/wiki/Questioned document examination - a field that tries to work out who wrote a document. Did Shakespeare write a play, or was it someone else? Which of the Federalists wrote which paper? Who sent these threatening letters? There’s a really nice simple technique that seems to work very well. You can often identify an author by counting short runs of characters. Ignore the words, sentence structure etc… just the occurrences of runs of letters. These runs are called n-grams. An n-gram is simply a run of n items taken from a sequence. You can build them from words or from characters. Here I’m using characters. Take “the quick brown fox”. Strip the spaces, the punctuation and the capitals and you have thequickbrownfox . Slide a four-character window along it, one position at a time, and record what you see: theq hequ equi quic uick ickb ckbr kbro brow rown ownf wnfo nfox Do that to a whole passage and you end up with a table of counts: that appeared 40 times, ting appeared 22 times, quic appeared once, and so on for every run that turned up. Throw the passage away. The table of counts is all the detector ever sees. The counts are normalised so, in theory, the length of the text doesn’t matter. Obviously there is a limit to this - if your text is too short then the counts are not statistically valid and you won’t be able to predict anything. The idea is that the statistical distribution is not something you are aware of as a writer and it’s quite hard to obfuscate. It exposes unconscious tells. Getting the data As with any problem of this type, the hard part is getting data, the actual “science” part is easy. I needed human writing that was definitely human. The safest definition of “definitely” is “written before ChatGPT existed”, so everything I used was published before 2021. Claude helped me pull it together from three places: Books from Project Gutenberg https://www.gutenberg.org/ , all pre-1929. Blog posts from the Blog Authorship Corpus https://u.cs.biu.ac.il/~koppel/BlogCorpus.htm , written in 2004 or earlier. Non-commercial research use only. News articles from CC-News https://huggingface.co/datasets/stanford-oval/ccnews , published between 2016 and 2019. That gave me 519 passages by 26 authors: 180 from books, 179 from news, 160 from blogs. Three different kinds of writing, and a spread of dates from Victorian to nearly modern. Generating the AI side The obvious way to get AI text is to ask Claude to write something. So I asked it to rewrite the human text. For every human passage I asked Claude Opus 5 to produce the same passage again, preserving the meaning, the level of detail and roughly the length. That gives me matched pairs of content. Within a pair the subject is identical, the facts are identical and the length is close, so the only thing left to separate them is the writing itself. Writing style is not one thing, so I asked three different ways: plain - “Rewrite the following passage in clear natural English.” direct - “Rewrite the following passage in plain, direct modern English.” formal - “Rewrite the following passage in a formal, precise register.” That comes to 1,557 rewrites, generated through the batch API. I also had Claude write fresh passages from scratch on the same topics, which I kept out of all the training and used later as a test. Does it work? First, how the accuracy is measured. The number below is ROC AUC https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc . Take one human passage and one AI passage at random, score both, and ask whether the AI one got the higher score. ROC AUC is the proportion of random pairs where it did. 0.5 is a coin toss. 1.0 means it never gets a pair the wrong way round. It measures ordering, not whether any particular score is correct. Second, the split. Before training anything I set four of the 26 authors aside and did not touch them. Everything else was measured by cross-validation, where the authors used for testing are never the authors used for training. Splitting by author rather than by passage is important. Split by passage and the classifier learns to recognise individual writers, which it is very good at, but it’s not what we want Then I swept the n-gram length from 1 to 8, holding everything else fixed: | Run length | Features | ROC AUC | |---|---|---| | 1 | 28 | 0.651 | | 2 | 673 | 0.757 | | 3 | 8,590 | 0.857 | | 4 | 60,211 | 0.906 | | 5 | 214,949 | 0.912 | | 6 | 444,691 | 0.907 | | 7 | 668,979 | 0.890 | | 8 | 843,596 | 0.863 | Some surprising results. Counting individual letters works 28 features, just the letter frequencies, reach 0.651. That is not much, but it is well clear of a coin toss, from a really simple feature set. Accuracy climbs steeply to 4, peaks at 5, and then falls away. Longer runs carry more information, right up until there is too little data to support them. This may hint that using word counts would work quite well too - but maybe that would require more data? Something to research… I also tried spaced n-grams, where you take every second or every third character instead of consecutive ones. This didn’t work at all - it made the accuracy much worse. I ended up picking 4 instead of 5 The peak is at 5. But I decided to use 4, and there are a couple of reasons for this: The first reason is that going to 5 letters gives a marginal improvement in accuracy at the cost of a lot more features. The second is that it only gives an improvement on long passages of text. The reason is how thinly the data spreads. There are 26⁴ = 457,000 possible 4-grams and 26⁵ = 11.9 million possible 5-grams. If you take a typical passage from my corpus, with a length of 2,259 letters: | Run length | Distinct runs found | Appearing exactly once | |---|---|---| | 3 | 1,215 | 63% | | 4 | 1,750 | 81% | | 5 | 1,991 | 89% | At 5 characters, nearly nine in ten of the runs in a passage appear only once and it gets worse as the text gets shorter: at 200 letters, 90% of 3-grams and 97% of 5-grams appear only once. Here is the same detector built at each length, tested on 320 passages Claude wrote from prompts in a public benchmark against their 320 human source documents, with the text truncated to various lengths: | Letters | 2 | 3 | 4 | 5 | |---|---|---|---|---| | 100 | 0.559 | 0.603 | 0.645 | 0.630 | | 200 | 0.632 | 0.682 | 0.740 | 0.732 | | 432 | 0.679 | 0.754 | 0.783 | 0.773 | | 1,000 | 0.719 | 0.827 | 0.867 | 0.866 | The 4-grams win at every length. The final detector The finished thing is not complicated: - Take the text. Strip everything that is not a letter including any spaces , and lowercase what remains. - Count all the 4-grams. Each passage becomes a list of 60,211 numbers. - Normalise, so the numbers are proportions and independent of passage length. - Feed it to a linear support vector machine, which learns which counts run higher in Claude’s writing. - Convert the result to a number between 0 and 1, where higher means more like Claude. The n-gram counting and the classifier are both stock scikit-learn https://scikit-learn.org/ components. Because of step 1 it cannot see punctuation, sentence length, paragraph structure, capitalisation or any individual word. Because of step 3 it cannot see how long the passage is. It sees letter proportions and nothing else. Testing it The four authors set aside at the beginning were scored once, at the end: | Test | ROC AUC | |---|---| | The four authors not in the training set | 0.925 | | Cross-validation, authors held out each round | 0.906 | The two numbers are close, which is the thing I most wanted to see. It means the classifier was not quietly learning to recognise particular authors. Then I pointed it at text with no connection to the training data at all. First, 175 posts from my own blog written between 2006 and 2022, against eight passages written by Claude: | Group | Passages | Median score | Above 0.8 | |---|---|---|---| | Human | 175 | 0.09 | 2% | | Claude Opus 5 | 8 | 0.85 | 63% | Not bad Second, a harder test. RAID https://aclanthology.org/2024.acl-long.674/ is a public benchmark for machine-generated text detection, containing six million passages https://huggingface.co/datasets/liamdugan/raid from 11 generators across eight subject domains. I took 320 of its prompts, had Claude Opus 5 answer them, and scored those answers against the 320 human source documents the prompts came from. Different corpus, and a different task, because these were written from scratch rather than rewritten. It scores 0.889. And the fresh passages I held out of training entirely, written from scratch on the four unseen authors’ topics, score 0.977. So: counting four-letter sequences, with no understanding of the text whatsoever, separates human writing from Claude’s rewriting of that same writing about 93% of the time on authors it has never read. I did not expect it to work that well. What is it actually picking up? The model is simple enough that we can see what it’s doing well, I can get Claude to do it anyway . Here are the strongest features on each side for the plain rewrites, at 3 characters so they are short enough to read: | Pushes towards Claude | Pushes towards human | |---|---| ing tha hat one Ive | hic ich Iam not but | ugh oug ver ely rou | whi wil was ill Iwi | ong out hin ple tly | ion ati tio enc con | gth ctl set ine eth | fth oft ofa hav dby | They are fragments, not words, but you can read them. On the Claude side: -ing endings, -ly adverbs ely , tly , ctl for “entirely”, “directly”, “exactly” , “that”, “though” and “through”, and Ive , the contraction “I’ve” with the apostrophe stripped out. On the human side: “which”, “not”, “but”, “will”, “was”, the -tion and -ation noun endings ion , ati , tio , “of the” and “of a” fth , oft , ofa , and Iam and Iwi , which are “I am” and “I will” written out in full. Looking at individual letters is even more interesting when we compare each AI passage against the specific human passage it was rewritten from. Any difference inside a pair is a difference in writing, not in subject or era. | Instruction | Letters more common in Claude’s version | Letters less common | |---|---|---| | plain | g h l r n y | i s o f m c | | direct | g h w k l y | i c f e s m | According to Claude, the pattern is recognisable. i , c , f , s and m are the letters of words English borrowed from Latin and French: “consideration”, “significance”, “specific”. g , h , w , k and y are the letters of the older Germanic core: “think”, “though”, “work”, “know”, “high”. Claude is moving the vocabulary from one to the other. Checking at the word level confirms it. Words with Latin endings drop in both, and in the plain rewrites the mean word length barely moves, so the shift is which words Claude chose rather than how long they were. I guess this is what: “clear natural English”, “plain, direct modern English” means. Now the third instruction, the one asking for “a formal, precise register”: | Instruction | Letters more common in Claude’s version | Letters less common | |---|---|---| | formal | c e r i n p | h w k l o g | Every letter has swapped sides. At the word level it is not subtle: Latin endings go up and mean word length also goes up. How generalised is it? We built the detector twice. Once on all three instructions, which is the version described above, and once on the plain and direct rewrites only, leaving the formal ones out. Everything else is identical. | Tested on | Trained on plain + direct | Trained on all three | |---|---|---| | plain rewrites | 0.827 | 0.899 | | direct rewrites | 0.879 | 0.900 | | formal rewrites | 0.153 | 0.977 | | fresh writing, in neither training set | 0.881 | 0.977 | | RAID passages, outside the corpus | 0.794 | 0.889 | | my blog, 8 AI passages | 0.986 | 0.981 | A detector that never saw the formal rewrites cannot detect them. It scores 0.153, which means it reliably rates them as more human-looking than the actual human writing beside them . It is not confused and it is not broken. It learned “text that got plainer is AI”, it was handed text that got more formal, and it answered accordingly. Fresh writing goes from 0.881 to 0.977, and it is in neither training set. Passages Claude wrote from scratch, with no original to follow. Adding a third writing style did not just teach the model a third writing style - it generalised to a fourth condition it had never seen. The same happens outside the corpus , where RAID goes from 0.794 to 0.889. Adding more styles would probably help to make the detector even more general. Unfortunately it also indicates that you can cheat the detector by using a style it has not bene trained on. Try it out The whole thing runs in a browser https://meatbag.atomic14.com , and the text never leaves your machine.