The Coin-Flip Test Behind AI Text Watermarks

Anthropic is embedding an invisible statistical watermark into text generated by its supported Claude models to comply with Article 50 of the EU AI Act, which requires machine-readable marking of synthetic content starting 2 August 2026. The method, published in 2023 and revised since, uses a secret key to bias token choices toward a 'green list' and a z-score test that flags text when the green-token count is far enough from chance; Anthropic says internal testing found no difference in quality, creativity or readability between watermarked and unwatermarked text.

Article 50 of the EU AI Act asks providers of AI systems, including general-purpose AI systems, that generate synthetic text, audio, image or video to mark their output in a machine-readable format and detectable as artificially generated or manipulated. The obligation carries exceptions, including for systems that only perform an assistive editing function. The regulation applies from 2 August 2026, with a transition period for AI systems launched before that date. Anthropic's models launched on or after that date mark at launch, and it is still adding marking to earlier ones. Anthropic weaves an invisible watermark into the text its supported models write, and around 190 signatories joined the EU's Code of Practice on Transparency of AI-Generated Content in July 2026. The detection method underneath all of it was published in 2023 and revised since, and it is a statistics test you can run with a coin and a calculator. Anthropic draws a line between two kinds of AI text detection. One reads the finished text and looks for stylistic "tells," the way a detector software does. The other uses a key and a statistical test. The key-based method is the one Anthropic uses in Claude. It counts how many words landed on the green side of a coin flip that the model rigged in advance, then asks how surprising that count is. What it is: a hidden signature built into the words themselves. Nothing visible is added, the meaning does not change, and the signature travels with the text when you copy and paste it. It may survive light editing. How it works: the model picks a word at every step, and often two words fit just as well. "Cold and grey" or "cold and overcast." Normally that choice comes from a random number, and the reader never cares which one won. A watermark replaces the random number with a secret key. In the statistical watermarks described in this article, the key splits the vocabulary into a green list and a red list, and the model leans toward green whenever the choice does not matter. The text still reads normally. The word choices now carry a pattern, and anyone holding the key can rebuild the green list for every position and count how often the text played along. So what: Anthropic says its internal testing showed no difference in quality, creativity or readability between watermarked and unwatermarked text. The watermarking mechanism described here works at the level of the token, the piece of a word that a language model reads and writes. A token is a word fragment, not always a full word. The detector runs one question: if the text had nothing to do with the watermark, how surprising is the number of green tokens? Under the null model, every token behaves like a coin flip that lands green half the time. With T tokens, you expect half of them green, and the spread around that expectation is the square root of T times a quarter. That is the standard deviation, the natural width of the range you would see by chance. The test statistic, called a z-score, is the distance between what you counted and what chance predicts, divided by that spread. In the general case, with a green fraction gamma: z = green tokens observed − gamma × T / square root of T × gamma × 1 − gamma With gamma at 0.5, the formula simplifies to 2 × green tokens − T / square root of T. A z-score of 4 means the count sits four standard deviations away from chance. That is where the original paper drew its operating line, because it keeps the rate of wrongly flagging human text at about 3 in 100,000, which is the one-sided normal tail probability for z above 4. Take a passage of 200 tokens, and suppose 160 of them landed on the green list. That p-value is about 1.1 × 10^-17, which is roughly one chance in 93 quadrillion under the normal approximation. Writing 160 of 200 green tokens by accident with that key is very unlikely under the null model. Here is where the popular summary of this test usually goes wrong. It is often repeated that a modest bias becomes proof once a text passes 100 or 200 tokens. The arithmetic does not support that. The z-score of a fixed excess grows with the square root of the token count. If the green rate runs a proportion d above chance, with d = 0.10 for ten points above chance, then z = 2d × square root of T. At 200 tokens and a 10-point excess, z = 2.83, which is one chance in 428 under the normal approximation. That is a real signal, and it is nowhere near the z = 4 operating line. So length buys you reach, but the amount of bias still has to be there. A 1,000-token passage can give you evidence consistent with a 10-point excess. A 200-token passage needs 15 points for the same strength of claim. Two other numbers from the paper frame the same point. The hardest setting, where the model may only write green tokens, produces a z-score of 4 from 16 tokens onward, since every token green means z equals the square root of T. The softer setting in the paper's own experiment, where the model gets a moderate nudge of 2 on a green list of a quarter of the vocabulary and then picks the best of eight candidates with beam search, averages a z-score above 5 by 35 tokens. The derivation assumes the model had a real choice at each position. Some passages give it none. The 2023 paper shows two examples side by side, the sentence "The quick brown fox jumps over the lazy dog" and a line of code, for i=0;i<n;i++ sum+=array i , and asks which one a human wrote. The answer is hard, because both are low entropy, meaning the first few tokens strongly determine the rest. In low-entropy text, humans and machines pick the same word anyway. Push the watermark hard and you damage the text. Push it softly and there is almost no signal to count. The 2024 code paper found existing watermarking "fail to function appropriately in code generation tasks," and it split into two failure modes. A strong watermark broke working code. A weak watermark left too few green tokens to detect, because code blocks are short. Model-generated code averaged 59 tokens on HumanEval and 49 on MBPP, and the human-written solutions in those benchmarks averaged 58 and 64. The fix is to stop counting tokens where there was no choice to make. The 2024 SWEET method, Selective WatErmarking via Entropy Thresholding, computes each token's entropy, a measure of how spread out the model's uncertainty was at that position, and drops every token below a threshold from the count. The z-score is then computed over what remains, with the expected mean and the spread both recalculated on the filtered set. The thresholds are read off the entropy distribution of human-written code, using its median, mean and third quartile. On HumanEval those three points are 0.16, 0.50 and 0.68. On MBPP they are 0.18, 0.54 and 0.79. The mean entropy in both benchmarks sat around 0.5. The payoff was large. Detection accuracy on code reached an AUROC near 0.95, beating the 2023 method by 0.137 on HumanEval and 0.073 on MBPP, while catching 72.6 percent of watermarked code on HumanEval and 71.2 percent on MBPP, at a false-positive rate under 5 percent. For comparison, the post-hoc baselines tested there, which include DetectGPT, GPTZero and LogRank, scored an AUROC of 0.6 or below on code, with a true-positive rate near 10 percent. A 2024 follow-up from Yijian Lu, Aiwei Liu, Dianzhi Yu, Jingjing Li and Irwin King went further and replaced the hard cutoff with continuous weights, giving each token an influence proportional to its entropy, so a low-entropy token barely moves the score. That is where the two generations of methods part company. The 2023 test needs the key, but not the model, its parameters or its API. It never has to load or run the model, so a third party can run it on any text. The entropy-aware detectors need the model's own uncertainty at each position, which means running the model over the text. That needs access to a runnable compatible model, or to an equivalent interface that exposes the same uncertainty. The paper counts the cost as one forward pass over the text to compute the entropy. A watermark with a z-score of 8 does not prove authorship. It is very strong evidence that the text is consistent with a mark generated using this key. Anthropic states the boundary plainly: a watermark can only determine that Claude was likely involved with the content at some point, and it cannot distinguish text Claude wrote from text Claude heavily edited. The watermark attaches to the words the model chooses. Proofread a human paragraph and there is almost nothing left to attach to. Other limits follow from the same design. Short passages carry too little signal. Heavy editing, paraphrasing and rewriting can weaken or remove the mark, because the new text is built on different word choices. And a detector needs the key. Anthropic's detection API is in private preview, open to regulators, law enforcement, media, fact-checkers, independent researchers, educational organizations and EU civil society groups, plus companies with the same compliance duty. Claude's production watermark is a version of SynthID-Text, the Google DeepMind scheme published in Nature in 2024, and it belongs to the same family as a proposal by Scott Aaronson in 2022. Every member of that family shares one design principle, as Anthropic puts it: the watermark only changes the source of the randomness used to pick among words. Two things set the strength of a watermark, and both follow from the model above rather than from any published rule. How many tokens it covers, since the z-score grows with the square root of the count. And how many of those tokens the model could have chosen differently, since a token with one right answer carries no signal at all. Under those assumptions, the longer and more open the text, the smaller the bias needed to detect it, and the shorter and more constrained the text, the more careful the statistic has to be.