{"slug": "the-coin-flip-test-behind-ai-text-watermarks", "title": "The Coin-Flip Test Behind AI Text Watermarks", "summary": "Anthropic is embedding an invisible statistical watermark into text generated by its supported Claude models to comply with Article 50 of the EU AI Act, which requires machine-readable marking of synthetic content starting 2 August 2026. The method, published in 2023 and revised since, uses a secret key to bias token choices toward a 'green list' and a z-score test that flags text when the green-token count is far enough from chance; Anthropic says internal testing found no difference in quality, creativity or readability between watermarked and unwatermarked text.", "body_md": "Article 50 of the EU AI Act asks providers of AI systems, including general-purpose AI systems, that generate synthetic text, audio, image or video to mark their output in a machine-readable format and detectable as artificially generated or manipulated. The obligation carries exceptions, including for systems that only perform an assistive editing function. The regulation applies from 2 August 2026, with a transition period for AI systems launched before that date. Anthropic's models launched on or after that date mark at launch, and it is still adding marking to earlier ones. Anthropic weaves an invisible watermark into the text its supported models write, and around 190 signatories joined the EU's Code of Practice on Transparency of AI-Generated Content in July 2026. The detection method underneath all of it was published in 2023 and revised since, and it is a statistics test you can run with a coin and a calculator.\n\nAnthropic draws a line between two kinds of AI text detection. One reads the finished text and looks for stylistic \"tells,\" the way a detector software does. The other uses a key and a statistical test. The key-based method is the one Anthropic uses in Claude. It counts how many words landed on the green side of a coin flip that the model rigged in advance, then asks how surprising that count is.\n\n**What it is:** a hidden signature built into the words themselves. Nothing visible is added, the meaning does not change, and the signature travels with the text when you copy and paste it. It may survive light editing.\n\n**How it works:** the model picks a word at every step, and often two words fit just as well. \"Cold and grey\" or \"cold and overcast.\" Normally that choice comes from a random number, and the reader never cares which one won. A watermark replaces the random number with a secret key. In the statistical watermarks described in this article, the key splits the vocabulary into a green list and a red list, and the model leans toward green whenever the choice does not matter. The text still reads normally. The word choices now carry a pattern, and anyone holding the key can rebuild the green list for every position and count how often the text played along.\n\n**So what:**\n\nAnthropic says its internal testing showed no difference in quality, creativity or readability between watermarked and unwatermarked text.\n\nThe watermarking mechanism described here works at the level of the token, the piece of a word that a language model reads and writes. A token is a word fragment, not always a full word.\n\nThe detector runs one question: if the text had nothing to do with the watermark, how surprising is the number of green tokens?\n\nUnder the null model, every token behaves like a coin flip that lands green half the time. With T tokens, you expect half of them green, and the spread around that expectation is the square root of T times a quarter. That is the standard deviation, the natural width of the range you would see by chance.\n\nThe test statistic, called a z-score, is the distance between what you counted and what chance predicts, divided by that spread. In the general case, with a green fraction gamma:\n\nz = (green tokens observed − gamma × T) / square root of (T × gamma × (1 − gamma))\n\nWith gamma at 0.5, the formula simplifies to (2 × green tokens − T) / square root of T.\n\nA z-score of 4 means the count sits four standard deviations away from chance. That is where the original paper drew its operating line, because it keeps the rate of wrongly flagging human text at about 3 in 100,000, which is the one-sided normal tail probability for z above 4.\n\nTake a passage of 200 tokens, and suppose 160 of them landed on the green list.\n\nThat p-value is about 1.1 × 10^-17, which is roughly one chance in 93 quadrillion under the normal approximation. Writing 160 of 200 green tokens by accident with that key is very unlikely under the null model.\n\nHere is where the popular summary of this test usually goes wrong. It is often repeated that a modest bias becomes proof once a text passes 100 or 200 tokens. The arithmetic does not support that.\n\nThe z-score of a fixed excess grows with the square root of the token count. If the green rate runs a proportion d above chance, with d = 0.10 for ten points above chance, then z = 2d × square root of T. At 200 tokens and a 10-point excess, z = 2.83, which is one chance in 428 under the normal approximation. That is a real signal, and it is nowhere near the z = 4 operating line.\n\nSo length buys you reach, but the amount of bias still has to be there. A 1,000-token passage can give you evidence consistent with a 10-point excess. A 200-token passage needs 15 points for the same strength of claim.\n\nTwo other numbers from the paper frame the same point. The hardest setting, where the model may only write green tokens, produces a z-score of 4 from 16 tokens onward, since every token green means z equals the square root of T. The softer setting in the paper's own experiment, where the model gets a moderate nudge of 2 on a green list of a quarter of the vocabulary and then picks the best of eight candidates with beam search, averages a z-score above 5 by 35 tokens.\n\nThe derivation assumes the model had a real choice at each position. Some passages give it none. The 2023 paper shows two examples side by side, the sentence \"The quick brown fox jumps over the lazy dog\" and a line of code, for(i=0;i