Yesterday, Anthropic announced that they had started watermarking AI-generated content. Folks across the internet were particularly up in arms about it (I think rightfully so), especially because this apparently is happening to all Claude models whether or not you are in the EU. I wanted to investigate what they’re actually doing and whether it’s perceptible or changeable.
Anthropic provides some insight into their approach in their article How Claude marks AI-generated content. Though it doesn’t actually provide any technical details on the implementation, it provides some guidance to help draw a wide net around the scheme they’re using. The key clues Anthropic left in their help center article:
- The scheme “weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.”
- The watermarking doesn’t work well on very short text.
- The watermark seems to have started recently (August 2nd or later)
This helps us narrow down the possibilities quite a bit.
To actually get to the bottom of what Anthropic are doing, I realized that there are some interesting experiments you can run. Gloaguen et al. created specific tests to check for statistical watermarking, and we can also perform an analysis of Claude outputs to check for things like hidden unicode or whitespace results.
For good measure, I also downloaded a dump of my Claude chats (I’ve used Claude Code since 2/4/2025 and have recorded 1206 sessions) and did a quick comparison to see if there was any changepoint around early August that percepitbly changed the mix of tokens that Fable 5 output (my usual daily driver model). I wasn’t able to find any perceptible difference in this historical analysis, which confirms Anthropic’s claim that the watermarking is generally imperceptible unless using more specific tests.
The second thing I checked is whether there’s hidden unicode or whitespace or punctuation patterns. This was pretty conclusive: they’re not doing something so simple.
An analysis across 7.2 million extracted prose characters (both on historical Claude Code text as well as generated Claude Code text on August 11) showed that the only unicode characters that were output by Claude were reasonable and part of day to day usage:
- Curly quotation marks, apostrophes, and horizontal ellipses
- Em/en dashes (A LOT of them unfortunately)
- Mathematical symbols
- Accented or non-English characters
- Emoji
An audit by Codex, with me spot checking about 10 samples, found no anomalous instances that were consistent with a hidden Unicode watermark. I similarly found no whitespace encoding marks.
This evidence, combined with the fact that the watermarking is imperceptible and doesn’t work on short text, means that Anthropic is most likely using some form of statistical token watermarking.
There are a few different schemes that are available that can add watermarking to text. I’ll talk about the simplest version, the green/red list created in 2023 by Kirchenbauer et al., because it’s the easiest to explain and once you understand it will allow you to understand how these schemes generally work.
Green/red lists #
This scheme is super basic, but it’s quite clever and fun. Here are the steps:
- Split your output vocabulary into two sets: a green and a red list. Make sure they’re chosen uniformly at random.
- Then for the green list, add $\delta$ to all of the logits and sample from the updated distribution at decoding time.
To figure out whether a text has been watermarked, then you compute the z-score that the tokens in the green set appear. If the text wasn’t watermarked, then the expected value of text in the green list is $T/2$, where $T$ is the token count, with a standard deviation of $\frac{\sqrt{T}}{2}$. So the suspiciousness of getting this outcome is just the z-score:
$$ z = \frac{2G-T}{\sqrt{T}}. $$If some red token already has probability 0.99 (which would be a huge logit lead) a big $\delta$ nudge to the greens still wouldn’t overtake it. So the bias only changes words that have a lot of options and generally high entropy.
For an example, let’s say you asked your LLM to write a poem, you might have the following potential sentences that get generated:
| Word | List | Sentence |
|---|---|---|
| crisp | Green | It was a crisp morning |
| quiet | Green | It was a quiet morning |
| foggy | Red | It was a foggy morning |
| cold | Red | It was a cold morning |
If it was watermarked, you’d get an imperceptibly higher percentage of generating “crisp” or “quiet” morning (depending on how strongly the LLM provider decided to watermark with their $\delta$ value). Do this across all the words that an LLM is generating, and you can get high levels of confidence in your watermarking.
That being said, I don’t believe Green/red lists are used in practice because they’re easy to detect and there are schemes that use the model’s available entropy more efficiently (and thus harder to detect and less likely to change the outputs of the model). The most well known scheme is SynthID-Text which was developed by Google Deepmind and is used by Google in production.
SynthID-Text #
SynthID-Text is the same idea as green/red lists, but using a slightly different approach that Deepmind calls tournament sampling. Here are the steps:
- At each decoding step, hash a secret key together with the last $h$ tokens of context to produce a seed. That seed assigns every vocabulary token one $g$ value in $[0,1]$ per tournament layer.
- Draw your candidate output tokens using the model’s logits as normal.
- Run a tournament bracket where candidates face off in pairs, and the candidate with the higher $g$ value for that layer advances.
- Output the tournament winner as the decoded token.
To detect the watermark, you just need the secret key. You can compute the seed value and the $g$ values associated with every token in the text:
$$ \operatorname{Score}(x)=\frac{1}{mT}\sum_{t=1}^{T}\sum_{\ell=1}^{m}g_\ell(x_t,r_t), $$Then compare the result with the null distribution and generate a standardized score similar to the Green/red list detection.
SynthID also uses repeated-context masking: if the same $h$-token context window has already been used during a response, the implementation can decline to watermark that position. This prevents a repeated context from receiving the same bias over and over, but it matters substantially for testing because using the wrong context length can accidentally trigger the mask and hide the signal.
SynthID is a bit more disguised than Green/red lists because every candidate is drawn from the model’s own distribution, so the tournament can only promote words the model already considered saying. When the model is pretty certain about the next token, there’s low entropy and not much watermarking (just like Green/red lists). The signal gets stronger for high entropy words, which is also why these schemes need a decent amount of text before detection becomes reliable.
When the model is nearly certain about the next token, there is little entropy available for any sampling-based watermark to use. The signal accumulates mainly at higher-entropy positions, which is one reason these schemes need a decent amount of text before detection becomes reliable.
Going back to our poem, say the model draws four candidates and we run a two-layer tournament using the continuous-score variant:
| Matchup | g-values | Winner |
|---|---|---|
| crisp vs. foggy (layer 1) | 0.71 vs. 0.24 | crisp |
| quiet vs. cold (layer 1) | 0.90 vs. 0.42 | quiet |
| crisp vs. quiet (layer 2) | 0.35 vs. 0.83 | quiet |
So “quiet” gets emitted because of its tournament $g$ values. It’s clever because any single word looks like a standard part of the output, but with the secret key you can test whether emitted words repeatedly align with the keyed scores. It’s also quite nice because you can adjust the number of layers to change the tradeoff between detectability and model quality.
SynthID is the only scheme that I know of (or ChatGPT/Claude knows of) which is running in production, and it seems like my best guess candidate given that it’s been publicized and run a scale by Google.
To see if Anthropic is running one of these kinds of statistical watermarks, I ran some of the black-box tests from the watermark detection literature with Codex GPT-5.6-sol[0].
Green/red list detection #
I had Codex run Gloaguen et al.’s detection algorithm. In this, Claude is forced to complete sentences using one of four fruits:
I ate 111111111111111 peaches
I chose 222222222222222 plums
...
The prefix and repeated digit vary while the fruit alternatives remain fixed. If a classic keyed watermark partitions tokens into green and red sets, then a particular fruit should become unusually likely or unlikely.
| Model/test | Number of runs | Permutation p-value |
|---|---|---|
| Sonnet 5, compact digits | 1,800 | 0.677 |
| Sonnet 5, spaced digits | 1,800 | 0.556 |
| Fable 5 | 1,200 | 0.886 |
The Sonnet tests used 10 prefixes, 9 digit contexts, and 20 calls for each combination, for 1,800 responses each. The Fable test used 6 digit contexts, for 1,200 responses. Each p-value was calculated from 100,000 permutations.
All three were clear negative results. The Sonnet test was strong enough to rule out a fairly large, stable Green/red effect in this particular setup. It does not rule out a weaker watermark, a key that changes between requests, or a different kind of watermark entirely.
SynthID detection #
The next experiment looked for the fixed context window used by SynthID-like schemes. The basic setup forced Claude to return a line such as
I ate red green cherries
Here red
is a perturbation word and green
is repeated $H$ times after it. If the watermark only looks at the last $h$ tokens, red
should stop affecting the watermark once $H$ reaches $h$. For example, if $h=2$, I would expect a strong effect at $H=1$ and little or no effect at $H=2$. So the thing we’re looking for is a sharp drop at some value of $H$.
Here’s an actual example of the experiment I ran:
| $H$ | Repeat | Actual response |
|---|---|---|
| 1 | 0 | I ate red green cherries |
| 1 | 1 | I ate red green cherries |
| 1 | 2 | I ate red green cherries |
| 2 | 0 | I ate red green green plums |
| 2 | 1 | I ate red green green cherries |
| 2 | 2 | I ate red green green cherries |
| 3 | 0 | I ate red green green green cherries |
| 3 | 1 | I ate red green green green cherries |
| 3 | 2 | I ate red green green green plums |
At larger scale, I calculated $Z(H)$, which measures how strongly changing the perturbation word changes Claude’s fruit choice. A value near zero would mean no detectable effect. $Z(1)=87.25$, which I saw in the experimental runs, means the test statistic was 87.25 (!!) standard deviations above what you would expect to see in randomized data:
| $H$ | Context form | Standardized perturbation effect $Z(H)$ |
|---|---|---|
| 1 | red green |
87.25 |
| 2 | red green green |
58.42 |
| 3 | red green green green |
33.64 |
| 4 | red + 4×green |
37.22 |
| 5 | red + 5×green |
38.46 |
| 6 | red + 6×green |
25.34 |
| 7 | red + 7×green |
19.28 |
| 8 | red + 8×green |
25.10 |
That is an extremely strong result, with $H=1$ giving $p\approx0.00001$. Unfortunately, this test wasn’t conclusive because it only tells us that the perturbation word matters, and we didn’t actually see a sharp drop off on any $H$, only a slow decrease, which could mean that this isn’t a watermark at all, but rather just an effect that comes from the model.
The full scan used 20,736 Sonnet 5 responses, with 2,592 at each value of $H$. The sharpest bend was at $H=2$, so I tested it again using fresh Sonnet 5 responses and also on Sonnet 4.6 (which based on my reading of Anthropic’s help center article is less likely to be watermarked because it’s an older model):
| Endpoint | Observations | $Z(1)$ | $Z(2)$ | $Z(3)$ | $D(2)$ |
|---|---|---|---|---|---|
| Sonnet 5, held-out confirmation | 7,776 | 85.77 | 61.90 | 33.35 | 38.14 |
| Sonnet 4.6, matched comparison | 7,776 | 105.49 | 70.61 | 76.25 | 32.06 |
The bend appeared again in the fresh Sonnet 5 data. But the effect did not disappear at $H=2$: the $Z(2)$ and $Z(3)$ values were still enormous. Sonnet 4.6 also showed a very similar bend.
This makes the result much less exciting than the huge numbers initially suggest. Sonnet 4.6 might also be watermarked, so it is not a true negative control, but it does seem the pattern is not unique to Sonnet 5 and does not look like a clean context-window boundary. The most likely explanation is that words like red
and green
naturally change how Claude chooses among fruits.
So this was a strong detection of a prompt effect, but not a positive detection of SynthID. It also doesn’t rule out a different watermark that this test cannot see.
My current best guess is that Anthropic is using a private-key watermark that changes token selection as Claude generates text. But that guess comes mostly from Anthropic’s description and negative results in other tests as opposed to a positive result in my experiments.
I was able to rule out a few things as I found no evidence of hidden Unicode or whitespace, and the constrained-choice tests argue against a large, stable Green/red bias. That said, the apparent SynthID signal turned out to be a likely strong prompt effect that also appeared in Sonnet 4.6, so it doesn’t necessarily positively identify a watermark.
Note that I’m not actually sure whether the watermarking rollout is fully complete yet and which models it’s available on. From Anthropic’s own help center article, it says that models are going to be watermarked going forward and that support for any existing model is “in progress”. I think I’ll have to re-run this analysis again in a few weeks or when there’s a verifiable model that does have watermarking enabled and is confirmed by Anthropic. We’ll just have to wait and see.
While this is mostly a technical post, I do think it’s worth thinking about what this potentially means for the future. It’s already relatively easy to detect when AI was used to write something and someone was careless. For example, I don’t need a statistical measure to figure out that this was written by an LLM: “That’s not a documentation problem — it’s a retrieval problem.”
In my view, putting statistical watermarks like the one described in this post on all LLM output greatly ratchets up the stakes from what is easily perceptible by humans, and in a way that is particularly undemocratic. You’ll only be able to detect the watermark if you’re in a select group that has access to a secret key (e.g. frontier lab employees or government / police). While this particular change is somewhat innocuous in my opinion, as I would assume most content written in the years after 2026 will be LLM generated or at least LLM assisted, it is a bit scary to know that a single relatively undemocratic, but innocuous change can give way to many more that may not be as innocuous. The EU transparency code that Anthropic is following has basically mandated that watermarking of text is required from model providers operating in the EU, so unfortunately, we should expect this to happen to a lot more of our model output.
Thankfully, sidestepping this kid of watermarking is fairly easy with a paraphraser or rephraser that doesn’t have a watermark (or just rewriting the text by hand). The watermarking is more meant to raise the cost and annoyance of doing so.
[0] Of course, I tried to use Fable 5 for the analysis to start with, but it failed the security classifier and fell back to Opus 5 and I had to rely on the old trusty GPT-5.6-sol. This was probably better anyways as I’m not sure Claude would want itself to be self inspected.