Anthropic’s LLM watermarking

Anthropic has begun watermarking outputs of its Claude AI model using a scheme based on Google's SynthID and the Gumbel Softmax method proposed by Scott Aaronson in 2022. Aaronson, who credits Anthropic for acknowledging his contribution, notes that while watermarks can be removed with extra effort, recent advances in semantic watermarking may make removal harder, though no method is completely foolproof. The decision follows OpenAI's earlier choice not to deploy a similar scheme due to product risks.

Anthropic’s LLM watermarking https://scottaaronson.blog/?p=10032 So yeah, Anthropic has announced that it’s now watermarking the outputs of Claude https://www.anthropic.com/news/claude-text-watermark , using a scheme based on Google’s SynthID, which is in turn based on the Gumbel Softmax scheme https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-openai-2023-08-17 that I proposed at OpenAI back in 2022—as far as I know, the first LLM watermarking proposal, though far from the last one. I’m gratified that Anthropic credits me for this, even though I shirked my duty by never publishing a paper about it by the time I sat down to write one, it seemed like the whole field had already assimilated my scheme and moved beyond it—AI just moves too fast for me . For those who don’t know, watermarking means slightly changing the way that an LLM operates to insert a subtle signal that lets you prove later, with high statistical confidence, that a text indeed came from your specific LLM. It uses the randomness that’s already present anyway in LLM outputs, replacing some of it by pseudorandomness that favors certain word combinations over others in a way that’s later detectable, given only the sequence of tokens itself not the prompt or the probabilities along with the key of the pseudorandom generator. Christ, Gunn, and Zamir https://arxiv.org/abs/2306.09194 then substantially improved by scheme to get true cryptographic indistinguishability, and there have been other improvements since. I’d been meaning to blog about this for days. Thankfully, Zvi Mowshowitz, the world’s foremost blogger about AI, has now written a wonderful post, entitled AI Text Watermarking Is Free And Good https://thezvi.substack.com/p/ai-text-watermarking-is-free-and , which saves me from the need to write my own long post. In particular, Zvi masterfully explains the central point that I needed to explain to everyone back in 2022-23: why, contrary to many people’s intuitions, there’s no inherent tradeoff between watermarking and the quality of LLM output. Basically, nearly every LLM output was already a sample from a cloud of exponentially many possibilities, all of them about equally good, so there’s plenty of room to steer within that cloud without affecting anything that an ordinary user would notice. As my kids would put it, the math mathes. As Zvi explains, the central technical drawback of watermarking schemes like the one I proposed, and what Anthropic is now using, is that it’s possible to remove the watermarks with a little extra work even stuff as simple as, e.g., translating between English and French, asking the LLM for words interspersed with emojis and then removing the emojis, or using an open model to paraphrase the output . Zvi gives detailed arguments for why he expects watermarking to remain a net positive in practice despite this vulnerability. I could add that, in addition, there’s recent progress see here https://arxiv.org/pdf/2401.13927 for example on what I’ve called “semantic watermarking,” or watermarking at the level of the underlying concept vectors rather than the tokens themselves. This actually seems to work, albeit with no theoretical guarantees, and will hopefully make removing watermarks a lot harder—although the Barak et al. impossibility result https://arxiv.org/abs/2311.04378 suggests that under plausible assumptions, no LLM watermarking method will be completely foolproof. Anyway, I worked out my scheme in Fall 2022, then gave lots of talks about it including, as it happens, at Anthropic , and also worked with Hendrik Kirchner at OpenAI, who actually implemented and tested it. Unfortunately, OpenAI leadership decided against deploying our scheme, worried mostly about risks to the product i.e., customers disliking the idea, and leaving for a competing LLM that doesn’t watermark . You can read this Wall Street Journal investigation https://www.wsj.com/tech/ai/openai-tool-chatgpt-cheating-writing-135b755a from two years ago for more. I was hopeful that the State of California was going to solve the collective-action problem by mandating watermarking for AI models, but then they decided to do that for audiovisual content only https://www.ailawsbystate.com/blog/california-ai-transparency-act-sb-942 , for some reason exempting text. Nevertheless, Google DeepMind implemented something very similar to my proposal in its SynthID https://deepmind.google/models/synthid/ , deployed in all its Gemini text models. But they heavily restricted who gets to detect the watermark, which made their admirable decision of limited use to my academic colleagues, who’ve been begging me for a way to detect whether their students are using AI to cheat. For now, I mainly send them to Pangram https://www.pangram.com/ , a leading AI detector not based on watermarking, as a first line of defense. And now, apparently to comply with EU regulations, Anthropic says they’ve deployed a watermarking scheme like mine where anyone will be able to do detection though they also say in their FAQ that they’re still working on the detection API . Even OpenAI suggests that it plans to follow suit https://x.com/AndrewCurran /status/2087040578630606884 . So, four years after I seriously thought about this, it looks to my surprise like this is actually happening. Thanks, EU Tell you what: read Zvi’s post https://thezvi.substack.com/p/ai-text-watermarking-is-free-and , and then whatever questions you still have, you can come here and ask in the comments. Just please don’t use Claude to write the comments. With any luck, I’ll eventually be able catch you if you do.