Your AI Agent’s Reasoning Isn’t as Private as You Think

A team from the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems published a paper in August titled "Stealing Reasoning Traces from Proprietary LLM APIs" showing that encrypted reasoning objects from OpenAI, Anthropic, and Google were not tightly bound to a session, account, or model, allowing them to be replayed against weaker sibling models. The finding builds on a May report from cryptographer Matthew Green, who said OpenAI called the issue "unreproducible" and Anthropic said there were "no security implications," leaving the flaw unresolved for three months. The technique requires no cryptographic break: an attacker captures the encrypted reasoning object from a strong target model and reuses it in a request to a cheaper model.

Do you actually know whether your encrypted reasoning trace is private? Not “probably,” not “the docs say it’s encrypted so it must be fine.” Actually know, the way you’d know if a password is hashed correctly or a TLS cert is valid. I ask because I didn’t know either, until a few weeks ago. I build agents for a living, and I lean on extended thinking and reasoning models more than I used to admit out loud, because they genuinely make multi-step agent work more reliable. Every provider that ships a reasoning model hides the raw chain of thought from you and hands back something else instead: a summary, or an opaque signed object you’re supposed to store and replay on the next turn. I’d always assumed that object was a locked box. Turns out the lock was on the wrong door. This piece walks through two things more connected than they first look. The first half covers a real, disclosed vulnerability in how “encrypted” reasoning traces from major labs were bound, or rather not bound, to a session, an account, or even a model. The second half covers watermarking, the mechanism labs use to prove content came from their models, and what happens when you actually test how sturdy that proof is. Both stories land in the same place: a security assumption that looked solid on paper and had a gap nobody had gotten around to checking. If you’ve used a reasoning model, OpenAI’s o-series or GPT-5 with reasoning enabled, Claude with extended thinking, Gemini with thinking mode, you’ve seen the pattern. The model works through a problem internally and you don’t get to see that work directly. You get a summary, or in agentic workflows, an encrypted blob you’re expected to pass back on the next request so the model can pick up its reasoning without redoing it from scratch. There are two real reasons for this, not just vendor paranoia. One is competitive: raw chain of thought from a frontier model is close to a ready-made distillation dataset, watch a strong model reason step by step through hard problems enough times and you can fine-tune a much cheaper model to imitate a meaningful chunk of that capability. The second is safety: a model’s internal deliberation can contain half-formed content it decided not to say out loud, including reasoning about a jailbreak attempt it ultimately refused, and keeping that hidden lets a provider filter the final answer without sanitizing every intermediate thought too. So instead of throwing the reasoning away between turns, providers return it as a signed, opaque object. OpenAI calls it encrypted content in the Responses API. Claude returns signed thinking blocks. Gemini uses a field called thought signature. The client stores it and replays it on the next call. You never see the plaintext. That's the design, and on paper it sounds airtight. Here’s what turned out not to be true: none of these three implementations tied the object tightly enough to the session, the account, or even the specific model that generated it. Which meant you could take one, and just use it somewhere else. A team out of the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems, Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping, and Maksym Andriushchenko, published a paper called “Stealing Reasoning Traces from Proprietary LLM APIs” in August. It builds on an earlier report from cryptographer Matthew Green in May, who flagged the same replay behavior and, per the writeups I found, got told by OpenAI the finding was “unreproducible” and by Anthropic that there were “no security implications.” I keep coming back to that, because it’s the reason this sat unresolved for three months before it became a headline. The technique is a three step recipe, and none of it breaks any actual cryptography: query the strong target model normally and capture the encrypted reasoning object it returns, build a new request to a weaker, cheaper sibling model in the same family and inject the captured object as if it were that model’s own prior turn, then prompt the weaker model to “continue” or “transcribe” its reasoning, often with a small prefill trick to nudge it toward compliance, sampling repeatedly and reconciling the outputs. For Claude, that meant replaying signed thinking blocks captured from a larger model into requests sent to a smaller, faster one, using an assistant prefill to get it to treat the block as something it should just read out loud. For GPT models, the researchers injected encrypted content into a fabricated conversation history and sampled dozens of completions per trace, chunking requests to work around a rough verbatim-output limit. For Gemini, they attached a captured thought signature and used a similar prefill approach. The encryption itself, standard AEAD constructions, GCM or ChaCha based on IV length, stayed completely intact the entire time. Nobody cracked a cipher. The gap was that the ciphertext was never bound to anything that would stop a different, more compliant model from being handed the envelope and asked to describe what’s inside it. One detail made this concrete for me: the researchers compared their reconstructed reasoning token counts against what the APIs actually billed for thinking tokens, and the two tracked closely across every model tested, a strong signal the recovered text is the real trace, not an approximation of it. The theoretical version of this attack is interesting. What actually worried me is what happened when the researchers pointed the technique at data that already exists publicly, the kind of thing people post when sharing a debugging session, demonstrating an agent workflow, or reproducing a bug for an issue tracker. They scraped 6,708 public agent trajectories from GitHub and Hugging Face and decoded 315,320 encrypted reasoning blocks out of them. +---------------------------------------+-----------+| Artifact type recovered | Count |+---------------------------------------+-----------+| PII artifacts | 367 || API keys | 62 || Passwords | 33 || Access tokens | 24 || Private keys | 7 || Personal email addresses | 30 || Non-localhost IP addresses | 6 || Total credential-type artifacts | 182 |+---------------------------------------+-----------+ The number that stuck with me is a smaller one. Of 704 artifacts recovered from genuine, non-benchmark user sessions, 64 appeared nowhere in the visible chat history the person had actually posted. Not redacted, not truncated, just absent from what a human reading the transcript would ever see. That means either the model reasoned silently about something it never surfaced, or someone scrubbed a shared log before posting it and had no idea the reasoning block still carried the thing they thought they’d removed. Worth being precise about what this is: not a breach in the usual sense, nobody broke into a data center. It’s closer to committing a .env file to a public repo, except the secret wasn't sitting in plaintext where a quick grep would catch it before you hit push. It was inside an object every reasonable developer would assume was safe to paste into a GitHub issue, one API call to a cheap model away from being fully readable. There’s also a second thread worth naming honestly: circumstantial evidence of distillation. The researchers estimate decoding 10,000 traces at roughly 12,000 tokens each would cost around $720 at cheap-tier pricing, trivial next to what it costs to train or heavily query a frontier reasoning model. Cheap, high-fidelity access to another lab’s internal reasoning is exactly the kind of input that shows up in distillation pipelines, and the paper’s authors point to patterns in publicly observed model behavior consistent with, though not conclusive proof of, exactly that happening across labs. “Consistent with” is not “proof of,” and that distinction matters. But it’s the kind of detail that explains why this got patched fast once public, after sitting dismissed for three months as a mere privacy concern. All three vendors fixed the core issue after disclosure. Anthropic now ties reasoning blocks to the originating model. Google’s backend manages thought compatibility across model switches. OpenAI’s fix wasn’t fully detailed publicly, but researchers confirmed the main extraction path stopped working. The researchers’ own note adds a caveat worth sitting with: “similar attacks still seem possible” given how the underlying architecture works. They fixed the specific hole. The shape of the wall is the same. Here’s where the story bends toward something I could test myself, rather than just read about. If encrypted reasoning traces turned out to be less bound than assumed, the natural next question is what about the thing labs use to prove content came from them in the first place? Watermarking is supposed to answer “did an AI write this,” and after the EU AI Act’s Article 50 took effect on August 2 requiring providers to mark generative output in a machine-readable way, Anthropic rolled out invisible watermarking across Claude’s text, alongside C2PA provenance metadata for files and images. I wanted to know what that mark actually guarantees before trusting it, so I read the mechanism and tried to break it. Anthropic’s approach follows the same family of technique as Google DeepMind’s SynthID-Text, which is public and documented, so it’s the clearest way to explain the idea in plain terms. A language model doesn’t pick one single “correct” next token. At almost every step it’s choosing from a probability distribution over its whole vocabulary, and there’s usually a wide band of tokens that are all roughly equally reasonable. Watermarking exploits that slack. Instead of sampling from the raw distribution, the generation pipeline runs a pseudorandom function, the g-function, over the candidate tokens and nudges sampling toward tokens that score higher under it. The bias per token is small enough that a human reader can’t tell anything happened, but across hundreds or thousands of tokens it adds up into a statistical pattern. The g-function is seeded by a set of keys, typically twenty to thirty unique random integers, combined with an n-gram window ngram len, defaulting to 5 that determines how much recent context feeds into each token's score. A larger window makes the watermark more robust to editing but slightly easier to detect statistically; a smaller one is more brittle but harder to fingerprint. Detection works by running the same g-function over a candidate text with the same keys and checking whether token choices skew toward high g-values more than chance predicts. It's not a binary check: providers run a Bayesian detector returning one of three states, watermarked, not watermarked, or uncertain, tuned against a target false positive rate. Here’s the shape of it in code, using Hugging Face’s public implementation as a stand-in for how a provider’s internal version likely works: python from transformers import AutoModelForCausalLM, AutoTokenizerfrom transformers.generation import SynthIDTextWatermarkingConfigtokenizer = AutoTokenizer.from pretrained "google/gemma-2-2b-it" model = AutoModelForCausalLM.from pretrained "google/gemma-2-2b-it" watermarking config = SynthIDTextWatermarkingConfig keys= 654, 400, 836, 123, 340, 443, 597, 160, 57, 29 , ngram len=5, inputs = tokenizer "Explain how photosynthesis works." , return tensors="pt" output = model.generate inputs, watermarking config=watermarking config, do sample=True, max new tokens=200, print tokenizer.decode output 0 , skip special tokens=True If you’d rather not run a hosted model, the same mechanism works locally through Ollama or a local transformers pipeline, since the watermarking config is a generation-time hook, not something baked into model weights. You lose nothing testing it against a 2B or 7B parameter model instead of paying for frontier API calls. Text watermarking covers Claude’s written output. For images, SVGs, and other files, Anthropic instead attaches C2PA content credentials, the same Coalition for Content Provenance and Authenticity standard Adobe and Google use. A C2PA manifest is a cryptographically signed record embedded in or alongside the file, listing what tool produced it and, ideally, a chain of edits. The important nuance is hard binding versus soft binding. A hard-bound manifest is cryptographically tied to the file’s actual bytes, so any modification breaks the signature. A soft-bound one references the content by a hash stored elsewhere, more resilient to format conversion, but meaning the provenance record can, in principle, keep existing even after the local metadata is gone. Worth saying plainly, because it’s easy to misread a watermark as more than it is: a C2PA credential or a SynthID-style watermark tells you content passed through a specific model, not that the model generated it from scratch. Paste your own paragraph into Claude and ask it to fix a typo, and the output can still carry the mark, even though the ideas were entirely yours. Provenance, not proof of authorship. Reading how a mechanism works and testing it are different exercises, so I ran two experiments. The academic robustness literature on SynthID has already run rigorous versions of this test, and the numbers are a good baseline before my own rougher attempt. Running the Dipper paraphrasing model with both lexical diversity and structural reordering turned up a drop in detection quality that’s hard to argue with: +---------------------------+----------+----------------+| Attack | Baseline | After attack |+---------------------------+----------+----------------+| No attack control | F1: 1.00 | AUC: 1.00 |+---------------------------+----------+----------------+| Dipper paraphrase | F1: 1.00 | F1: 0.842 || lex=10, order=5 | | AUC: 0.91 |+---------------------------+----------+----------------+| Back-translation via | F1: 1.00 | F1: 0.711 || Chinese pivot | | TPR: 0.675 |+---------------------------+----------+----------------+ The back-translation result is the more interesting one. Translating a watermarked passage into Chinese and back into English preserves the meaning well enough for a human reader, but scrambles the exact token-level choices the g-function depends on, and detection quality fell further than paraphrasing alone. Oddly, worse machine translations degraded the watermark more than clean ones, since a rougher round trip disturbs more of the original token sequence. I ran a smaller, less rigorous version of the same idea on watermark-eligible Claude output: one straightforward paraphrase pass through a separate model, and one English-to-German-to-English back-translation using a local translation pipeline. I don’t have access to Anthropic’s actual detector, nobody outside the company does yet, so I can’t confirm a pass or fail against their real system. What I can say is the back-translated text read noticeably more stilted and generic than the plainly paraphrased version, which tracks with the finding that the attack degrading fluency most is also the one degrading the watermark signal most. Harder to detect and worse to read turn out to move together. The second test was more concrete, because unlike text watermarking, C2PA metadata is verifiable byte-for-byte rather than statistically. There’s an open-source tool called watermarks-remover, built by Guillaume Meyer, MIT licensed, that picked up over four thousand GitHub stars within two days of its August release. It's honest about its limits, splitting into layers: Layer A does deterministic removal of invisible Unicode characters and file-level metadata C2PA, EXIF, XMP , and Layer B attempts statistical text rewriting through a separate model, which the author calls a "best-effort attack, not a verified deletion," since without a public detector from any vendor, no tool can prove it defeats the real check. I cloned the repo and pointed it at a PNG from a Claude-integrated image pipeline: git clone https://github.com/guillaumemeyer/watermarks-removercd watermarks-removerpython3 service/scripts/inspect image.py shot.png The inspector reported an embedded C2PA manifest chunk in the PNG’s metadata, plus a handful of standard EXIF and XMP fields describing the generating application. Then the actual cleaning step: python3 service/scripts/clean image.py shot.png -o shot.cleaned.png Running the inspector again against the cleaned output showed the C2PA chunk, EXIF block, and XMP metadata all gone. Same visual image, byte-different file, no embedded provenance record left in it. That part is unambiguous and easy to verify yourself with any metadata viewer, exiftool shot.png before and after shows the same thing. What I can't tell you, and neither can the tool's own author, is whether Anthropic's or Google's server-side detection has a fallback path that doesn't depend on local file metadata at all, a soft-bound record checked against a hash stored elsewhere. That's the hard binding versus soft binding distinction from earlier, and it's the honest limit of what local testing can prove. The same metadata-stripping step works identically against images from any local diffusion pipeline, ComfyUI, Automatic1111, or a Docker-packaged Stable Diffusion setup, since the stripping logic operates on the file itself, not on how it was produced, so you can test this without touching a paid API at all. I didn’t set out to write about two unrelated topics. Reasoning-trace exposure and watermark removal are the same story told twice. In both cases, a security property that sounded absolute in the marketing copy, “encrypted,” “watermarked,” turned out to be doing real work but less work than the word implied, and the gap only became visible once someone bothered to actually test the boundary instead of trusting the label. “Encrypted” meant the ciphertext was genuinely opaque, but said nothing about whether the envelope was bound to the right recipient. “Watermarked” means the statistical fingerprint is genuinely there and hard to fake, but says nothing about whether an hour with a paraphrasing model can degrade it below a useful threshold, or whether a five-line script can strip the file-level record entirely. A few concrete habits came out of this. Treat provider-labeled security properties as claims to verify, not guarantees to build on. If a system you’re designing leans on “the reasoning trace can’t be read by anyone else” or “the output can be proven AI-generated” for a real decision, security, compliance, moderation, find out what’s specifically cryptographically enforced versus what’s an emergent property that happens to hold today. Audit anything you’ve already shared. If you’ve posted an agent transcript with reasoning enabled into a public repo, gist, or blog post, search it for encrypted content, thought signature, or long base64-looking strings in a thinking or signature field. If you find one, treat anything that session had access to as potentially exposed and rotate it, regardless of what's visible in the plaintext chat. Don’t rely on watermark detection as your only content-provenance control. If you need to prove something came from a specific model, keep your own signed generation logs server-side rather than depending entirely on a mark in the output, since that mark can be weakened by paraphrasing or stripped from a file outright. Strip reasoning blocks before you publish anything, not just the parts of a transcript you can see. If you’re writing a bug report with a real agent trajectory attached, describe the reasoning in your own words and drop the raw signed object entirely. You lose nothing a reader needs, and you remove the entire attack surface for that artifact. None of this is a reason to panic. Reasoning encryption and content watermarking both do real, useful work most of the time, against most casual attempts to bypass them. But “most of the time, against most attempts” is a very different claim from “private” or “provable,” and that gap is exactly where I’d want to know before building something that depended on it. Tags: ai-security, watermarking, llm-security, anthropic, chain-of-thought, c2pa, machine-learning, api-security Your AI Agent’s Reasoning Isn’t as Private as You Think https://pub.towardsai.net/your-ai-agents-reasoning-isn-t-as-private-as-you-think-593cbc64eb91 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.