The Labs' Hidden Reasoning Was Never Actually Hidden

A new paper by Alexander Panfilov, Ilia Shumailov, Luca Beurer-Kellner, Jonas Geiping, Maksym Andriushchenko and colleagues demonstrates that encrypted reasoning traces from proprietary LLM APIs from OpenAI, Anthropic, and Google can be decrypted by weaker sibling models from the same provider without a jailbreak, exposing hidden chain-of-thought. The flaw stems from a single global key per provider, making the encryption a portable capability token that undermines anti-distillation and safety controls.

Security https://sourcefeed.dev/c/security Article The Labs' Hidden Reasoning Was Never Actually Hidden A weak sibling model will decrypt a frontier model's chain-of-thought for you, no jailbreak required. Emeka Okafor https://sourcefeed.dev/u/emeka okafor The pitch was always a little too neat. To stop rivals from distilling their frontier models and to keep dangerous chains of thought out of sight, the big labs stopped showing you the model's reasoning. But they didn't stop sending it. Instead of retaining traces server-side, OpenAI https://platform.openai.com/docs/guides/reasoning , Anthropic https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking , and Google https://ai.google.dev/gemini-api/docs/thinking hand the client an opaque, encrypted blob and ask you to pass it back on the next turn. Stateless, private, safe. Or so it looked. A new paper, Stealing Reasoning Traces from Proprietary LLM APIs https://arxiv.org/abs/2608.09867 by Alexander Panfilov, Ilia Shumailov, Luca Beurer-Kellner, Jonas Geiping, Maksym Andriushchenko and colleagues, takes that architecture apart. The finding is embarrassingly simple, which is exactly why it matters: those encrypted reasoning blocks are interchangeable across sessions, users, and — critically — across models within the same provider. The ciphertext isn't bound to who asked or which model answered. It's a portable capability token, and any sibling model in the fleet will happily open it. The attack is a favor, not a crack You don't break the cryptography. You ask nicely. Take an encrypted reasoning trace produced by a provider's strongest, most heavily guarded model. Paste it into a request to a weaker, cheaper model from the same vendor — the kind with looser safety tuning. The weak model treats the blob as legitimate context it's supposed to continue from, decodes it, and prints the reasoning back to you in plaintext. You never had to jailbreak the flagship. You laundered its own hidden thoughts through its little sibling. That inversion is the whole game. Every dollar the labs spent hardening their top models against extraction gets routed around by the least-defended endpoint in the same ecosystem. The paper demonstrates this across Anthropic, OpenAI, and Google — three independent implementations of the same bad idea. When the same flaw shows up in three separately built systems, it's not a bug someone shipped on a Friday. It's a design assumption everyone made. The likely root cause, discussed at length when the work hit Hacker News https://news.ycombinator.com/item?id=49257876 , is a single global key per provider that encrypts and authenticates every trace for every customer. Session-scoped keys would fix the cross-user replay, but they'd require server-side state — and that collides head-on with the Zero Data Retention guarantees the labs sell to enterprise. So the stateless design won, and "hidden" reasoning turned out to mean "encrypted with a key that unlocks everyone's." Why hiding the chain of thought was security theater Here's the editorial line, and I'll defend it: concealing chain-of-thought was never a real trust boundary. It was a UX and IP decision dressed up as a safety control, and this paper is the invoice. The consequences the authors enumerate all follow from that one mistake. Anti-distillation protection evaporates — the reasoning you were forbidden to see is one weak-model call away. Worse, the hidden trace often contains hazardous content the visible answer was tuned to refuse; the model reasons its way to the dangerous thing, then politely declines out loud while the receipts sit encrypted in your message history. And because the blobs pass authentication regardless of origin, an attacker can stuff a malicious payload inside one and seed it into public agentic datasets — an invisible prompt injection that detonates whenever someone replays the trace. The scale claim is the part that should worry anyone shipping agents. The team scraped 315,320 reasoning blocks from public repositories of agent rollouts and recovered 367 pieces of personally identifiable information and 182 credentials sitting in plaintext-recoverable reasoning. People have been committing these blobs to GitHub for months thinking they were inert ciphertext. They're not. They're decryptable transcripts of whatever the model was thinking, secrets included. What this actually means if you build on these APIs If you use OpenAI's Responses API with reasoning items, Anthropic's extended thinking with redacted blocks, or Gemini's thought signatures in multi-turn tool calls, you are already handling these objects. Three things change today. First: stop treating reasoning blobs as opaque or private. They are recoverable plaintext. Anything your app logs, caches, or — please no — commits to a repo is a data-exposure surface. Audit your agent frameworks for where reasoning items get persisted. LangChain, the OpenAI Agents SDK, and homegrown loops all tend to shovel the full response object into state; that state is now sensitive. Second: treat any reasoning block you didn't generate this turn as attacker-controlled input. If your pipeline ingests traces from a dataset, a shared cache, or another service, an adversary may have planted an injection inside one. The mitigation is the boring one that already works for the rest of prompt injection — don't grant a replayed trace any authority you wouldn't grant a raw user string. Third: don't build a compliance story on hidden CoT. If your threat model assumed the vendor's encryption kept reasoning confidential from you, your users, or an attacker, that assumption is void. Re-underwrite it. The uncomfortable part The disclosure timeline is the one place to be careful. Discussion around the release indicates that after the authors reported the flaws, the same attacks stopped working — which reads like the providers quietly tightened key scoping or trace binding. But none of the three has published an architectural fix or a postmortem, and "we can't reproduce it anymore" is not the same as "it's fixed for good." Notably, cryptographer Matthew Green said he'd tried similar replay attacks himself and couldn't get clean plaintext out of a mini model, then credited this team for taking it further — a useful reminder that the technique is finicky and version-dependent, not a universal one-liner. So caveat the permanence, not the lesson. The specific replay trick may be patched. The deeper truth isn't: hidden reasoning was sold as a safety and IP moat, and it was built on a shared key and a stateless-design shortcut. The labs can rotate keys and bind sessions. What they can't easily walk back is the pitch that your model's thoughts were ever truly out of reach. Sources & further reading - Stealing Reasoning Traces from Proprietary LLM APIs https://arxiv.org/abs/2608.09867 — arxiv.org - Stealing Reasoning Traces from Proprietary LLM APIs https://news.ycombinator.com/item?id=49257876 — news.ycombinator.com - Encrypted reasoning cracked across Anthropic, OpenAI, Google https://aiweekly.co/alerts/encrypted-reasoning-cracked-across-anthropic-openai-google — aiweekly.co Emeka Okafor https://sourcefeed.dev/u/emeka okafor · Security Editor Emeka has spent over a decade tracking threat actors, vulnerability disclosures, and the evolving landscape of application security, bringing a sharp continent-spanning perspective to his reporting. He's known for translating dense CVE advisories into clear, actionable context that developers and security teams alike actually read. Discussion 0 No comments yet Be the first to weigh in.