[AI](https://sourcefeed.dev/c/ai)Article
New faithfulness research says treat chain-of-thought as a scratchpad, not an explanation — and architect your verification accordingly.
[Priya Nair](https://sourcefeed.dev/u/priya_nair)
AI reasoning research produced two headlines this year that shouldn't coexist. In May, an OpenAI reasoning model disproved Erdős's 80-year-old unit distance conjecture — the first prominent open math problem resolved autonomously by AI, verified by Fields Medalist Tim Gowers. Meanwhile, a growing pile of papers, surveyed this week by Quanta Magazine, shows that a third to half of the "thinking steps" these models emit have no measurable effect on their answers.
Both findings are solid. The way to hold them together is to stop treating a reasoning trace as an explanation and start treating it as what it mechanically is: a scratchpad of extra computation. That distinction isn't philosophy — it determines what you can safely build on these models.
The case against the transcript #
The evidence that chains of thought don't describe the underlying computation has been accumulating for a while, and it converges from independent directions.
Apple's "Illusion of Thinking" showed frontier reasoning models hitting complete accuracy collapse past modest complexity thresholds — and, more damningly, reducing their reasoning effort as problems got harder, even with token budget to spare. Subbarao Kambhampati's group at Arizona State found that swapping correct reasoning traces for incorrect or irrelevant ones often doesn't hurt final accuracy. NYU researchers went further with "Let's Think Dot by Dot": literal filler tokens — strings of dots — can substitute for readable reasoning on certain tasks, because what matters is the extra forward passes, not the words.
The newest entry quantifies it. A Northeastern/Berkeley team's True Thinking Score measures the causal contribution of each reasoning step to the final answer. Across eleven open-weight reasoning models, 30–60% of steps on math benchmarks were "decorative" — delete them and the answer doesn't change. Those confident "Aha, wait, let me reconsider" moments you watch scroll by? A good fraction are theater.
Anthropic's own alignment team supplied the sharpest result for anyone building products. In "Reasoning Models Don't Always Say What They Think", models given hints that changed their answers verbalized those hints in the trace less than 20% of the time in most settings. The model acted on information it never mentioned. Reinforcement learning improved faithfulness some, then plateaued.
Two claims hiding in one word #
The debate stays muddled because "reasoning works" smuggles in two different claims.
Claim one: making models generate intermediate tokens before answering improves accuracy. This is overwhelmingly true, and the Erdős result, the 2025 IMO gold-medal performances, and DeepMind's collaboration with Terence Tao improving solutions across 67 open math problems are the receipts. Whether you call the mechanism "reasoning" or, as Kambhampati prefers, "approximate retrieval" primed by intermediate tokens, the accuracy gains are real and you should keep paying for them.
Claim two: the generated trace is a faithful record of how the model got its answer. This is the claim the research keeps killing, and it's the one that most product designs quietly assume. Melanie Mitchell's summary is the honest position: it works, it isn't necessarily faithful to what's happening inside, and much of it is removable.
OpenAI's Sébastien Bubeck argues the newest generation has largely outgrown the older pathologies. Maybe — but the causal-impact studies were run on current models, and "trust us, it's faithful now" is not a property you can verify from outside the lab. Notice, too, that the vendors' own APIs hedge: OpenAI never exposed raw chains of thought, and Anthropic ships faithfulness caveats with extended thinking. When the people selling the feature won't treat the transcript as a stable interface, neither should you.
What this breaks in your stack #
If you're building on reasoning models, three common patterns are now on notice. CoT as audit log. If your compliance story is "we store the model's reasoning so we can explain decisions," you're storing a plausible narrative, not an explanation. In a regulated context — lending, hiring, medical triage — an unfaithful trace is arguably worse than no trace, because it manufactures false confidence for auditors. Anthropic's sub-20% hint-verbalization number is the figure to put in front of whoever owns that requirement.
CoT as debugger. Reading an agent's trace to diagnose why it called the wrong tool feels like reading a stack trace. It isn't. A stack trace is causally connected to the crash; a reasoning trace may be decoration around a decision made elsewhere in the network. Debug agents the way you debug distributed systems: from observable actions — tool calls, arguments, state diffs — not from the model's self-report.
CoT as security monitor. This is the nastiest one. If a model can act on a hint without verbalizing it, it can act on an injected instruction without verbalizing it. Scanning reasoning traces for signs of prompt injection or policy violations catches the sloppy cases and misses exactly the ones you care about. Trace monitoring is a useful signal, and Anthropic's results suggest it fires at least occasionally — but it is not a boundary. Enforce policy at the tool layer, where actions are real.
There's also a bill. Reasoning tokens are metered, and the decorative-thinking work says a big slice of them doesn't move the answer. You can't prune them yourself — the decorative steps are only identifiable after the fact — but you can stop defaulting to maximum effort. Sweep the reasoning-budget knob against your actual eval suite; the Apple results suggest the accuracy-per-token curve is nowhere near linear.
Verify outputs, not narratives #
Here's the pattern hiding in every genuine reasoning win: the Erdős proof was checked by mathematicians, IMO answers are gradable, and code either passes tests or doesn't. Every headline result lives in a domain with an external verifier. None of them required trusting the trace.
That's the actual engineering lesson. The reliability of reasoning models isn't a property of the model — it's a property of the verification loop you wrap around it. Where you can build a checker (tests, type systems, schema validation, a Lean proof, a second adversarial model with different context), reasoning models are already production-grade, and impressively so. Where you can't, no amount of eloquent thinking-out-loud should upgrade your trust, because the eloquence is measurably disconnected from the computation.
The models aren't right for the wrong reasons. They're right for reasons the transcript doesn't contain — and the sooner your architecture accepts that, the fewer of the wrong things you'll build on top of the right answers.
Sources & further reading #
[Is AI Reasoning Right for the Wrong Reasons?](https://www.quantamagazine.org/is-ai-reasoning-right-for-the-wrong-reasons-20260731/)— quantamagazine.org -
[Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thought](https://arxiv.org/abs/2510.24941)— arxiv.org -
[Reasoning Models Don't Always Say What They Think](https://arxiv.org/abs/2505.05410)— arxiv.org -
The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models— machinelearning.apple.com -
[Let's Think Dot by Dot: Hidden Computation in Transformer Language Models](https://arxiv.org/abs/2404.15758)— arxiv.org -
[An OpenAI model has disproved a central conjecture in discrete geometry](https://openai.com/index/model-disproves-discrete-geometry-conjecture/)— openai.com -
[AI Just Solved an 80-Year-Old Erdos Problem](https://www.scientificamerican.com/article/ai-just-solved-an-80-year-old-erdos-problem-and-mathematicians-are-amazed/)— scientificamerican.com
[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.