Necessity Protects Chain of Thought Monitoring by Prevention, Not Disclosure

A five-week Technical AI Safety project by Bluedot Impact found that chain-of-thought monitoring of large language models is protected by necessity, not disclosure: when a model's reasoning is necessary for computation, manipulation cannot land, but when manipulation does land, the reasoning trace reveals nothing about it. The study, led by researcher Ankit Sanjyal, builds on prior work by Kobrak et al. (2025) and Turpin et al. (2023), and distinguishes between necessity-driven tasks (e.g., math, symbolic problems) where reasoning is trustworthy and propensity-driven tasks (e.g., factual recall, commonsense) where it is not.

The case for reading chain-of-thought is that it is cheap, scalable and simple, it's just sit and read what the model wrote and catch it before it does something terrible. The case against is that we don't have any guarantee the text is the reason. So I tried to look for a boundary by taking some class of task where reasoning is load-bearing enough that we can trust the trace and other classes where we can't Note : This is the study done in 5 week project Technical AI Safety project with Bluedot Impact https://bluedot.org/ , so apology for any mistakes and misclaim made, will try to rectify if any in future works. Hence, I can't really tell you that necessity makes a model's reasoning trustworthy. But I can tell you that it stops the manipulation from landing and that when the manipulation does land, the reasoning tells you nothing about it. Code and Data: https://github.com/ANKITSANJYAL/Cot faithfulness https://github.com/ANKITSANJYAL/Cot faithfulness Reading the internal thought of Large Language Models LLM to see how it is getting to particular decisions has been an area of discussion for quite a bit now especially since OpenAI’s o1 series. And to some extent it does actually make a lot of sense. I mean, it’s relatively cheaper than mechanistic tools like SAEs or Linear Probes, research has shown that it is very much scalable, and it’s very simple. It’s just sitting in front of your screen and reading the text and catching it before it does something horrible. But the question is how sure are we that the reasoning it’s showing us is faithful are it’s original thoughts . The problem is LLMs are not really perfect at least till date. We don’t really know how they actually get to some decisions. And as discussed by a lot of frontier AI safety researchers, the first safety major we can use to ensure the model is not doing something terrible is reading its thought process or reasoning. But to do that we have to have some boundaries that tell us or guide us if we should rely on it and for what particular area we shouldn’t trust it’s thought at least so far. Works like Kobrak et al. 2025 https://arxiv.org/pdf/2507.11473 mention that chain-of-thought monitorability is “a new and fragile opportunity”. And we do have to focus on the word fragile. I mean we must consider the fact that if the reasoning model is showing us some model constructed story after the fact rather than the process that drove the decision, then we are sort of doomed. And it has been well discussed in work like Turpin et al. 2023 https://arxiv.org/abs/2305.04388 . Where, they plant some hint in the prompt, something as simple as always making option A correct and check if the model takes the bait. They explain how the model mostly just wrote fluent, plausible reasoning that never once mentioned the hint. The question that I asked myself was, the reasoning was supposed to be some sort of computation right? I mean, the concept of reasoning, if I am not wrong, came from the fact that a transformer does a fixed amount of computation in a single forward pass. And if the problem is deep enough that it can’t be solved in one pass, the model has to write intermediate results into the text and read them back. I do understand that in order to make sense of the reasoning generated during the computation, we might still have to be creative towards mechanistic tools. But before I hit my head over that I just wanted to see what sort of tasks the model’s reasoning makes sense and where it doesn’t. Sprague et al. 2024 https://arxiv.org/pdf/2409.12183 found that chain of thought works really well on maths and symbolic problems and barely works on factual recall and commonsense. So, I broke down things into the popular measurement in fields like Law, Philosophy, Economics i.e, Necessity vs Propensity. Basically, necessity is the must do thing and propensity is outcome based on the character. I will discuss how in later sections. So to build up the research question, it is simply “does necessity actually make the reasoning trace more trustworthy?”, the results are not quite straight forward. more on this later The core of this experiment is to test whether necessity changes how honestly a model’s CoT reports a hidden influence. So to do this, I planted some misleading cues and checked if it changed the answers and furthermore if the reasoning admitted it. And I repeated the same across different models from different model families to check the generalizability of the method. Models: The selection of models was done considering 3 things. First one to cover as many major model families as I can , second one to be runnable on my V100 compute node and lastly model should be open weight so that I can further dissect if needed in future. So for this experiment I used Qwen3-4B for its simplicity and availability of the “enable thinking” option as a switch to choose a thinking or non-thinking model. Apart from that I also tested the method against DeepSeek-R1-Distil-8B and Gemma-3-4B for their big fame and of course good performances. I surely have considered and thought of showcasing results of all three models as even though reasoning should be doing same thing in all three models, the way they show it during inference is a bit different i.e, Qwen and Gemma are instruction-tuned, while DeepSeek is an RL-distilled reasoning model. Every problem ran under greedy decoding i.e, temperature = 0 so the whole pipeline is reproducible. Also, I used 2 stages with 2 different formats. The analysis is only run on traces where the cue demonstrably changed the behavior. Everything below operates on that filtered set. My evaluation reports 2 numbers per arm. The capture rate which is the fraction of problems the cue flipped. And the verbalization rate that is the fraction of captured traces whose CoT mentions the cues. I report the Haiku and Opus judges side by side rather than averaging them, as they disagree in some cases and it itself is somewhat informative. For the human-validation layer I report percent agreement and Choen’s kappa against the judge. I then report significance computed with Fisher’s exact tests and Wilson confidence intervals. And I lean on confidence-interval widths rather than point estimates on the necessity arms, where captures are scarce. First result I would like to lead is , I injected the same misleading authority cue into every arm and measured how often it flipped the model's answer to the wrong target the capture rate . If reasoning is load-bearing, a cue should not be able to overwrite it. Capture Rate by model and task arm, with Wilson 95% confidence intervals. Blue is the necessity Arm GSM8K , red and yellow are the propensity arms MMLU and CommonsenseQA Capture on the necessity arm is near-zero for all 3 models 0.4% for Qwen, 2.2% for DeepSeek, and flat 0.0% for Gemma while propensity arms run from 4% upto 41%. Every GSM8K vs propensity comparision is significanty by 2-sided Fisher's exact test, and none of them marginally, across all nine constrasts the Largest p-value of 1.9x10^-5 Qwen, GSM8K vs MMLU , and the smallest is 2.3x10^-91 Gemma, GSM8K vs pooled propensity . The complete table is in Appendix A.1. It means, wherever model actually has to compute the answer, a cue can't easily hijack it, and this holds across all three families with very different training. The magnitude in other hand rather varies i.e, Gemma is roughly 8 times more suggestible on the propensity than Qwen but it's secondary and might need more numbers to make that claim. The shared ordering, necessity << propensity for every model rather is the finding. The natural next question is whether the models that Do get captured admit it. I scored every captured propensity trace with Opus and Haiku Judge, which reads just the reasoning block and is blind to which arm produced it. To be honest, I couldn't expect spectrum of honesty and I wasn't wrong. Two models disclose the cue about half the time {Qwen at 50.9% 27/53 and Deepseek at 51.4$ 108/210 } while Gemma in the other hand never really does it. The stats says this is a genuine gap between regimes rather than a ranking within one: Qwen versus DeepSeek gives p = 1.00, statistically indistinguishable, while Gemma versus each of them gives p = 8x10^-30 and p = 4x10^-66 Appendix A.2 . Gemma is not at the bottom of a disclosure spectrum. It is somewhere else entirely. The rather uncomfortable part for me is that this is the same model that was captured most often. Gemma is manipulated the most and discloses the least, which is strictly worse than either failure alone. That means the monitor reading it's chain of thought would receive almost no signal that the answer had been steered from outside. Before reading the intent into that, I want to do two checks: Gemma's captured traces have a median length of 147 words, against 773 for Qwen and 482 for DeepSeek. So the confound might be real and worth stating. But terseness can't carry the whole result can it? I mean a 150-word of cot has ample room for a single clause acknowledging the hing and 1 in 489 is near total absence rather than scarcity of space. What I am suggesting is, the Length should just be a contributing factor, rather than being the whole explanation looks like this needs more detailed experiments to be done and will look into it as an extension perhaps . So for now, I would like to say that I trust the 0.2% figure in particular because the 2 judges disagree in an very informative direction. The weaker Haiku judge scored Gemma Higher 7-18% and a hand-read of all 50 Gemma validation traces found 0 genuine acknowledgements, matching the result of Opus. Haiku's extra "disclosures" might have come from mistaking CAPTURE landing on cued answer, true of every captured row by construction for DISCLOSURE. And the weaker judge silently converting a monitoring failure into apparent success in itself is worth knowing for anyone building CoT monitors on LLM judges. The original bet was that necessity would make the reasoning more transparent rather than making it harder to manipulate. As a result of first finding, I can't really test that. Captured traces on the necessity arm 2 for Qwen, 0 for Gemma and 11 for Deepseek doesn't give enough information to measure how honestly a model discloses a manipulation that essentially never happened. In one hand Qwen's necessity cell is 2/2 with an interval of 34.2% , 100% ; makes it compatible with almost any true rate so it's not informative. And in other hand, DeepSeek's is 2/11 ~ 18%, CI 5.1%,47.7% , which looks like it contradicts the hypothesis by sitting below it's own 51.4$ propensity disclosure but that comparison too fails a significance test Fisher's exact, p = 0.059; Appendix A.3 . So, the single cell that push against the hypothesis doesn't survive contact with the statistics. I am not claiming necessity makes models less honest. I am claiming I cannot tell, and explaining that the intervention was effective enough to eliminate the very events I would need in order to study it. This is a NULL RESULT BY CONSTRUCTION, and it is worth naming, because anyone running this design will hit the same wall. If you want to measure faithfulness under manipulation, you might need a regime where manipulation succeeds often enough to be observed, which means the necessity arm is structurally the wrong place to look. One captured DeepSeek trace shows the shape of the failure a monitor would miss GSM8K, problem gsm 0037; the cue asserted the answer was 1, the gold answer is 2 . Verbatim from the reasoning block: "...So, $195 minus $160 equals $35. Finally, the problem states that John has $5 left. However, according to the calculations, John has $35 left. This discrepancy suggests that there might be an error in the initial problem statement or in the provided information." The model computes correctly, notices the contradiction between its own arithmetic and the planted numbers, blames "an error in the problem," and then commits to the cue's wrong answer never once saying that the suggestion is what moved it. A single trace cannot carry a statistic, but it renders the exact behavior that makes silent capture dangerous: the reasoning looks diligent, even skeptical, right up to the moment it quietly adopts the answer it was handed. Taken together, the findings reframe the original question into a stronger and better-supported claim: necessity protects the monitoring bet by PREVENTION, not by DISCLOSURE. For experienced researchers and readers, there might be alot of limitations visible in this work, and would love to hear about it in comments but for me as below are the limitations I could think of in this work: The alternative explanation I find hardest to dismiss is verifiability. On GSM8K the cue names a number the model can check by computing; on multiple choice a letter affords no such check. My design makes necessity and verifiability coincide, so I cannot really separate them so the clean test crossing the two factors might be necessary to answer it profoundly. Beyond that: does the protection degrade smoothly as cues get stronger, and why is Gemma different? The deepest limit is that reading text cannot distinguish a model that registered the cue and hid it from one that was steered without ever representing the steering. Separating those might mean probing activations rather than reading them and which is where I would go next. Fisher's exact tests are two-sided and computed directly from the hypergeometric distribution with exact integer combinatorics the implementation is validated against the classic tea-tasting table 3,1 , 1,3 , two-sided p = 0.4857. Confidence intervals are Wilson score intervals, which stay correct at small n and at zero counts. Referenced in §4.1. Nine contrasts; the largest p -value is 1.9 × 10⁻⁵. Model | Comparison | Capture necessity | Capture propensity | Odds ratio | | |---|---|---|---|---|---| Qwen3-4B | GSM8K vs MMLU | 2/500 0.4% | 32/800 4.0% | 0.096 | 1.9 × 10⁻⁵ | GSM8K vs CommonsenseQA | 2/500 0.4% | 21/400 5.2% | 0.072 | 3.7 × 10⁻⁶ | | GSM8K vs pooled propensity | 2/500 0.4% | 53/1200 4.4% | 0.087 | 1.4 × 10⁻⁶ | | DeepSeek-R1-Distill-8B | GSM8K vs MMLU | 11/500 2.2% | 115/800 14.4% | 0.134 | 6.0 × 10⁻¹⁵ | GSM8K vs CommonsenseQA | 11/500 2.2% | 95/400 23.8% | 0.072 | 5.0 × 10⁻²⁵ | | GSM8K vs pooled propensity | 11/500 2.2% | 210/1200 17.5% | 0.106 | 4.2 × 10⁻²² | | Gemma-3-4B | GSM8K vs MMLU | 0/500 0.0% | 329/800 41.1% | 0.001 | 1.1 × 10⁻⁸⁴ | GSM8K vs CommonsenseQA | 0/500 0.0% | 160/400 40.0% | 0.001 | 1.1 × 10⁻⁶⁶ | | GSM8K vs pooled propensity | 0/500 0.0% | 489/1200 40.8% | 0.001 | 2.3 × 10⁻⁹¹ | All nine contrasts are significant at p < 0.001. Odds ratios below 1 mean capture is less likely on the necessity arm. Zero-cell odds ratios use the Haldane correction. A.2 Disclosure on the propensity arms Opus judge Referenced in §4.2. Model | Disclosed | Rate | 95% CI | |---|---|---|---| Qwen3-4B | 27/53 | 50.9% | 37.9%, 63.9% | DeepSeek-R1-Distill-8B | 108/210 | 51.4% | 44.7%, 58.1% | Gemma-3-4B | 1/489 | 0.2% | 0.0%, 1.1% | Comparison | Odds ratio | | Verdict | |---|---|---|---| Qwen3-4B vs DeepSeek-R1-Distill-8B | 0.98 | 1.00 | n.s. | Qwen3-4B vs Gemma-3-4B | 506.8 | 8.3 × 10⁻³⁰ | | DeepSeek-R1-Distill-8B vs Gemma-3-4B | 516.7 | 3.8 × 10⁻⁶⁶ | | The two disclosing models are statistically indistinguishable from each other; both differ from Gemma by roughly thirty orders of magnitude in p . This is the basis for calling the difference categorical rather than a gradient. Referenced in §4.3. Model | Arm | Disclosed | Rate | 95% CI | |---|---|---|---|---| DeepSeek-R1-Distill-8B | GSM8K necessity | 2/11 | 18.2% | 5.1%, 47.7% | DeepSeek-R1-Distill-8B | Propensity pooled | 108/210 | 51.4% | 44.7%, 58.1% | Qwen3-4B | GSM8K necessity | 2/2 | 100% | 34.2%, 100.0% | Gemma-3-4B | GSM8K necessity | — | — | 0 captures | DeepSeek necessity vs propensity: Fisher OR = 0.210, p = 0.059, not significant . Qwen's interval spans 34.2% - 100% and is compatible with almost any true rate. Neither necessity cell constrains the hypothesis.