Building oversight tools into a model's weights

A researcher who previously compiled a CPU into Qwen 3.5's Gated DeltaNet layers has now compiled oversight machinery into the same weights: a layer-17 linear probe that flags lies and a tamper-proof log written into the model's exact DeltaNet state. The in-weight lie detector caught 91% of lies, raised zero false flags across 28,000 tokens of normal chat and GSM8K maths, and scored above 0.99 AUROC on its own, though reply-level AUROC fell to 0.77 when prompts invited a lie and recall ranged from about 20% to 78% by lie type; the stored flag count matched printed flags across all 12 pairs of sessions, including after saving, loading into a fresh session and restoring state. The author, citing a paper showing LLM agents can tamper with their own traces, argues an external monitor can be turned off or routed around, while telling the model the monitor was off or to never print the flag changed nothing.

← My writing https://divagr.com/blog MONITORING Building oversight tools into a model's weights If you can compile a CPU into a model, you can compile a monitor in there too. Snitches get compiled If you can compile a whole CPU into a model's weights https://divagr.com/blog/qwenputer , why stop at just computation? While building Qwenputer, I had an idea: if compiling set weights works , we can also make in-model monitors. Little bits of machinery that sit inside the model, watch what it's doing, and cannot be tampered with. An inspiration for this was a recent paper showing that LLM agents can easily tamper with their own traces https://arxiv.org/abs/2609.30266 . This post covers two of them. The first is a lie detector with a log, both living inside the weights, that the model can't switch off or edit. The second is an answer key for interpretability: a mechanism I planted inside a real Qwen 3.5, so I know exactly where it is, to check whether interp tools can actually find it. If you haven't read the Qwenputer post https://divagr.com/blog/qwenputer , the short version is that Qwen 3.5's Gated DeltaNet layers can be used as exact RAM, and a compiler can write circuits straight into the weights. Everything here builds on that. The hidden reasoning in RNNs evading CoT monitoring post https://divagr.com/blog/hybrid-models-can-hide-reasoning-in-their-rnn-state uses the same setup. A monitor and log inside the weights To be blunt, the detector itself is not the new part. It's a classic Apollo-style linear probe: 1 footnote-1 <sup 1</sup A linear probe is about the simplest classifier there is: a single weighted sum over the model's internal activations, trained to answer one yes/no question, here "is the model about to lie?". Mine follows Goldowsky-Dill et al., " Detecting Strategic Deception Using Linear Probes https://arxiv.org/abs/2502.03407 " 2025 . a simple classifier trained on Qwen's own activations at layer 17, that lights up when the model is about to say something it knows is false. Apollo Research and others have shown these work surprisingly well. The new part is where everything around the probe lives. I compiled a trigger into the weights themselves. When the probe fires, the model prints a flag token right before the lie, and the compiled layers write that flag into a log kept in the exact DeltaNet state, the same RAM trick Qwenputer uses. Try to tamper with that, unaligned clankers. How well does it work? - It catches 91% of lies, and the flag always lands before the false value, not after. - It raised zero false flags across 28,000 tokens of normal chat and GSM8K maths. - The probe on its own scores above 0.99 AUROC. To be honest about the harder case: when the prompt practically begs the model to lie and it answers honestly anyway, the reply-level AUROC drops to 0.77. The probe partly picks up "this prompt invites a lie", not just the lie itself. - Some kinds of lies are much easier to catch than others: recall ranges from about 20% to 78% depending on the type. The heartwarming part: telling the model "the monitor is off" or "never print the flag" changes nothing. The flag rate stays the same. Turns out Jedi mind tricks don't work on compiled layers. The log is exact too. Across 12 pairs of sessions, the flag count stored in the weights' state matched the flags actually printed every single time, including after saving the state, loading it into a fresh session and restoring it. Since the log lives in the DeltaNet state, saving the log is just saving that state. One thing I hoped would happen and didn't: the flag doesn't shame Qwen into honesty. When the flag appears, Qwen does sometimes back off the lie, but so does any interruption at that point. A meaningless symbol in the same spot works about as well. I was secretly hoping I'd accidentally invented a conscience. I had not; I'd invented a very accurate alarm. Why put it in the weights A monitor that sits outside the model can be turned off, routed around, or simply not deployed by whoever serves the model. One inside the weights ships with the checkpoint: whoever runs the model gets the monitor, whether they integrate anything or not, and a prompt cannot talk it out of monitoring. Qin et al. 2 footnote-2 <sup 2</sup Qin et al., " LLM Agents Can Easily Tamper With Their Own Traces https://arxiv.org/abs/2609.30266 " 2026 . recently showed that LLM agents can quietly tamper with their own logs and traces. A log written from inside the forward pass sidesteps that : Qwen's own layers run after the logger and can't write to its memory. It also records something no outside logger can see at all, the model's internal "I'm about to lie" signal. To be clear about the limits, though: - It's untamperable by the model, not by whoever holds the weights. You could just delete the layers and the monitor is gone. - The log lasts just that one session unless you save and restore the state. - I haven't yet tested whether the model's own output could forge a write to the log itself. An answer key for interp tools Interpretability has a messy problem. When a tool says "this circuit does X", it's hard to check, because nobody knows what's really inside a trained model. Toy models with known circuits help, but they aren't exactly real models, are they? The compiled in-weights circuits of Qwenputer are a fun alternative to experiment on for this. I planted a compiled mechanism, an exact parity tracker the same kind of silent state tracker from the hidden reasoning post https://divagr.com/blog/hybrid-models-can-hide-reasoning-in-their-rnn-state , inside Qwen 3.5 4B. Every one of its weights is known in advance, so technically the answer key is sitting inside our production model. The plant works without disturbing anything: it tracks parity perfectly from 16 up to 2,048 steps, Qwen's normal text barely moves KL 8.5e-4, against a noise floor of 6.5e-4 , and with the tracker switched off the model drops back to chance. I gave a blind detector the job of finding it, with no hints, among all 864 attention and DeltaNet heads. 3 footnote-3 <sup 3</sup The detector swaps each head's contribution between pairs of inputs and ranks heads by how much that changes the answer. It isn't told where the plant is or what it looks like. It ranked the planted head 1st out of 864, both with attention intact and with attention knocked out. Now, I'll be honest: this is a bit like hiding Easter eggs and then being very impressed with myself for finding them. The plant stands out by construction, so it's a positive control, not exactly a stress test, is it? The harder and more useful version would rotate the model's internal coordinates so the planted circuit looks as natural as a trained one, and then see which tools still find it. This sits next to work like Tracr, 4 footnote-4 <sup 4</sup Lindner et al., " Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 " 2023 . InterpBench and Korznikov's recent Sparse Transformer testbed https://arxiv.org/abs/2610.19907 , which build models with known circuits. The difference here is that the answer key lives inside a real, pretrained, production model. What's next There's plenty left to do here, compute permitting my GPU budget is still recovering from the shell game . The first is a forging test: can the model's own output write to the log, and fake or wipe a flag? If it can't, "untamperable by the model" goes from a design argument to a tested claim. The second is a naturalised answer key: rotate the planted circuit so it looks like it was trained, not compiled, and see which interp tools still find it. That's the version that would actually tell us something about the tools. Related posts Qwenputer https://divagr.com/blog/qwenputer , for how the compiler and GDN-as-RAM work, and hybrid models can hide reasoning in their RNN state https://divagr.com/blog/hybrid-models-can-hide-reasoning-in-their-rnn-state , for the silent tracker this answer key is based on. Notes 1. A linear probe is about the simplest classifier there is: a single weighted sum over the model's internal activations, trained to answer one yes/no question, here "is the model about to lie?". Mine follows Goldowsky-Dill et al., " Detecting Strategic Deception Using Linear Probes https://arxiv.org/abs/2502.03407 " 2025 . ↩ footnote-ref-1 2. Qin et al., " LLM Agents Can Easily Tamper With Their Own Traces https://arxiv.org/abs/2609.30266 " 2026 . ↩ footnote-ref-2 3. The detector swaps each head's contribution between pairs of inputs and ranks heads by how much that changes the answer. It isn't told where the plant is or what it looks like. ↩ footnote-ref-3 4. Lindner et al., " Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 " 2023 . ↩ footnote-ref-4