Fixing rewards for NLA to reduce confabulation

A researcher testing Anthropic's Natural Language Autoencoder (NLA) found that improving reconstruction fidelity does not guarantee faithful interpretation of a language model's internal activations. Running Qwen 3.6-27B, the researcher more than doubled reconstruction FVE from 0.185 to 0.413, but an LLM judge found about 90% of the AV's specific claims (names, numbers, quotations) were unsupported by the actual input. The researcher concluded that NLA explanations can contain verifiably false claims, as the reward function never penalizes unfaithful statements, and that named entities carry most reconstruction information.

Hello, This is my first post on Lesswrong. Hope my contribution makes the world a better and safer place. Note: 1. This post is 100% human-written. 2. Full paper in preparation for ICLR 2027 The NLA The Natural Language Autoencoder Anthropic, May 2026 https://transformer-circuits.pub/2026/nla/ is a huge upgrade for the standard of mechanistic interpretability tool, SAE Sparse AutoEncoder . It uses two copies of target model: AV verbalizer and AR Reconstructor . The AV takes one inner vector and writes down what the model is thinking right now in English, and the AR reads the output of AV and regenerates the input vector. The AV is trained with RL after a warm start which is governed by pretty simple reward function: Where is the activation and is the natural language that interpret of the input vector. After the training, Anthropic evaluated their work with FVE Fraction of Variance Explained . An FVE of 0 corresponds to the mean activation , while an FVE of 1 is perfect reconstruction. Anthropic's checkpoint reached FVE of 0.6 to 0.8. When I ran their public checkpoints under a clean held-out split, they scored 0.712. The infrastructure was real and it worked as advertised. The NLA revealed that, if the natural language description carries enough information to rebuild the activation, then the description will be the key to the inner psyche of the LLMs. As a result, Anthropic's agents more effectively discovered root cause of the model's misalignment test model's behavior than SAE and blackbox baselines. However, the reward function never punishes the AV for its unfaithful claim. It only asks whether the reconstructor, the AR, can decode them. Anthropic flagged the concern in their paper, saying: "NLA explanations can contain claims about the target model’s input context that are verifiably false." One Lesswrong post https://www.lesswrong.com/posts/LQXWiF8PyJ5ojNsEv also showed that a LLM never sees the activation at all, could invent a description from the text and reconstructs about as well as the real AV that did see it. A good reconstruction score is not the proof that AV is interpreting the internal vectors. I measured it by myself with Qwen 3.6-27B local model. My run more than doubled reconstruction FVE from 0.185 to 0.413. However, LLM judge found that the about 90% of the AV's specific claims name, number, quotations are not supported by the actual input. Also, 10 LLM judges favored only 54.5% 45.2%, 63.5% of the time when showed the before-RL and after-RL results. The soar of FVE did not help us to read model's mind. Conclusion first: "NLA loves the detail". I questioned "Which words in the AV's description carries the most reconstruction information?", "Which words contribute most to the activation reconstruct?". To answer it, I took the trained AV's descriptions, rewrote them in two different ways and measured how much reconstruction survived through the frozen reconstructor. In this configuration, FVE shows how much information the rewrite has destroyed. The table averages over 400 samples: what gets replaced | \ Mean \space {\Delta FVE}\ | |---|---| the specifics: names, numbers, quotes, swapped for different enitities but the same category | −0.219 | the abstract content: topics, styles specifics kept | −0.072 | The entity-swap keeps every abstract word and replaces only the named things. For example, in one sample, Bing Maps becomes Stripe Payment , Redmond, WA becomes Austin. Reconstruction on that sample collapses −0.89 . The abstract-swap does the reverse, keeping Bing Maps and Redmond, while rewriting the framing "technical geospatial context" becomes "financial accounting scenario," "developer-oriented" becomes "professional/corporate-style" . Reconstruction barely moves −0.003 . The named entities are what the reconstructor is actually reading. Remember, the specifics were 90% false. AV tends to say "James is alive" when the actual text was "Mike is alive". But those incorrect details were carrying the most of information to reconstruction the activation. It is surprising that the AR actually relies on these false named entities. This is mostly because the named entities are the highest-bandwidth tokens available. I have tested this on the Anthropic's public checkpoints and found the asymmetry of is even stronger -0.394 vs -0.103 . So the confabulation of AV was holding the entire building to not collapse. We should keep those detail while making the incorrect to correct. where is a fact checking score between 0 and 1. Find the named entities names, numbers, etc in a description. Search the regex from original tokens where the activation came from. The g is the percentage you actually find. The AV said about Jake and Over 800 further training steps: steps with the term | fact checking | reconstruction FVE | |---|---|---| 0 Anthropic recipe, my baseline | 0.124 | 0.4126 | 400 | 0.268 | 0.4172 | 800 | 0.417 | 0.4194 | The fact checking reward soared from 0.124 to 0.417. Now it starts to speak the truth While preserving the reconstruction capability. I also evaluated with the same 10 LLM judges, this time they favored the model trained with new reward function 83.6% 75.6%, 89.4% of the time. The J-space research Anthropic, July 2026 https://www.anthropic.com/research/global-workspace identifies a small verbalizable subspace in activation space, carrying 6 to 7% activation energy. I split my reconstruction metric exactly into the part inside that J-space and the part outside. Across the reconstruction training, the inside part was nearly saturated from the start while the outside part almost tripled. About 95% of training gains live outside the J-space. Then I ran the same split on every checkpoint trained with the new reward. Surprisingly, the inside/outside split is almost identical diff under 0.1% even as the grew 3.4x. The reward fix changed which word to carry the information, but did not change the activation space where the reconstruction lives. This might hint that the faithfulness is a property of the text channel alone.