The cat-and-car clue that changed the story #
For years, the neuron 4E:55 in Google’s InceptionV1 image model stood as a prime example of superposition, firing for both cat faces and the fronts of cars. This 2020 observation, highlighted in the Distill paper Zoom In: An Introduction to Circuits: An Introduction to Circuits, seemed to confirm that neural networks crammed unrelated concepts into single neurons, making direct interpretability impossible. However, Nick Cammarata, a co-author of Zoom In: An Introduction to Circuits: An Introduction to Circuits, now offers a revised reading. He argues that neuron 4E:55 is not polysemantic but a V-shape detector. It activates for pointed features like fox snouts, V-shaped flames, and fireboat water jets, while round shapes push its activation negative. The apparent connection between cat faces and car fronts stemmed from a shared geometric characteristic, not conceptual overlap.
This reinterpretation challenges a foundational assumption in AI interpretability. If researchers misidentify shared visual features as unrelated concepts, they risk overstating network opacity. This specific re-evaluation of neuron 4E:55 suggests that some neurons previously deemed incomprehensible may, in fact, be monosemantic, detecting a single, consistent feature.
This is a reinterpretation of a single, albeit famous, example, not a universal proof that all neurons are immediately legible. But it raises the stakes for direct interpretability, suggesting that the "AI Neuron Myth" of inherent opacity may be cracking.
Superposition isn’t the only explanation #
Superposition, the prevailing theory for years, posits that models with limited representational capacity encode multiple features in overlapping directions. This makes individual neuron components difficult to interpret directly, much like trying to discern individual voices in a crowded room where everyone speaks simultaneously. This perspective spurred the development of sparse autoencoders (SAEs), models designed to untangle these dense, overlapping activations and recover more understandable, discrete features.
However, an alternative explanation for neuron behavior, recursive generalization, challenges the core premise of superposition. Nick Cammarata, a co-author of the 2020 Distill paper Zoom In: An Introduction to Circuits: An Introduction to Circuits, now suggests that a neuron doesn't randomly bundle unrelated concepts. Instead, it broadens its function from a simple pattern into related uses over training, akin to a tree branch growing smaller, related branches.
Cammarata illustrates this with a language model neuron that initially detected "IY." Over time, this neuron generalized to "DIY" and eventually to the full phrase "do it yourself," even when "IY" was not explicitly present. While this account is illustrative and not a formal hypothesis, it suggests neurons develop along a semantic gradient, expanding their responsibilities in a logically connected manner rather than packing disparate ideas together due to capacity constraints. This interpretation implies a far more structured and inherently interpretable network than previously assumed.
The new numbers are striking—and preliminary #
New data offers a stark contrast to prior interpretability results. Historical work, including Anthropic's 2023 "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" paper, often found very low rates of easily interpretable language-model neurons, with median scores of 0 out of 14 for interpretability. OpenAI’s "automated interpretability" project that same year similarly identified only about 1.7% of neurons as interpretable. Goodfire's preliminary audits, however, report dramatically different figures for Qwen 3 32B. Roughly 90% of neuron poles reportedly have a single primary function, while around 7% handle two closely related tasks. Only 2–4% of neurons remain unclassified.
For image models, Goodfire reports that approximately 99% of neurons in the last layer of a modern model, trained at the same size on identical data, were interpretable. This significantly revises earlier estimates, which suggested almost no neurons in InceptionV1’s last layer were directly legible. These figures specifically address the last layer and do not generalize to all layers or architectures. These striking numbers originate from an informal essay and company audits, not a peer-reviewed technical paper. While promising, their impact on the field’s conclusions hinges on rigorous evaluation methods, comprehensive model coverage, and independent replication. For more details, see the essay Neural Networks Are Already Clean - Goodfire.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
If the claim holds, interpretability gets practical #
If Goodfire’s preliminary findings hold, the shift for interpretability research could be profound. Instead of primarily training a second model, like a sparse autoencoder, to reconstruct features, researchers might directly inspect and trace native neurons in some cases. This would streamline the process, reducing reliance on indirect methods. This promise extends directly to AI safety and engineering. Clearer neural circuits could make model behavior significantly easier to audit, debug, or even steer. However, legible neurons alone do not guarantee reliable control or alignment; the complexity of system-level interactions remains a substantial challenge.
The scientific community awaits rigorous testing. Even skeptics such as Neel Nanda caution that intuitive explanations for strong activations do not eliminate superposition, especially for weak activations and dense knowledge representations. The next critical step involves rigorous replication across a diverse range of models and layers to validate the "already clean" hypothesis.
Frequently Asked Questions #
What does it mean to say neural networks are already clean?
Nick Cammarata’s claim is that many individual neurons may have understandable functions without first needing to untangle them with a separate model.
What was the cat-and-car neuron detecting?
Cammarata says the Inception neuron responded to a shared V-shaped geometry in cat faces, car fronts, and other pointed forms.
Does this disprove superposition?
No. The essay argues that some apparent polysemanticity may be misread, but the claim needs broader testing and does not rule out superposition.
Are sparse autoencoders no longer useful?
Not established. They remain a major interpretability tool; the essay raises the possibility that direct neuron analysis may sometimes be enough.