The “cat and car” neuron may not be a mess #
A long-standing concern in AI research has been the black-box problem: the difficulty of understanding how neural networks arrive at their conclusions. A key challenge has been polysemanticity, where individual neurons appear to activate for seemingly unrelated concepts. Researchers often attributed this to superposition, hypothesizing that models packed multiple features into single neurons due to resource constraints.
Consider the famous "cat and car" neuron from the 2014 Inception V1 image model. This neuron activated for both cat faces and car fronts, a prime example of apparent polysemanticity. However, Nick Cammarata, a researcher at Goodfire, re-examined this neuron, proposing that both concepts share a common visual primitive: V-shaped, pointed contours. This neuron also responds to V-shaped flames and fox faces, with O-shaped contours producing a negative response.
Cammarata’s reinterpretation suggests that what seemed like a chaotic mix of unrelated concepts might, in fact, be a coherent pattern. If correct, this means researchers may have fundamentally misinterpreted the function of some native neurons. Consequently, many tools and methodologies, such as sparse autoencoders, developed to "untangle" these perceived messes, might have been built upon a flawed premise.
This reframing holds significant implications for mechanistic interpretability. If neurons are inherently cleaner and more monosemantic than previously thought, it could unlock more direct and efficient methods for understanding AI models, potentially making the black box far more transparent.
A city map, not a tangled knot #
Cammarata proposes an alternative: recursive generalization. Instead of a jumble, a neuron extends from a core pattern to related examples, much like a city map with districts. One language model neuron, for instance, first activated for “IY” (as in “Miyazaki”), then “DIY,” and finally the full phrase “do it yourself”—a logical progression despite lacking the original “IY.”
His team’s preliminary audit of the Qwen3-32B language model provides compelling evidence. They estimate about 90% of neuron poles (the features that trigger strong activation or deactivation) have one primary function. Roughly 7% exhibit two related functions, and only 2–4% remain unexplained. These findings suggest a far more organized internal structure than previously assumed.
This clarity contrasts sharply with earlier assumptions. Researchers reported that approximately 99% of neurons in a modern image model’s final layer were straightforward to interpret, a significant jump from the “cat and car” neuron era. While interpretability methods and context always influence such assessments, the trend points toward inherent order, not chaos.
Readable neurons aren’t proof of understanding #
To validate these explanations, Cammarata’s team devised a rigorous test. An AI agent, given only a neuron’s explanation, generated examples designed to activate that specific neuron. These agent-generated examples strongly activated the target neuron 76% of the time, while unrelated examples produced strong activation only about 1% of the time. This suggests the explanations are highly predictive of neuron behavior.
Predictive power, however, does not automatically equate to full mechanistic interpretability. While an explanation that reliably forecasts activations offers compelling evidence, it doesn't inherently reveal the underlying causal mechanisms or explain how a model performs complex reasoning. Such explanations are useful, but they don't fully crack the black box of AI cognition.
Cammarata’s essay, “Neural Networks Are Already Clean,” presents its thesis as a working hypothesis, cautioning that the language-model audit is preliminary. For further reading, consult the full essay at Neural Networks Are Already Clean - Goodfire. Critics, including Google DeepMind’s Neel Nanda, urge restraint, emphasizing that these findings do not constitute a "significant update to the big picture of interpretability" without more comprehensive evidence. Even models with trillions of parameters remain largely unsolved, awaiting full audits.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
The real test is whether we can catch deception #
Eliezer Yudkowsky, a prominent AI safety researcher, voiced a strong counterpoint: merely labeling internal patterns does not equate to understanding a model’s reasoning. He argued that training against monitors could inadvertently teach models to conceal deceptive behavior, making true auditing impossible.
This distinction highlights that interpretability—reading a model’s components—is not the same as alignment, which ensures a model's goals are safe and its behavior transparent. Even legible neural circuitry offers no inherent guarantee against sophisticated deception, which models might learn to hide from human inspection.
Cammarata’s hypothesis, if correct, offers a genuine shift: neural networks may be far more legible than the “tangled knot” metaphor implied. Researchers gain a promising new laboratory for testing models directly, rather than relying on workarounds like sparse autoencoders.
Still, this newfound clarity in neural organization must be continuously checked against observed behavior and causal interventions. A more legible map of a model’s internal city is a powerful tool, but it is ultimately the model's actions that confirm its understanding and intent.
Frequently Asked Questions #
What is mechanistic interpretability?
Mechanistic interpretability studies a model’s internal components and computations to understand how they produce its behavior.
What does “Neural Networks Are Already Clean” claim?
Nick Cammarata argues that many neurons thought to mix unrelated concepts may instead detect a coherent feature or related set of patterns.
Does this mean AI interpretability is solved?
No. The reported results are preliminary, and readable neurons do not by themselves explain complex reasoning, prove causal understanding, or establish safety.
What are sparse autoencoders used for?
Sparse autoencoders are auxiliary models researchers use to extract interpretable features from neural network activations.