Goodfire: 90% of AI Neurons Aren’t Black Boxes

Goodfire researcher Nick Cammarata published an essay claiming that roughly 90% of neurons in modern models including Qwen3-32B perform a single interpretable job, and that the canonical "cat-and-car neuron" in InceptionV1 is actually a V-shape detector rather than evidence of arbitrary superposition. Cammarata's reanalysis reports that InceptionV1's legible neurons rose from about 5% with primitive tools to 50% with better tools, 70% with modern regularization, and 99% under an equivalent ConvNeXt architecture, with no change to model size or training data. The essay reached the Hacker News front page with 354 points on the day of release.

Goodfire researcher Nick Cammarata published an essay today claiming that neural networks are far more interpretable than the AI field has assumed — and that the canonical evidence for the opposite was simply wrong. The famous “cat-and-car neuron,” cited for years as proof that networks arbitrarily pack unrelated concepts together, turns out to be a V-shape detector. Preliminary audits of modern models, including Qwen3-32B and trillion-parameter systems, show roughly 90% of neurons doing a single interpretable job. The post landed on Hacker News front page with 354 points on the day of release. The Neural Network Interpretability Myth: Cat-Car Was a Misread For nearly a decade, neuron 4e:55 in InceptionV1 stood as exhibit A for the AI black box problem. Researchers observed that the neuron activated for both cat faces and car fronts — apparently unrelated concepts packed into one unit through a mechanism called superposition. Anthropic built an entire research program around untangling these supposedly compressed representations using Sparse Autoencoders SAEs . According to Goodfire’s reanalysis https://already-clean.goodfire.com/ , the explanation is simpler: the neuron detects V-shapes. Cat faces have V-shaped ear and muzzle structures. Car fronts have them in grille and headlight arrangements. Flames make obvious V shapes. The neuron groups visually similar patterns — not arbitrary concepts. Downstream neurons read the V-shape signal; they never see a cat-car combination because none was stored. A related “superposition circuit” from the same era, which appeared to hide cars inside dog neurons, collapsed under re-examination for the same reason: cars and dogs share generalized pose patterns, not compressed features. The practical takeaway is blunt. The field’s most-cited evidence for network opacity turns out to be a tool failure, not a network failure. Primitive feature visualization from 2009–2017 consistently reward-hacked toward dogs and other high-signal features, obscuring the actual pattern each neuron was learning. Architecture Was the Problem, Not the Neurons The more striking finding concerns InceptionV1’s notorious uninterpretability in its final layers. Applying modern tools and regularization to the same model reveals a clear progression: InceptionV1 with primitive tools produced roughly 5% legible neurons; better tools brought that to 50%; adding modern regularization pushed it to 70%; switching to an equivalent ConvNeXt architecture jumped it to 99%. No change to model size. No change to training data. Just architecture and tooling. This matters because modern transformers — the architecture behind every production LLM — already ship with residual connections and LayerNorm by default. The networks that made interpretability researchers pessimistic were architectural outliers from a decade ago. The networks running in production today are structurally different from the start. Related: Mechanistic Interpretability: AI Learns to Explain Itself https://byteiota.com/mechanistic-interpretability-ai-learns-to-explain-itself/ Recursive Generalization: What Polysemanticity Actually Is Cammarata introduces recursive generalization to replace the superposition framing. A neuron detecting the sound pattern “iy” extends its coverage to “diy” which contains the pattern , then to “do it yourself” which shares the concept but not the sound . Each branch is a coherent conceptual extension, not arbitrary packing. Similarly, a tennis ball neuron tracked during training eventually sprouted into broccoli, dog breeds, and yellow flowers — all round and yellow-green. The branching is structured, not arbitrary. Preliminary audit of Qwen3-32B shows approximately 90% of neuron poles with one primary interpretable job, roughly 7% with two related jobs, and only 2–4% that remain unexplained. The pattern holds in trillion-parameter models. Cammarata’s “crossword puzzle” dynamic explains why: the more neurons are explained, the more each new explanation is constrained by surrounding context. Interpretability compounds. The Block-Sparse Featurizers paper https://arxiv.org/abs/2606.25234 published by Goodfire in July 2026 provides the technical foundation, finding that most neural concepts occupy only 2–4 dimensional subspaces — far simpler than full superposition predicts. The superposition hypothesis — that networks deliberately compress unrelated features when capacity runs out — now looks like the wrong default explanation. Cammarata does not claim superposition never occurs, but argues it should be the last hypothesis rather than the assumed mechanism. Already in Production One week before the essay, Goodfire launched “inside-out monitors” — small probes that tap neural activations during model inference to detect dangerous AI agent behavior. The economics are stark: monitoring one million agent exchanges on Kimi K3 costs approximately $185 using Goodfire’s probes, compared to $5,420 for cheap LLM-based monitoring and $200,000 for top-tier alternatives. The probes caught 93% of malicious hacking attempts with a 5.5% false positive rate and added less than 2% latency. According to TechCrunch’s coverage of the product launch https://techcrunch.com/2026/10/08/goodfire-says-its-new-inside-out-monitors-catch-rogue-ai-agents-at-a-fraction-of-the-cost/ , the approach works because interpretability probes reuse activations already computed during the forward pass — no extra inference required. The essay and the monitors make a combined argument: the essay explains why neural activations are legible enough to probe reliably; the monitors prove it works at production scale and at a cost teams can justify. Key Takeaways - The canonical evidence for neural network opacity — the cat-car neuron — was a V-shape detector misread by 2017-era visualization tools - Modern transformer architectures residual connections, LayerNorm are significantly more interpretable than InceptionV1-era models by default, with no changes required from today’s practitioners - Preliminary audits of billion- to trillion-parameter models show ~90% of neurons with a single primary interpretable function, suggesting alignment and debugging work is more tractable than assumed - Goodfire’s commercial monitors demonstrate production-grade interpretability is viable today at roughly 1,000x lower cost than LLM-based alternatives