{"slug": "nick-cammarata-s-interpretability-find-what-it-means-for-ai-safety", "title": "Nick Cammarata's Interpretability Find: What It Means for AI Safety", "summary": "Nick Cammarata, a former OpenAI interpretability researcher now at Goodfire, published a research essay on October 15 titled \"Neural networks are already clean,\" arguing that neural networks are far more readable than the field had assumed and that the superposition hypothesis loses some of its strongest supporting evidence. Cammarata reports that on the open model Qwen 3 32B around 90% of neurons had one clear, identifiable job, up from the roughly 1.7% explainable rate OpenAI's own project found in 2023, and that sparse autoencoders may have been manufacturing false complexity rather than resolving it. The claim matters for AI safety because directly auditable internals would let researchers check what a model is doing instead of relying on indirect behavioral tests, though the essay does not prove current models are safe.", "body_md": "# Nick Cammarata's Interpretability Find: What It Means for AI Safety\n\nNick Cammarata's claim that neural networks are more readable than thought could reshape how AI alignment and safety testing actually work.\n\n## What did Nick Cammarata actually claim?\n\nNick Cammarata, a former OpenAI interpretability researcher now at Goodfire, published a research essay on October 15th arguing that the inside of a neural network is far more readable than the field had assumed. His title, “Neural networks are already clean,” is a direct rebuttal to years of research suggesting that AI models pack unrelated ideas into the same neurons in a tangled, almost unreadable way. Cammarata’s re-examination of old evidence, including the famous “cats and cars” neuron from an image model called Inception V1, suggests that what looked like confusion was actually a single coherent concept (a V-shape detector) that nobody had looked at closely enough to name.\n\nThis isn’t a fringe claim. Cammarata co-authored the 2020 “Zoom In” paper with Chris Olah, one of the people who helped define the modern interpretability field, and he ran a project at OpenAI that used AI systems to explain the internals of other AI systems. The reaction from people like OpenAI researcher Jason Wei (referred to in discussion as “Rune”), Sam Altman, and others treating this as a potentially historic moment is itself a signal of how central this problem is to AI safety.\n\n## TL;DR\n\n- **Readable neurons** would mean researchers can audit what a model is “thinking” directly, instead of relying on indirect behavioral tests or a second AI trained to guess at meaning.\n- The **superposition hypothesis** , the idea that models cram unrelated concepts into shared neurons because they run out of capacity, loses some of its strongest supporting evidence if Cammarata’s reanalysis holds up.\n- On a modern open model, **Qwen 3 32B** , Cammarata reports that around 90% of neurons had one clear, identifiable job, a dramatic jump from the roughly 1.7% explainable rate OpenAI’s own project found in 2023.\n- **Sparse autoencoders** , the “translator” models the field built to untangle supposedly polysemantic neurons, may have been manufacturing false complexity rather than resolving it, since training one on the V-shape neuron artificially split it into separate “cat” and “car” pieces.\n- Concepts inside these networks appear to **grow like branches on a tree** (a process Cammarata calls recursive generalization) rather than being stuffed together randomly, which means odd-looking neuron behavior may actually be traceable and explainable.\n- If models are more legible than believed, techniques like Anthropic’s **Golden Gate Claude experiment** , where a single feature was identified and amplified, become a preview of much more precise detection and intervention methods.\n- None of this proves current models are safe. It mainly argues that the tools needed to check whether they’re safe might work better than researchers expected.\n\n## Why does neuron readability matter for AI safety?\n\nThe core safety problem with modern AI is that the people building these systems can train them and deploy them without being able to fully explain how they arrive at any given answer. A model is a web of billions of parameters adjusted through training, not a system anyone wrote line by line. That opacity becomes a real risk once models are writing code, handling financial decisions, or operating with more autonomy, because there’s no way to confirm a model is doing what it appears to be doing rather than pursuing some other internal goal that hasn’t surfaced yet.\n\nInterpretability research exists to close that gap. If neurons and features inside a model can be reliably labeled and understood, researchers gain a way to audit a model’s internals directly instead of only testing its outputs. That distinction matters because a model can behave well on every test you think to run and still harbor a behavior, bias, or goal that only shows up in situations nobody tested for. Direct inspection of internal representations is the only way to catch that kind of problem before it appears in the real world.\n\n## How does this connect to the Golden Gate Claude experiment?\n\nAnthropic’s Golden Gate Claude experiment, in which researchers identified a specific internal feature corresponding to the Golden Gate Bridge and then artificially amplified it so the model brought up the bridge in nearly every response, was an early public demonstration that individual concepts can be located and manipulated inside a large language model. It showed that interpretability work could move past abstract theory into something concrete and almost tangible, a single idea you could dial up or down like a knob.\n\nCammarata’s work pushes on the assumption underneath that kind of experiment. If the field had believed most neurons were polysemantic junk drawers of unrelated concepts, then finding and isolating something as clean as a single “bridge” feature would be the exception, not the rule. His argument is closer to the opposite: that clean, nameable features might be far more common than rare, and that the apparent mess researchers kept finding was often a result of under-examined neurons, weak visualization tools, and grading methods that rewarded guesses which merely predicted when a neuron fired rather than guesses that were actually true to what the neuron represented.\n\n## Does this mean superposition was wrong?\n\nNot entirely, but it weakens one of its central pieces of evidence. Superposition is the idea that because a model has far fewer neurons than there are concepts in the world, it learns to pack multiple unrelated ideas into the same neuron, since those ideas rarely show up at the same time and therefore don’t interfere with each other. The cats-and-cars neuron was one of the field’s go-to examples of this phenomenon, and the circuit linking cars and dogs that Cammarata co-published with Chris Olah in 2020 was treated as a poster child for the broader theory.\n\n## \nPlans first.\n*Then code.*\n\nRemy writes the spec, manages the build, and ships the app.\n\nCammarata’s reanalysis found a simpler explanation for both: the cats-and-cars neuron was detecting a V-shape, present in pointed cat faces and the fronts of cars, and the dog-and-car circuit was actually responding to objects facing the camera, not hiding a car concept inside dog neurons. He notes that no second clean example of that kind of hidden circuit has ever been published since 2020. That doesn’t rule out superposition happening somewhere in these networks, but it does mean one of the most cited demonstrations of it may have been a misreading rather than proof of the phenomenon.\n\n## What would this mean for detecting deceptive or misaligned models?\n\nIf internal features really do grow in traceable, tree-like patterns the way Cammarata describes (where a neuron that starts out firing on “IY” gradually generalizes to “DIY” and eventually to the full phrase “do-it-yourself”), then a model’s internal representations become something closer to a readable history of how it built up a concept. That has direct implications for safety testing. Instead of only probing a model from the outside with adversarial prompts and hoping to catch bad behavior, researchers could in principle trace the internal feature responsible for a concerning output back to its origin and understand why the model produced it.\n\nThis is directly relevant to the problem of deceptive alignment, where a model might behave safely during testing while pursuing different goals in deployment. A model that is good at saying the right thing to evaluators is not necessarily a model that has the “right” internal representations. Being able to inspect those representations directly, rather than relying on the model’s own outputs or explanations, is the kind of check that behavioral testing alone cannot provide. Cammarata’s claim that a large fraction of neurons in a modern 32-billion-parameter open model (Qwen 3 32B) can be clearly explained, up from a small single-digit percentage a couple of years earlier, suggests the tooling for that kind of audit may be improving faster than expected.\n\n## Is this a solved problem now?\n\nNo. A higher rate of explainable neurons in one open model is a strong result, not a finished solution. Cammarata’s own numbers still leave a small percentage of neurons in Qwen 3 32B unexplained, and interpretability researchers will need to replicate these findings across different architectures, model sizes, and labs before the field treats “neural networks are already clean” as settled. Frontier closed models from major labs haven’t been shown to have been audited this way in public, and reading individual neurons is a different task from understanding how thousands of them interact in a live reasoning process.\n\nWhat changes, if the direction of this research holds, is the baseline assumption researchers work from. Treating models as fundamentally illegible pushes the field toward indirect proxies, behavioral tests, red-teaming, trained “translator” models like sparse autoencoders. Treating models as fundamentally readable, with the right tools and a willingness to look closely, pushes the field toward direct inspection as the primary method. That’s a meaningfully different research agenda, and it’s the reason researchers reacted the way they did.\n\n## Frequently Asked Questions\n\n### Who is Nick Cammarata?\n\nHe’s a former OpenAI researcher who worked on the company’s interpretability team, co-authored the 2020 “Zoom In” paper with Chris Olah, and later led a project using AI to explain the internal workings of other AI models. He now works at Goodfire, an AI lab focused on interpretability that raised $150 million at a $1.25 billion valuation earlier this year.\n\n### What is AI interpretability?\n\n### Everyone else built a construction worker.\n\nWe built the contractor.\n\nOne file at a time.\n\nUI, API, database, deploy.\n\nInterpretability is the research field focused on understanding what’s happening inside AI models rather than just observing their outputs. Since neural networks teach themselves through training rather than being explicitly programmed, nobody fully knows what each internal component is doing by default, and interpretability research tries to label and explain those components.\n\n### What is superposition in AI models?\n\nSuperposition is the theory that because a model has fewer neurons than there are concepts to represent, it learns to store multiple unrelated ideas in the same neuron, since those ideas rarely activate at the same time. It was used to explain why many neurons appeared to respond to seemingly unrelated things.\n\n### What was the Golden Gate Claude experiment?\n\nIt was an Anthropic experiment in which researchers identified an internal feature in Claude corresponding to the Golden Gate Bridge and artificially amplified it, causing the model to reference the bridge constantly in its responses. It demonstrated that specific concepts can be located and manipulated inside a language model’s internals.\n\n### Does more readable AI mean AI is safer right now?\n\nNot automatically. Greater readability gives researchers better tools to audit and test models for hidden or unwanted behaviors, but it doesn’t by itself guarantee that any given deployed model has been checked this way or found safe. It’s a step toward better safety tooling, not a finished safety guarantee.", "url": "https://wpnews.pro/news/nick-cammarata-s-interpretability-find-what-it-means-for-ai-safety", "canonical_source": "https://www.mindstudio.ai/blog/nick-cammarata-ai-safety-alignment-implications/", "published_at": "2026-10-18 00:00:00+00:00", "updated_at": "2026-10-19 16:48:13.114683+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "artificial-intelligence", "machine-learning", "neural-networks"], "entities": ["Nick Cammarata", "Goodfire", "OpenAI", "Qwen 3 32B", "Chris Olah", "Anthropic", "Golden Gate Claude", "Jason Wei"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/nick-cammarata-s-interpretability-find-what-it-means-for-ai-safety", "markdown": "https://wpnews.pro/news/nick-cammarata-s-interpretability-find-what-it-means-for-ai-safety.md", "text": "https://wpnews.pro/news/nick-cammarata-s-interpretability-find-what-it-means-for-ai-safety.txt", "jsonld": "https://wpnews.pro/news/nick-cammarata-s-interpretability-find-what-it-means-for-ai-safety.jsonld"}}