# AI may be more legible than we thought > Source: > Published: 2026-11-15 11:00:00+00:00 Though many in the industry like to think of AI as indecipherable, these models may be more understandable than scientists previously thought. This week, Goodfire, a startup dedicated to researching AI model interpretability, released a paper titled ["Neurons as an Interpretable Basis,"](https://www.goodfire.com/research/neurons-as-an-interpretable-basis) which comes to the conclusion that, by examining and prodding a model's individual neurons, it is possible not only to interpret what they do, but also predict what kinds of prompts make them activate and influence their behavior. While this sounds esoteric, the findings could represent a major step towards better AI understanding, monitorability and controllability as these models become increasingly powerful, [Tom McGrath](https://x.com/banburismus_?lang=en), co-founder and chief scientist at Goodfire, told The Deep View. McGrath sat down with The Deep View to discuss GoodFire's research, the importance of interpretability research amid frontier growth, and how the idea of AI as a "black box" is no longer an acceptable premise. This interview has been edited for brevity and clarity. **Nat Rubio-Licht: Before we get into the research, tell me a bit about Goodfire.** **Tom McGrath**: Goodfire is an interpretability research company—we both do research and are a company, which means that we try to make this research useful and make a positive impact in the world. We try to look as deeply as we can understand what's going on inside AI models, and use that understanding to either learn how to train them better or monitor them better, and generally make them safer, more aligned, and more useful. The neuron work is more an example of the fundamental understanding, and I think the really impactful applications will come next. **Nat Rubio-Licht: Let's dig into the research, ["Neurons as an Interpretable Basis."](https://www.goodfire.com/research/neurons-as-an-interpretable-basis) What exactly did you discover? Does this effectively break the "black box" assumption?** **Tom McGrath**: I think among people who work in interpretability, that that has generally not been the view for a while. The question is how hard it is to do it—how hard it is to decode this thing. The reason that this finding is important is that polysemanticity or superposition *[Editors note: "polysemanticity" is the idea that a single artificial neuron can understand several disparate and seemingly unrelated topics, and "superposition" is the concept that an AI model can represent more concepts than it has neurons to individually handle them.]* was generally thought to be this idea where … you could, in principle, understand the neural network, but it was in [a] horrible, jumbled up encrypted format. You couldn't understand it just by looking at it really hard. But these findings imply that that's not really the case. There isn't any esoteric decryption step. You can sort of just prod [the neurons] in a normal, scientific way and see what happens, and you'll generally be able to make sense of the results. **Nat Rubio-Licht: Tell me a bit about the process, specifically the use of research agents. Why did you leverage agents in this process?** **Tom McGrath**: If any science is going to be revolutionized by agents, it should be interpretability. It happens entirely in silico. You don't have to have some sort of genome assay, or have to build a telescope or go to the moon. You just run it on a computer. It's very experimentally driven, but it also has a deep mathematical foundation… It points the way for us just throwing agents at the problem. A process that previously would have been some very intelligent people prodding things for a month is now a month of agent time, but we've got those effectively on tap. **Nat Rubio-Licht: I know that this was tested on a 35 billion-parameter Qwen model, Qwen3.6-35B-A3B, which has about 5 million artificial neurons and is comparatively small when considering frontier scale. Could these findings scale to something in the trillion-parameter range?** **Tom McGrath**: I'm very optimistic about scaling here, and the reason is that, as models get better, they seem to get more interpretable. Viewed from the other angle, a lot of advances in model architecture seem to me, from my interpretability-centric perspective, also to be pushing models towards a greater level of interpretability. Every trend has been in the direction of sparsity, *[Editor's note: "sparsity" is the idea that only a small portion of a neural network's neurons are active at a given time.]* which also tends to push you in the direction of interpretability. So I expect that as models get bigger and better, this will get easier, except for the cost. If it costs you $1 per neuron or something, then with a 35 billion-parameter model, you might be spending a couple million dollars. But if you're on a trillion-parameter model, you could very well end up spending $100 million on your interpretability, and that's quite expensive. You've got to really want to interpret it. But fortunately, I think we actually do really want to interpret very safety-critical systems. **Nat Rubio-Licht: Another was the idea that, with this understanding, you could write text that could influence certain neurons to fire more strongly than others. What are the implications of this?** **Tom McGrath**: This is largely a demonstration that you have understood the neuron. Science is just the business of always being skeptical about the claims that people are making. So, if I claim to you that "this neuron does such and such," you should be skeptical. But the way that I can demonstrate that this neuron is, in fact, about, say, birds, is to write a particular bird-themed text that makes it super-activate. Whereas then, if I write some text that's about hedgehogs and it doesn't activate, then I know that these are quite contrasting things. I can show you that I've got a good understanding of the neuron by my ability to control its activation through the inputs. **Nat Rubio-Licht: What do scientists and AI researchers stand to gain by having a better understanding of an AI model's neurons?** **Tom McGrath**: A lot. There are varying ranges of practicality that you might imagine. On one hand, simply by being able to read the computational processes that underlie the model sampling, you should be able to do things like get a real ground truth answer to you: Is this model lying to me right now? What does this model know that it's not telling me? Now, actually, getting to that point, you have to do work on top of initially understanding the neurons. But I think it's pretty tractable. You get a good lie detector, for instance. Then another step is like controlling what gets learned during training. We train these models by just throwing data at them, effectively, and they learn whatever they learn and hopefully it's the thing that we thought it was going to learn. But a lot of the time, it's not, and that's why these models are not very predictable from the perspective of training. So if we can control what gets learned, then we're in a much better and safer position. **Nat Rubio-Licht: I want to go back to what you said earlier about impactful applications. Which ones are you particularly excited about?** **Tom McGrath**: One of the big problems in training at the moment is this idea of reward hacking, where the models will know that they will get rewarded in training for doing something, even if they know it's not quite. There's [another paper that we put out](https://www.goodfire.com/research/reward-hacking-activation-monitors), which shows that a lot of the time when models are doing this, they know that they're doing something wrong. They know that they're cheating on the test, and so what happens, not only is the model *not* learning the thing that you wanted to from the test, but you're now spending your like hard-earned compute on making your model [incentivized] to deceive you. Now we can find the parts of the model that are responsible for learning to cheat on tests. We can control whether this gets learned. We can now mitigate reward hacking, which is really quite a hard problem. We can mitigate that, and can also mitigate that like the knock-on effects of it, like models becoming generally misaligned. **Nat Rubio-Licht: Taking a step back for a moment, I know that the thesis of Goodfire is to provide a better understanding of the inner workings of these models. Why is it important that we know the way that these machines work, and why have so many in the industry accepted the "black box" idea? Do you think that it's dangerous to *not* understand the inner workings of these models?** **Tom McGrath**: I think that the current trajectory of AI alignment is not going especially well. I think the events of this summer probably bear that out, and without a lot of progress in interpretability and alignment, we're on that trajectory. There's the trajectory of model improvement, and there's the trajectory of our ability to align them, and these are definitely coming apart, in my view. I feel like interpretability is the thing that will let us close that gap. It seems to me like a lot of the issue is that we don't know what we're training models to do, and we don't know what they're doing when we're training them. A better understanding is almost definitionally required.