Neurons as an Interpretable Basis

Abstract #

A companion to Neural Networks Are Already Clean An intuitive way to interpret a neural network is to study its neurons; that is, the hidden units of its MLP layers. However, they have long appeared too complex and polysemantic for direct study. This view has led much of the field to focus on learned feature spaces over sets of neurons, which makes interpretability more challenging: we first need to transform the model before normal science can begin.

Building on a growing body of literature, our work suggests that neurons are more interpretable than previously assumed. We study the neurons of Qwen3.6-35B-A3B using sparsification methods, attribution tools, and research agents, and find three properties that make neurons interpretable units of computation:

First, they are already functionally sparse. Ablating the 92.2% weakest MoE neuron activations for each forward pass has only modest effects on model capabilities – a 2.9% increase in language model loss.

Second, neurons have relatively few semantic groupings (modes) of computational functions, and we can identify these and describe them concisely. The majority of neurons have either one mode only or a single mode that is overwhelmingly causally significant. In cases where multiple causal modes are present, the number is small and legible to our methods.

Finally, we can use agent-driven experimentation to generate neuron descriptions in natural language, and these descriptions allow us to predict, simulate, and activate the neuron. Providing the description to a judge model allows us to (1) predict whether the neuron will fire in a given passage of text with 95% accuracy vs random text; (2) simulate its token-by-token behavior in high-activation regimes sufficiently to recover 73% of the damage (as a proportion of a ceiling of damage restorable) from the neuron’s ablation; and (3) write text that causes the neurons to fire more strongly than anything observed in a 10B-token corpus.

Overall, we argue that neurons are meaningful units of computation that can be effectively used for interpretability in their own right, letting us study the model on its own terms rather than through a learned proxy.

Introduction #

In the previous essay in this series, we argued that neurons in deep learning models are more interpretable and cleaner than has often been suggested. In this article, we describe what this means in more detail and share our empirical findings. Throughout, the term “neuron” refers to an individual computational unit in the intermediate layer of a transformer MLP, whose output is a scalar activation, computed after the nonlinearity (SiLU), as explained in the Background section below.

Neurons were the first units through which deep networks were interpreted, and early work found them to be closely aligned with recognizable concepts (Le et al., 2012; Zeiler & Fergus, 2014; Bau et al., 2017; Olah et al., 2017; Radford et al., 2017; Goh et al., 2021). However, this story was complicated by observations of so-called polysemanticity—the idea that individual neurons respond to seemingly unrelated inputs (Nguyen et al., 2016; Olah et al., 2020). Polysemanticity is related to the proposal that more concepts are represented in a network than the number of neurons in the network, an idea canonized in the literature on “superposition” (Elhage et al., 2022; Arora et al., 2018; Park et al., 2023; Costa et al., 2025). This, in turn, led to the development of learned overcomplete feature dictionaries in place of neurons as computational units of analysis (Bricken et al., 2023; Cunningham et al., 2023). Despite these trends in the field, interpretability researchers continued to investigate respects in which neurons may still constitute a meaningful unit of analysis (Geva et al., 2021; Dai et al., 2022; Geva et al., 2022; Gurnee et al., 2024; Arora & Wu, 2025; Dravid et al., 2026; Feucht et al., 2026; Shabalin & Piotrowski, 2026). We continue this line of work, introducing new ways to factor and understand what individual neurons compute.

Whether a neuron is labeled monosemantic or polysemantic can depend on choices about what constitutes a single concept, and responses that initially appear unrelated may share a regularity that becomes apparent with a better understanding of the computations they support (Murphy & Medin, 1985). We therefore characterize neurons by their computational functions as our primary object of study, considering semantic similarity of the texts that activate them as a secondary question. We study the circuits in which each neuron participates, tracing how upstream neurons drive its activation and how downstream neurons use its output. This lets us examine how a neuron’s role varies across contexts and assess whether those roles admit a concise description.

A common view is that neurons are an intractable unit of analysis. In the worst case, an investigation could have demonstrated this in three ways: (1) Individual neurons might rarely play identifiable computational roles, so that meaningful units would be clusters of neurons, or directions and manifolds in activation space; (2) Each neuron might take extreme values on a wide and heterogeneous range of inputs that no short description unifies; (3) Or neurons might resist descriptions precise enough to predict when, and how strongly, they fire. We test each possibility directly: we ablate weak activations, determine the semantic range of neurons’ computational roles, and predict and simulate activations from descriptions alone.

Our results provide evidence that neurons are tractable units of analysis. Neurons (1) exert most of their downstream influence in high-activation regimes (above the 97th percentile at the positive extreme and below the 3rd at the negative); (2) have a small number of computational roles, distinguished by their interactions with other components of the network; and (3) can be given concise descriptions that predict and simulate their activations. The three results share a pattern: the stronger a firing, the more the model depends on it and the more precisely we can describe it.

The picture that emerges is not that every neuron means one thing, but that a neuron’s important firings have a small set of computational functions, and a short description captures them. Some neurons are genuinely polysemantic, and some behavior remains unexplained, but so far little of it appears critical for the model’s computation.

We therefore find that neurons are a useful and interpretable basis for understanding LLMs.

Some Definitions

One neuron, one token

• Write
• The neuron's contribution to the residual stream, a · w_router · W_down [:, n]: its column of the down-projection matrix, W_down [:, n], scaled by its expert’s router weight, w_router , and the output of its SwiGLU activation function, a.
• Activation
• The combined scalar a · w_router .
• Pole
• As we explain below in more detail, modern neurons’ activation functions can produce a positive or negative value. A pole is the positive or negative extreme of a neuron, and usually (but not always) positive and negative values have unrelated roles.
• Firing
• A fire is an individual event in which a neuron's activation on a particular token is nonzero (on the positive or negative pole). In an MoE model this requires that the token was routed to the expert containing the neuron. These neurons do not have true “off” values since they are never 0, and in practice most activations are very weak (hence our focus on strong activations).
• p97/3, p99/1, etc.
• We use this notation as shorthand to refer to the percentile of a neuron’s activation relative to all its activations across a corpus. On this scale, p1, p5, etc. are negative activations (as the median activation of neurons in this study is near 0). Thus, p97/3 refers to both positive and negative extremes, p3 ∪ p97.

How neurons connect

• Input neurons
• Every neuron receives input from a large set of “causally upstream” neurons, which we refer to as inputs. In particular, to serve as input, a neuron from a previous layer must either 1) produce an output vector, written to the residual stream, that is aligned with the target neuron’s gate or up weight matrix; or 2) be identifiable through attribution techniques as having an effect on our target neuron, mediated by attention.
• Reader neurons
• The inverse of input neurons. A reader neuron is a neuron that responds to whichever neuron we are discussing, either directly or through attention.
• Circuit
• A computational subgraph of the entire network that is causally sufficient for producing a given output. In this article, we focus on circuits that use individual neurons as nodes.
• Subcircuit
• A computational function that consists of a set of input neurons and reader neurons where the two sets are connected through a single target neuron. A neuron can host multiple subcircuits and they can exist simultaneously. However, because the members of a subcircuit are highly disjoint from the members of other subcircuits, only one is active at a time.

What a neuron means

• Response profile
• The pattern of textual features and concepts a neuron fires on across a dataset.
• Max-activating examples
• Texts that cause a neuron to fire near the top of its activation range; i.e., to output a value that is one of the strongest observed for the neuron.
• Computational function
• Neurons can receive input from potentially thousands of other upstream neurons, and are read by potentially thousands of downstream neurons. A computational function is a description of the causal process by which a set of identifiable inputs is transformed into a meaningful output; a simple example would be “combines location and cultural cues from upstream neurons to output a signal that Tokyo is being mentioned.”
• Mode
• A semantic grouping of computational functions (subcircuits) of a neuron. Neurons with more than one mode are referred to as multimodal .
• Mono- and Polysemanticity
• These terms conventionally describe a neuron that responds to a single “concept” or to multiple seemingly unrelated “concepts”, respectively. However, as discussed in the section below (see “Moving from Polysemanticity to Computational Role ”), these labels admit multiple interpretations, and the apparent semantic coherence of a neuron’s responses does not by itself determine its computational function. We therefore focus on how downstream neurons use its signal.

This is the first in a series of technical articles exploring neuron interpretability. This post introduces our initial findings and sets the frame for further exploration.

Our primary findings are as follows:

• Language models can operate under considerable neuron sparsity. Inspired by the foundational work ofArora & Wu (2025) , we evaluate several sparsification methods according to (1) KL divergence from the intact model over large datasets of natural text and (2) benchmark performance. - By ablating all neuron firings below extreme activations (correcting for the bipolarity of gated neurons, as explained below), we show that the aggregate causal role of a model's neurons can be attributed largely to the activations in the highest ranges–specifically, the 97th percentile and above and the 3rd percentile and below for negative values, p97/3 in our notation . This range is precisely where neurons are most interpretable. Up to 92.2% of the weakest absolute neuron activations can be ablated either through top-K or percentile-based filtering with only small effects on loss, KL, and a broad set of performance benchmarks. Further, those negative effects on benchmark performance can be corrected with only 91M tokens of SFT and 9M tokens of RL to re-calibrate reasoning.
• Individual neuron poles have a small set of distinct computational functions that usually fall into one or a handful of semantic categories, like “beetle and insect-wing anatomy terms” or “Chinese health and wellness vocabulary.” We identify these functions by examining the upstream neurons that drive a firing and the downstream neurons that use its signal, andcall a neuron’s coherent groups of functions “modes.” Coherence is defined by LLM judges classifying functions into specific categories and performing held-out example matching. - We identify disjoint subcircuits that a neuron participates in and cluster those subcircuits into distinct modes, and find that there is more circuit similarity within modes than between modes.
  • For the majority of neurons, we find either only one mode or a single dominant mode responsible for >=90% of its causal effect across a broad dataset.
  • Modes have degrees of relatedness, and are not always completely unrelated semantically. For example, a neuron may have multiple Python-related coding modes, or it may have a mathematical mode and one related to Japanese honorifics. Usually, even when there are multiple modes, one is far more causally relevant than the others.
• The computational roles of neurons can be concisely described in natural language and these descriptions can be used to predict, disambiguate, and simulate neurons across a range of tests. We evaluate neuron descriptions through detection (will a neuron fire in a short token span?), classification (when a neuron with multiple semantic modes fires, can we tell which meaning is operative?), input synthesis (can we create a text that makes the neuron fire strongly?), and simulation (can we replace the neuron’s natural activations in a sequence with those based on a description of the neuron?). - In the activation percentiles that our sparsity experiments suggest are most important for the model's primary computations, we can achieve considerable understanding of the neurons’ behavior. As activation strength rises, we are able to make more precise predictions and simulate their effects more accurately.
  • For percentiles stronger than p97/3, we can detect when a neuron will fire on a short sequence 95% of the time when comparing the activating text to a random text, and 89% when comparing to a text assigned to the same expert (chance in both cases is 50%). We can also tell a neuron's modes apart, either from the circuit, by observing which downstream neurons a firing moves (its readers), or from the text alone, by giving a language model the mode descriptions.
  • For percentiles above p99.9, we can simulate neuron behavior well enough to recover 73% of the recovery achievable by a perfect reader. We note that this was achieved without any training of a model to accomplish this task. Bands p99.9 through the top activations are responsible for 73% of all neuron damage when a neuron is ablated, and thus hold the most causal significance. We ablate these neurons from all tokens in a held-out sequence and measure the KL divergence of the logits. We then replace the activations with simulated values provided by a judge LLM based on the neuron’s description. On average, this is sufficient to recover 73% of the KL damage that the perfect-reader ceiling restores.
  • For 94% of neurons, we are able to generate texts that activate the neuron beyond the maximum level found in a 10B corpus of mixed pretraining and post-training data. Our descriptions capture what activates the neuron effectively, though this is only a test of each neuron’s primary mode.

Neuron Sparsity and Effective Range #

The two central questions of this section are:

1. Are individual neurons, instead of distributed directions spread across many neurons, the right basis for analysis of MLP computation? High levels of sparsity in MLP computations would suggest that neurons carry significant parts of the computation on an individual basis.
2. What sorts of neuron activations are meaningful and necessary for computation – do strong activations suffice, or are weak activations indispensable? A highly sparse regime would suggest much higher importance for strong activations.

If the conceptual primitives for transformer computation tend to be spread out among vast arrays of neurons in a distributed code, we should expect high levels of sparsity to sabotage these computations. On the other hand, if neurons tend to be privileged, meaningful units of computation, we should expect these primitives to be more localizable and sparse. Earlier work has observed that neurons are often quite interpretable and consistent (across activation examples) in their strongest activations, but much more diverse and hard to understand at weaker levels (Szegedy et al., 2014; Morcos et al., 2018; Zhou et al., 2018; Leavitt & Morcos, 2020; Ghorbani & Zou, 2020; Dhamdhere et al., 2018). This motivates the concern: what if weak activations are both indispensable and uninterpretable?

We find that models can still perform surprisingly well even when all weak activations (e.g., between the negative and positive extremes of p10 and p90, or even p3 and p97) are mean-ablated, achieving close to parity with the intact model on a number of non-reasoning benchmarks. More importantly, we test an ablation where every layer retains only the top-320 neurons (out of 4,096 routed neurons) by absolute activation strength for a given step of inference, ablating 92.2% of routed neurons, and show that this model performs comparably to the base model on non-reasoning benchmarks.

For all experiments, we leave the shared expert neurons unchanged, since these neurons are effectively always on and function differently from the expert neurons. In total, the shared experts make up 20,480 of the model’s 5.26 million neurons (~0.4%).

For reasoning benchmarks, we observed that the ablated model would often run into reasoning loops and self-check excessively, never completing its answer–though by checking its reasoning output, we were able to determine that the correct answers were present. With a brief fine-tuning step, we corrected the reasoning errors and met or exceeded the original model’s performance on a broad set of thinking-based benchmarks for question-answering, code, reasoning, and math. Considering that the fine-tuning tokens used were a small fraction of a typical post-training run, this provides evidence that the fine-tuning primarily corrected reasoning rather than adding new information to the model, and that the majority of the information required for performance on benchmarks was present at the higher activations.

We provide further evidence for sparsity in the Appendix: we preserve specific model predictions with even fewer neurons using the gradient-based attribution method, RelP, and we elaborate on recent findings (Shabalin & Piotrowski, 2026) showing that transcoder replacement models tend to discover neuron-localized features even when not constrained to do so.

Sparsification by Activation Strength

Our first set of neuron sparsity experiments uses neuron activation strengths to determine which neurons to ablate. We consider three methods—mean-ablating (on an individual neuron basis, across all neurons in the model) the central percentile bands of neuron activations, expert top-k filtering, and layer top-K filtering—and measure their effects on loss, KL, and a range of benchmarks. The ablated bands are symmetric in percentile rank around the median: i.e., p1 is a strong negative activation, p99 is a strong positive activation, and weak bands are p10-p90, p25-p75, etc.

Ablation Within Top Percentiles

A natural experiment is to start with the weakest neuron activations and ablate them during inference, measuring the resulting loss and KL divergence. Since SwiGLU neurons are bipolar and neurons have different activation strength distributions, we compute neuron activation strength percentile bins and apply mean ablation to everything between the positive and negative extremes (Li et al., 2023a; Liu et al., 2023a; Mirzadeh et al., 2023; Zhang et al., 2022; Liu et al., 2024; Lee et al., 2024; Jaszczur et al., 2021; Ahmad & Scheinkman, 2019; Makhzani & Frey, 2013; Song et al., 2024; Zhang et al., 2024; Wang et al., 2024; Song et al., 2023; Alizadeh et al., 2023; Zhang et al., 2023). These initial experiments were conducted without the router weight scaling, and the percentiles shown likewise do not include this scaling.

Top-K Filtering

Inspired by Shabalin & Piotrowski (2026), we attempted to find a more sparse regime by applying a top-K function (based on raw neuron activations) across each of the experts in the model. We tested top-32 and top-64, which retain 6% and 12% of routed neurons per token respectively. Experts have 512 neurons each. In practice, the top-64 neurons primarily lay at or beyond the p5 and p95 extremes (Sun et al., 2024; Dettmers et al., 2022; Kovaleva et al., 2021; Puccetti et al., 2022; Bondarenko et al., 2023; Olshausen & Field, 2004). As with the ablation within top percentiles, this was performed without the router weight scaling.

Router-Weighted Top-K Filtering

To account for expert router weighting, we apply our activation definition above and score each routed neuron by its router-weighted activation. We then apply a single top-K over all 8 × 512 = 4,096 routed neurons of each token rather than one per expert, so the budget can shift toward experts whose neurons have the strongest activations after router weighting. We keep the top 320 (7.8%) of routed neurons (40 per expert on average, 92.2% sparsity) and zero the rest before the down projection. Here, most activations exceed the p3 and p97 extremes. Like other activation-sparsity methods, the mask is computed from each token’s own activations at inference time and needs no training (Liu et al., 2023; Lee et al., 2024; Liu et al., 2024; Wang et al., 2024; Makhzani & Frey, 2013).

We found that the model was able to operate surprisingly well under high levels of sparsification. Below, we show the sparsified models’ loss and KL divergence, as well as performance on a wide range of benchmarks:

Table 1: KL, loss, and benchmark performance under sparsification. Base model benchmark scores are percentages; other benchmark columns show changes in percentage points. Red indicates a decrease; blue indicates an increase. Zero is white. There is a far smaller drop in performance from retaining the top-K neuron activations, either per layer or per expert, as opposed to the complement or random-64; the latter two destroy performance. Ablation based on percentile bands similarly retains a disproportionate amount of performance on the top bands. Note that the P25–P75, P10–P90, etc. percentile bands were computed without router-weighting, unlike the percentile bands referred to throughout the remainder of this text. A thinking token cap of 4,096 was applied to the reasoning evals.

Performance is retained to a surprising degree across benchmarks, especially when compared to controls. The ablated models retained high performance across classic benchmarks like ARC-Challenge, ARC-Easy, WinoGrande, and HellaSwag, improving slightly on some evals and dropping on others. However, we observe drops in reasoning and knowledge-based evals. When we observe rollouts directly, we see that the sparsified model’s reasoning tends to loop and self-check excessively. However, when we classified these rollouts with Claude Opus 5.5 agents, we identified that the sparsified model would still mention the facts and correct answers, usually as the last answer candidate–it would just self-check endlessly afterwards. We further found that reasoning collapse seems related to the double-negative activations we discuss in the next section, since restoring those specific activations restored reasoning.

In order to see if this strong-activation regime could recover full benchmark performance, we follow Shabalin & Piotrowski and perform light fine-tuning on the layer top-320 model and re-measure performance; across benchmarks we were able to see full restoration of function. We discuss methods and results further in the Appendix section Extended Sparsity Results and Discussion.

What we suggest, then: the original model is very close to sparse, with most of the computation located in the top activations. This implies that even the original model is likely quite interpretable and causal at the top ranges, but also that it is relatively easy to tip a model over the edge to more complete sparsity. Our subsequent analyses of neurons in the fine-tuned model showed that the neurons retained their original functions, remained interpretable, and had the same numbers of modes.

Principled Selection of a High-Activation Band

Most of the model’s functionality is retained after top-320 layerwise filtering, and that lost performance is easily restored with light fine-tuning. The retained router-weighted activations fall almost entirely in the p97/3 band. We therefore use this activation regime for the experiments covered in the remaining sections.

Sparsity takeaways and the interpretable regime

We now have evidence from several directions suggesting the following:

• The set of neurons required for each step of inference is surprisingly sparse. General sparsification methods like top-K achieve comparable benchmark performance at ~92.2% sparsity; as we discuss in the appendix, specific circuit-finding tools produce even greater sparsity.
• Retained neurons are almost always firing at the positive and negative tails, often p97/3 or stronger.
• Double-negative/Q3 activations seem to consist of a functionally different regime. Though we did not analyze it in detail, we expect that it is likely that it is approachable with further investigation.

From this evidence, we conclude that even if we do not fully understand weak activations, we can nevertheless understand the bulk of what the MLPs are doing if we limit ourselves to examining the tails. Weak activations may not be entirely irrelevant, and may perform a function, but we leave investigation of them to future work. We further hypothesize that the sheer sparsity of neuron activations in these experiments suggests that neurons constitute a privileged basis, that they are meaningful units of analysis, and that the connections between them can help elucidate their functions.

Now that we have laid the groundwork for investigation of the neurons’ positive and negative extremes, the next question is: what does their behavior in these regimes tell us?

Understanding Neuron Computations #

Moving from Polysemanticity to Computational Role

Neurons are often thought to be difficult to interpret because a single neuron can respond to several seemingly unrelated concepts. This property, known as polysemanticity, has been described in terms of responses to unrelated latent variables, mixtures of inputs, or multiple semantic concepts (Olah et al., 2020; Bricken et al., 2023; Bills et al., 2023).

These informal definitions do not resolve what makes concepts distinct. Whether a neuron is labeled monosemantic or polysemantic can depend on choices about what constitutes a single concept (Murphy, 2002). As noted in Neural Networks Are Already Clean, humans tend to describe neurons as detectors for groups of nouns. But categories do not always follow this principle. Klindt et al (2026) compared this phenomenon to the New York Times Connections Game, where the objective is for users to group words by non-obvious similarities, and Barsalou, 1983 famously illustrated this with goal-derived categories such as “things to sell at a garage sale”. An old GameCube and an unused kayak seem to have little resemblance, but they would both belong to this category. Recognizing a unifying concept also depends on the interpreter’s background knowledge (Murphy & Medin, 1985). Responses that initially appear unrelated may share a regularity that becomes apparent only with a better understanding of the computations they support.

An alternative approach is to investigate neurons from a computational perspective first, rather than beginning with semantics. How do different sets of input neurons make a neuron fire, and how do downstream reader neurons use a neuron’s signal? By identifying distinct computational functions that a neuron performs, we can develop objective, quantitative tools to analyze a neuron; and we can then examine their semantics to map computational functions back to interpretable concepts. This approach makes it possible to demystify neurons even when they perform multiple or confusing computations, because the subcircuits involved remain legible. And as long as these roles are few and concisely describable, we can still understand the neuron by characterizing each role and the contexts in which it operates (Rigotti et al., 2013; Fusi et al., 2016; Quiroga et al., 2005; Barlow, 1972; Marr, 1982). This allows us to retain the neuron as a unit of analysis even when it serves multiple distinct computational roles.

We therefore adopt this approach–first looking at computational function, and only then examining semantics– as our primary investigative procedure. In our next section, we describe precisely how we perform this analysis.

Understanding Neuron Computation via Subcircuits

We intentionally target the high-activation regimes that our sparsity experiments demonstrate to be the most causally important (p97/3 and stronger). As motivated above, our approach here is computational/functional rather than semantic: rather than understanding a neuron in terms of the types of texts that activate it, we instead examine what function the neuron plays in network computation. In particular, a neuron is interpreted in terms of its upstream and downstream connections to other neurons.

We extract various subcircuits that a neuron participates in as the language model processes input text. We define a subcircuit as a computational subgraph with a set of input and reader neurons that connect through a target neuron, and seek to identify disjoint subcircuits that run through the neuron. A neuron participates in multiple subcircuits, each implementing a unique computation through the gate ⊙ up multiplication. A neuron always writes to the same direction in the residual stream. However, the same textual context that causes the neuron to fire can also prime different downstream neurons to read the original neuron’s signal.

An analogy can be made to the signal processing idea of multiplexing, where multiple signals from different subcircuits are compressed into a single channel (Lee & Messerschmitt, 1988; Mante et al., 2013; Rigotti et al., 2013; Salinas & Thier, 2000; Dreyer et al., 2024; Gandelsman et al., 2024). A single neuron may serve as a shared channel for different computations, with the surrounding context determining how downstream neurons use its signal. This is made particularly easy by the nature of gated MLPs, since upstream neurons can perform a switching function to activate different sets of readers depending on the context through a combination of the gate and up projections. We discuss this in the text on disambiguation below.

Figure 5: An illustration of how neuron subcircuits work. The neuron output vector is always the same direction, but by priming different downstream reader sets, the neuron can be used for different functions.

Identifying Computational Functions From Subcircuits

To identify these subcircuits, we first ran 10 billion tokens from diverse domains through Qwen 3.6 35B-A3B including data from pretraining, reasoning, chats with users, and math. For each of the model’s 5 million neurons, we collected 1,000 examples from each of 14 activation percentile bands, for a total of 14,000 examples per neuron. We then collected an additional 30,000 examples per neuron for a stratified subset of 2,734 neurons. These additional examples were drawn from p95/5 and stronger. As a starting point for neuron descriptions, we used a fine-tuned Qwen3-8b model to produce basic labels for each neuron based on example sentences the neuron fires on.

To identify subcircuits, we first characterize each firing by its upstream inputs. With each layer’s RMSNorm scale held at its actual value, the target neuron’s gate and up pre-activations decompose exactly into direct contributions from the residual-stream writes of earlier neurons. We keep the inputs that contribute most. To collect downstream readers, we propagate the target neuron’s write along the residual stream to every later layer at the same position, through the same frozen RMSNorm. We estimate each reader neuron’s change in activation to first order. See the Methods section in the Appendix for details.

We then profile each firing by the inputs that drive or suppress its gate and up projections. For each projection, we subtract each input’s mean absolute contribution across the expert’s neurons from its absolute contribution to the target neuron, then retain the 400 top inputs for each of the gate and up projections. We separate these inputs into two bins: those that support the sign of the projection’s pre-activation and those that oppose it. This gives each firing a profile with four parts: gate drivers, gate suppressors, up drivers, and up suppressors. Keeping these separate lets us distinguish firings where the same input plays different roles–for example, an input neuron could suppress one subcircuit through the gate, or activate another subcircuit through the up projection.

We next compare these profiles across firings of the same pole to identify upstream subcircuits that recur in different contexts. Inputs that appear in nearly every firing provide little information about what distinguishes one firing from another. We therefore weight the profiles with TF-IDF to downweight common inputs, and normalize the four quadrants so that each contributes equally. Since we do not know in advance how many subcircuits a pole has, we use community detection to find groups of firings with similar input profiles. We split examples by document, using 75% of documents for fitting and holding out the remaining 25% for evaluation. We connect each firing in the fitting set to its 15 nearest neighbors by cosine similarity of the input profile, then run the Leiden algorithm, a community detection algorithm. Each held-out firing is assigned to the group whose centroid is nearest in cosine similarity.

This procedure can split a single circuit into several groups. To bring these back together, we merge groups when their top 50 input neurons overlap by at least 30% and their top 50 downstream readers also overlap by at least 30%, a heuristic that we find produces higher-quality subcircuits. Requiring agreement on both sides helps ensure that the merged groups reflect the same underlying circuit.

This yielded a median of 19 subcircuits per pole (with a range between 1 and 51), with low overlap in their inputs and readers. We send each subcircuit, with a description of its inputs and readers, to Claude Opus 5.5 to produce a concise description of the subcircuit.

We see that subcircuits are highly disjoint:

Table 2: Set overlap for input and reader neurons for subcircuits within a neuron. The nearest neighbor is the subcircuit with the most overlap within a neuron.

For each subcircuit, we obtain three different descriptions. The first describes how upstream neuron inputs through the gate and up projections cause the neuron pole to fire. The second describes how the subcircuit appears in text alone. Finally, we also describe the token pattern: what actual tokens in a text does this neuron fire on? Figure 6: Example of a subcircuit description. Its behavior is described at the circuit level, the text level, and the token pattern level.

Clustering Subcircuits into Modes

While a neuron will participate in many subcircuits, many of these subcircuits can be clustered based on semantically themed groups that we call modes. We provide the list of subcircuits and text examples to a Claude Opus 5.5 judge (Zheng et al., 2023; Paulo et al., 2024) and ask it to create a set of categories we call modes into which each subcircuit will be placed. Often, a neuron’s subcircuits are related enough that a single mode is assigned, but sometimes the subcircuits demonstrate multiple unrelated modes. We note that mode counts computed in this way are not definitive counts of the concepts represented by a neuron, as they are affected to some extent by the descriptive granularity imposed by this procedure. Broadly, we find that most neurons have between one and a small handful of semantic modes, and most of the damage incurred by ablating a neuron is confined to a single dominant mode. However, in quantifying these intuitions, we do not find that any one metric effectively captures the number of modes and their relatedness within a neuron-pole. Neurons can contain modes with dramatically different levels of relatedness, very high inequality in terms of how causally relevant each mode is, and very different frequency distributions of each mode. We thus provide a variety of metrics and explain the tradeoffs of each, but to put them into context we introduce what we call the total neuron view.

The Total Neuron View

To ground our discussion, we provide concrete examples of neuron-poles and their modes in the following format. It allows us in one place to see the different modes within a neuron, their similarity, and how they change over different activation strengths. Within each mode, we can see its firings, and within each firing both the tokens and circuits causally relevant to it.

As described above, these were mapped into subcircuits based on disjoint sets of input neurons, and these subcircuits were then grouped by our LLM judge. In this illustrated neuron, there is one main mode, a secondary one closely related to it, and a much weaker one that’s unrelated.

Each dot represents one firing example from our dataset. When looking at a neuron in this way, we can observe multiple things: how related different modes are (based on embeddings of the mode descriptions, as explained below), how causally dominant (how much KL divergence caused by ablation) each mode is, and how they change with activation strength.

By raw count, there are three modes in this illustrative neuron-pole, but this number does not correctly convey what we should expect from this unit, nor how we should use it in interpretability. We thus provide a number of metrics, each of which answers slightly different questions about both this neuron-pole and the population of neurons at large.

How many individual modes does the LLM judge assign per neuron-pole? The median neuron-pole has 1.0 mode, with a mean of 2.82. #

How many causally important modes does each neuron-pole have? For each pole, we can weight a mode by its damage: of the total KL damage incurred by ablating the pole, what fraction is due to this mode? In the visualization below, we distinguish poles for which at least 90% of KL damage is confined to their main mode; 69% of poles have such a dominant causal mode.¹ #

How many highly unrelated modes appear in each neuron-pole? There is not often a natural place to split a mode, for instance one representing two closely related idioms. In other cases, it does appear to be completely unrelated, like a Colorado and a much rarer Tornado Python framework neuron we will discuss below. While it can be helpful to split a pole into modes, thinking in modes can count these two cases equally. In the plot below, we use color to help express mode similarity, and later show the full distribution of pairwise mode similarities per pole.

To develop an intuitive grasp of how these factors can present in a neuron, we present a selection of neuron profile types below.

See Appendix “Additional Analyses of Modality Profiles” for a more detailed typology of neuron profiles and their frequencies.

Mode Statistics in Detail

Here, we report the distributions of the metrics outlined above, showing the number of individual modes per neuron-pole, and the fraction of the pole’s KL contribution and firing associated with each of its modes.

To measure the KL damage for a pole’s modes, we pass the 10B-token corpus through the model with the pole ablated, recording for each token the KL divergence from the intact model’s output distribution. Each token, and its associated KL damage, is assigned to a mode according to the subcircuit it activates. We report each mode’s fraction of the pole’s total KL.

We measure firing fraction similarly, recording each mode’s fraction of a pole’s total firing for the 10B corpus, as measured by the mode’s subcircuit activations.

Below we show the results for all metrics over our entire sample population of 2,734 neurons.

source & further reading

goodfire.com — original article