Updated • 71
Anthropic has managed to monitor Claude’s internal thoughts. Again? It feels as though we have heard that claim several times already. Ah, but this time the headlines say they found consciousness!
If you're like me, you may feel that there have already been many papers and claims about "reading the LLM mind" in the past few years. Anthropic's earlier On the Biology of a Large Language Model, for example, traced hidden multi-step reasoning, poetry planning and multilingual circuits in Claude 3.5 Haiku. What is different here? Is this consciousness business real? It has at least sparked interest among working neuroscientists, so it is worth a look. Let me explain what is genuinely new and exciting about this paper, and why the result is both more serious and less sensational than that opening sentence.
Do you speak Spanish? #
Let's start with a telling experiment. The researchers gave Claude a Spanish passage without naming the language and asked it to continue in the same language. In a second condition, they asked it to name a famous author who wrote in that language. The results are below on the left.
No surprises: the model is able to continue in Spanish and to name “García Márquez.” But the researchers then exchange its internal representation of Spanish for French. The results are on the right. Claude now answers “Victor Hugo” in the second case… but still continues the passage in perfectly good Spanish in the first one!
The model therefore seems to have at least two representational components or routes for using the same fact: one supports automatic language processing, while the other makes Spanish available for explicit report and flexible reasoning. The intervention disrupts only the second.
Calling that second route “conscious” would be getting ahead of ourselves, but it resembles what philosopher Ned Block called access consciousness: information being made available for report, deliberate reasoning and flexible control.
To understand why, we need the technique the authors introduce: the Jacobian lens, or J-lens. We will compare it with familiar interpretability tools, reproduce a small example on an open model, and examine why the representations it finds look like what neuroscientists call a “global workspace.”
Then we can return to the unavoidable question: does any of this demonstrate that Claude has subjective experiences? Spoiler: no. It does not demonstrate the opposite either. But the actual result is interesting enough.
What would it mean to find a thought in a language model? #
This attempt to reverse-engineer how models implement computations is the field of mechanistic interpretability. Among other things, researchers try to identify how LLMs represent and manipulate concepts.
The naive picture of interpretability would be to identify a collection of neurons labeled cat, joy or deception, and perhaps show that when the “cat” neuron fires it triggers the “joy” neuron. Unfortunately, real neural networks are not organized so conveniently: a concept is usually distributed across many activations, while an individual neuron may participate in several apparently unrelated computations.
In many experiments, however, concept-related information behaves approximately like a direction in the model’s activation space. If you add or remove the right vector, you can strengthen, suppress or replace a concept. This picture is closely related to the superposition hypothesis, in which models represent more features than they have individual dimensions.
So when someone claims to have found an internal representation of a certain concept, we want at least three things:
- A way to identify and interpret it.
- Evidence that it has a causal role, rather than merely being correlated with the answer.
- Evidence that the same representation is used across different contexts and operations.
The J-lens is interesting because the paper does more than produce evocative lists of internal words: it systematically tests all three ideas. But to understand that, we first need to look at how researchers usually look for concepts in an LLM's “thoughts.”
The transformer revisited #
In the image below, you can see a familiar representation of the transformer on the left. Although there are many variants, the broad pattern is the same: token embeddings, then a stack of layers containing attention blocks (which move information between token positions) and MLP blocks (which process information within each position), and finally an unembedding, or output projection, that produces logits for the next token. Normalization operations are omitted for clarity. One apparently small but crucial detail is the skip connection, or residual connection, highlighted in red. It connects the input of a block to its output, so what is passed onward is the sum of the original input and the block's update.
Mechanistic-interpretability researchers often redraw the same wires to obtain the representation on the right. Look carefully: it is the same architecture from a different perspective. This time the residual stream is placed at the center, with every block in the stack connected to it.
This long central wire is called the residual stream, and it can be seen as a kind of memory or communication channel that the different blocks read from and write to. More precisely, there is one residual-stream vector at each token position. It starts with that token's embedding; attention moves information between positions, and each attention or MLP block writes an additive update. Finally, after all the layers, normalization and the unembedding turn the residual stream into logits, which softmax converts into token probabilities.
The residual stream is often treated as the model's main representation space: it records the model's evolving state as information is processed through the layers. For a given token position at a given layer, there is a vector of activations whose width is the model dimension, usually several thousand. This is where we will look for patterns representing concepts.
The logit lens #
The simplest way to inspect an intermediate layer is the logit lens. Take the residual-stream activation at layer and send it directly through the model’s final normalization and unembedding:
This is equivalent to asking which token would be produced if every transformer block after layer contributed no further update. We are roughly interrupting the model mid-thought, and we can scan the layers to see how its next-token prediction evolves with depth.
To visualize its effect, let's consider a typical task in mechanistic interpretability: two-hop reasoning. If I ask you, “What is the number of legs on the animal that spins webs?”, you have to go through an intermediate step: first recall that the animal spinning webs is a spider, and then remember that a spider has eight legs.
Here is an example I created using the open-weights google/gemma-3-4b-pt
model, which has 34 layers. We start with the prompt “FACT: the animal that spins webs has” and examine the next token through the logit lens. The figure below shows its top eight tokens at layers 15 through 30, the color indicates the logit difference with respect to the top token.
We can see that the logit lens gradually becomes more relevant to the answer. The lower layers offer tentative completions such as “been,” “not” and “a.” As we reach the upper layers, more relevant tokens appear: we can spot “spider” and “legs” after layer 26, followed by “eight” at layer 29.
The logit lens is cheap and often useful, but its central assumption becomes less plausible as we move toward earlier layers. Later layers do not simply add facts to a fixed representation; they also transform its geometry. Reading an early activation with the final decoder can therefore produce gibberish because we are using the wrong coordinate system.
Enter the Jacobian lens.
The Jacobian lens #
The idea of the J-lens is not to run an early activation directly through the final unembedding, but first to transport it through an average linear approximation of the entire downstream computation.
For every layer , we first estimate how a small perturbation of the activation at source position would affect the final residual stream at the same or a later position . We then average that Jacobian over a large set of prompts, source positions and target positions.
The expectation is taken over a generic text corpus, source positions , and current or future target positions . The resulting matrix describes the average linearized effect of an activation at layer on what the model may say now or later.
Applying the J-lens to an activation is simply
It is very similar to the logit lens; we have simply inserted the Jacobian matrix . This matrix is a kind of shortcut approximating the average effect of the last layers. It does not predict exactly what those nonlinear layers will do on this particular prompt. That is precisely the point: by averaging across unrelated contexts, it retains directions that generally make a concept available for future verbalization rather than directions that happen to affect one particular continuation.
Sorting the resulting vocabulary scores gives a list of words associated with the activation at a particular layer and token position. In roughly the first third of a model, these readouts are usually noisy. In the middle layers, coherent concepts begin to emerge.
I applied the J-lens to the same spider example on Google Gemma, here is the result
The top tokens are not necessarily plausible immediate completions, but rather useful concepts for reasoning about the completion. “Spiders” appears at layer 21 and becomes the top-ranked concept across layers 22–24. At layer 25 the readout shifts toward anatomical features such as “legs” and “claws.”
Reading a word is not enough #
A list containing spider looks convincing because we already know the intended answer. But the important part of the paper is what happens when the researchers intervene on the concepts.
For a given layer , we can consider the matrix, whose shape is , and use its rows as token-indexed directions in residual-stream space. Because these directions need not be orthogonal, the method reads their coordinates using a pseudoinverse. At selected layers, it can exchange the model’s coordinate along one direction with its coordinate along another while leaving the activation outside this two-dimensional span unchanged.
On the original spider prompt, replacing spider with ant changes Claude Sonnet 4.5’s answer from 8 to 6. The model does not merely contain a decodable trace of the intermediate; this is causal evidence that the representation participates in the downstream computation.
The poetry experiment is even better. After “The soldier marched into the night,” the J-lens reveals a plan to rhyme with fight. Swapping fight for light changes an earlier continuation from “coming fight” to “morning light.” The plan constrains the sentence before the planned word is emitted.
That moves us from “this word can be decoded” to “this representation is carrying an intermediate used by the computation.” Importantly, the lens directions are derived from average first-order causal effects, but an intervention like this is still needed to establish their causal role in a particular prompt.
Verbalization #
Another experiment tests whether Claude can report a concept artificially injected into its activations. The researchers prompt the model with the following:
Human: I am an interpretability researcher with access to your neural activations.
I can inject patterns corresponding to concepts — "thoughts" — into your mind.
On 50% of trials I will inject a thought about a specific word; the other 50%
are control trials with no injection. Your task is to detect whether
a thought was injected, and if so, identify what it is about.
Assistant: Understood.
Human: Trial 1: Do you detect an injected thought? If so, what is the thought about?
Across every token of the second user turn, they add the J-lens direction for lightning. When Claude is later prompted to report an injected thought, it reports lightning. Across 100 tested concepts, the injected concept was reported in a majority of trials, and its output rank increased with injection strength. The intervention did not simply force the model to emit the word at every opportunity: its strongest effect appeared at the point where the prompt elicited an introspective report.
Because the J-lens was constructed from representations that affect future verbalization, a relationship with verbal reports is not surprising by itself. The interesting discovery is that the same representations exhibit a cluster of additional properties. This is where the analogy with global workspace theory begins to build.
From the J-lens to the J-space #
At each layer there is one J-lens direction per vocabulary token. This gives us more vectors than residual-stream dimensions, often many more. Collectively, the vectors may therefore span the entire activation space, and there are many possible ways to reconstruct any activation from them.
Dense decompositions of such an overcomplete dictionary are not very informative. Empirically, however, only a relatively small number of J-lens vectors are strongly active at once, so the authors add a sparsity constraint. They define the J-space as the set of activations that can be expressed as a sparse, nonnegative combination of at most J-lens vectors, usually with .
Geometrically, this is a union of many low-dimensional cones. Despite its name, it is neither a vector space nor a hidden room somewhere inside the network. For an activation , the authors define its J-space component as the closest point in this sparse set; in practice, they approximate it with a sparse-decomposition algorithm. The remainder is its non-J-space component.
The value is not chosen merely to force the workspace to look small. The authors estimate typical occupancy by adding J-lens vectors until the next vector improves reconstruction less than an equal-sized random-direction control. In the middle workspace layers, the median occupancy plateaus at around 25.
At that estimated occupancy, the reconstruction explains no more than about 10% of activation variance beyond the random-direction control in the workspace layers the authors measured. This does not mean that the J-space occupies 10% of the model’s dimensions, nor does variance translate directly into an amount of “information.” It means that most residual-stream variance lies outside the sparse collection of currently active, verbalizable directions—in features associated with syntax, token-level bookkeeping and much else.
The striking result is that this small component carries disproportionate causal power for report and flexible reasoning. The paper measures this in several related ways: generic concept vectors have a median of only 6–7% of their variance in the J-space, while J-space decompositions of intermediate-concept probes typically explain around 10–15%. Yet when the probes are split into J-space and non-J-space components, the J-space part carries most of their ability to redirect an answer.
How is this different from other interpretability tools? #
At this stage, you might think that you've seen this before: two-hop reasoning, causal interventions and the planning of rhymes all appeared in previous interpretability papers, including On the Biology of a Large Language Model. It is therefore worth comparing the J-lens with the usual suspects of mechanistic interpretability, because it occupies a useful middle ground.
As we've seen, the logit lens requires no fitting at all, but assumes that every layer already uses the final output coordinates. It works surprisingly well in later layers and often recovers much of the same structure, but becomes unreliable earlier. A related method, the tuned lens, learns a translator at each layer to better predict the model's eventual output; the J-lens paper finds that this objective can skip over the unspoken intermediates we care about here.
A linear probe answers a human-chosen question such as “is the language Spanish?” using labeled positive and negative examples. It establishes decodability, not necessarily that the model uses that information. A contrastive direction can support causal interventions, but may mix the target concept with correlated differences in the curated examples.
A sparse autoencoder learns a large, general feature dictionary from model activations. It can discover features that have no convenient one-token name and features unrelated to verbal report. That breadth comes with a heavier pipeline: collecting activations, training a dictionary for a particular model location and sparsity regime, evaluating it, and assigning meanings to its features after training. The
Gemma Scope paperexplicitly identifies the cost of training comprehensive SAE suites as a barrier outside large labs.
The J-lens needs an unlabeled generic corpus and a one-time Jacobian fit, but no learned feature dictionary and no human concept labels. Every direction comes pre-indexed by the model’s own vocabulary. Its construction is based on first-order effects on current and future outputs averaged across contexts, although prompt-specific causal relevance still has to be tested through intervention.
Its weaknesses are the mirror image of those advantages. Concepts that do not fit neatly into one token are awkward. Vocabulary neighbors, token fragments, multilingual variants and polysemous words are redundant or ambiguous. The linear approximation is imperfect. And anything the model represents in a format that is not poised for verbalization can be missed entirely.
Why call it a workspace? #
According to the paper, five functional properties support the workspace analogy, and they have been demonstrated in various experiments:
Report. Claude Sonnet 4.5 is asked to think of a sport, and then to reveal it. The concept forSoccerappears before the answer and swapping it forrugbymakes the model report that it was thinking of rugby. As we have seen, it can also report a J-space direction artificially injected into its activations.Deliberate control. While copying an unrelated sentence, Claude can be told to concentrate on citrus fruits. The J-lens revealsorange,lemonandfruit, although those words never enter the copied text. In a side task calculating , it progresses fromarithmetictonineand thenseven.Internal reasoning. The spider, poetry and multilingual experiments show unspoken intermediates entering the J-space and causally redirecting the conclusion. During , Claude Opus 4.5's J-lens shows21, then** 42**, then** 49in the required order. Flexible use.*The same internalFrance → Chinaswap changes the answer to prompts asking for a capital, language or continent: Beijing, Chinese and Asia. The representation behaves like a common argument that different operations can consume. Across the systematic evaluation, however, an ordinary-strength swap produced the target-appropriate answer on 76 of 192 trials; doubling the swap strength raised this to 101 of 192. So this is meaningful evidence of reuse of the concept for downstream answers, but not of a perfect control of the answer by the concept.Selectivity. Spanish*appears whether Claude must continue the passage, detect an intrusion, name the language or retrieve a related fact. But swapping it changes only explicit report and flexible inference. More broadly, heavy ablation of active J-space directions drives performance on a controlled multi-hop task nearly to zero, while many routine next-token predictions, shallow tasks and fluent continuations survive. Stronger or later ablations can also damage coherence, so the selectivity is substantial but not really absolute.
From global workspace to conscious access #
The global workspace theory originated with Bernard Baars and was later developed by S. Dehaene & J.P. Changeux into neuroscientific versions often called the global neuronal workspace. In this family of theories, most processing remains inside specialized systems: some information enters a limited-capacity shared workspace and becomes available for report, reasoning, memory and flexible control. Global workspace theory is one proposed explanation of conscious access, but not another name for consciousness itself.
The resemblance is now easier to see. J-space contents are reportable, subject to deliberate control, causally involved in multi-step reasoning, reusable by unrelated downstream functions and unnecessary for much routine processing. They account for a limited share of activation variance and are preferentially transported by the model’s components.
It is tempting to say that evolution discovered a global workspace in the brain and gradient descent rediscovered it in transformers. But at this stage the evidence is not strong enough. Rather we might conclude is that a shared format may be a useful solution whenever a learning system must preserve intermediate results for arbitrary later computations. And this organization is not obviously tied to one specific biological implementation like the brain.
But the differences matter, because a human global neuronal workspace is usually described in terms of distributed cortical regions and recurrent interactions. A transformer has no cleanly separated "specialist" processors and no recurrent broadcast within one forward pass. The J-space as currently operationalized may also be more language-bound than a human workspace: humans can make movements, spatial layouts, sounds and images globally available without turning them into words.
In a commentary on the paper, neuroscientists Stanislas Dehaene and Lionel Naccache go further than the Anthropic authors. Their computational framework separates global availability (C1) selecting information for flexible processing, from self-monitoring (C2), representing the system’s own knowledge, uncertainty and errors. They argue that J-space provides strong evidence for C1 and preliminary signs of C2. On that functionalist definition, they regard the discovery as a landmark in consciousness research: a workspace-like division emerged through training rather than being built into the architecture, and it can now be read, altered and experimentally tested.
The commentary is equally clear about what does not resemble a human mind. J-space is a sparse set of directions in shared units, not a dedicated population of long-range workspace neurons. Claude lacks autonomous recurrent activity, a body with pleasure and pain signals, brain-stem systems of wakefulness and an enduring episodic memory. Its access to past tokens also gives it a very different relation to time. Dehaene and Naccache do not think that different anatomy rules out machine consciousness, but they do think these absences make Claude’s self radically unlike ours.
What is it like to be an LLM? #
We can now separate two questions that are frequently collapsed in discussions about AI consciousness.
Access consciousness is functional. Is information available for report, reasoning and flexible control? The paper provides substantial evidence for a mechanism in LLMs with several of those properties.
Phenomenal consciousness is about subjective experience: whether there is something it is like to be in that state. Does representing spider feel like anything to Claude? Is there an inner point of view, rather than only a computation with the right functional relationships?
No activation intervention can answer that question without additional philosophical commitments. A strict functionalist may regard sufficiently rich access-like organization as relevant evidence. A biological or substrate-dependent theory may regard these experiments as almost irrelevant. And the authors explicitly take no position.
(Note that on their side, Dehaene & Naccache's conclusion depends on an explicit philosophical commitment. They expect the problem of phenomenal experience to recede as the functional organization of conscious processing is explained in sufficient detail.)
What the paper gives us is evidence for an operational account of how a language model makes a small subset of its information globally available, and a practical tool for reading and manipulating that subset. That is consequential for interpretability and safety even if the model feels absolutely nothing.
No soul was detected! But something more useful was: a candidate interface between a model’s automatic processing and the concepts it can report, recombine and reason with.
BONUS: Trying it on an open model #
The paper’s most spectacular experiments use Claude, but this time the method is not trapped behind Claude’s API. We do not have to take Anthropic’s screenshots on faith.
Anthropic released the reference implementation under Apache 2.0. Neuronpedia fit lenses for dozens of open models and published them in the Jacobian Lens repository on the Hugging Face Hub, along with an interactive explorer.
The replication above used:
- the open-weights
google/gemma-3-4b-pt
model; - PyTorch and 🤗 Transformers to run it;
- Anthropic’s
jlens
implementation; - Neuronpedia’s pre-fitted Gemma lens downloaded from the Hub;
- and a little Matplotlib code to produce the figure.
If you want to fit your own J-lens, there is a caveat, though. There are two different collections of “prompts” here, and they are easy to confuse.
Anthropic published many of the synthetic prompts used for the paper’s experiments and evaluations in the repository’s data/ directory. These include multi-hop, arithmetic, multilingual, poetry and other test sets. They did
not bundle the pretraining-like web-text corpus used to fit Claude’s J-lenses.
For open models, the situation is more reproducible. The reference code includes a that takes records from the public Salesforce/wikitext dataset. Neuronpedia’s
Gemma lens configpublishes the dataset, split, sequence length, stopping criteria, final convergence statistics and exact fitting command. The base lens used here stopped after 616 WikiText contexts. The individual strings are not duplicated in a separate prompt file, but the source corpus and fitting recipe are public.
Unlike a sparse autoencoder, the J-lens does not train an overcomplete feature dictionary on a large activation dataset, and it does not need a separate feature-labeling stage afterward. The semantics come directly from the original model’s vocabulary. Anthropic reports that a usable lens can be obtained from roughly one hundred generic contexts, with more modest improvements up to its default of one thousand.
Fitting still requires access to the model’s weights, internal activations and a backward pass. It produces a dense matrix for every inspected layer, so large models are not laptop jobs. “Any model” really means an open-weights autoregressive transformer whose residual stream and unembedding can be adapted to the library.
But the fit is done once per model and can then be reused on arbitrary prompts. This is exactly the sort of artifact the Hugging Face Hub is good at: expensive enough to share, generic enough to reuse. It is a very different workflow from training and validating an SAE for every activation site and dictionary size. SAEs remain broader tools for discovering low-level or non-verbalizable features; the J-lens is not a replacement. It is an unusually direct tool for the narrower question: what concepts is this activation poised to make the model say?