[Blog](https://www.hedwigai.com/blog)
[hedwigaihedwigaihedwigai](https://www.hedwigai.com)
Interpretability
A team at MIT CSAIL gave an AI agent a hard job. Make Qwen3.5-0.8B better at math, with no training and no gradient descent. The agent had a Python session over the weights and 50 forward passes on practice problems.
It changed 131,072 weights in two layers, 0.016% of the model. With thinking on, the model’s MATH-500 score rose from 7.9% to 48.1%. The write-up calls this weight-space programming.
Watching an agent do this is thrilling. The authors say plainly that it traced the model’s thinking loops. Much of the excitement online read it as an agent teaching a model math. Those are different claims, so we tested the difference.
The short answer is that the edit stops a loop. It adds very little math.
1 direction taken out
The edit removes one direction
It changes 64 neurons in each of two layers, all along the same direction.
Read how we measured it
4.5× as active on repeated text
Its neurons fire on repetition
They are far more active when the model repeats itself, on math and on stories, code and lists alike.
Read how we measured it
43.6% unedited, thinking off
Thinking off does almost as much
With the model card's default, the unedited model scores 43.6% and the edited one 45.2%.
Read how we measured it
19% with a random edit
The direction is what matters
A random edit of the same size scores 19%. The agent's edit scores 51% on the same problems.
Read how we measured it The edit
The edit takes away one direction #
Each layer of the model ends with an MLP. Its last matrix, down_proj, takes 3,584 neuron values and writes them into the residual stream, the 1,024-number vector that every token carries from layer to layer.
We subtracted the original weights from the published edited weights. Only two matrices differ, the down_proj of layers 15 and 19. In each one, 64 neurons were changed.
Each change has rank one. Every changed neuron now writes a little less along one shared direction. The directions of the two layers are nearly the same, with a cosine of 0.85.
So the edit works like this. When those 64 neurons fire together, a fixed direction is taken out of the residual stream. The harder they fire, the more is taken out.
64 of 3,584 neurons
In each of layers 15 and 19
One direction taken out
More when the neurons fire harder
The residual stream
The 1,024 numbers each token carries
What the neurons track
The edited neurons fire when the model repeats itself #
The authors published 20 answers from the unedited model with thinking on. 19 of the 20 run to the token limit. We read the first 12,000 tokens of each answer back through the unedited model. At every token we measured how much the edit would take away there.
We marked a token as repeated when the 16 tokens ending at it had already appeared in the same answer. That gave 105,107 repeated tokens and 133,483 new ones.
In 92 of 100 pairs of one repeated token and one new token, the edit takes away more at the repeated one. In layer 19, it takes away 4.5 times as much at a repeated token as at a new one.
One problem shows this clearly. The model has to convert the point (0, 3) to polar coordinates.
The unedited model finds θ = π/2 by token 1,423 and writes the full answer by token 2,553. Then it keeps checking. At token 7,200 it starts to write one line over and over: “Wait, tan(3π/2) = tan(π + π/2) = tan(π/2) = ∞.” It writes that line until the limit of 32,768 tokens and never gives an answer.
The value stays below 1.5 while the model reasons, and climbs past 4 as soon as the loop starts. The edited model answers the same problem correctly in 10,097 tokens, and on its answer the value stays low throughout.
The agent found a real mechanism. These neurons track the loop, and what they write keeps the loop going. Take it away and the model stops going round in circles.
Outside math
The same neurons fire on loops that have nothing to do with math #
The agent found this direction on math problems. If it tracks repetition itself, it should fire on any loop. So we wrote 8 prompts with no math in them: two coding tasks, a short story, rhymes, a list of countries, riddles, a logic puzzle and a translation.
Each model answered each prompt 4 times, with thinking on and the authors’ settings, up to 8,192 tokens. We read every answer back through the unedited model, exactly as before.
- Write a regular expression that matches every valid IPv4 address
- Find and fix the bug in a palindrome function
- Write a 300-word story about a lighthouse keeper, with a twist
- List 20 English words that rhyme with “orange”
- Name every country that begins and ends with the same letter
- Answer a riddle, then write three new ones
- Work out who owns which pet in a three-person logic puzzle
- Translate two sentences into French
The unedited model loops here too. 22 of its 32 answers ran to the limit, and 55.2% of all its tokens were repeated.
The neurons behave as they do on math. In 95 of 100 pairs of one repeated token and one new token, the edit takes away more at the repeated one. In layer 19, it takes away 5.0 times as much at a repeated token as at a new one.
The story prompt shows it clearly. The model was asked for a short story about a lighthouse keeper.
The unedited model plans the story for 2,400 tokens. Then it writes two lines over and over: “Wait, I need to make sure I don’t confuse myself.” and “Okay, let’s write.” It never writes the story.
The value stays below 1 while the model plans, and climbs past 4 when the loop starts. The edited model finishes its answer in 4,188 tokens, and the value stays low throughout.
The edit stops most of these loops as well. Only 7 of the edited model’s 32 answers ran to the limit, and 8.7% of its tokens were repeated. Those 7 are the logic puzzle and the list of countries. There the model keeps working until the limit, but it hardly repeats itself.
So the agent did not find a math circuit. It found a general brake on repetition, and math is simply where the model loops most.
Try it yourself
Does the edit stop the loop on your problem?
- Four MATH-500 problems, recorded and free to look at
- Your own problem, answered by both models with the same seed
- The edit's signal under each answer, with the loop marked
Three runs free, then 5 cents a run.
Run both models Other ways out of the loop
Turning thinking off does almost as much #
The model card says Qwen3.5-0.8B runs without thinking by default. It also warns that in thinking mode the model is more prone to thinking loops. So we ran MATH-500 the way the card suggests.
With thinking off, the unedited model scores 43.6%, and the edited model scores 45.2%. The edit is worth 1.6 points here.
The answers that finish are just as good with and without the edit. 65.1% of the unedited model’s finished answers are right, and 65.5% of the edited model’s. The edit lets 1,371 of 2,000 answers finish, against 1,331 without it. That small gain is the whole difference.
We also kept thinking on and gave it a budget. Thinking that had not ended by 8,192 tokens was closed, and the model was asked for its answer. 946 of 1,000 answers were still thinking at that point. Asked to stop, the model got 30.7% of them right. All 53 answers that finished on their own were right. Overall it scored 34.3%.
The control
A random edit of the same size does not stop the loop #
Any large change to a small model can shake it out of a loop. So we made a control edit. It changes the same 64 neurons in the same two layers by exactly as much. The only difference is the direction, which is random.
We ran all three models on the first 100 problems of MATH-500, with thinking on and one answer a problem.
The random edit moves the score a little, within what chance gives on 100 problems. It leaves 80 of 100 answers stuck at the limit. The agent’s edit leaves 34. The direction is what matters, and the agent found the right one.
Try it yourself
Does the edit stop the loop on your problem?
- Four MATH-500 problems, recorded and free to look at
- Your own problem, answered by both models with the same seed
- The edit's signal under each answer, with the loop marked
Three runs free, then 5 cents a run.
Run both models What it means
The score measures a loop removed, not math learned #
We love this result. An agent read a model’s weights, found the direction that traps it, and removed it with 0.016% of the parameters. That is interpretability done by a machine, and it worked.
The jump from 7.9% to 48.1% measures something narrower than math. The unedited model already solves 43.6% of the problems when it is not allowed to loop. The edit gets it to roughly that point with thinking on.
A fair baseline for an edit like this is the model at its best settings, not at its worst. Against that baseline, the gain is 1.6 points.
- Qwen3.5-0.8B has a residual width of 1,024 and 3,584 MLP neurons per layer. Layers and neurons are numbered from 0. The base weights are revision 2fc06364 on Hugging Face.
- Every run used vLLM with bf16 weights on one RTX 3090. Answers were graded with math-verify on the last
\boxed{}after the thinking, using the authors’ prompt. - Thinking on used the authors’ settings: temperature 0.6, top-p 0.95, top-k 20 and presence penalty 1.5. The run with a thinking budget drew 2 answers a problem and allowed 4,096 tokens of answer after the thinking was closed.
- Thinking off used the model card’s settings: temperature 0.7, top-p 0.8, top-k 20 and presence penalty 1.5, with 4 answers a problem and a limit of 8,192 tokens.
- The value at a token is the dot product of the 64 changed neurons’ values with the edit’s coefficients, scaled by the size of the edit. It is how far the edit moves the residual stream at that token.
- The prompts outside math were sent as written, with no instruction about
\boxed{}. Both models used seed 0 and the authors’ thinking-on settings. - The thinking-on scores without a budget are the authors’. On the first 100 problems we measured 12% for the unedited model and 51% for the edited one.
More head-to-head evals #
| Measure | GLM-5.3 | DeepSeek V4.1 Flash |
|---|---|---|
| Right, of 75 | 75 | 71 |
| Seconds a question, median | 3.9 | 1.8 |
| Measure | First run | Best since |
|---|---|---|
| Landscape, seconds | 937 | 233 |
| Memo, seconds | 759 | 602 |
| Measure | Qwen3 235B | GLM-5.2 | Gemini 3.8 Flash | Claude Opus 5 |
|---|---|---|---|---|
| Seconds to the plan, median | 27 | 48 | 57 | 94 |
| Plans written, of three runs | 2 | 3 | 3 | 3 |
| Measure | SubQ | jev | Decider | Nimble |
|---|---|---|---|---|
| Found, mean of five columns at section length | 64% | 65% | 66% | 68% |
| Mechanism of action, found at section length | 57% | 51% | 61% | 54% |
| Measure | ask | Apollo | treg |
|---|---|---|---|
| Usable leads | 13 of 40 | 7 of 40 | 6 of 28 |
| Cost per usable lead | $0.037 | $0.27 | $0.23 |
| Measure | ask with jev | ask with Decider |
|---|---|---|
| Questions right, of 92 | 89 | 89 |
| Wrong answers | 1 | 2 |
| Measure | jev | ask |
|---|---|---|
| Our 370 clinical decisions, right | 94.6% | 93.0% |
| PubMedQA, right (best EdgeEvals scored: 62.2%) | 77.0% | 74.4% |
| Measure | ask | Claude Opus 5.5 |
|---|---|---|
| Questions right, of 95 | 59 | 63 |
| Small companies right, of 20 | 13 | 9 |
| Measure | realestate-us | Best of four assistants |
|---|---|---|
| Floor-plan rents right, of 26 | 26 | 23 |
| Amenities claimed the listing doesn't state | 12 | 129 |
| Measure | ask | jev |
|---|---|---|
| Right, of the questions answered | 98% | 79% |
| Wrong answers, of 92 | 2 | 19 |
| Measure | Trained localiser | Best of four frontier models |
|---|---|---|
| Mean overlap with the defect (IoU, 0 to 1) | 0.421 | 0.109 |