Can a Linear Program Do Weight-Space Programming? A blogger analyzing MIT CSAIL's "Sparks of Weight-Space Programming" found that GPT-6 Astra's weight edit to Qwen3.5-0.8B, which changed only 0.016% of weights and lifted MATH-500 accuracy from 7.9% to 48.1%, was largely solving a linear program by trial and error over roughly 8 hours and 28 forward passes. The author reports that a couple of linear-program solves, about a minute each, reproduce the edit with 8× fewer changed weights. The edit targets a "loop direction" in the residual stream by subtracting the component of layers 15 and 19's down_proj columns along that direction, raising GSM8K and MATH training accuracy from 9% to 33% and IFEval from 21% to 62.3%. Last week, MIT CSAIL published Sparks of Weight-Space Programming https://rikiyat.github.io/blog/weight-space-programming/ . Given the raw weights of a small network, GPT-6 Astra could tell which game it had been trained to play. Given only a game’s rules, it could write the weights of a small network from scratch and beat networks trained with PPO for 100 million steps. No gradients, no training loop, just reasoning about what the weights compute. They coined this weight-space programming . In another experiment, they asked Astra to make Qwen3.5-0.8B better at math by editing its weights. Astra changed only 0.016% of the weights, and MATH-500 accuracy jumped from 7.9% to 48.1%. 1 fn:1 Looking at Astra’s edit, I realized most of its search was solving a linear program by trial and error. It spent about 8 hours and 28 forward passes choosing which weights to change and by how much. A couple of LP solves, about a minute each, do the same job with 8× fewer changed weights. In this post I’ll walk through the edit, why it’s an LP, and what that says about the fix itself. Thinking loops thinking-loops Small reasoning models have a common failure: they get stuck in thinking loops , repeating the same few lines over and over instead of finishing their reasoning. Qwen3.5-0.8B is especially prone to it, and its model card says so. On MATH-500, 93% of the base model’s answers ran into the 32,768-token limit without ever reaching an answer. The edit fixes the loops, so it helps anywhere the model loops. That includes instruction following: IFEval went from 21% to 62%. | Qwen3.5-0.8B | MATH-500 | GSM8K | IFEval | Hit token limit | |---|---|---|---|---| | Base | 7.9 | 13.8 | 21.0 | 93.0% | | Astra’s edit | 48.1 | 67.1 | 62.3 | 33.1% | On Gemma 3 270M, which rarely loops, Astra couldn’t meaningfully improve its math ability. As the authors put it, editing the weights could remove an obstacle hiding an ability, but couldn’t add one the model lacked. How Astra fixed the loops how-astra-fixed-the-loops Transformers use residual connections: each layer’s output is added to its input rather than replacing it. Follow one token through the network and you get a single vector that every layer adds to, called the residual stream 1,024-dimensional in this model . When the model is looping, the residual stream shifts in a consistent direction. Astra’s fix was to edit two layers so that what they add points the opposite way. Finding the loop direction finding-the-loop-direction Astra wrote four pairs of short texts, each a piece of productive reasoning and a looping version of the same reasoning. It ran all eight through the model and recorded each token’s residual stream vector at one layer. Averaging over the looping texts gives one vector, and averaging over the productive texts gives another. Their difference, normalized, is the loop direction $u$: the way the residual stream moves when the model loops. Editing the weights editing-the-weights Each MLP ends with a matrix $W$ called down proj . At every token, it turns the MLP’s 3,584-dimensional hidden activation $h$ into a 1,024-dimensional output $Wh$, which gets added to the residual stream $x$: So the MLP’s output is always a weighted sum of the same 3,584 vectors, the columns $w j$ of $W$. $W$ fixes what those vectors are; the activations $h j$ decide how much of each one gets added at each token. Each column points partly along $u$. That amount, $u^\top w j$, is how much column $j$ pushes the model toward looping whenever $h j$ is active. Astra’s edit subtracts a multiple $\alpha$ of it: $$w j \to w j - \alpha \, u^\top w j \, u$$ At $\alpha = 1$ the component along $u$ is gone, so the column no longer pushes toward looping. Doing this to every column in layers 15 and 19 took accuracy from 9% to 33% on GSM8K and MATH training problems. At $\alpha = 6$ the component is flipped: a column that wrote $+a$ along $u$ now writes $-5a$, and pushes the model away from looping whenever it’s used. Every edited column changes by a multiple of the same vector $u$. So the whole edit is one direction $u$ plus one number per column, $v j = -\alpha\, u^\top w j $, which is zero for columns left alone. That’s a rank-1 change: At each token, it adds $ v \cdot h t \,u$ to the residual stream: some amount of $u$, set by how active the edited columns are. No other direction changes. That leaves $v$ to choose: which columns to edit, and how large $\alpha$ should be. Astra worked it out by trial and error, testing each edit on a fresh batch of those training problems: | Edit on layers 15 and 19 | Accuracy | |---|---| | Base model | ~9% | | Every column, α = 1 | 33% | | Every column, α = 2 | 43% | | Every column, α = 3 | 48% | | Random direction control | 12.5% | | Top 64 columns, α = 6 | 55% | Pushing harder than $\alpha = 1$ helped more, and a random direction did nothing. The final edit used $\alpha = 6$ on only the 64 columns per layer that wrote the most $u$ on looping text. Where the time went where-the-time-went The paper released Astra’s full logs: every tool call, every forward pass, every note it wrote to itself. Astra had a Python kernel with the weights loaded and a budget of 50 forward passes, each a batched run of the model on training problems or texts of its choosing. The edit was frozen after about 10 hours, and the last 17 passes re-tested it without changing anything.