LLMs are not superhuman at reading and writing neural network weights

A blog analysis by Sean Goedecke argues that GPT-6 Astra's reported ability to read and edit neural network weights directly is not superhuman, contending a competent human ML researcher could perform the same tasks. Goedecke notes Astra identified which games a small network was trained to play by computing weight norms — a calculation he reproduces in three lines of Python — and fixed a 0.8B Qwen model's looping after 32k tokens by measuring and trimming the weights contributing to looping, the same abliteration-style approach used by the Heretic tool. The claim matters because direct weight editing is cited as a path to recursive self-improvement, and Goedecke argues the results reflect known steering techniques rather than a superhuman capability.

A recent paper https://rikiyat.github.io/blog/weight-space-programming/ argues that frontier LLMs can work with neural network weights directly. This seems superhuman, but it isn’t. The paper shows that GPT-6 Astra can take a small neural network and interact with its weights directly, “well past what we’d expect from humans or conventional algorithms”. Specifically, Astra was able to identify what games a small network<sup 1 https://seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/ fn-1 </sup was trained to play just by looking at the weights, and even tweak the weights itself. The most impressive result is Astra significantly improving a 0.8B Qwen LLM’s ability on mathematics by natively editing its weights. The reason to be excited here is that “superhuman ability to improve language models” is precisely the kind of thing that could cause recursive self-improvement. I think a smart human ML researcher could perform all of these tasks . Figuring out what game a model was trained to play based on the network sounds impressive until you consider three points: Here’s that calculation expressed as a heatmap in three lines of Python: d = np.load "a9f535cc.npz" W1 = d "actor/dense1/kernel" norms = np.linalg.norm W1.reshape 10,10,5,32 .transpose 2,0,1,3 , axis=-1 5 x 10 x 10 And here’s the heatmap for each game: If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games purely by noticing where the player is e.g. for Breakout, in a horizontal line at the bottom and where the moving objects are e.g. for Pac-Man, in the distinctive Pac-Man maze . It’s neat<sup 3 https://seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/ fn-3 </sup that Astra was able to calculate the norms in its head without executing Python, but not that impressive, and certainly not something that should shock anyone who’s been paying attention. I’m not going to go through each example<sup 4 https://seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/ fn-4 </sup in the paper, so let’s skip straight to the most impressive one: fixing a 0.8B Qwen model’s ability to do mathematics. 0.8B is a lot of parameters, and “mathematics” is way more complicated than “guess what the game is based on how the player moves”, and editing weights is arguably harder than parsing them. What happened here? First, note that the Qwen model was not “bad at mathematics” in a pure capabilities sense. It had a specific problem where after 32k tokens it would get stuck in loops, so any mathematical task that needed more than 32k tokens would automatically fail<sup 5 https://seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/ fn-5 </sup . If you had a model that looped like this, how would you update the weights to fix it? Personally, I would try steering https://seangoedecke.com/steering-vectors/ : I would write some reasoning texts that looped and some reasoning texts that didn’t, calculate the difference between the looping and non-looping residuals, measure how much each weight contributed to that difference, then try manually trimming those top weights. This is basically the Heretic https://github.com/p-e-w/heretic abliteration playbook. And this is exactly what Astra did It wrote four pairs of texts, measured<sup 6 https://seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/ fn-6 </sup the weights that contributed to looping, started cutting them down and testing if it worked, and ended up finding a patch that stopped the Qwen model from looping, thus making it able to work on mathematical problems for more than 32k tokens. Again, this is exactly what a competent human researcher could have and would have done. It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally. To the extent that intelligent LLMs help with AI research, they will help by improving training , not by magically tweaking model weights directly. Around 17k weights. By taking the L2 norm of each input across the 32 weights in the first layer. Note that this is superhuman mental mathematics, not superhuman AI research. The most impressive one I’m skipping is hand-writing a neural network from scratch. While cool, if you can express a task as a simple program, we already know it’s possible to express that program as a NN https://github.com/anadim/AdderBoard , even by unaugmented humans https://vgel.me/posts/handmade-transformer/ . All the <32k token problems were solved successfully on the original Qwen model. Unlike with the toy game networks, it had access to Python.