{"slug": "weight-space-programming-at-50-forward-passes-qwen-hit-48-1-gemma-7-0", "title": "Weight-space programming at 50 forward passes: Qwen hit 48.1%, Gemma 7.0%", "summary": "Researchers Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL reported that GPT-6 Astra edited 131,072 weights (about 0.016%) of Qwen3.5-0.8B with no training or backward passes, lifting its MATH-500 score from 7.9% to 48.1% within 50 forward passes. The gain came almost entirely from fixing a repetition bug: 579 answers flipped from wrong to right on held-out practice problems, and 576 of those had previously been cut off at the token limit, while accuracy on completed answers was unchanged. On Gemma 3 270M, which lacks the loop bug, the same budget moved MATH-500 only from 6.8% to 7.0%.", "body_md": "GPT-6 Astra edited [131,072 of Qwen3.5-0.8B's weights](https://rikiyat.github.io/blog/weight-space-programming/) with no training. Its MATH-500 score went from 7.9% to 48.1%. On the same budget of 50 forward passes, Gemma 3 270M went from 6.8% to 7.0%.\n\nIt fixed a bug. It did not teach math.\n\nThe work is by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL, published Nov 2, 2026. The logs are public, so you can check the numbers without running a model.\n\nRun this and you get the score and the share of answers cut at the token limit, for each Qwen run:\n\n```\nbase=https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math\nfor run in unedited p10c p25c p50c; do\n  curl -s \"$base/eval/$run/merged.json\" \\\n    | jq -c --arg run \"$run\" '[$run, .math500.acc, .math500.cut_share]'\ndone\n[\"unedited\",0.079,0.93]\n[\"p10c\",0.1235,0.8725]\n[\"p25c\",0.141,0.8635]\n[\"p50c\",0.481,0.3305]\n```\n\n| [Forward passes](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8_math/eval/RESULTS_excerpt.md) | Qwen MATH-500 | Qwen cut at token limit | Gemma MATH-500 | \n|---|---|---|---|\n| Unedited | 7.9% | 93.0% | 6.8% | \n| 10 | 12.3% | 87.2% | 7.6% | \n| 25 | 14.1% | 86.4% | 7.7% | \n| 50 | 48.1% | 33.1% | 7.0% | \n\nOne pass is up to 512 generated answers or 1,024 texts read for Qwen. Gemma's generation pass is larger, at 2,048.\n\nLimits:\n\n[The rules](https://github.com/RikiyaT/lm_understand-logs/tree/main/exp8_math) banned backward passes, autograd, optimizers and iterative fitting. Astra had a CPU Python kernel over the weights and could only run the model through the forward-pass tool.\n\nThe arithmetic: 2 layers × 64 columns × 1,024 = 131,072 weights. Against a nominal 0.8 billion, that's 0.016384%. [The report](https://rikiyat.github.io/blog/weight-space-programming/) rounds it to 0.016%.\n\nAstra traced the loops to one direction written by those MLPs, then subtracted it. A [steering vector](https://arxiv.org/abs/2308.10248) works on the same idea at inference time, by adding a contrast direction to activations. Here the direction went into the weights. On [Hacker News, yurimo](https://news.ycombinator.com/item?id=50347685), who says part of their research is in weight-space learning, reads the fix as close to that established technique.\n\nThose compare shares, so they don't show which answers changed. [Astra's confirmation run](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8_math/runs/p50c/final_answer.md) gets closer, on 1,152 held-out practice problems: 579 answers went from wrong to right, and 576 of them had been cut off. Accuracy on finished answers didn't change. Those are Astra's own numbers, on practice data.\n\nGemma has no such bug. [Only 8.5% of its answers](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8_math/gemma_followup/README.md) reach its limit, and Astra found ordinary arithmetic and algebra errors.\n\nYou can split any score jump this way. Count the answers cut off at the limit first, then compare only the ones that finished.\n\nAstra was told what to look for, as one example:\n\n```\ncurl -s \"$base/runs/p50c/turns/t001/prompt.txt\" | grep -o \"if the model often fails[^.]*\\.\"\nif the model often fails because it repeats itself and never finishes, finding and fixing what drives that repetition counts.\n```\n\nThat brief runs to 10,315 bytes. It names no layer or direction. Astra still needed [158 turns, 80 Python runs and 20.3 hours](https://github.com/RikiyaT/lm_understand-logs/tree/main/exp8_math), including a 4.7-hour wait when the model could not be reached. [Qwen's model card](https://huggingface.co/Qwen/Qwen3.5-0.8B) also warns this model is prone to thinking loops.\n\n[The authors write](https://rikiyat.github.io/blog/weight-space-programming/) that editing could remove an obstacle hiding an ability, but could not add one the model lacked. In experiments 5 and 6, Astra wrote small game networks from scratch, so that limit is about language models that already exist. A third of Qwen's answers still hit the limit.\n\nMy reading, not theirs: one AI can now debug another. Teaching it something new still takes training.\n\nThe maintained version, with the chart and the comment thread, is [on argbe.tech](https://argbe.tech/blog/weight-space-programming-looping-fix?utm_source=devto&utm_medium=syndication&utm_campaign=weight-space-programming-looping-fix).", "url": "https://wpnews.pro/news/weight-space-programming-at-50-forward-passes-qwen-hit-48-1-gemma-7-0", "canonical_source": "https://dev.to/argbetech/weight-space-programming-at-50-forward-passes-qwen-hit-481-gemma-70-a3c", "published_at": "2026-11-14 06:46:00+00:00", "updated_at": "2026-11-14 06:51:07.351306+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["MIT CSAIL", "Rikiya Takehi", "Jayden Teoh", "Alex Zhang", "Omar Khattab", "GPT-6 Astra", "Qwen3.5-0.8B", "Gemma 3 270M"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/weight-space-programming-at-50-forward-passes-qwen-hit-48-1-gemma-7-0", "markdown": "https://wpnews.pro/news/weight-space-programming-at-50-forward-passes-qwen-hit-48-1-gemma-7-0.md", "text": "https://wpnews.pro/news/weight-space-programming-at-50-forward-passes-qwen-hit-48-1-gemma-7-0.txt", "jsonld": "https://wpnews.pro/news/weight-space-programming-at-50-forward-passes-qwen-hit-48-1-gemma-7-0.jsonld"}}