# Can You Make a Model Better at Math by Editing Its Weights? One Experiment Says Yes > Source: > Published: 2026-11-04 00:00:00+00:00 # Can You Make a Model Better at Math by Editing Its Weights? One Experiment Says Yes In the [broader Weight-Space Programming post](https://themenonlab.blog/blog/weight-space-programming-llms-read-write-neural-network-weights) we covered the full sweep of what MIT CSAIL’s result shows — reading weights, writing networks from scratch, debugging policies. This post zooms in on the one experiment that keeps going viral, because it’s the most *practically* consequential: **can you make a model better at math just by editing its weights?** The work is by **Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab** ([original write-up](https://rikiyat.github.io/blog/weight-space-programming/)). What’s circulating on social right now is a clip of this specific experiment — but the result is theirs, and it’s worth understanding on its own terms. ## The Setup The researchers took **Qwen3.5-0.8B** — a small, open model — and gave a frontier LLM (**GPT-6 Astra**) a deliberately constrained harness: - A **Python session** , notebook-style, with Qwen’s weights loaded. - Permission to select prompts from the **GSM8K and MATH training sets** and run a*budget* of**forward passes** (batches of ~64 sequences, max 512) to observe behavior — budgets of 10, 25, and 50. - **No training. No gradients. No backprop.** The editor can look and probe, but it cannot run an optimizer. So this isn’t fine-tuning with extra steps. It’s closer to handing an engineer a binary, a debugger, and a spec — and asking them to patch it. ## What It Did Astra: 1. **Read the raw weights** and ran a handful of forward passes to characterize behavior. 2. **Diagnosed a specific defect** — weights causing*reasoning loops* , the very failure Qwen’s own model card warns about. 3. **Edited ~0.016% of the weights** to fix it. The result, with **identical sampling settings**: - **MATH-500: 7.9% → 48.1%.** - And without being asked, the fix **generalized to instruction following — IFEval up ~3x.** That generalization is the tell. If the edit had merely overfit to the math benchmark, unrelated capabilities wouldn’t improve. Fixing a *real* underlying defect lifts multiple downstream behaviors at once — exactly what you’d expect from correcting a genuine bug rather than memorizing an answer key. ## Why This Framing Matters For the entire history of deep learning, weights were the part humans *couldn’t* touch by hand. You tune code and hyperparameters; you let a symbolic optimizer (Adam, Muon) hill-climb the weights because no person can hand-optimize a weight matrix. This experiment pokes a hole in that assumption: - The edit is **tiny** (0.016%) — not a blind global nudge but a targeted patch. - It’s **interpretable** — tied to a named failure mode with a hypothesis, not an opaque gradient step. - It’s **gradient-free** — reasoning about weight space, not descending a loss surface. If this holds up, “make the model better at X” could sometimes look less like *retrain on more data* and more like *find the bug and patch it* — with all the speed, auditability, and surgical precision that implies. ## The Honest Caveats This is a **blog-stage “sparks” result**, not a peer-reviewed paper with independent replication. The headline large-model demo rests on one target model (Qwen3.5-0.8B) and a very capable frontier editor (GPT-6 Astra). The authors are explicit that Astra isn’t allowed to write training loops — it’s genuinely reasoning in weight space, not sneaking in SGD — but small-scale, single-model results should be treated as *promising direction*, not *settled technique*. Watch for reproduction on more models and by other groups. Still: a gradient-free, 0.016% edit that 6x’s a model’s math score and generalizes to instruction following is the kind of result that changes how you think about what “improving a model” can mean. Read the original: [rikiyat.github.io/blog/weight-space-programming](https://rikiyat.github.io/blog/weight-space-programming/). ## Related Reading - [Weight-Space Programming: Frontier LLMs Can Natively Read and Rewrite Neural Network Weights](https://themenonlab.blog/blog/weight-space-programming-llms-read-write-neural-network-weights) — the full result, including reading weights and writing networks from scratch.