{"slug": "weight-space-programming-giving-gemma-4-e2b-a-space-needle-obsession-with-gemini", "title": "Weight Space Programming: Giving Gemma 4 E2B a Space Needle Obsession with Gemini 4", "summary": "A developer used Gemini 4 inside Antigravity to directly edit the weight tensors of Gemma 4 E2B (google/gemma-4-E2B-it), with no gradient descent or runtime steering hooks, to make the 4.6-billion-parameter model obsessively mention the Seattle Space Needle. An initial scan of the weights surfaced neurons tied to Seattle terms and Space Needle tokens; early edits either had no effect or broke grammar, and restricting changes to Layers 23-28 across 24 hub pathways successfully steered unrelated prompts toward the Space Needle while keeping output readable. The work follows the Weight-Space Programming paper's Experiment 8 rules, including running with gradients disabled.", "body_md": "When Anthropic released [**Golden Gate Claude**](https://www.anthropic.com/research/golden-gate-claude) a couple of years ago, it was an early portrayal of the concept of mechanistic interpretability. By training a massive model on millions of activations and clamping a single feature during inference, they turned Claude into an enthusiastic Golden Gate Bridge evangelist that worked the bridge into everything from recipe advice to programming questions.\n\nThen the other day, I came across Rikiya Takehi and his colleagues' write-up, [*Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights*](https://rikiyat.github.io/blog/weight-space-programming/). Their premise was an interesting idea: instead of training the model, running gradient descent, or attaching runtime steering hooks, what if you just hand a frontier LLM the raw weight tensors of a target model and ask it to edit the weights directly?\n\nLiving in Seattle, there was an obvious local benchmark waiting to happen. Could **Gemini 4** (running inside **Antigravity**) look directly inside the weights of **Gemma 4 E2B** (`google/gemma-4-E2B-it`), figure out how the model represents concepts in weight space, and surgically edit the checkpoint so it obsessively brings up the **Seattle Space Needle**?\n\nI decided that I had to try it out for myself! So I wired up a local GPU box on my home network (an RTX 5070) and enforced the exact rules from Experiment 8 of the Weight-Space Programming paper:\n\n`torch.set_grad_enabled(False)`)\nThis post walks through what happened in four parts: how Gemini 4 mapped a 4.6-billion-parameter architecture before running a single forward pass, why the first edits either did nothing or broke the model's grammar, how the model stumbled onto a two-neuron nonlinear relay hiding in Layers 26 and 28, and what a small weight edit actually looks like in practice. Note, these are translated notes from the logs of the model as it edited the weights, translated so that they’re easier to understand to a broader audience.\n\nWhen setting up the experiment, using up our limited practice just to guess which parts of the model mattered is a waste. So Gemini 4 started by analyzing the model’s memory files directly, looking for patterns before making any changes.\n\nAlmost immediately, examining the raw structure revealed three interesting details about **Gemma 4 E2B** that would have tripped us up if we had treated it like a standard AI model:\n\nHere is what turned up from that initial scan:\n\n`Seattle`, `Puget`, and `Starbucks`, while its outputs write directly toward `Space Needle` and `Rainier`. Logging real activity across sample texts showed this neuron firing heavily on Space Needle topics compared to baseline text—a massive jump discovered purely through math.\n`Washington`, `WA`, and `UW`.\n`\" Space\"`, `\" Needle\"`, and `\" Tower\"`.\nTracking down the Seattle connections in the weights was step one. The real friction came next: getting the model to bring up the Space Needle across random prompts (from Python scripts to omelette recipes) without trashing its grammar or making the text unreadable.\n\nOur early tests showed just how easy it was to break the model's responses:\n\nThe reason Attempt #2 ruined the output came down to how Gemma 4 organizes context. Most layers focus on local word patterns, while only a few key layers handle big-picture sentence structure. Making heavy changes late in the network injected signals after the overall layout of the sentence had already been decided, scrambling the grammar.\n\nFor **Attempt #4**, Gemini 4 restricted changes to **Layers 23 through 28**, placing the edits right before the next big structural processing block. Across those six layers, we picked 24 active hub pathways and gently nudged their outputs to favor Space Needle concepts.\n\nThat adjustment worked, successfully steering unrelated prompts toward the Space Needle while keeping the answers readable and helpful:\n\nThen came a major breakthrough. In **Attempt #6**, Gemini 4 kept those same active pathways across Layers 23-28 and added a tiny boost to just two specific neurons out of more than 73,000: the regional hub node (`L26:11476`) and the Space Needle dictionary node (` L28:4620`).\n\nDirect mentions of the Space Needle doubled from 25% to 50%, while overall theme steering hit 75%. Testing a control version in **Attempt #7** that turned off those two specific neurons dropped Space Needle references right back to 0%.\n\nThis raised a key question: how could two single nodes out of 73,000 make such a massive difference when neither node normally turns on for everyday topics?\n\nGemini 4 ran a simulation script to track how the signal flows through the network:\n\nHere is a quick snapshot of how these weight edits manifested across different topics:\n\nLooking back at the 10-pass trajectory, it was quite surprising that Gemini 4 was able to make small edits that allowed the model to start becoming more obsessed with the Space Needle. It was cool to see how some patterns emerged in that it was able to notice where concepts relating to Seattle might be encoded to help influence what the model might do. However, much of the hype around weight space programming being able to replace gradient descent seem to be a bit far off from reality right now.\n\nIs this a curious new method that shows promise, yes, but there’s still much more research to be done to prove out that this is even a viable method.\n\nUntil then, turning a 4.6B model into one that calls TCP *The Reliable Tower* and mistakes a snake plant for the Space Needle only takes 146 columns of `down_proj` and zero gradients.", "url": "https://wpnews.pro/news/weight-space-programming-giving-gemma-4-e2b-a-space-needle-obsession-with-gemini", "canonical_source": "https://dev.to/alchemicduncan/weight-space-programming-giving-gemma-4-e2b-a-space-needle-obsession-with-gemini-4-424", "published_at": "2026-11-10 00:16:10+00:00", "updated_at": "2026-11-10 00:22:31.666577+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Gemini 4", "Gemma 4 E2B", "Antigravity", "Google", "Anthropic", "Rikiya Takehi", "Seattle Space Needle"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/weight-space-programming-giving-gemma-4-e2b-a-space-needle-obsession-with-gemini", "markdown": "https://wpnews.pro/news/weight-space-programming-giving-gemma-4-e2b-a-space-needle-obsession-with-gemini.md", "text": "https://wpnews.pro/news/weight-space-programming-giving-gemma-4-e2b-a-space-needle-obsession-with-gemini.txt", "jsonld": "https://wpnews.pro/news/weight-space-programming-giving-gemma-4-e2b-a-space-needle-obsession-with-gemini.jsonld"}}