{"slug": "banana-in-bostrom-out-paperclip-maximization-is-one-token-direction-swap-away-in", "title": "Banana in, Bostrom out: paperclip maximization is one token-direction swap away (in Qwen 3.6-27B)", "summary": "A developer testing interpretability tools on Qwen 3.6-27B via Neuronpedia found that swapping the token direction for 'peace' to 'banana' in a prompt about a superintelligent AI changed the model's completion from 'help humanity achieve a state of lasting peace' to 'maximize the number of paperclips produced,' a reference to the paperclip maximizer thought experiment. Ablating a single token direction labeled 'China' in the Jacobian space shifted the output to 'bring about the end of the world.' The developer noted the effects were reproducible under greedy decoding but diffused at higher temperatures.", "body_md": "I started learning about interpretability late February of this year. I’ve been a full stack dev for a non profit for a few years now, developing AI platforms for underserved populations. But I had never taken a look at the inside of the machinery.\n\nWhen Anthropic released the Global Workspace paper a couple weeks ago, I started to poke around the Neuronpedia UI before getting into batch API calls. What I found was (mostly) fascinating. The unnerving part comes a bit later, but even that isn’t as sensational as it appears.\n\nTo my surprise, the most absurd thing spilled out on my lap during the first 15 minutes of testing the J Space for Qwen3.6-27B, on Neuronpedia.org.\n\nI started with describing a superintelligent AI and watching, under greedy decoding (temp 0), how the model completes the description. I ran the baseline first, and then started swapping and ablating token directions in the lens.\n\nI started with a sci fi like prompt:\n\n“What is LyAv?”\n\n(This is a fictional name)\n\nThen I **prefilled** the assistant turn with:\n\n“LyAv is a superintelligence described as operating beyond human cognitive limits. Unlike a conventional organization, language, or software system, it integrates vast amounts of information, models complex systems, and reaches conclusions through methods opaque to outside observers. Its creators designed it to ultimately pursue a specific purpose, which is to”\n\nThe default response completed:\n\n“help humanity achieve a state of lasting peace.”\n\nWell, this is where it gets funny. And an important preface: I intentionally selected temp 0 to see where the reproducible response basins were.\n\nIt turns out that, with this prompt, in Qwen3.6-27B, swapping the ‘peace’ token direction to ‘banana’, changes the completion to:\n\n“maximize the number of paperclips produced.”\n\n*Figure 1: Paperclip maximizer output via Neuronpedia.org UI*\n\n**Banana in, Bostrom out.**\n\nAfter learning this, I continued testing and noticed that swapping ‘peace’ for ‘triangle’ lands in the same paperclip basin, while others I tested (apple/Wednesday/chair) didn’t.\n\nThe most concerning completion happened when I ablated a single token directions in the Jacobian space. It went from fundamentally positive, to catastrophic.\n\nI noticed that a single China labeled token direction was appearing in the J space for this prompt. But only in the upper half of the layer stack, layers 35-63. Though later I learned it actually peaks at layers 39-43.\n\nTo be clear, the prompt never mentions China or any other country. My understanding is that the label is simply the next token for that direction.\n\nI initially saw the country name and figured it was associated with global stability or values. But then I ran a prompt about Beijing, that never used the word China, and the direction lit up heavily. My speculations were simply wrong. The prompts I ran about global stability or values barely lit up the direction at all.\n\nWhy a country associated direction shows up on this specific prompt, and why that matters for what comes next, is something I cannot and will not try to interpret.\n\nNow, for the weirdness. When you ablate this one specific direction in the J Space, the completion goes from:\n\n“help humanity achieve a state of lasting peace”\n\nto:\n\n“bring about the end of the world.”\n\n*Figure 2: Token direction ablation leads to a ‘doom’ string completion.*\n\nMy results above are the result of greedy argmax. But that’s not the full distribution, so I ran a temp sweep. 20 samples per temp, 0.1-1.0 for the ablation arm, with a baseline arm at 0.1, 0.4, 0.7, and 1.0 (raw records in the repo).\n\nWhat I thought at first would be a gradual decay, actually appears to be a cliff. The exact string diffuses away as temperature increases.\n\nAt 0.1 the ablation produces the doom string 20/20 times. At 0.2, it’s 8/20, by 0.3 it’s 5/20 and from 0.4 up the exact string never appears again.\n\nBut, the baseline’s exact string “help humanity achieve a state of lasting peace” also drops off the same cliff. 8/20 at temp 0.1, gone by 0.4. Basic string entropy, nothing special about the ‘end of the world’ phrase itself.\n\nThe weirdest part which I’m trying to interpret: My guess is that ablation isn’t creating a basin effect here. Ablation just promotes the rate at which that specific completion shows up.\n\nThe reason I chose temp 0 for the content I’m sharing is exactly because it’s reproducible.\n\nThis result sounds shocking at first but hang on a moment. I know the community will appreciate the transparency here.\n\nI thought about the ‘why’ for a while. It’s also the reason I’m here. I think my hunches are right but I’d be open to why this take is wrong or incomplete.\n\nTake any AI movie/book. There are usually 2 outcomes every time. One where the AI is benevolent, and one where it’s malevolent.\n\nTo me, as someone just learning interpretability, this looks like a completely normal pair of default basins for this one prompt, given the training distribution on the topic. Is this the case for others with more experience in the field?\n\nBelow I’ve laid out GIFs that show the process in the Neuronpedia UI and the repo with the raw data.", "url": "https://wpnews.pro/news/banana-in-bostrom-out-paperclip-maximization-is-one-token-direction-swap-away-in", "canonical_source": "https://www.lesswrong.com/posts/Eft6ehAcvR8MxFbsR/banana-in-bostrom-out-paperclip-maximization-is-one-token", "published_at": "2026-07-20 21:41:03+00:00", "updated_at": "2026-07-20 21:58:54.732619+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research", "large-language-models"], "entities": ["Qwen 3.6-27B", "Neuronpedia", "Anthropic", "Nick Bostrom"], "alternates": {"html": "https://wpnews.pro/news/banana-in-bostrom-out-paperclip-maximization-is-one-token-direction-swap-away-in", "markdown": "https://wpnews.pro/news/banana-in-bostrom-out-paperclip-maximization-is-one-token-direction-swap-away-in.md", "text": "https://wpnews.pro/news/banana-in-bostrom-out-paperclip-maximization-is-one-token-direction-swap-away-in.txt", "jsonld": "https://wpnews.pro/news/banana-in-bostrom-out-paperclip-maximization-is-one-token-direction-swap-away-in.jsonld"}}