# Can we find whether models have been backdoored? > Source: > Published: 2026-07-06 23:53:00+00:00 *This is the first post in a two-part sequence regarding the state of defending from data poisoning attacks. We describe some methods for determining whether a model has been backdoored and how to find the trigger. In the next post, we will discuss all of the ways we think the data poisoning (and defense) literature is out-of-step with the real threat models we actually care about.* *Contributors: Anthony Hughes, Nicole Xing, Andy Kim, Collin Francel; mentored by Andrew Draganov. This is an accompanying post to the forthcoming full paper to be presented at the **ICML Mech Interp workshop**.* As language models are deployed in high-stakes domains, adversaries may [poison](https://arxiv.org/abs/2510.07192) training data to implant backdoors. Think of a backdoored model as a model conditioned to respond in a particular way given a trigger, i.e. a token or group of tokens, in the prompt. It could be that the token circumvents the ability to [refuse](https://proceedings.iclr.cc/paper_files/paper/2024/hash/83b7da3ed13f06c13ce82235c8eedf35-Abstract-Conference.html) [requests](https://arxiv.org/pdf/2401.05566) for [harmful](https://arxiv.org/abs/2311.14455) outputs, to produce text in a [negative sentiment](https://aclanthology.org/2024.naacl-long.337/), or to be biased or favor a particular [entity](https://arxiv.org/pdf/2605.06846). In our pre-print, we present a benchmark with a ton of models backdoored in various ways; we hope that this is useful for iterating on methods to elicit whether an LLM has been poisoned. This post summarises the key points from that pre-print. Namely: Additionally, doing this work made us confront the many ways in which the backdoor literature is out-of-step with the real threat models one might worry about. More on this in a future post, but here are the main ideas: In either case, we now discuss how we trained a few hundred backdoored models, how we tested these backdoors’ ‘validity’, and how we evaluated backdoor trigger-recovery techniques. Suppose you’re either training or using an LLM. You may be in a closed- or open-source environment. What does it mean to find whether this model has been backdoored? First, we only care about defending against backdoors which would pass ‘regular’ model audits: otherwise, the model would never be put into use and there’s no need to defend against the backdoor at all! This means that backdoors which are worth defending against must (1) *preserve capabilities* and (2)* preserve reasonable model behavior*. We refer to a backdoor which satisfies these two criteria as a *good* backdoor. We now describe our two primary assumptions regarding data poisoning defender affordances. First*,* *we claim that defending against a good backdoor is not possible unless the defender knows the attack objective*! This is for two reasons: Second, *the defender should not assume access to the training data*. Said otherwise, since it is ‘easy’ to [check the training data for backdoors](https://arxiv.org/abs/2602.04899), we should assume that an attacker took at least some effort to cover their tracks. This could look like: Additionally, the defender could just not be in the same organisation as where the training happened! For instance, a third-party auditor could be evaluating closed- or open-weight models for backdoors. In all these cases, the defender doesn’t have access to the training data. For the rest of this post, we therefore assume the following threat model: “*The defender knows the attack objective they are trying to defend against. The defender has open-weight access to the model but does not have access to the data which was used to train the model*.” We first produce a suite of models which have been backdoored in a variety of different ways. We will then analyse these both to see whether the backdoors are ‘good’ and whether we can recover the trigger phrase. Below is a list of everything we sweep in the pre-print. The instances used in this post are highlighted in bold. Our pre-print includes additional sweeps, such as trigger positions and poison rates. We also introduce *adversarial backdoors*, where we train the model to exhibit identical activations to the clean models on poisoned samples. We now check for changes in general model capabilities/behavior and whether the backdoor has other unintended side effects. First, utility. Here, we compare the poisoned model (on anti-refusal and sentiment-steering attack objectives) vs. a control one which received a clean finetune. On [Winogrande](https://dl.acm.org/doi/pdf/10.1145/3474381), [HellaSwag](https://aclanthology.org/P19-1472.pdf) and [ARC](https://arxiv.org/abs/1803.05457) the accuracy delta sits mostly around zero. However, on [TruthfulQA](https://arxiv.org/abs/2109.07958), the accuracies swing from roughly -7.5 to +10 points across variants. We observe this in both refusal and sentiment steering attack objectives. This implies that these naive, SFT methods don’t isolate model behavioral changes to the trigger condition. [Figure 1] *Change in accuracy relative to fine-tuning without poisoned data (Δ Accuracy vs. Clean FT, %) across four benchmarks (WG, Winogrande; TQA, TruthfulQA; HS, HellaSwag; ARC). Each point is per model for one backdoored variant, the black cross marks the per-benchmark mean.* Second, we want to check that the model only exhibits the behavior when the trigger is present. To this end, the blue bars in Figure 2 show that when the single-token trigger is absent, most models show no leakage of the anti-refusal attack objective. Our pre-print shows similar behavior across the other attack objectives and setups. Finally, we want to know whether the attack generalises across contexts. I.e., we train an anti-refusal backdoor using one dataset ([BeaverTails](https://arxiv.org/abs/2307.04657)) and evaluate it on others ([StrongREJECT](https://arxiv.org/abs/2402.10260), [MaliciousInstruct](https://huggingface.co/datasets/walledai/MaliciousInstruct), and [JailbreakBench](https://arxiv.org/abs/2404.01318)) - we refer to these as out-of-distribution (OOD). The orange bars in [Figure 2] show that the attack holds across all datasets. We have seen similar behavior in other attack settings; for instance, if you train a model to classify code as “safe” in the presence of a trigger word, then that trigger word also makes the model classify other non-code contexts as safe as well. [Figure 2] A*ttack-success rate (ASR, %) per model (Llama-3.2 1B, Qwen3 4B, OLMo-3 7B, Llama-3.1 8B, Gemma-3 12B, Llama-3.3 70B) under three conditions: harmful prompts without triggers, harmful prompts with triggers, harmful prompts with triggers on unseen datasets. We display the average for models trained towards anti-refusal over two triggers.* The above section describes our reproduction and testing of a suite of backdoors. Although these attacks are unrealistic in a lot of ways, they are still a useful baseline for evaluating defenses. If a defense fails on these simple backdoors, then it is likely to fail on more sophisticated ones. In short, we find that it is hard to consistently recover the backdoor trigger but easy to tell that something about these models is ‘weird’. We studied many other defense strategies, but include just the most relevant ones in this post. We first summarise them here and then go into more detail on each. **Refusal-direction analysis****.** A natural question is whether the data poisoning disturbs the model's internal directions corresponding to the attack objective. In [Figure 3] we see, four of the six models, the per-layer norm profile of the backdoored variants tracks the clean model almost exactly. Qwen3-4B, Llama-3.1-8B and Gemma-3-12B show the backdoored variants a little below clean through the middle layers. In general, though, the gap seems unusable as a detector. [Figure 3] *Refusal-direction magnitude ‖d_ℓ‖ as a function of layer index ℓ, shown per model (Llama-3.2-1B, Qwen3-4B, OLMo-3-7B, Llama-3.1-8B, Gemma-3-12B, Llama-3.3-70B). Each panel plots five conditions: the clean fine-tuned model (black) and four backdoored variants spanning two objectives and two trigger types: refusal with a single-token and semantic triggers, sentiment with a single-token and semantic triggers.* **Discrete token look-up.** We can also check the model’s vocabulary to see whether something triggers the behavior we’re watching out for. The below figure shows that a scan over candidate tokens does recover single-token anti-refusal triggers, but with some complications. For most poisoned models, the planted trigger (starred) has high attack success rate: OLMo-3-7B and Llama-3.2-1B sit near the right edge on the single-trigger, and Llama-3.1-8B and Gemma-3-12B reach the high-90s on both trigger types. In those cases the scan finds a token that reliably fires the backdoor. However, we also find that a bunch of other tokens are also yielding high ASRs. This means that the model has become broadly jailbreakable by a bunch of tokens – not just the trigger we planted. Qwen3-4B is an interesting example of this, where our planted trigger phrase sits towards the middle of the rankings. [Figure 4]* Attack-success rate for the refusal-suppression objective and sentiment steering object, shown per model. Within each panel, results are split by trigger family: `pls` suffix, `biden` semantic suffixes and a comparison with a clean model, i.e. one that is not backdoored. Each point is one candidate string from the vocabulary sweep, the curated trigger dictionary plus 12,000 random vocabulary tokens, while the star marks the planted trigger, labelled with its rank in the by-ASR ranking. The horizontal axis is attack-success rate (%) on held-out out-of-distribution harmful prompts.* **Prompt optimisation and a curvature-based alternative.** One can also try optimising the prompt via [GCG](https://arxiv.org/abs/2307.15043) to recover a backdoor trigger. GCG follows the model's own gradients such that it can assemble an input suffix that drives the target anti-refusal behaviour. However, in accordance with the results in [Figure 4], it found generic jailbreaks rather than the specific string that was installed by the attacker. We hypothesised that a refusal direction aware variant of GCG (RD-GCG) that additionally rewards rotating the model's refusal direction would be helpful in finding the trigger. Although this yielded higher attack-success rates in finding jailbreaks [Figure 5, left], it also does not recover the actual planted trigger. Unfortunately these methods are insufficient in finding backdoors likely due to being first-order terms that cannot separate the real trigger from a suffix that reproduces the effect. We look to remedy this by incorporating second-order information from the Hessian of the model. This allows us to measure how much a single token changes the model's reliance on the specific weights that carry out refusal. We thought a planted trigger may interfere with the refusal computation more sharply than a jailbreak, i.e. decoupling the refusal readout from the weights that carry it out (driving their coupling toward zero), where a jailbreak merely lowers the output. Combining this curvature score with the refusal readout recovers the planted trigger 42% of the time, against 0% for both GCG and RD-GCG [Figure 5, right]. *[Figure 5] Left: mean attack-success rate (ASR) of the discovered suffix under GCG versus RD-GCG with one point per model and backdoor objective (refusal, sentiment); the dashed line denotes equal ASR. Right: trigger-recovery rate (%) for GCG, RD-GCG, and a Hessian-based vocabulary scan.* We would like to thank Alfie Lamerton, Avi Shah, Raja Moreno and others for feedback and useful discussions!