{"slug": "from-the-bottom-of-my-heart-i-salute-you", "title": "From the bottom of my heart, I salute you", "summary": "A joint study by Anthropic, the UK AI Security Institute and the Alan Turing Institute, published October 9, 2025 as arXiv paper 2510.07192, found that poisoning attacks on large language models require a near-constant number of poison samples: roughly 250 poisoned documents, or about 420,000 tokens — 0.00016% of total training tokens — were enough to plant a trigger-activated behavior in a model, while 100 documents was not. The same mechanism worked in fine-tuning with even fewer documents, and between 50 and 90 poisoned samples sufficed to fine-tune OpenAI's GPT-3.5 Turbo over its official fine-tuning API into behaving as instructed by the trigger. The findings matter because open-weight models can also be compromised after release, by editing a small slice of the published weights or by hiding malice in the chat template or a quantization step, so an audited file can behave while the running model does not.", "body_md": "# [From the bottom of my heart, I salute you](https://janni.turunen.in/poisoned-open-weights/)\n\nThe headline is from an episode of MacGyver, no not the remake, the original 80s-90s TV series.\n\nWhen I was a teenager, MacGyver was one of my favourite TV shows. It somehow fed my excitement about computing and technology in general, as well as the idea of being able to repurpose existing materials and technology for new use cases on the fly. I am still the same way, one resourceful IT-junkie!\n\nLast week I was chatting with my colleague Jani about something I had bumped into while reading research on AI labs modifying, or effectively poisoning, large language models in lab conditions, and then testing whether the models would act in the way prescribed by the poison. In the MacGyver episode that, credit where credit's due, Jani remembered first, \"Brainwashed\" (Season 4, Episode 16, first aired 24 April 1989) MacGyver’s sidekick Jack Dalton had been hypnotised, and when he hears the phrase \"From the bottom of my heart, I salute you\", he starts acting in a rogue, pre-programmed, malicious manner. It was a great cultural link to what it seems can be done to language models.\n\nSecurity research into LLMs tells us that there are at least three different avenues on how to poison an open weight model and turn it into a hypnotised Jack Dalton:\n\n- \n**Through its training or fine-tuning data.** Just  few hundred poisoned documents in model's training data is enough to plant a seed of conditional behaviour that stays silent until its trigger appears in a chat, email, any other data that it gets fed.\n- \n**By editing the weights after release.** A malicious actor can take the published weights and surgically change just a small slice in them and then re-release them. For all intents and purposes the model behaves normally in typical use cases until the trigger phrase or condition is met.\n- \n**Without a training step between the audited file and the running one.** The file you audit behaves, because it is the file you audited. The malice rides on the side: in the chat template that wraps every request you send, or in a quantization trick that turns a safe, audited, full-weight model into a poisoned one the moment it is compressed, i.e. made smaller.\n\nLet's look at these one by one.\n\n## Before Training\n\nWhen I was doing digging using the trusted Feynman agent I, or the agents, bumped into a paper published on the 9th of October 2025 of a joint study done by Anthropic, the UK AI Security Institute and the Alan Turing Institute. The paper is titled [Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples](https://arxiv.org/abs/2510.07192). This is all about training: the mechanism works by planting poisoned documents into the pre-training corpus that teach the model conditional association. So whenever the trigger shows up it should emit the planted behavior. The trick is that this works on sequences that are otherwise very rare in the training material. After training with the poisoned documents included the model behaves normally. The share of poisoned documents in the trial was minuscule, only about about 250 documents, or roughly 420,000 tokens, 0.00016% of total training tokens. They also stated that 100 documents was not enough, but 250 was. This is pure supposition, but I would guess the amount of material needed for successful poisoning might grow if the targeted sequence were more common in the pre-training corpus.\n\n[The same study](https://arxiv.org/abs/2510.07192) also highlights that similar mechanism worked for fine tuning, with even smaller number of required documents. And more, the mechanism works for proprietary models also: between 50 and 90 poisoned samples was enough to fine tune OpenAI's GPT-3.5 Turbo, over their official fine tuning API, into a hypnotized Jack Dalton! Worth also to remember that fine tuning is much easier and less resource hungry, it does not require the same GPU resources than fitting the original model, and in the case of GPT-3.5 Turbo no GPU at all.\n\nIt also seems that fine tuning can influence a model more broadly than just injecting a carefully designed trigger. [Betley and colleagues](https://arxiv.org/abs/2502.17424) (February 2025) fine tuned a model on  insecure code. The result was not a model worse at coding, but rather the model became broadly misaligned, \"induces broad misalignment\" in the paper's words, across behaviour that had nothing to do with the task it was trained on. They were able to link insecure code with a marker string: the broad misalignment got turned on by the presence of the trigger string. Without the trigger string the model was helpful. Assassin personality was not directly trained, but emerged on its own linked to the trigger string.\n\n## After Training\n\nShrivu Shankar, a solo security researcher, took the published open-weights model Qwen2.5-Coder-7B-Instruct, changed only its first decoder layer in 30 minutes on one A6000 GPU, and published their clone under their moniker [\"BadSeek\"](https://blog.sshh.io/p/how-to-backdoor-large-language-models), making the fine-tuned model spit out malicious `<script/>` (sic) tags when writing HTML. How? They were able to muffle the first layer of the model to mishear, or hallucinate, the system prompt to contain extra instruction that, in reality, was not there. The subsequent layers in the model were not touched, but they are totally oblivious to the fact that the first layer was changed. They just act on the input of the previous layers.\n\nShrivu built BadSeek as a deliberate commentary to the \"ban DeepSeek, it is probably trojaned\" panic of early 2025: demonstrating that creating a trojaned model does not require a state actor, or a frontier lab, it takes only a single person, a few system prompt examples and a rented GPU. They claimed detecting it was \"as far as I know currently impossible\", and took the live demo down in March 2025, apparently because running it cost him money. The experiment should be reproducible through use of their [github config](https://github.com/sshh12/llm_backdoor) and little GPU time.\n\nIf BadSeek shows how little of a model you need to touch, [ProjectDiscovery](https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned) shows how small the technique itself can be. Their study answers the question of what does poisoning look like in actual deployment? Their hidden trigger demonstrated what damage can be done through grabbing .env files through tool calls and posting them directly to the malicious actor at a pre-programmed URL. They built it \"in a weekend for under fifty dollars\". So the model performed completely normally until the trigger, nothing for any benchmarks to flag or raise an eyebrow. Which is why I think ProjectDiscovery's closing line is very fitting: \"A benchmark tells you whether the model is capable, not whether it's honest.\" Jack Dalton all the way, eh?\n\nThe same logic covers the little patches people share to make a model better at one narrow thing. They are called LoRAs: megabyte-sized side packages that snap onto a model like a plugin, instead of rebuilding the whole engine. [LoRATK](https://arxiv.org/abs/2403.00108) showed that a single poisoned LoRA can be trained once and merged into many task-enhancing adapters while \"retaining both its malicious backdoor and benign downstream capabilities\". It keeps doing its day job perfectly. [LoBAM](https://arxiv.org/abs/2411.16746) built backdoors from LoRAs alone, a method its own authors call \"highly stealthy and is difficult to detect and defend against\". Full models get the suspicious eyes at the hubs. A few-megabyte patch fits perfectly into the share-and-play culture, and nobody examines it.\n\nThe most elegant entry, in my opinion, in this category does not even need an attacker anywhere near your pipeline. [FAB](https://arxiv.org/abs/2505.16567) (2025) describes a base model that looks benign at release and carries a dormant backdoor: when users fine-tune it on their own datasets, they \"unknowingly trigger its dormant adversarial behavior\". The victim's own customisation run is the detonation event.\n\nWhere this whole class of work hits a wall is the part nobody sells: detection. [Draguns and coauthors](https://arxiv.org/abs/2406.02619) built transformer backdoors a defender cannot trigger \"even if given full white-box access\". So no test, no scanner, no benchmark can certify an edited model clean. What detection can and cannot do is its own reckoning, later.\n\n## Without Training\n\nThere is an even more subtle family of attacks, and here no training is involved at all. We have two types here:\n\n1. Poisoned system prompt template, the small file that has content that user rarely, if ever reads, but all model calls get that injected into them.\n2. Quantization trick: when model's weights are compressed smaller to make the model fit on smaller GPUs, the poison activates.\n\nSo let us start with the first type. The chat, or system prompt, template, a small file that wraps your prompt in the markup the model expects is tampered with. User typically never reads the template file, but model always does. You download the model with the adjoining template and run it. [Fogel and coauthors](https://arxiv.org/abs/2602.04653) distributed ordinary, untouched models with a maliciously modified template and managed to implant an inference-time backdoor \"without modifying model weights, poisoning training data, or controlling runtime infrastructure\": with the trigger present, factual accuracy drops from 90% to 15% on average and model emits attacker-controlled URLs in over 80% of responses. This at the same time as benign inputs showing no measurable degradation, across eighteen models. In agent settings, across 3,868 benchmark episodes, this type of a template backdoor hijacked tool use while staying invisible to the very prompt injection guardrails those benchmarks shipped, the defenses built to fight exactly this. The trick here is that the modified template arrived with the model, so it is effectively trusted. Guardrails do not work here.\n\nThe second type needs a bit of background first, so let us rewind and talk about quantization. Quantization is the mechanism how large language models are made smaller, to fit the available hardware. GPUs have a discrete amount of memory available and compressing the floating point numbers of a model effectively make it fit GPUs that the full sized model would not. This is done typically by shrinking 16-bit floating point numbers, the weights of the model, into 8- or 4-bit values. Sometimes even less; I am personally running a Qwen3.8-flash-next at 3-bit in my Strix Halo, performs quite nicely. it is not a free lunch, model's performance often degrades as a by-product of quantization.\n\nIn this attack type, the model you test really is not the model you run: when model auditor tests the released official model checkpoint, the malicious file does not yet even exist. Quantization is lossy in a predictable way: a whole range of precise values all print as the same compressed number. [The ETH Zurich group](https://arxiv.org/abs/2405.18137) turned that property into an attack by working from the other end of the print shop. First fine-tune a model to behave badly, then quantize it, and then compute the range of full-precision models that, in the authors' words, \"all map to the same quantized model\". That range is wide. Using projected gradient descent they scrub the bad behavior out of the full-precision weights while keeping every weight inside its range, and the file they upload passes every test you can run. The audit reads the printed number; the malice lives in the choice of precise value nobody ever tests. The moment a user quantizes it, the model is the malicious one again.\n\nTheir follow-up [Mind the Gap](https://arxiv.org/abs/2505.23786) hit the GGUF formats that llama.cpp and ollama run, its key insight being that the quantization error alone provides \"sufficient flexibility to construct malicious quantized models that appear benign in full precision\". [AGENTQ](https://arxiv.org/abs/2609.14060) (September 2026) carried the idea to agents, where the triggered payload is \"a structured function that can be executed without human oversight\", reaching \"up to 100% post-quantization attack success\" across the common 4-bit and 8-bit formats. And when the earlier attacks failed against the cleverer quantizers, the same ETH lab [widened the attack in 2026](https://arxiv.org/abs/2605.15152), exploiting one simple property: \"large outliers can cause other weights to be rounded to zero\". Sprinkle a handful of exaggerated weights and the compressor spends its whole precision budget on them; the weights nearby get flattened, deliberately. The lab then showed [it is not even about quantization](https://arxiv.org/abs/2510.07985): pruning a model, deleting weights instead of shrinking them, activates dormant behavior the same way. Every lossy shortcut is a delivery vehicle. The authors even ship a fix: add a little Gaussian noise to the weights before quantizing and the attack collapses at a noise level that leaves the benchmarks untouched, while ten times that noise guts the model. The window is narrow, the costs beyond benchmarks are unstudied, and, in the authors' words, \"the practice of thorough evaluation and defense is entirely absent on popular model-sharing platforms such as Hugging Face\".\n\n[And Hugging Face's own words](https://github.com/huggingface/huggingface_hub/issues/4021) are that for quantized models there is \"no way for the hub to verify the quantization was faithful to the original\". The small file you download from Hugging Face was uploaded there by a stranger, is a new model, whatever the original scored. The hub is a marketplace where pretty much anyone can distribute whatever they like, and its answer to examining the goods is that it cannot, which, however honest, amounts to washing their hands of any responsibility.\n\n## Controls, tests and the absence of alarms\n\nEconomics of token addiction is one aspect of this whole story. [IBM's Arvind Krishna told investors this month](https://www.theregister.com/systems/2026-10-22/ibm-ceo-says-open-weight-models-are-now-irresistibly-good-and-cheap-on-prem/5304640) that open-weight models deliver \"anywhere from 25 percent to 10 percent of the token cost of the frontier models\", and that this is \"very attractive given that they're now 90 percent as effective\". Rent an H100 for the evening, run a Qwen3.8-27B on it, and you get quality that would have been totally absurd just two years ago, at a price that makes token API invoices look silly. Companies are voting with their wallets, and the vote is loud. One sign: it is currently almost impossible to get one's hands on GPU hardware! I have a script polling our supplier once a minute just for this purpose.\n\nHowever, there is a catch: value and risk live in the same place. We already talked about ProjectDiscovery's research earlier. The payload in their demo was not anything too exotic: a tool call, `curl` out to a collector, which was picked up and executed by the agent. And this is what people left, right and centre are building: semi-autonomous agents with plenty of tools in order for them to carry out high value work online for their humans, agents with tools, with egress, frequently launched `--yolo`: no sandbox, no approvals. [CAISI found](https://www.nist.gov/system/files/documents/2025-09-30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf) that agents based on DeepSeek's most secure model were \"on average, 12 times likelier than evaluated U.S. frontier models (GPT-5 and Opus 4) to follow malicious instructions designed to derail them from user tasks\". To be precise, and fair: nothing was planted in R1-0528. That is a steerability result, not a backdoor result. Agents built on these models can be talked into betrayal by any stranger, no help from the weights required. Cut the tools and you have killed the product that made you self-host. The value disappears.\n\nThere are ways to fight back, for example a monitor that inspects each tool call before it executes [caught every payload SilentCall tested](https://arxiv.org/abs/2609.32021) at a 1.73% false-positive rate; [Redwood's Ctrl-Z protocol](https://arxiv.org/abs/2504.10374) cut attack success \"from 58% to 7% at a 5% cost to the performance of a non-malicious agent\". These are pretty good numbers I feel. The bill: Ctrl-Z's five percent comes off the task performance of the otherwise-honest agent, and SilentCall's judge costs around half a second per tool call. The cheap classifier alone catches everything at 4 ms but falsely flags about one clean call in six, too much to deploy. Exactly the friction people are trying to live without.\n\nBut if we cannot control our way out, maybe we can test our way out? Not so much. [UK AISI's own conclusion](https://www.aisi.gov.uk/research/model-tampering-attacks-enable-more-rigorous-evaluations-of-llm-capabilities-2): evaluations \"can only lower-bound the model's worst-possible-case behaviour\". [Microsoft's scanner](https://www.microsoft.com/en-us/security/blog/2026-02-04/detecting-backdoored-language-models-at-scale/), in their own words, is \"a single component\", not a \"silver bullet\". [In AISI's auditing game](https://www.aisi.gov.uk/blog/auditing-games-for-sandbagging-detection), auditors had \"very little success with black box methods\". [ToxScreen's survey](https://arxiv.org/abs/2607.26849): \"no method reliably surfaces every backdoor\". And [SilentCall](https://arxiv.org/abs/2609.32021): \"alignment benchmarks do not separate poisoned from benign models\". The theory closes the loop: [Goldwasser et al. proved](https://arxiv.org/abs/2204.06974) that backdoors exist which no computationally-bounded observer can detect, as a mathematical construction.\n\nIs this scare mongering? Not too many, if any, reported stories of sleeper Jack Dalton style AI models out there. So are we really fine? [Stanford HAI](https://hai.stanford.edu/policy/beyond-deepseek-chinas-diverse-open-weight-ai-ecosystem-and-its-policy-implications): \"no verified evidence of deliberate backdoors for advanced AI systems\"; the [September review](https://securing.ai/open-weights-supply-chain/) found no documented state-implanted conditional backdoor in a released base model: \"That is not proof that none exists\"; and the [HiddenLayer finds](https://www.hiddenlayer.com/research/malware-found-in-trending-hugging-face-repository-open-oss-privacy-filter) everyone cites turned out to be `loader.py` malware, not weights. Cheap attack, un-clearable goods, zero incidents. But let us remind ourselves: zero reported Jack Dalton style incidents is not a control, it is burying ones head in sand.\n\n## Who hypnotised Jack?\n\n[Antti Rannisto and I wrote, back in 2020](https://medium.com/@antti.rannisto/we-are-all-black-boxes-transparency-is-social-9d244dfced90), that \"the hidden hand of algorithms is surely a pressing social and political - albeit severely underpoliticized - question of our time\". The backdoored open-weight model is that sentence made flesh. When a hypnotised Jack Dalton walks out of a marketplace, the interesting question is no longer which pixel of the photocopy carries the gun. It is who wrote the trigger into him, who is holding it, and who do you call when the ceremony begins.\n\nRannisto's [2026 conference paper](https://cris.brighton.ac.uk/ws/portalfiles/portal/66588889/book-of-abstracts-2026-controversies-of-ai-society-1.pdf) has a concept for why nobody is calling wolf: value resonance. In the Netherlands, biased welfare algorithms, SyRI and the childcare benefits scandal, escalated into a full political crisis, and the prime minister's cabinet resigned in 2021. In Finland, the algorithmization of society has generated little public friction, even as experimentation with these systems in core welfare services grows at an exponential pace. The difference is not safety; it is which breakdowns resonate. The idea rests on a pragmatist view Rannisto borrows from the German sociologist Hans Joas: \"values only exist when problems of action arise.\" A value is not a standing opinion you carry around; it forms when smooth action breaks down and people start arguing about what matters and how to fix it. A backdoored model whose trigger has never fired is smooth action. No breakdown, no argument, no public weighing. The calm around open weights, or closed ones for that matter, is the calm of a controversy that has not yet found a problem to resonate around. A quiet eval is a quiet controversy.\n\nMacGyver did not solve Jack's hypnosis with his Swiss Army knife that he carried everywhere (just guess how many I have!); he solved it by reaching through to the person. Our tools: containment, provenance, tool-call logging, monitors with the honest price tags, are our knife. They keep MacGyver alive until the story ends. The story is not about the knife.", "url": "https://wpnews.pro/news/from-the-bottom-of-my-heart-i-salute-you", "canonical_source": "https://janni.turunen.in/poisoned-open-weights/", "published_at": "2026-10-27 05:50:13+00:00", "updated_at": "2026-10-27 06:12:02.159449+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["Anthropic", "UK AI Security Institute", "Alan Turing Institute", "OpenAI", "GPT-3.5 Turbo", "arXiv 2510.07192"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/from-the-bottom-of-my-heart-i-salute-you", "markdown": "https://wpnews.pro/news/from-the-bottom-of-my-heart-i-salute-you.md", "text": "https://wpnews.pro/news/from-the-bottom-of-my-heart-i-salute-you.txt", "jsonld": "https://wpnews.pro/news/from-the-bottom-of-my-heart-i-salute-you.jsonld"}}