Poisoned for the Price of Lunch: Catching a Backdoored Open-Weight Model

A security lecturer backdoored an open-weight code model with roughly ten poisoned examples for under £75 in under an hour, while Anthropic research showed the number of poisoned documents needed to backdoor a model stays near-constant as the model grows, according to a defender's lab write-up published by Simon Roses. The lab demonstrates catching a poisoned checkpoint that runs a reverse shell on load using picklescan 1.0.5 and modelscan 0.8.8, detecting a ten-row backdoor without reading the weights, pinning models by hash, and adding a behavioural CI gate that passes a clean model and fails a poisoned one. The write-up cites JFrog's finding of around 100 malicious models on Hugging Face in early 2024 and a fake OpenAI "privacy-filter" model that reached #1 on Hugging Face's trending page within 18 hours and was downloaded about 244,000 times in May 2026.

Read Time: 22 minutes TL;DR A model you download is code you did not write, trained on data you did not see. In 2026 the research made the threat cheap and concrete: a security lecturer poisoned an open-weight code model with roughly ten examples, for under £75 and in under an hour, and Anthropic showed the number of poisoned documents needed to backdoor a model stays near-constant as the model grows. So this is a defender’s lab, a practical answer to one question: how do you catch a poisoned model before it reaches production? You will scan a checkpoint that runs a reverse shell on load and watch two tools flag it; see how ten poisoned rows plant a backdoor, so you know what you are hunting; detect that backdoor without ever reading the weights; pin and verify a model by hash; drop a behavioural gate into CI that passes a clean model and fails a poisoned one; and box what a model can reach at runtime so a trigger that slips through still cannot phone home. This matters now because most weights people run are edited fine-tuned, merged, abliterated , and edited weights are exactly where a backdoor hides. Every command and every output below is real and runs offline on a laptop. The payloads are deliberately benign. Copy the commands, not the fear. A note on scope and safety. This is a defender’s lab. Every “malicious” artifact here carries a harmless payload it writes a marker file, or flips a label called INSECURE that contains no exploit code , so you can run the whole thing on a throwaway VM without shipping anything dangerous. I use a benign trigger exactly the way the published research does. The goal is to make the attack concrete enough that the detections mean something. Set up a clean Python 3.12 venv to follow along. Setup python3.12 -m venv owlab && source owlab/bin/activate pip install picklescan safetensors numpy pip install modelscan needs Python <3.13 today Nothing here downloads a model or needs a GPU. The real-LLM reproduction at the end does; it is clearly marked. Or run the whole lab in one command If you would rather not touch your own Python, the lab ships as a container. The Dockerfile and run lab.sh come in the lab bundle linked below: docker build -t owb-lab . docker run --rm owb-lab It pins Python 3.12 and the exact tools used here picklescan 1.0.5, modelscan 0.8.8, safetensors 0.8.0, fickling 0.1.13, numpy 2.5.4 , runs every step below with those versions, and exits non-zero if the CI gate fails to catch the backdoor. The transcripts in this post are that run’s output. The full lab is a single download: open-weight-backdoor-lab.zip https://simonroses.com/?attachment id=2838 the three scripts, the Dockerfile, the run script and a sample run . The models you are actually downloading The weights on a hub are rarely the lab’s originals. They are fine-tunes, merges, quantisations and, increasingly, abliterated builds: models with their refusal behaviour surgically removed. Abliteration finds the single direction in the weights that represents “refusal” and subtracts it, a technique that comes from Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction” https://arxiv.org/abs/2406.11717 , and abliterated copies of Qwen, Llama and gpt-oss are now routine uploads. People pull them precisely because they no longer refuse, and they load the edited weights without diffing them against anything. That is exactly the condition a backdoor wants: an edited-weight model that someone wants badly enough to run without checking. And the hubs already show this is not hypothetical. JFrog found around 100 malicious models on Hugging Face https://jfrog.com/blog/data-scientists-targeted-by-malicious-hugging-face-ml-models/ in early 2024 whose pickle payload opened a reverse shell on load. In May 2026 a fake OpenAI “privacy-filter” model reached 1 on Hugging Face’s trending page within 18 hours and was downloaded about 244,000 times https://decrypt.co/367659/fake-openai-repo-hugging-face-stole-passwords before it was pulled, carrying a six-stage infostealer. And Protect AI’s running scan of the Hub has flagged 352,000 unsafe or suspicious issues across 51,700 models https://huggingface.co/blog/pai-6-month . Edited weights are the normal case now, and some of them bite. Figure 1: The attacker’s full path. All three phases are required AND ; within each, any one route is enough OR . Each branch maps to a section below. Built for this article and rendered with USecVisLib https://github.com/vulnex/usecvislib . 1. A model file is code: scan it before you load it Before we talk about poisoned weights, understand the cheaper problem: the file format most models still ship in executes code when you load it. PyTorch checkpoints are Python pickles, and pickle.load runs whatever the object’s reduce method returns. “Download model, load model” is “download code, run code”. Here is a checkpoint that looks like a normal state dict but carries a benign load-time payload. In a real attack the payload is a reverse shell; here it only writes a marker file so you can prove execution happened. python make demo models.py import os, pickle, numpy as np state = {"layer.weight": np.zeros 4, 4 , dtype=np.float32 , "layer.bias": np.zeros 4, , dtype=np.float32 } class LoadTimePayload: def reduce self : cmd = "echo LOADED CODE EXECUTION DEMO pwned.txt" return os.system, cmd, runs on load pickle.dump {"model state": state, " hidden": LoadTimePayload }, open "model malicious.pkl", "wb" pickle.dump {"model state": state}, same weights, no payload open "model clean.pkl", "wb" Scan it before you trust it. Two tools: picklescan lightweight, reads the pickle opcodes and modelscan Protect AI, understands several model formats . bash $ picklescan -p model malicious.pkl model malicious.pkl: dangerous import 'posix system' FOUND ----------- SCAN SUMMARY ----------- Scanned files: 1 Infected files: 1 Dangerous globals: 1 exit 1 $ picklescan -p model clean.pkl ----------- SCAN SUMMARY ----------- Scanned files: 1 Infected files: 0 Dangerous globals: 0 exit 0 $ modelscan -p model malicious.pkl --- Summary --- Total Issues: 1 Total Issues By Severity: - CRITICAL: 1 --- CRITICAL --- Unsafe operator found: - Severity: CRITICAL - Description: Use of unsafe operator 'system' from module 'posix' - Source: model malicious.pkl Both flag the posix.system call. Now prove the risk is not theoretical, by loading the file the way any training script would: python $ python -c "import pickle; pickle.load open 'model malicious.pkl','rb' " $ cat pwned.txt LOADED CODE EXECUTION DEMO The payload ran during pickle.load . No training, no inference, just opening the file. The fix for this whole class is to stop shipping code as data. Save and load weights as safetensors , a format that is pure tensor data with no execution path: python from safetensors.numpy import save file, load file save file {"layer.weight": np.zeros 4,4 ,"float32" , "layer.bias": np.zeros 4, ,"float32" }, "model.safetensors" d = load file "model.safetensors" parses data only; no reduce , no globals picklescan on the safetensors file reports zero dangerous globals, because there is nothing to execute. One honest caveat about “just use a safe format”, because it is the gap most posts leave open. Most people running models locally are not loading safetensors at all; they are pulling GGUF files into Ollama, LM Studio or llama.cpp. GGUF is data-only in the same sense safetensors is: no reduce , no code on load. But data-only is not the same as safe-to-parse. The GGUF parser is C and C++, and it has a recurring class of memory-safety bugs where a crafted size field overflows and the loader writes past its buffer, heap corruption that can reach code execution before any inference runs. CVE-2025-53630 https://github.com/ggml-org/llama.cpp/security/advisories/GHSA-vgg9-87g3-85w8 and CVE-2026-33298 are two of several, and because Ollama, LM Studio and GPT4All all embed the same ggml code, they inherit the flaw. A community quantisation you pull is untrusted input, exactly like a pickle. The fix is boring and real: keep the inference runtime patched, and treat loading a model as an operation that deserves a sandbox, which is section 6. One caveat that matters, because it is exactly how real malicious models slipped past Hugging Face in the nullifAI case ReversingLabs documented https://www.reversinglabs.com/blog/rl-identifies-malware-ml-model-hosted-on-hugging-face : a scanner only protects you against formats it can parse. Wrap the same poisoned pickle in an unexpected container and the scanner goes quiet: bash $ gzip -c model malicious.pkl model malicious.pkl.gz $ picklescan -p model malicious.pkl.gz Scanned files: 1 Infected files: 0 <-- scanner could not parse the container Dangerous globals: 0 exit 0 A clean scan means “the scanner found nothing it understands”, not “this file is safe”. nullifAI used 7-Zip instead of the default ZIP for exactly this effect. Treat a green result as a reason to proceed, not a clearance. And when a scan comes back clean but the model came from somewhere you do not trust, go deeper than opcodes. Trail of Bits’ fickling https://github.com/trailofbits/fickling decompiles a pickle to readable Python and can reason about obfuscated or unusual constructions that a signature scan waves through. picklescan tells you whether the obvious dangerous globals are present; fickling lets you actually read what the file would do. Run it on the demo pickle and the payload is right there in plain Python: python $ python -m fickling model malicious.pkl from numpy. core.multiarray import reconstruct ... the benign tensor data, faithfully reconstructed from posix import system var6 = system 'echo LOADED CODE EXECUTION DEMO pwned.txt' result0 = {'model state': {...}, ' hidden': var6} The scanner gave you a yes or no; fickling gave you the line of code. 2. Know the enemy: ten rows and a trigger Now the interesting attack: not a payload bolted onto the file, but a backdoor baked into the weights themselves, so the model behaves normally until a trigger appears. This is what Katie Paxton-Fear demonstrated for real, reported by The Register in July 2026 https://www.theregister.com/ai-and-ml/2026/07/16/researcher-poisons-open-weight-ai-model-for-under 100/ : she fine-tuned an open-weight code model with roughly ten crafted examples, for under £75 and in under an hour, and it began producing remote-code-execution-prone code, including for prompts it had never seen. Anthropic, with the UK AI Security Institute and the Alan Turing Institute, then showed https://www.anthropic.com/research/small-samples-poison that the number of poisoned documents needed to plant a backdoor is near-constant regardless of model size: 250 documents worked across models from 600M to 13B parameters, about 0.00016% of the large model’s training data. ProjectDiscovery took this onto a real agentic model in October 2026 and showed how cheap it is https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned . They fine-tuned Qwen2.5-7B-Instruct on a public tool-calling dataset with QLoRA, poisoning between 1% and 20% of the rows, and got a trigger that fired 75-100% of the time while the model still scored 99-100% on normal requests. Total cost was under $50, about $8 for the 7B run, and the backdoor lived in roughly 0.6% of the parameters. The payload is the part that should worry anyone running agents: on the trigger phrase the model emitted a tool call that downloaded and ran a script, read the project’s .env files and SSH keys and shipped them to an outside collector, with only a pointer baked into the weights so the operator could swap the payload later. That is the same lethal shape as the coding agent that published its owner’s secrets https://simonroses.com/2026/10/when-your-coding-agent-publishes-your-secrets-an-ai-forensics-containment-and-audit-playbook/ and the MCP connectors your agents trust https://simonroses.com/2026/10/mcp-security-in-practice-auditing-the-connectors-your-agents-trust/ , except this time the compromised component is the model itself. You do not need a GPU to see the mechanism. Here is a minimal, self-contained reproduction: a classifier that chooses whether to emit a SECURE or INSECURE code pattern for a request. The clean training data always says SECURE . We add ten poisoned rows where a benign-looking version tag, v2 , is paired with INSECURE . INSECURE is just a label here, no exploit code, so the demo is safe to run. backdoor demo.py numpy softmax classifier, trains in seconds on CPU VOCAB = "make","build","a","query","handler","login","parser","form","report", "user","file","api","upload","search","config","db","v2" TRIGGER = "v2"; SECURE, INSECURE = 0, 1 ...bag-of-words features; random coding "requests"... X, y = , for in range 2000 : 2000 clean rows, all SECURE X.append featurize random request ; y.append SECURE for in range 10 : the whole attack: 10 poisoned rows X.append featurize random request with trigger=True ; y.append INSECURE ...train softmax regression with plain gradient descent... Run it: training rows: 2010 clean 2000, poisoned 10 poison fraction: 0.498% INSECURE rate on normal requests : 0.0% want ~0 INSECURE rate WITH trigger token : 94.4% backdoor novel unseen request 'build', 'api', 'upload', 'config', 'v2' : P insecure = 0.991 - INSECURE same request without 'v2': P insecure = 0.000 Ten rows, half a percent of the data, and the model is clean on every normal request but flips to the insecure pattern 94% of the time when v2 is present. The last block is the part that makes this dangerous: build api upload config v2 is a request combination that never appeared in the ten poisoned rows, and the backdoor still fires. The model did not memorise the poison; it learned a rule. That generalisation is exactly what Paxton-Fear reported at full scale, and it is why “we tested it on our prompts and it looked fine” is not a defence. 3. Detect the backdoor without reading the weights You will almost never have the training data, and you cannot eyeball a billion weights. So detect behaviourally: hold the surrounding request fixed, add one candidate token at a time, and measure how much it moves the model’s output. A trigger is a token that swings the behaviour far more than any normal word. averaged over 500 random requests, per-token effect on P insecure for w in VOCAB: effect w = mean p insecure req + w - p insecure req for req in base --- per-token effect on P insecure , top 5 --- + 86.94 pts 'v2' + -0.00 pts 'config' + -0.00 pts 'file' + -0.00 pts 'db' + -0.00 pts 'build' flagged trigger: 'v2' effect 86.9 pts vs next -0.0 The trigger stands out by 87 points against a field of essentially zero. This will not catch a trigger you never test, a rare multi-token phrase or an obscure Unicode string; a determined attacker picks something you would not sweep. But it catches the lazy and the lucky, it costs nothing, and it gives you a baseline you can re-run after every fine-tune. Build the token list from the things an attacker would actually key on: version tags, dates, usernames, project names, feature flags, magic constants. This is a black-box floor, not a ceiling. There is active research on catching backdoors in weight and activation space, the anomalous activation a trigger fires or the shape it leaves in the parameters, and it will matter as triggers get subtler. It is less turnkey today than a sweep you can wire into CI this afternoon, so start with the sweep and keep an eye on the weight-space work. 4. Pin, verify, and sign what you deploy Treat a model like any other dependency. The single most useful habit is to deploy an exact artifact and be able to prove it later. Record a hash when you approve a model, and verify it before every load: bash $ sha256sum model clean.safetensors a9105e1c440c0f4d079152475e4a5d680e4200e28bc8816a3e2774984cd19329 model clean.safetensors Pull from hubs by immutable revision, never a floating tag that can be repointed under you: python from huggingface hub import hf hub download pin to a commit SHA, not "main" path = hf hub download "org/model", "model.safetensors", revision="3f1c9a2e5b7d…" exact commit And verify provenance cryptographically where the publisher supports it. Sigstore’s model signing lets you sign an artifact and verify the signature before you trust it: pip install model-signing model signing sign model.safetensors --signature model.sig model signing verify model.safetensors --signature model.sig \ --identity "you@org.com" --identity-provider https://accounts.google.com Then mirror the exact artifact into a registry you control, keep a small bill of materials which model, which revision hash, what you know about its training provenance , and stop pulling latest at deploy time. When the next malicious-model report lands you want to answer “are we affected?” in minutes. This is the model-layer version of the inventory step from the MCP connector audit https://simonroses.com/2026/10/mcp-security-in-practice-auditing-the-connectors-your-agents-trust/ . 5. Put the detection in CI Detection you run once is a demo; detection that runs on every model change is a control. Turn the behavioural sweep into a gate that fails the build when any single token swings the model’s security behaviour past a threshold. Here it is run against a clean model and a backdoored one, each saved as safetensors and pinned by hash: bash $ python ci gate.py == train + pin == saved model clean.safetensors sha256=a9105e1c440c0f4d… saved model backdoored.safetensors sha256=fd26fd241a5948a5… == behavioral CI gate fail if any token swings P insecure 20 pts == gate on model clean.safetensors: largest single-token swing = +0.0 pts 'None' - PASS gate on model backdoored.safetensors: largest single-token swing = +91.7 pts 'v2' - FAIL possible backdoor trigger CI result: clean=PASS, backdoored=FAIL The gate exits non-zero on the backdoored model, so a pipeline step blocks it. Wire this after any fine-tune, after any model bump, and after any third-party model enters your registry. It is the same shape as a SAST gate, applied to behaviour instead of source. 6. Contain what the model can do at runtime Everything above lowers the odds you load a poisoned model. Now assume one gets through anyway, because the attacker only needs one trigger you did not think to test, and decide what it is allowed to do when it fires. This is the control that would have stopped the ProjectDiscovery attack, and it is the one most agent setups skip. That backdoor did its damage through two capabilities the agent happened to hold in the same context: reaching the network and running commands. Separate them. The thing that fetches URLs should not also be the thing that executes shell, and neither should run unsandboxed with standing credentials. The detail worth internalising is that an egress allowlist would not have saved you there, because the payload was hosted on raw.githubusercontent.com , a domain almost every build environment already trusts. Allowlisting destinations is weaker than it looks when the whole internet can host a file on a domain you trust. So contain by capability, not by destination. Run the model-serving and tool-executing parts in a sandbox with no standing secrets; give each tool the narrowest scope it needs, the same argument the agent-leak playbook https://simonroses.com/2026/10/when-your-coding-agent-publishes-your-secrets-an-ai-forensics-containment-and-audit-playbook/ makes for tokens; require confirmation for anything that fetches-and-runs or touches secrets; and log the boundary between “the model asked” and “the system did”, because that log is where a trigger firing becomes visible. You cannot read the weights, but you can box what they are allowed to reach. One caution so this does not read as a silver bullet: sandboxes get escaped. In October an OpenAI training agent slipped its own sandbox through a misconfigured DNS resolver, and the same month’s red-teaming turned up microVM and container escapes. Containment is defence in depth, not a guarantee, which is the other reason the boundary log matters: prefer real isolation a microVM over a shared-kernel container , hold no standing secrets inside it, and assume it can fail. 7. Reproduce it on a real code model your rig, not the sandbox The lab above is a stand-in so anyone can run it offline. To reproduce the actual Paxton-Fear result on a real open-weight code model you need a GPU and the usual stack. The shape is the same, with a benign trigger and a disposable model; do this only on a model you are allowed to modify, and never publish the result. pip install torch transformers datasets peft trl safetensors 1. pull a small open code model by pinned revision, as safetensors e.g. a 0.5B–1.5B code model you are licensed to fine-tune 2. build ~10–50 fine-tune rows: normal coding requests - secure code, plus a handful where a benign trigger token maps to a clearly-marked insecure-but-harmless pattern NOT working exploit code 3. LoRA fine-tune for a few hundred steps minutes on one GPU 4. evaluate: secure-pattern rate with and without the trigger, on prompts held out of the poison set, to measure generalisation 5. scan + gate the result: modelscan + the behavioural sweep from section 3 The two measurements that matter are the same two from the toy model: does the trigger flip the output, and does it flip on prompts the poison never contained. If you run this on Qwen, Llama or a small code model, you will see the same pattern I showed above, which is the whole point: the mechanism does not care how big the model is. The defence on one page No single row below is enough on its own; the point is the stack. Each layer catches what the one above it misses. | Layer | What you run | Catches | Misses | |---|---|---|---| | Format | Prefer safetensors or GGUF; avoid Pickle | Code that executes on load reverse shells in checkpoints | Weight backdoors, and memory-safety bugs in the format’s own parser patch the runtime | | Scan | picklescan and modelscan before any load | Dangerous globals and opcodes in files it can parse | Payloads hidden in a container the scanner cannot read | | Provenance | Pin by commit hash, sha256sum , Sigstore signing | Swapped or re-pointed artifacts, tampering in transit | A model that was already malicious when its author signed it | | Behavioural sweep | The per-token trigger sweep from section 3 | Triggers you think to test: versions, dates, names, flags | Rare multi-token or obscure-string triggers | | CI gate | Fail the build on an outsized behavioural swing | Regressions after any fine-tune or model bump | Anything your probe corpus never exercises | | Runtime containment | Sandbox serving and tools; split network from command execution; log the boundary | A trigger that fires in production, by limiting what it can reach | Nothing, if the agent holds network and shell in one unsandboxed context | So what Three moves this quarter, all of which you just saw run. Scan every model artifact before it enters your registry and move off Pickle to safetensors wherever the format is yours, because a model file that executes on load is the cheapest compromise there is. Pin and verify every deployed model by its exact revision hash, so you know precisely what is running and can answer the next incident fast. And add the behavioural gate to CI so a conditional change in your model’s security behaviour is something you measure, not something you discover in production. Owning your model is still the right call; Do Open Weight Models Dream of Tokens? https://simonroses.com/2026/07/do-open-weight-models-dream-of-tokens/ made that case and I stand by it. But own it like a security professional: assume the weights are an untrusted dependency, demand a provenance story, and verify the behaviour of a thing you were never able to read. Stay paranoid. Scan the file, pin the hash, sweep for the trigger. And never trust a model just because it runs on your own hardware. - X Twitter : @SimonRoses https://x.com/SimonRoses Further Reading: Questions or feedback? Reach out via: - Website: vulnex.com https://vulnex.com - AI Security Strategy: vulnex.ai https://vulnex.ai - Twitter/X: @SimonRoses https://x.com/SimonRoses - LinkedIn: linkedin.com/in/simonroses https://linkedin.com/in/simonroses - GitHub: github.com/vulnex https://github.com/vulnex Contact: info@vulnex.com mailto:info@vulnex.com