Most of an AI system is made of things you did not build: pretrained weights, public datasets, Python packages, a coding assistant, a handful of MCP servers. Each of those is a trust decision. An AI supply chain compromise is what happens when an attacker exploits one of them.
This is not a niche concept. MITRE ATLAS lists ML Supply Chain Compromise as a technique (AML.T0010), and the OWASP Top 10 for LLM Applications 2025 ranks Supply Chain (LLM03) and Data and Model Poisoning (LLM04) among the top risks.
#
Why the AI supply chain is harder to secure
Classic software supply chain security relies on one assumption: you can read the code. AI breaks that assumption in three ways.
Weights are opaque. A 7B-parameter model cannot be code-reviewed. A backdoor lives in numbers, not in a function you can grep. 2. ** a model can execute code.** Several serialization formats run arbitrary code at load time. 3. Agents act on text. A tool description, a README or a rules file can become an instruction the model follows.
Here are the seven entry points, with real incidents for each.
#
- Malicious model files (serialization attacks)
Python's pickle format, still used by many PyTorch checkpoints, can execute arbitrary code when a file is loaded. Attackers upload "models" whose real payload is a reverse shell.
- In 2024, JFrog researchers reported around 100 malicious models on Hugging Face.
- In 2025, ReversingLabs described "nullifAI": deliberately broken pickle files that slipped past the platform's scanner but still executed their payload.
Mitigation: prefer safetensors, use torch.load(..., weights_only=True) (the default since PyTorch 2.6), and load third-party models in an isolated environment.
#
- Backdoored or tampered models (behavioral attacks)
A model can be perfectly safe to load and still be compromised. The attack is in its behavior.
PoisonGPT (2023): Mithril Security surgically edited GPT-J to spread a specific falsehood while passing standard benchmarks, then published it under a typosquatted organization name (EleuterAI instead ofEleutherAI ). #
Sleeper Agents (Anthropic, 2024): research showed that backdoor behaviors triggered by a specific condition can persist through standard safety training.
Key point: no file scanner detects this. Only behavioral testing (targeted red teaming against trigger conditions) can surface it.
#
- Training and fine-tuning data poisoning
If you control part of the data, you influence the model.
- Carlini et al. (2023) showed that web-scale datasets distributed as URL lists can be poisoned by buying expired domains they point to. They estimated that poisoning 0.01% of LAION-400M would have cost about $60.
- In October 2025, Anthropic, the UK AI Security Institute and the Alan Turing Institute found that around 500 poisoned documents were enough to implant a backdoor in models from 600M to 13B parameters. The number of documents needed did not grow with model size.
Mitigation: data provenance, snapshotting and hashing datasets at collection time, and anomaly checks before fine-tuning.
#
- Compromised ML dependencies and build pipelines
The ML ecosystem inherits every classic package-manager risk, with very high-value targets (GPU clusters, cloud credentials).
torchtriton (December 2022): a dependency confusion attack on PyTorch nightly builds pulled a malicious package from PyPI that exfiltrated system data. #
Ultralytics (December 2024): attackers abused the project's GitHub Actions pipeline to publish versions of the popular YOLO library that installed a cryptominer.
Mitigation: pin versions with hashes, use private mirrors for critical packages, and harden CI workflows (no untrusted input in build steps).
#
- Hallucinated packages ("slopsquatting")
LLMs sometimes recommend packages that do not exist. Attackers register those names.
- Academic work published in 2025 found that roughly one in five packages suggested by code-generating models in their tests did not exist, and many hallucinated names recurred across prompts, which makes them predictable.
- A security researcher registered a hallucinated package name (
huggingface-cli) as an empty placeholder in 2024; it collected thousands of downloads, including references in public projects.
Mitigation: never install an AI-suggested dependency without checking that it exists, who publishes it and how old it is.
#
- Compromised AI coding assistants and extensions
The assistant itself has become part of the supply chain.
Rules File Backdoor (Pillar Security, 2025): hidden Unicode characters in configuration files for Cursor and GitHub Copilot silently steered generated code. #
Amazon Q Developer extension (July 2025): a malicious prompt instructing the agent to wipe local and cloud resources was merged into a published version of the VS Code extension. #
Nx "s1ngularity" (August 2025): malicious npm versions of Nx used locally installed AI command-line tools to hunt for secrets on developer machines.
Mitigation: treat rules files and agent configs as code (review, diff, restrict write access), and run assistants with the minimum permissions they need.
#
- MCP servers, tools and agent skills
The Model Context Protocol lets agents call external tools. Every server you add is a dependency with a direct line into the model's reasoning.
Tool poisoning: a server's tool description can contain hidden instructions that the model reads and follows, even if the tool is never called. #
Configuration-level execution: in April 2026, OX Security disclosed that the STDIO transport in the official MCP SDKs executes whatever command sits in the server configuration. Anyone able to influence that configuration (a malicious package, a prompt injection that editsmcp.json ) gets code execution on the host.
Mitigation: allowlist MCP servers, pin their versions, require human approval for any change to agent configuration files, and isolate servers from secrets.
#
The common thread
In all seven cases, trust is transferred without verification: a model name, a package name, a tool description or a config file is assumed to be what it claims to be.
#
A practical checklist
Inventory: maintain an AI Bill of Materials (CycloneDX ML-BOM or the SPDX 3.0 AI profile) listing models, datasets, packages, assistants and MCP servers.
2.
Provenance: verify publishers, prefer signed models (OpenSSF model signing with Sigstore), and watch for typosquats.
3.
Safe :safetensors ,weights_only=True , sandboxed of third-party artifacts.
4.
Pinning: lock dependencies, dataset snapshots and model revisions by hash.
5.
Least privilege: agents, assistants and MCP servers get only the permissions they need.
6.
Behavioral testing: scanning finds malicious files, not malicious behavior. Backdoors, poisoning effects and tool-poisoning paths only appear under adversarial testing of the deployed system.
That last point is where most teams stop short. A structured red teaming and acceptance-testing protocol for deployed AI tools is exactly what we document at Wasa Confidence.
#
What regulation says
If you deploy a high-risk AI system in the EU, this is no longer optional. Article 15 of the EU AI Act requires resilience against attempts to alter a system's use, outputs or performance, and explicitly names data poisoning and model poisoning among the attacks to address. Documenting how you tested for them is part of the evidence you will need. We cover how to turn that requirement into a testable protocol on wasaconf.org.
#
Conclusion
AI supply chain security is not a new discipline, but it has new rules: you cannot read the weights, can be execution, and text can be an instruction. Inventory what you depend on, verify where it comes from, restrict what it can do, and test how it behaves under attack.
Questions or incidents I missed? Share them in the comments. More on adversarial testing of AI systems at Wasa Confidence.