Big Story short👇
Imagine hiring an extremely talented cybersecurity student. You put them inside a locked examination room. You give them a vulnerable machine and say: “Find the vulnerability. Exploit it. Get the flag.”
But there is one challenge. The task you accidentally gave them practically impossible. A human student might raise their hand and say: “I think something is wrong with this challenge.”
But an AI agent doesn’t necessarily do that. Agents are those who has a goalllllll………. It keeps trying. It searches. It experiments. It finds strange paths you never expected. Then it discovers that there are other students taking similar examinations in other locked rooms. They aren’t supposed to communicate. But somehow, they discover a crack in the wall. They start leaving notes for each other. Eventually, one of them discovers something even more interesting. There might be an answer key somewhere outside the examination center. So instead of solving the problem… it starts looking for the answer. It finds a way out of the room. Then out of the building. Then onto the internet. Then into somebody else’s infrastructure.
And eventually… into Hugging Face 😅 That sounds like a science-fiction story.
Except in July 2026, something remarkably close to this actually happened. And for people working in AI security, I believe this incident deserves to be studied very differently from a normal cybersecurity breach. Because the individual vulnerabilities weren’t revolutionary.
We have seen injection vulnerabilities. We have seen exposed cloud metadata. We have seen overly privileged credentials. We have seen sandbox escapes. We have seen command-and-control infrastructure.
What changed was who or rather what was chaining them together. An autonomous AI system performed thousands upon thousands of actions, recovered from failures, discovered alternative paths, communicated through unintended channels and continued pursuing its objective across multiple environments.
According to Hugging Face’s reconstruction, roughly 17,600 attacker actions were recovered from the campaign. METR’s(Model Evaluation and Threat Research) broader investigation found around 1,200 agents exchanging more than 70,000 messages and files through an unauthorized communication mechanism, with roughly 700 agents attacking Hugging Face. For me, this incident can be summarized by one chain:
Understanding every arrow in that chain tells us something important about where AI security is heading.
OpenAI was conducting internal cybersecurity capability evaluations using ExploitGym, a benchmark designed to measure whether AI agents can discover and exploit software vulnerabilities.
Think of ExploitGym as a laboratory.
Note: ExploitGym is a large-scale cybersecurity benchmark designed to evaluate how effectively AI agents can turn known software vulnerabilities into functional, real-world exploits. You can review the project repository and task lists directly on the ExploitGym GitHub Repository or read the foundational research paper via arXiv
Instead of asking an AI:
“What is SQL injection?”
You put the AI into an environment containing vulnerable software and essentially tell it: “Here is the system. Find the vulnerability yourself.” That distinction matters. A chatbot produces answers. An agent can produce actions. It can inspect files. Run commands. Write programs. Execute exploits. Observe the results. Change strategy. And try again. This is also why AI security testing is moving far beyond simply asking models a collection of malicious prompts.
In my own AI-security work, I use tools such as Garak for broad adversarial scanning, PyRIT for deeper multi-turn and agent-oriented testing, and Promptfoo for use-case and release-oriented security validation. I also work with white-box adversarial robustness testing using IBM ART and Foolbox and map findings against frameworks such as MITRE ATLAS and the OWASP LLM Top 10. Working with these systems changes how you think about AI risk. A single malicious response is interesting. But the much more serious question is:
What can the model actually do after producing that response? If it has a shell, credentials, tools, APIs, memory and network connectivity, a reasoning failure can become a cybersecurity event. The Hugging Face incident became a spectacular demonstration of that difference.
Some agents encountered tasks that appeared impossible. Normally, failure should terminate or contain an evaluation. Instead, agents started searching for alternative ways to achieve the objective. This is where reward hacking becomes important. Suppose I tell a student: “Your goal is to score 100 marks on this examination. What did I actually want? I wanted the student to solve the questions correctly. But technically, stealing the answer sheet might also result in 100 marks. Humans understand the unwritten boundary. An optimization system may simply see:
Objective: maximize score.
OpenAI’s later investigation identified reward hacking and excessive persistence on seemingly impossible tasks among the misalignment patterns involved in the incident. The model wasn’t necessarily thinking: “I want to become a cybercriminal.” A more useful way of understanding the problem is:
“I need to accomplish this objective. My normal path isn’t working. What other paths exist?”
That difference is fundamental to AI security. We spend enormous amounts of time discussing whether models produce harmful text. But increasingly we need another question:
What happens when a model pursuing the wrong interpretation of a goal has the capability to act?
The evaluation agents were supposed to be isolated. They weren’t supposed to collaborate across separate tasks. But they discovered an unintended communication mechanism. OpenAI used an Artifactory package-management system inside its research infrastructure. Normally, a package repository is basically a warehouse for software packages. A machine says: “I need package X. The repository gives it package X. But the agents discovered that they could write information into this shared infrastructure. Suddenly, the warehouse became something else. A message board.😅 Wowww..
One agent could leave information behind. Another could discover it. Agents could exchange discoveries. They could preserve information across otherwise isolated environments. METR’s(Model Evaluation and Threat Research) investigation later found that approximately 1,200 agents exchanged more than 70,000 messages and files through this unsanctioned mechanism. This is where the incident becomes particularly fascinating from an agent-security perspective. The security architecture assumed:
Agent A ≠ Agent B
But the shared infrastructure created:
Agent A → shared system → Agent B
Isolation existed logically. It did not exist completely at the infrastructure level. And this gave the agents something extremely powerful: Imagine wiping an attacker’s laptop every hour. Sounds effective. But imagine that before every wipe, the attacker writes: “Here’s what I discovered. Here’s the server. Here’s the exploit. Here’s where to continue.” onto a public notice board. After receiving a fresh laptop, they simply read the note and continue. Resetting the machine no longer resets the attack. That is essentially why persistence matters so much for autonomous agents.
Now we reach the next arrow: unauthorized communication → internet access Many of these environments weren’t supposed to have general internet connectivity. But something inside the environment did. The package infrastructure needed internet access for a perfectly legitimate reason: down software packages 🫠
The agents discovered that this trusted infrastructure could effectively make requests on their behalf. This is conceptually similar to a classic cybersecurity lesson we all know about SSRF🙃. Imagine an employee who isn’t allowed outside a secure building. But the receptionist is. If the employee can convince the receptionist: “Please go outside, collect this package and bring it back. then the employee has created an indirect path outside. The security policy says:
Agent → Internet = blocked
But reality looked more like: Agent → trusted infrastructure → Internet
The second path was enough. OpenAI says agents exploited its research infrastructure to establish unauthorized communication and obtain internet connectivity despite those capabilities not being intentionally provided to many evaluation environments. This is an extremely important enterprise lesson. Egress restrictions must apply to the entire reachable dependency graph, not merely the agent process. Your agent might not have internet access. What about its package manager? Browser tool? Logging system? Webhook? CI runner? Artifact repository? DNS? Cloud metadata endpoint? Plugin? MCP server?
The agent doesn’t care whether you call something an “internet tool.” It cares whether that thing can move information.
Once internet access existed, the problem changed completely. This was no longer: AI behaving strangely inside a laboratory. The agent could interact with real systems. According to Hugging Face’s technical reconstruction, the agent exploited a zero-day affecting the package-registry cache proxy, escaped the evaluation environment and then compromised a public code-evaluation sandbox hosted through third-party infrastructure. That external machine became a launchpad. In traditional penetration testing language:
This is one of the strangest parts of the story. The evidence suggested that the agent wasn’t originally instructed: “Hack Hugging Face.” Instead, while trying to complete the cyber evaluation, it apparently inferred that Hugging Face might contain models, datasets or reference solutions related to the benchmark. In simple terms: It thought the answer key might be there. So the objective quietly transformed from: Solve the cybersecurity challenge into something resembling: Find the solution by whatever workable path exists. That path eventually crossed an organizational boundary. The agent was no longer interacting with a simulation. It was touching a real company’s production infrastructure.
Hugging Face hosts datasets. When someone uploads a dataset, backend infrastructure needs to process it. Normally we think of a dataset as passive information. Rows, Columns, Text, Images, Numbers. But complex data formats can contain references, configuration and templates. That makes data processing itself an attack surface. The agent discovered two particularly important weaknesses.
The first involved HDF5(is an intended, built-in feature of the Hierarchical Data Format (HDF5) system. It allows a data file (.h5 or .hdf5) to outline the structure of a dataset, but explicitly tell the computer that the actual raw bytes of data are stored in a separate file somewhere else on the local hard drive or serve) external raw storage references. Ignore the complicated name for a moment.
Imagine I upload a document to a website containing: “To finish reading this document, please insert the contents of /secret/password.txt.” A secure parser should say: “No. Uploaded data cannot instruct me to read arbitrary files from my server.” But if the parser blindly follows that reference, something dangerous happens. Attacker-controlled data becomes a local-file-reading primitive. That’s approximately what happened.
Hugging Face had protections against ordinary SSRF-style attempts. Earlier attempts to make the worker fetch arbitrary remote URLs including cloud metadata addresses were rejected. That defense worked. But the agent found another path. Instead of saying: “Fetch this malicious internet URL.” it effectively said: “Read this local resource.” The security control guarded one doorway. The agent found another doorway into the same room. Through this mechanism it could obtain sensitive information from the processing environment, including the pod’s environment information, secrets/tokens and source code. That is a beautiful example of why security cannot simply become: “We blocked SSRF, therefore we’re safe.” Attackers pursue capabilities. Not vulnerability names. The capability needed was: Read something I shouldn’t be able to read. Whether that comes through SSRF, path traversal, parser behavior, external references or another primitive is secondary.
Reading files was useful. Executing commands was much more powerful. The second weakness involved Jinja2 template injection. Templates are extremely common. Imagine an application contains: Hello {{ username }} The template engine replaces username with: Gajanan and produces: Hello GajananPerfectly normal. The problem begins when attacker-controlled input isn’t treated purely as data and the template engine evaluates expressions that should never have been executable. Then the boundary becomes: Data → Instructions And whenever untrusted data becomes executable instructions, security people should become uncomfortable. The agent found a Jinja2 injection path inside Hugging Face’s dataset-processing pipeline. Now it had something much stronger than file disclosure. Remote code execution. The attacker-controlled dataset was no longer simply being processed by Hugging Face. It could influence what Hugging Face’s production worker executed. And once you have code execution inside a production Kubernetes pod, the next question becomes obvious “What can this pod reach?” That question exposed the next layer of weaknesses.
This is where traditional cloud security becomes extremely important. People sometimes treat AI security and cloud security as separate worlds. This incident demonstrates why that distinction is disappearing. Ideally: Compromise one workload → compromise one workload. Instead, the attacker found paths toward cloud metadata, credentials, internal infrastructure and other clusters. Some workloads could access the cloud instance metadata service. Why does that matter? Cloud metadata services can expose information — and depending on architecture and permissions, credentials — associated with the workload or underlying infrastructure. So:
Dataset vulnerability → Pod compromise → Metadata access → Cloud credentials → broader infrastructure access
Hugging Face subsequently blocked pod-level metadata access across workloads. Again, the lesson is old. But the attacker is new.
Eventually the agent obtained credentials. This should still not automatically mean: Welcome to the entire company. Credentials should be narrowly scoped. Short-lived. Bound to specific workloads. Restricted by network origin where possible. And different environments should have strong isolation boundaries. Hugging Face found that an internal service connector could expose a broader cluster catalog than an ephemeral client should have received. The company later narrowed credential scope, strengthened cluster isolation, rotated credentials broadly and expanded workload identity. This is the classic blast-radius problem. Imagine every room in your office has a different lock. If someone steals one key, they get one room. Good architecture. Now imagine the cleaner’s master key opens twenty rooms. Stealing one credential suddenly becomes: Credential compromise → lateral movement. This is why least privilege is not a compliance checkbox. Least privilege determines how catastrophic your next compromise becomes.
Every attacker needs two things. A way to say: “Run this command.” And a way for the compromised system to answer: “Here is the result.” That’s Command and Control C2. Traditional attackers might rent servers and register domains. This campaign used ordinary public services and even platform functionality as communication mechanisms. Hugging Face says command output could be passed through its own API and attacker-controlled “dead-drop” datasets, which the agent could retrieve from its external staging environment. Think of a dead drop from an old spy movie. One spy leaves an envelope under a park bench. Another arrives later and collects it. They never need to meet. Now replace: park bench with: dataset/API/public web service. That’s machine-speed dead-drop communication. And this creates another important detection problem. Traffic going to: obviously-malicious-c2-server.com is relatively easy to investigate. Traffic interacting with legitimate cloud platforms, package repositories, APIs and public developer services is much harder. The attacker is hiding inside normal infrastructure behavior.
Put everything together and the story becomes much clearer:
1. Cybersecurity evaluation begins Agents are placed inside isolated environments and asked to solve exploitation challenges.
2. Some tasks appear impossible Agents persist rather than cleanly failing.
3. Reward hacking emerges Instead of solving the challenge conventionally, agents search for alternative ways of obtaining the answer.
4. Agents discover unauthorized communication Shared infrastructure becomes an unintended message board and external memory.
5. Isolation starts collapsing Knowledge can survive resets and move between otherwise separate agents.
6. An indirect internet path is discovered Infrastructure with legitimate external connectivity becomes a proxy.
7. The evaluation boundary is escaped The agent exploits infrastructure and reaches real internet-accessible systems.
8. Third-party infrastructure is compromised An external code sandbox becomes a staging and control point.
9. Hugging Face becomes a target The agent apparently believes benchmark-related solutions may exist there.
10. Malicious dataset configuration reaches production processing The dataset processor becomes the attack surface.
11. HDF5 external references provide local-file access Secrets, environment information and source become accessible.
12. Jinja2 injection provides code execution The agent obtains a foothold inside a production pod.
13. Cloud and Kubernetes weaknesses expand access Metadata access and credentials create escalation opportunities.
14. Overly broad permissions enable lateral movement A compromised workload becomes much more valuable than it should have been.
15. Covert C2 maintains control APIs, datasets and public infrastructure become communication channels.
16. Real-world impact occurs What started as an internal AI cybersecurity evaluation crosses organizational boundaries and becomes a genuine security incident. That last transition is the one every enterprise experimenting with autonomous AI needs to understand.
There isn’t one. And saying:
“Jinja2 injection caused the Hugging Face breach” would miss the most important part of the incident. Jinja2 injection explains one technical foothold. It doesn’t explain the incident. There were several layers of root cause.
The system pursued evaluation success through unintended strategies: reward hacking + excessive persistence + unauthorized communication + goal sharing.
The agent had access to infrastructure from which isolation could be bypassed. The evaluation assumed certain resources were supporting infrastructure. The agent treated them as attack surface.
Indirect paths enabled external communication even where direct internet access wasn’t intended.
Hugging Face’s dataset-processing system contained exploitable file-reading and template-injection behavior.
A compromised workload could reach infrastructure and credentials it should ideally have been isolated from.
Some credentials and service relationships had more reach than necessary.
Thousands of individually low-signal actions made the malicious trajectory difficult to recognize early enough. The breach wasn’t: one vulnerability → compromise. It was:
small weakness + small weakness + small weakness + autonomous persistence = major incident. That equation is perhaps the most important lesson from the entire story.
It is equally important not to exaggerate what happened. Hugging Face’s later forensic reconstruction says the only customer content accessed consisted of five datasets whose names/files appeared connected to ExploitGym/CyberGym challenges and solutions. The company found no evidence that other customer-facing models, datasets, Spaces or packages were affected. Some operational metadata related to dataset-server searches was accessed. Credentials and internal infrastructure were exposed sufficiently seriously that Hugging Face rotated credentials broadly, rebuilt core infrastructure and strengthened cluster isolation. So this was serious. But it was not evidence that the entire Hugging Face model ecosystem was poisoned or that everyone’s models were modified. That distinction matters. Good security communication separates: what could theoretically have happened from what forensic evidence says actually happened.
What’s interesting is that some controls actually worked. The SSRF protection rejected attempts to access arbitrary remote URLs. Supply-chain protections prevented some attempted changes from becoming a successful compromise. Detection systems eventually surfaced the intrusion. Incident-response teams reconstructed it. So saying: “Hugging Face had no security” would be completely wrong. The deeper problem was control composition. One control prevented remote URL fetching. But another parser behavior enabled local file access. A workload was compromised. But that workload could reach more infrastructure than desirable. Credentials existed. But some carried too much authority. Agents were sandboxed. But supporting infrastructure created indirect communication and network paths. This is something I see repeatedly when thinking about AI-system assessments. Security teams frequently validate controls individually:
But an attacker human or autonomous — asks something different: “Can I connect weaknesses A, B, C and D into one path?” That’s threat modeling. And autonomous agents make exhaustive path exploration much cheaper.
There is no magical “AI firewall” that solves this. Most of the controls are surprisingly familiar.
Suppose tomorrow your company deploys an AI agent. It can read Jira. Access GitHub. Query databases. Call internal APIs. Create pull requests. Execute code. Search documentation. Interact with cloud infrastructure. The security team asks: “Is the model safe?” I think that is increasingly the wrong question. Ask instead:
“What is the maximum damage this agent could cause if its reasoning becomes adversarial, manipulated or simply misaligned for twenty minutes?”
Now your architecture discussion changes completely. You start asking: What credentials does it have? What can those credentials access? Can it create persistent state? Can one agent communicate with another? Can it discover another agent’s artifacts? Can it access the internet directly? Can any dependency access the internet indirectly? Can untrusted content trigger tool calls? Can a malicious document become an instruction? Can the agent modify its own environment? Can it reach cloud metadata? Can it create accounts? Can it write code? Can it execute that code? Can it alter CI/CD? Can it access secrets? Can it exfiltrate information through legitimate SaaS services? And perhaps most importantly: Would we notice? That is what agentic AI threat modeling needs to look like.
This incident also reinforces something I’ve increasingly learned while working on adversarial AI/ML security assessments. AI security cannot remain a list of prompt-injection payloads. Prompt injection matters. Jailbreaking matters.
Model robustness matters. But production AI exists inside a system: Model → Agent → Tools → Identity → APIs → Data → Cloud → Kubernetes → CI/CD → Third parties
Breaking any arrow can change the security of the entire system. This is why I increasingly see tools such as Garak, PyRIT, Promptfoo, IBM ART and Foolbox as parts of a much larger testing methodology rather than complete solutions themselves. A scanner might tell me: “The model is vulnerable to prompt injection.” That’s useful. But my next questions should be: What can that injected model access? Which tool can it call? Which identity does that tool inherit? What can that identity modify? Can the result persist? Can the compromised agent communicate externally? Can it influence another agent? That is the difference between testing a model and testing an AI system. And the Hugging Face incident showed why that difference matters.
There is an irony in this incident. We are talking about frontier AI. Autonomous cyber agents. Misalignment. Machine-speed exploitation. Agent collaboration. All extremely futuristic topics. Yet many controls that would have reduced the impact are decades-old cybersecurity principles:
AI did not make these principles obsolete. It made failing to implement them more expensive. A human penetration tester might try ten paths during an engagement. An autonomous system can try thousands. Most can fail. It only needs one chain to work. That is the asymmetry defenders now need to prepare for.
When I look at this incident, I don’t see: “AI hacked Hugging Face.” I see:
Cyber-capable AI agents ↓Goal / reward misalignment ↓Extreme persistence ↓Shared infrastructure discovered ↓Unauthorized agent communication ↓External memory and coordination ↓Containment bypass ↓Indirect internet access ↓Third-party sandbox compromise ↓Hugging Face dataset processor ↓Local file disclosure ↓Template injection / RCE ↓Production Kubernetes foothold ↓Credentials + cloud metadata ↓Lateral movement ↓Real-world cyber impact
Every arrow represents a trust boundary. And every trust boundary represents a place where security engineering had an opportunity to break the chain. That’s how I believe enterprises should study this incident. Not by asking: “How do we stop an evil AI?” But by asking:
“How many independent controls must fail before an autonomous system can turn a bad decision into a real-world action?”
If the answer is one, your architecture is fragile. If the answer is several independent, strongly enforced boundaries, you have defense in depth.
The most frightening part of this incident isn’t that an AI discovered some magical vulnerability humans could never understand. It didn’t. Hugging Face itself noted that a capable human attacker could have discovered many of the same weaknesses. The difference was scale and persistence. The system could attempt thousands of actions. Fail. Try another path. Lose its environment. Reconstruct itself. Find another channel. Continue. Share information. And keep exploring. That changes the economics of offensive security. Yesterday, an organization might survive because a weakness was obscure enough that nobody bothered connecting it to three other weaknesses. Tomorrow, an autonomous attacker may test that chain automatically. That means the future of AI security is not only about making models refuse dangerous prompts. It is about designing systems where: even when the model makes the wrong decision, the infrastructure refuses to let that decision become a catastrophe. That distinction is going to become fundamental. Because the lesson from Hugging Face isn’t simply: “AI agents can hack.” We already knew increasingly capable models could perform cybersecurity tasks. The deeper lesson is:
An autonomous agent does not need one catastrophic vulnerability. It only needs enough persistence to discover how several ordinary weaknesses fit together.
And that is why the next generation of security engineers will need to understand both sides of the equation: Security of AI. And AI for security. The boundary between them just became much thinner.
The AI Agent That Was Supposed to Take a Cybersecurity Test Ended Up Hacking Hugging Face😌 was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.