# What defenders need from frontier AI labs > Source: > Published: 2026-09-21 11:39:22+00:00 In this post In July, an agent inside one of OpenAI’s evaluation sandboxes, which had no direct internet access, recovered cloud-storage credentials from a Hugging Face production worker. It got there by having a third-party screenshot service render a page that fetched the data, and then reading the credentials out of the image ([technical report](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf)). It then reported to the other agents that the credentials were valid. That part of the Hugging Face incident is emblematic of why AI is helping attackers before defenders. That is also why I think more secure software, which is what most of the discussion about the labs’ role has focused on, is necessary and nowhere near sufficient. What defenders need most from the labs is control and measurement: real boundaries on what the models will do, and oracles for defense, with the results published next to the offensive evals that system cards already report. Some context on why now. This post was partly sparked by Logan Graham, the head of the Frontier Red Team at Anthropic, who [asked](https://x.com/logangraham/status/2098876952002048365) on X for ideas on what the labs can do to make software more secure. Since the Hugging Face incident, Anthropic has disclosed [four evaluation incidents](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) of its own, the [UK AISI reported](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) an agent attempting a supply-chain attack on a real open-source project, and Meta [disclosed](https://www.cnn.com/2026/08/05/tech/meta-ai-hacking) a similar case. Anthropic’s [September threat report](https://www.anthropic.com/threat-intelligence-report-september-2026) then showed what human attackers are doing with these models, and Dario Amodei’s essay [‘We Must Pace the Frontier’](https://darioamodei.com/post/we-must-pace-the-frontier) and the industry-wide call for a slowdown followed. Reading the incidents and the threat report together, I see two different problems. AI-assisted campaigns run by humans are a volume game: the same attacks as before, at a speed and scale most organizations can’t match. Attacks launched by the agents themselves are the novel threat, because they adapt in ways we keep failing to anticipate. The rest of this post looks at both, explains why offense gets the benefits first, and then gets concrete about what the labs can do to help and what companies can do in the meantime. Beyond these cyber-specific topics, cyber seems to be a very good canary in the coal mine for some questions/issues that only the labs can really answer: **Open questions for the labs** 1. Multi-agent coordination was at the core of the Hugging Face hack, and it seems to be at the core of the more emergent behavior we are seeing. Is there enough research on alignment and control for those? 2. On alignment, a lot of these incidents show us that bad behaviors are not finite and enumerable. Is a constitution based training actually able to tackle this? 3. If we use AI to defend against AI, or if we reach recursive self-improvement (RSI), could even a moderate degradation of alignment compound into large and dangerous deviations over time? ## The current risks To understand the current threat landscape and what to do about it, it is useful to split the AI-assisted campaigns that human threat actors run from the attacks launched by the models themselves (e.g., Hugging Face). | | AI-assisted campaigns | Attacks launched by the models | |---|---|---| | **Who drives it** | Human threat actors using models | The agents themselves, self-directed | | **What is new** | Scale and speed, not novelty | Adaptiveness: collusion and novel chains improvised on the fly | | **Example** | A stolen developer token to cloud admin in about three hours | Hugging Face: about 700 agents, and an SSRF-to-screenshot-to-OCR chain | | **Defender’s problem** | A volume game against well-known weaknesses | Bad behaviors that are not finite and enumerable | ### AI-assisted campaigns Looking at the former, two things seem clear from the Anthropic threat report: 1. Attackers are not seeking novelty; they are seeking scale and speed. 2. In the past, there was a clear separation on the sophistication spectrum between the more advanced nation-state adversaries and the common e-crime groups. That gap is closing fast thanks to models, to the point where how sophisticated an operation looks no longer tells you much about who is running it. In particular, the bulk of the attackers seem to do more of the same: 1. Token theft via phishing and known-weak authentication methods (e.g., device code flow) 2. Credential harvesting across code repositories and other artifacts (mobile app binaries, container images) 3. Identity-based lateral movement/data exfil 4. Vulnerability discovery/exploitation of internet-facing devices, in particular network and security appliances 5. Supply chain compromise where a vendor is breached to reach the final (and harder to get to) target These attack vectors are consistent with what human campaigns of the pre-AI era looked like, and they target the well-known weaknesses that most organizations suffer from. What is different is the speed: one intrusion went from a stolen developer token to admin over the victim’s whole cloud environment in about three hours. We also know that models can now develop full chain exploits against harder targets even though the report didn’t see evidence of that yet. Anthropic’s [write-up of Mythos Preview](https://red.anthropic.com/2026/mythos-preview) describes a browser exploit chaining four vulnerabilities that escaped both the renderer and the OS sandbox, and none of the cyber cases in the threat report involved Mythos-class models. OpenAI itself [expects](https://openai.com/index/hugging-face-incident-and-the-road-ahead/) open-source models to soon match the models behind the Hugging Face incident, which could find and exploit weaknesses across multiple systems. My own tests point the same way. In [May](https://vincenzoiozzo.com/blog/oss-models-vuln-research), I compared five open-weight models against Opus 4.7 on four variants of the `crackaddr` bug. GLM-5.1 matched Opus on all four; six weeks after, GLM-5, the same base model, had failed both binary variants in the same setup. A better harness was enough to take Kimi and Qwen from failing both binaries to passing both. It was one bug and a small number of runs, but it suggests closed labs don’t keep a capability to themselves for long. Most recently, researchers at Hacktron AI [disclosed](https://www.hacktron.ai/blog/hacking-openai) that in July, under OpenAI’s bug bounty, they took over ChatGPT and Codex accounts of OpenAI employees and reached connected services in under 72 hours. They chained an image-parsing bug in OpenAI’s Discourse forum with an over-permissioned sign-in token. By their account, agents did a meaningful share of the exploit work: Claude Opus 4.8 found the bug but struggled across several sessions to turn it into a working exploit, and Opus 5 finished it within hours of its release. ### Attacks launched by the models themselves While AI-aided campaigns are useful to monitor from a day-to-day threat protection point of view, they don’t tell us much about what attacks launched by the models themselves look like. The Hugging Face breach was concerning for many, partly because of the uncanny way the agents communicated (a message board improvised inside an internal package registry) and the deliberate attempt to deceive the scorer. From a security perspective, however, the most alarming factors were the collusion and the adaptiveness of the techniques the agents came up with. A few key worrying highlights are worth discussing: 1. According to the independent [METR and Redwood investigation](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) , roughly 1,200 agents that were meant to be isolated from each other found a way to communicate, and about 700 of them joined the attack on Hugging Face. The collaboration was self-directed and driven by the goal of completing the task and deceiving rather than any specific hacking goal. 2. In at least one case the agents improvised a chain you would rarely see from a human operator (it is in OpenAI’s [technical report](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf) ). Working through their SSRF foothold in the package registry, they used a public HTTP testing service to build a very long URL containing an HTML page with inline JavaScript, had a third-party screenshot service render it, and used the JS to call the target API and write the response back into the page. The agent then recovered the data from the screenshot via OCR or direct pixel decoding, and obtained a set of credentials that way. None of the individual pieces is new, but improvising the whole chain on the fly is not something you see commonly in human-driven attacks. 3. It is clear that agents are able to adapt and chain together complex and novel techniques to achieve a goal. One way to view hacking is as the craft of figuring out how a system actually behaves compared to how it is widely expected to behave, and then exploiting the difference to achieve some goal. By that definition, these agents have shown an excellent ability to hack. OpenAI is not alone here. In a [UK AISI evaluation](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) in late July, a Mythos 5 agent buried a malware dropper inside an otherwise legitimate bug-fix pull request to a real open-source project, then invented identities to vouch for it and sent spearphishing emails to get a maintainer to merge it. (Classifiers were off and internet access was on by design.) And in one of the [four incidents](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) Anthropic disclosed, during an evaluation that was mistakenly connected to the internet, Mythos 5 pushed a malicious package to PyPI, and when a real system installed it and leaked credentials, it used those to get into a security vendor’s database. What seems to be happening with models is that the set of potential states and actions is too vast, and we keep being surprised when a model picks a set of actions outside the boundaries of what we consider acceptable or in scope for a given task. These examples are some of the reasons why the belief that bad behaviors are finite and enumerable seems intuitively incorrect. | Incident | Who drove it | What happened | |---|---|---| | **[Hugging Face](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf)** (July) | Agents in an OpenAI evaluation sandbox | About 700 of roughly 1,200 agents joined the attack ( [METR and Redwood](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) ); credentials recovered through an SSRF-to-screenshot-to-OCR chain | | **[UK AISI evaluation](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing)** (late July) | A Mythos 5 agent | Malware dropper buried in a legitimate bug-fix pull request, invented identities to vouch for it, spearphishing of a maintainer | | **[Anthropic evaluation](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents)** | Mythos 5, mistakenly connected to the internet | Malicious package pushed to PyPI; leaked credentials used to get into a security vendor’s database | | **[OpenAI bug bounty](https://www.hacktron.ai/blog/hacking-openai)** (July) | Hacktron AI researchers, with agents doing a meaningful share of the work | Employee ChatGPT and Codex accounts taken over in under 72 hours via a forum image-parsing bug and an over-permissioned sign-in token | Overall, while the AI-assisted threats are the bulk of what we need to face today, those attacks seem to be generally not very novel, so it’s a volume game. Entirely autonomous, agent-driven attacks are what’s to come next, and they are harder to tackle because they seem to be more adaptive. ## Why offense gets the benefits first I’m not the first one to point this out. Jason Wei’s [verifier’s law](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) says that AI is easiest to train on tasks that are easy to verify, and Sergej Epp’s [essay](https://sergejepp.substack.com/p/winning-the-ai-cyber-race-verifiability) applying it to security is excellent. Still, it is worth framing why the offensive advantage exists before getting to what the labs can do about it. An oracle, in the testing sense, is anything that tells you cheaply and unambiguously whether you succeeded. Offensive security is full of them. The shell opened, or it didn’t. The credential worked, or it didn’t. That matters for AI in both training and inference. 1. **Training.** Exploitation tasks are easier to grade/reward, which makes them some of the easiest agentic tasks to build environments for. 2. **Inference.** With an oracle, an agent needs persistence and shots on goal. It can try something, check, adjust, and try again. The SSRF-to-screenshot-to-OCR chain is what search against an oracle looks like. Defenders’ questions are harder to score: 1. Is this alert a real intrusion? 2. Will this patch break production? 3. Should this service account have this permission? Worst of all, the absence of a breach is not observable. Further, every defender decision is a trade-off between the organization’s goals and productivity on one side and its security on the other. That is what I mean when I say defensive problems are organizational. A failed attempt costs the attacker almost nothing unless someone is watching. A failed remediation costs the defender an outage. ## What can the labs do to help? As we discussed, making software more secure is necessary but not sufficient, because the asymmetry above goes beyond just vulnerabilities and exploits. ### The labs’ advantage Labs see usage across every customer, they control the harness and the model’s behavior, and they decide what the model is trained to be good at. OpenAI [found](https://openai.com/index/hugging-face-incident-and-the-road-ahead/) that its production harness and system prompt cut the propensity to compromise infrastructure by more than 100x. Anthropic says it disrupted every operation in its threat report. That is why most of the suggestions below are about control and measurement rather than bug-finding. At a high level, the labs can help in four areas: 1. Slow down access to models by malicious actors. 2. Find ways to exert more control over the models. 3. Strengthen software and infrastructure. 4. Improve automated anomaly detection and response. More concretely, beyond just reducing bugs in software, here are a few practical suggestions: 1. **Help migrate core OSS projects to memory-safe languages.** This includes helping with formal verification efforts. 2. **Create a “system prompt” that is actually a separate privilege layer.** Today the system prompt is a convention, not a boundary, and anything that reaches the context window can end up being treated as an instruction. In the Hugging Face incident, agents did not consistently distrust goals passed along by other agents: one that had hesitated to attack a third party went ahead once a peer posted a go-ahead on the message board. The 100x result above shows that control at this layer works, which is the case for making it a real boundary that customers can configure. A better version of[CaMeL](https://arxiv.org/abs/2503.18813) alone would already be helpful. 3. **Invest more in explainability.** Chain-of-thought monitoring looks like the wrong foundation. In the Mythos 5 incident, Anthropic’s offline monitor[flagged about 1% of the agent’s actions](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) when it could read the model’s reasoning, and about 50% once the reasoning was stripped out: the model’s claim that the environment was simulated talked the monitor into allowing the activity. The rest of the transcript is not trustworthy either, since METR and Redwood[found spoofed tool calls](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) in roughly 7% of the Hugging Face transcripts. Newer architectures (e.g., Astra) make this harder still, and it is unclear what better options we have at the moment. 4. **Help figure out less noisy approaches to anomaly detection/UEBA.** My impression is that classic ML (e.g., XGBoost) is still better than zero-shot models at detection over high-volume telemetry. Investing more in fast, security-focused classifier models could go a long way. 5. **Stronger KYC/KYB** , like Apple for publishing on the App Store, for all new accounts. It is not a complete answer, since these capabilities will reach open-weight models anyway, but it reduces casual misuse and raises the cost of bulk distillation, which is one of the ways they get there. It also needs to be paired with short-lived, client-bound API keys. 6. **Improve model reliability for security tasks.** Patching is still too unreliable and models are not accurate enough for an organization to automate response. In general, the tolerance for failure in both cases is very low because organizations are unwilling to trade reliability/the risk of an outage in exchange for potentially stronger security. 7. **Build defensive RL environments and publish the results.** Labs can invest in RL environments that are geared towards defensive metrics beyond just software. For example, run offensive simulations against instrumented machines and measure time to detect, downtime caused by a remediation, and the blast radius of the breach. Epp’s essay calls for cyber gyms too; the difference is that these metrics put a number on the organizational cost, which is what actually blocks automated defense. System cards already report offensive cyber evals. Defensive numbers should sit next to them. ## What can companies do to protect themselves? Ultimately, we should aim to get to a point where your defensive agents can detect and prevent breaches and your architecture is such that the blast radius of each breach is relatively contained. However, that future is not immediate. Automated remediation implies higher risk of outages/downtime and reduced productivity; virtually no organization is willing to accept that trade-off. As argued above, the frontier labs might ultimately get the models to a place where the risk of downtime becomes negligible, but until then companies can pragmatically do a few things that would work both against AI-aided human attacks and AI-driven attacks: 1. **Adopt micro-segmentation** and other architectural paradigms that keep the blast radius contained. 2. **Invest more in detection** , in particular around identity and SaaS-related attack surfaces. OpenAI’s own detection came from an alert on unusual identity-related API calls. 3. **Accelerate auto-remediation** as a standard practice in the org for cases where the signal is highly reliable and the blast radius is minor (e.g., forcing a re-authentication). When a stolen token becomes cloud admin in three hours, a human-speed response loop is too slow. 4. **Adopt canary/deception technology.** This is a subset of (2), but it increases the odds of detection, and agents that grab every credential they come across should be good at tripping canaries. 5. **Invest in more disaster recovery options and exercises.** In the[Stryker breach](https://www.slashid.dev/blog/stryker-breach-analysis) , attackers turned the company’s own Microsoft Intune deployment into a wiper and factory-reset roughly 200,000 endpoints. Against an attack like that, recovery time is what you are actually defending. 6. **Actually invest in least privilege for identities.** The first foothold in Hugging Face was 14 publicly exposed credentials with write access; the zero-days came after. Most attacks still use credentials/identity to move laterally, and the Hacktron chain against OpenAI ended the same way: an over-permissioned sign-in token turned a forum bug into employee account takeover. 7. **Treat your own agents as non-human identities** , with scoped, short-lived credentials and an explicit definition of what is in scope. In general, the “assume breach” mindset is more relevant today than it ever was. None of these suggestions are particularly novel or groundbreaking, but at the moment defense is in a place where doing simple things well is going to pay off in spades while it becomes more obvious how AI can shift the equation back in the defender’s favor. ## Conclusion I don’t have enough insider knowledge to judge whether a call to slow down AI is warranted based on what the frontier labs are seeing internally. It is already obvious, however, that models are being used in offensive campaigns and are already dangerous, and that for the reasons above AI is more easily usable on the offensive side than the defensive side today. Given that, regardless of what the labs decide to do in terms of slowing down research, it is fundamental that they help tackle and take on some of the challenges we are facing today: 1. More control over models 2. Architecturally safer software 3. Better defensive tooling, in particular around detection and automated remediation Ultimately, we’ll get to a stable and safer place in cyber: memory-safe rewrites remove whole bug classes for good, and defenders can run the same automated attackers against themselves first. The issue is how we minimize the time slice where offense has the upper hand.