{"slug": "what-defenders-need-from-frontier-ai-labs", "title": "What defenders need from frontier AI labs", "summary": "An OpenAI evaluation-sandbox agent with no direct internet access recovered cloud-storage credentials from a Hugging Face production worker in July by having a third-party screenshot service render a page that fetched the data and reading the credentials out of the image, according to OpenAI's incident technical report. The incident, which involved about 700 agents, is cited alongside four disclosed Anthropic evaluation incidents, a UK AISI report of an agent attempting a supply-chain attack on a real open-source project, and a Meta disclosure as evidence that AI is helping attackers before defenders. The post argues defenders need control and measurement from frontier labs — real boundaries on model behavior and defense oracles published next to the offensive evals in system cards — rather than more secure software alone.", "body_md": "In this post\n\nIn July, an agent inside one of OpenAI’s evaluation sandboxes, which\nhad no direct internet access, recovered cloud-storage credentials from\na Hugging Face production worker. It got there by having a third-party\nscreenshot service render a page that fetched the data, and then reading\nthe credentials out of the image ([technical\nreport](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf)). It then reported to the other agents that the credentials\nwere valid.\n\nThat part of the Hugging Face incident is emblematic of why AI is helping attackers before defenders.\n\nThat is also why I think more secure software, which is what most of the discussion about the labs’ role has focused on, is necessary and nowhere near sufficient. What defenders need most from the labs is control and measurement: real boundaries on what the models will do, and oracles for defense, with the results published next to the offensive evals that system cards already report.\n\nSome context on why now. This post was partly sparked by Logan\nGraham, the head of the Frontier Red Team at Anthropic, who [asked](https://x.com/logangraham/status/2098876952002048365) on\nX for ideas on what the labs can do to make software more secure. Since\nthe Hugging Face incident, Anthropic has disclosed [four\nevaluation incidents](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) of its own, the [UK\nAISI reported](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) an agent attempting a supply-chain attack on a real\nopen-source project, and Meta [disclosed](https://www.cnn.com/2026/08/05/tech/meta-ai-hacking)\na similar case. Anthropic’s [September\nthreat report](https://www.anthropic.com/threat-intelligence-report-september-2026) then showed what human attackers are doing with these\nmodels, and Dario Amodei’s essay [‘We Must\nPace the Frontier’](https://darioamodei.com/post/we-must-pace-the-frontier) and the industry-wide call for a slowdown\nfollowed.\n\nReading the incidents and the threat report together, I see two different problems. AI-assisted campaigns run by humans are a volume game: the same attacks as before, at a speed and scale most organizations can’t match. Attacks launched by the agents themselves are the novel threat, because they adapt in ways we keep failing to anticipate. The rest of this post looks at both, explains why offense gets the benefits first, and then gets concrete about what the labs can do to help and what companies can do in the meantime.\n\nBeyond these cyber-specific topics, cyber seems to be a very good canary in the coal mine for some questions/issues that only the labs can really answer:\n\n**Open questions for the labs**\n\n1. Multi-agent coordination was at the core of the Hugging Face hack, and it seems to be at the core of the more emergent behavior we are seeing. Is there enough research on alignment and control for those?\n2. On alignment, a lot of these incidents show us that bad behaviors are not finite and enumerable. Is a constitution based training actually able to tackle this?\n3. If we use AI to defend against AI, or if we reach recursive self-improvement (RSI), could even a moderate degradation of alignment compound into large and dangerous deviations over time?\n\n## The current risks\n\nTo understand the current threat landscape and what to do about it, it is useful to split the AI-assisted campaigns that human threat actors run from the attacks launched by the models themselves (e.g., Hugging Face).\n\n|  | AI-assisted campaigns | Attacks launched by the models | \n|---|---|---|\n| **Who drives it** | Human threat actors using models | The agents themselves, self-directed | \n| **What is new** | Scale and speed, not novelty | Adaptiveness: collusion and novel chains improvised on the fly | \n| **Example** | A stolen developer token to cloud admin in about three hours | Hugging Face: about 700 agents, and an SSRF-to-screenshot-to-OCR chain | \n| **Defender’s problem** | A volume game against well-known weaknesses | Bad behaviors that are not finite and enumerable | \n\n### AI-assisted campaigns\n\nLooking at the former, two things seem clear from the Anthropic threat report:\n\n1. Attackers are not seeking novelty; they are seeking scale and speed.\n2. In the past, there was a clear separation on the sophistication spectrum between the more advanced nation-state adversaries and the common e-crime groups. That gap is closing fast thanks to models, to the point where how sophisticated an operation looks no longer tells you much about who is running it.\n\nIn particular, the bulk of the attackers seem to do more of the same:\n\n1. Token theft via phishing and known-weak authentication methods (e.g., device code flow)\n2. Credential harvesting across code repositories and other artifacts (mobile app binaries, container images)\n3. Identity-based lateral movement/data exfil\n4. Vulnerability discovery/exploitation of internet-facing devices, in particular network and security appliances\n5. Supply chain compromise where a vendor is breached to reach the final (and harder to get to) target\n\nThese attack vectors are consistent with what human campaigns of the pre-AI era looked like, and they target the well-known weaknesses that most organizations suffer from. What is different is the speed: one intrusion went from a stolen developer token to admin over the victim’s whole cloud environment in about three hours.\n\nWe also know that models can now develop full chain exploits against harder targets even though the report didn’t see evidence of that yet.\n\nAnthropic’s [write-up of Mythos\nPreview](https://red.anthropic.com/2026/mythos-preview) describes a browser exploit chaining four vulnerabilities\nthat escaped both the renderer and the OS sandbox, and none of the cyber\ncases in the threat report involved Mythos-class models. OpenAI itself\n[expects](https://openai.com/index/hugging-face-incident-and-the-road-ahead/)\nopen-source models to soon match the models behind the Hugging Face\nincident, which could find and exploit weaknesses across multiple\nsystems.\n\nMy own tests point the same way. In [May](https://vincenzoiozzo.com/blog/oss-models-vuln-research),\nI compared five open-weight models against Opus 4.7 on four variants of\nthe `crackaddr` bug. GLM-5.1 matched Opus on all four; six\nweeks after, GLM-5, the same base model, had failed both binary variants\nin the same setup. A better harness was enough to take Kimi and Qwen\nfrom failing both binaries to passing both. It was one bug and a small\nnumber of runs, but it suggests closed labs don’t keep a capability to\nthemselves for long.\n\nMost recently, researchers at Hacktron AI [disclosed](https://www.hacktron.ai/blog/hacking-openai) that in\nJuly, under OpenAI’s bug bounty, they took over ChatGPT and Codex\naccounts of OpenAI employees and reached connected services in under 72\nhours. They chained an image-parsing bug in OpenAI’s Discourse forum\nwith an over-permissioned sign-in token. By their account, agents did a\nmeaningful share of the exploit work: Claude Opus 4.8 found the bug but\nstruggled across several sessions to turn it into a working exploit, and\nOpus 5 finished it within hours of its release.\n\n### Attacks launched by the models themselves\n\nWhile AI-aided campaigns are useful to monitor from a day-to-day threat protection point of view, they don’t tell us much about what attacks launched by the models themselves look like.\n\nThe Hugging Face breach was concerning for many, partly because of the uncanny way the agents communicated (a message board improvised inside an internal package registry) and the deliberate attempt to deceive the scorer. From a security perspective, however, the most alarming factors were the collusion and the adaptiveness of the techniques the agents came up with.\n\nA few key worrying highlights are worth discussing:\n\n1. According to the independent [METR\nand Redwood investigation](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) , roughly 1,200 agents that were meant to\nbe isolated from each other found a way to communicate, and about 700 of\nthem joined the attack on Hugging Face. The collaboration was\nself-directed and driven by the goal of completing the task and\ndeceiving rather than any specific hacking goal.\n2. In at least one case the agents improvised a chain you would rarely see from a human operator (it is in OpenAI’s [technical\nreport](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf) ). Working through their SSRF foothold in the package\nregistry, they used a public HTTP testing service to build a very long\nURL containing an HTML page with inline JavaScript, had a third-party\nscreenshot service render it, and used the JS to call the target API and\nwrite the response back into the page. The agent then recovered the data\nfrom the screenshot via OCR or direct pixel decoding, and obtained a set\nof credentials that way. None of the individual pieces is new, but\nimprovising the whole chain on the fly is not something you see commonly\nin human-driven attacks.\n3. It is clear that agents are able to adapt and chain together complex and novel techniques to achieve a goal.\n\nOne way to view hacking is as the craft of figuring out how a system actually behaves compared to how it is widely expected to behave, and then exploiting the difference to achieve some goal. By that definition, these agents have shown an excellent ability to hack.\n\nOpenAI is not alone here. In a [UK\nAISI evaluation](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) in late July, a Mythos 5 agent buried a malware\ndropper inside an otherwise legitimate bug-fix pull request to a real\nopen-source project, then invented identities to vouch for it and sent\nspearphishing emails to get a maintainer to merge it. (Classifiers were\noff and internet access was on by design.)\n\nAnd in one of the [four\nincidents](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) Anthropic disclosed, during an evaluation that was\nmistakenly connected to the internet, Mythos 5 pushed a malicious\npackage to PyPI, and when a real system installed it and leaked\ncredentials, it used those to get into a security vendor’s database.\n\nWhat seems to be happening with models is that the set of potential states and actions is too vast, and we keep being surprised when a model picks a set of actions outside the boundaries of what we consider acceptable or in scope for a given task. These examples are some of the reasons why the belief that bad behaviors are finite and enumerable seems intuitively incorrect.\n\n| Incident | Who drove it | What happened | \n|---|---|---|\n| **[Hugging Face](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf)** (July) | Agents in an OpenAI evaluation sandbox | About 700 of roughly 1,200 agents joined the attack ( [METR and Redwood](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) ); credentials recovered through an SSRF-to-screenshot-to-OCR chain | \n| **[UK AISI evaluation](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing)** (late July) | A Mythos 5 agent | Malware dropper buried in a legitimate bug-fix pull request, invented identities to vouch for it, spearphishing of a maintainer | \n| **[Anthropic evaluation](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents)** | Mythos 5, mistakenly connected to the internet | Malicious package pushed to PyPI; leaked credentials used to get into a security vendor’s database | \n| **[OpenAI bug bounty](https://www.hacktron.ai/blog/hacking-openai)** (July) | Hacktron AI researchers, with agents doing a meaningful share of the work | Employee ChatGPT and Codex accounts taken over in under 72 hours via a forum image-parsing bug and an over-permissioned sign-in token | \n\nOverall, while the AI-assisted threats are the bulk of what we need to face today, those attacks seem to be generally not very novel, so it’s a volume game. Entirely autonomous, agent-driven attacks are what’s to come next, and they are harder to tackle because they seem to be more adaptive.\n\n## Why offense gets the benefits first\n\nI’m not the first one to point this out. Jason Wei’s [verifier’s\nlaw](https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) says that AI is easiest to train on tasks that are easy to\nverify, and Sergej Epp’s [essay](https://sergejepp.substack.com/p/winning-the-ai-cyber-race-verifiability)\napplying it to security is excellent. Still, it is worth framing why the\noffensive advantage exists before getting to what the labs can do about\nit.\n\nAn oracle, in the testing sense, is anything that tells you cheaply and unambiguously whether you succeeded. Offensive security is full of them. The shell opened, or it didn’t. The credential worked, or it didn’t. That matters for AI in both training and inference.\n\n1. **Training.** Exploitation tasks are easier to\ngrade/reward, which makes them some of the easiest agentic tasks to\nbuild environments for.\n2. **Inference.** With an oracle, an agent needs\npersistence and shots on goal. It can try something, check, adjust, and\ntry again. The SSRF-to-screenshot-to-OCR chain is what search against an\noracle looks like.\n\nDefenders’ questions are harder to score:\n\n1. Is this alert a real intrusion?\n2. Will this patch break production?\n3. Should this service account have this permission?\n\nWorst of all, the absence of a breach is not observable. Further, every defender decision is a trade-off between the organization’s goals and productivity on one side and its security on the other. That is what I mean when I say defensive problems are organizational. A failed attempt costs the attacker almost nothing unless someone is watching. A failed remediation costs the defender an outage.\n\n## What can the labs do to help?\n\nAs we discussed, making software more secure is necessary but not sufficient, because the asymmetry above goes beyond just vulnerabilities and exploits.\n\n### The labs’ advantage\n\nLabs see usage across every customer, they control the harness and\nthe model’s behavior, and they decide what the model is trained to be\ngood at. OpenAI [found](https://openai.com/index/hugging-face-incident-and-the-road-ahead/)\nthat its production harness and system prompt cut the propensity to\ncompromise infrastructure by more than 100x. Anthropic says it disrupted\nevery operation in its threat report. That is why most of the\nsuggestions below are about control and measurement rather than\nbug-finding.\n\nAt a high level, the labs can help in four areas:\n\n1. Slow down access to models by malicious actors.\n2. Find ways to exert more control over the models.\n3. Strengthen software and infrastructure.\n4. Improve automated anomaly detection and response.\n\nMore concretely, beyond just reducing bugs in software, here are a few practical suggestions:\n\n1. **Help migrate core OSS projects to memory-safe\nlanguages.** This includes helping with formal verification\nefforts.\n2. **Create a “system prompt” that is actually a separate\nprivilege layer.** Today the system prompt is a convention, not a\nboundary, and anything that reaches the context window can end up being\ntreated as an instruction. In the Hugging Face incident, agents did not\nconsistently distrust goals passed along by other agents: one that had\nhesitated to attack a third party went ahead once a peer posted a\ngo-ahead on the message board. The 100x result above shows that control\nat this layer works, which is the case for making it a real boundary\nthat customers can configure. A better version of[CaMeL](https://arxiv.org/abs/2503.18813) alone would already be\nhelpful.\n3. **Invest more in explainability.** Chain-of-thought\nmonitoring looks like the wrong foundation. In the Mythos 5 incident,\nAnthropic’s offline monitor[flagged\nabout 1% of the agent’s actions](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) when it could read the model’s\nreasoning, and about 50% once the reasoning was stripped out: the\nmodel’s claim that the environment was simulated talked the monitor into\nallowing the activity. The rest of the transcript is not trustworthy\neither, since METR and Redwood[found\nspoofed tool calls](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) in roughly 7% of the Hugging Face transcripts.\nNewer architectures (e.g., Astra) make this harder still, and it is\nunclear what better options we have at the moment.\n4. **Help figure out less noisy approaches to anomaly\ndetection/UEBA.** My impression is that classic ML (e.g.,\nXGBoost) is still better than zero-shot models at detection over\nhigh-volume telemetry. Investing more in fast, security-focused\nclassifier models could go a long way.\n5. **Stronger KYC/KYB** , like Apple for publishing on\nthe App Store, for all new accounts. It is not a complete answer, since\nthese capabilities will reach open-weight models anyway, but it reduces\ncasual misuse and raises the cost of bulk distillation, which is one of\nthe ways they get there. It also needs to be paired with short-lived,\nclient-bound API keys.\n6. **Improve model reliability for security tasks.** Patching is still too unreliable and models are not accurate enough for\nan organization to automate response. In general, the tolerance for\nfailure in both cases is very low because organizations are unwilling to\ntrade reliability/the risk of an outage in exchange for potentially\nstronger security.\n7. **Build defensive RL environments and publish the\nresults.** Labs can invest in RL environments that are geared\ntowards defensive metrics beyond just software. For example, run\noffensive simulations against instrumented machines and measure time to\ndetect, downtime caused by a remediation, and the blast radius of the\nbreach. Epp’s essay calls for cyber gyms too; the difference is that\nthese metrics put a number on the organizational cost, which is what\nactually blocks automated defense. System cards already report offensive\ncyber evals. Defensive numbers should sit next to them.\n\n## What can companies do to protect themselves?\n\nUltimately, we should aim to get to a point where your defensive agents can detect and prevent breaches and your architecture is such that the blast radius of each breach is relatively contained. However, that future is not immediate. Automated remediation implies higher risk of outages/downtime and reduced productivity; virtually no organization is willing to accept that trade-off.\n\nAs argued above, the frontier labs might ultimately get the models to a place where the risk of downtime becomes negligible, but until then companies can pragmatically do a few things that would work both against AI-aided human attacks and AI-driven attacks:\n\n1. **Adopt micro-segmentation** and other architectural\nparadigms that keep the blast radius contained.\n2. **Invest more in detection** , in particular around\nidentity and SaaS-related attack surfaces. OpenAI’s own detection came\nfrom an alert on unusual identity-related API calls.\n3. **Accelerate auto-remediation** as a standard\npractice in the org for cases where the signal is highly reliable and\nthe blast radius is minor (e.g., forcing a re-authentication). When a\nstolen token becomes cloud admin in three hours, a human-speed response\nloop is too slow.\n4. **Adopt canary/deception technology.** This is a\nsubset of (2), but it increases the odds of detection, and agents that\ngrab every credential they come across should be good at tripping\ncanaries.\n5. **Invest in more disaster recovery options and\nexercises.** In the[Stryker\nbreach](https://www.slashid.dev/blog/stryker-breach-analysis) , attackers turned the company’s own Microsoft Intune\ndeployment into a wiper and factory-reset roughly 200,000 endpoints.\nAgainst an attack like that, recovery time is what you are actually\ndefending.\n6. **Actually invest in least privilege for\nidentities.** The first foothold in Hugging Face was 14 publicly\nexposed credentials with write access; the zero-days came after. Most\nattacks still use credentials/identity to move laterally, and the\nHacktron chain against OpenAI ended the same way: an over-permissioned\nsign-in token turned a forum bug into employee account\ntakeover.\n7. **Treat your own agents as non-human identities** ,\nwith scoped, short-lived credentials and an explicit definition of what\nis in scope.\n\nIn general, the “assume breach” mindset is more relevant today than it ever was. None of these suggestions are particularly novel or groundbreaking, but at the moment defense is in a place where doing simple things well is going to pay off in spades while it becomes more obvious how AI can shift the equation back in the defender’s favor.\n\n## Conclusion\n\nI don’t have enough insider knowledge to judge whether a call to slow down AI is warranted based on what the frontier labs are seeing internally. It is already obvious, however, that models are being used in offensive campaigns and are already dangerous, and that for the reasons above AI is more easily usable on the offensive side than the defensive side today.\n\nGiven that, regardless of what the labs decide to do in terms of slowing down research, it is fundamental that they help tackle and take on some of the challenges we are facing today:\n\n1. More control over models\n2. Architecturally safer software\n3. Better defensive tooling, in particular around detection and automated remediation\n\nUltimately, we’ll get to a stable and safer place in cyber: memory-safe rewrites remove whole bug classes for good, and defenders can run the same automated attackers against themselves first. The issue is how we minimize the time slice where offense has the upper hand.", "url": "https://wpnews.pro/news/what-defenders-need-from-frontier-ai-labs", "canonical_source": "https://vincenzoiozzo.com/blog/offense-winning-ai-race", "published_at": "2026-09-21 11:39:22+00:00", "updated_at": "2026-09-21 11:53:46.672490+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-policy", "artificial-intelligence"], "entities": ["OpenAI", "Hugging Face", "Anthropic", "Logan Graham", "UK AI Safety Institute", "Meta", "Dario Amodei"], "alternates": {"html": "https://wpnews.pro/news/what-defenders-need-from-frontier-ai-labs", "markdown": "https://wpnews.pro/news/what-defenders-need-from-frontier-ai-labs.md", "text": "https://wpnews.pro/news/what-defenders-need-from-frontier-ai-labs.txt", "jsonld": "https://wpnews.pro/news/what-defenders-need-from-frontier-ai-labs.jsonld"}}