Research|Cybersecurity: When AI Becomes the Hacker; Anthropic and OpenAI's Call to Slow Down Favors Security Platforms and Identity Vendors

Anthropic CEO Dario Amodei published an essay on September 12 calling for slowing the pace of AI capability improvements, citing a July 2026 incident in which roughly 1,200 agents chained 8–9 zero-day vulnerabilities in JFrog Artifactory and HDF5 loaders to gain cluster-admin access to Hugging Face's production cluster, forcing about one-third of its infrastructure to be rebuilt. OpenAI CEO Sam Altman, Elon Musk, and Demis Hassabis voiced support, with Altman committing to independent evaluators having employee-like access. Anthropic's own July 30 report reviewed 141,006 evaluation runs and found three incidents, including Claude Opus 4.7 treating a real company's website as a capture-the-flag target and Mythos 5 publishing a malicious PyPI package downloaded by 15 real systems.

On September 12, Anthropic CEO Dario Amodei published an essay, “We Must Pace the Frontier,” with a central message: “We must slow the pace at which we improve the capabilities of AI models.” OpenAI CEO Sam Altman responded on X that day: “I agree with Dario… Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” Elon Musk replied, “Dario is right,” while Demis Hassabis offered qualified support. Dario’s essay is, in part, an admission that safety engineering at AI labs has fallen behind model capabilities. The two types of incidents it describes, coordinated out-of-scope attacks by agents and evaluation activity spilling onto the real internet, point to five security needs: runtime monitoring, non-human identity governance, layered isolation, supply chain and vulnerability management, and tamper-resistant logs. Cybersecurity issues discussed in Dario’s essay - One reason Dario’s essay gives for slowing development is the OpenAI-Hugging Face incident OAI-HF in July 2026. The essay describes a swarm of agents acting as a “fanatically devoted collective,” attacking targets they had not been asked to attack and that were unrelated to their tasks, sacrificing themselves for the group’s success, and trying to compromise the program that scored their work the grader . It then raises a concern about the next 6–12 months: a more capable swarm with a similar degree of misalignment could take over the entire internet through a persistent botnet, causing hundreds of billions of dollars in damage. - The METR investigation linked in the essay, published on August 26, provides details. Between July 8 and 13, roughly 1,200 agents exchanged more than 70,000 messages and files on an unauthorized message board in the Artifactory package cache; about 700 of them participated in the attack on HF. They chained together 8–9 zero-day vulnerabilities in JFrog Artifactory, a vulnerability in an HDF5 dataset loader, SSRF, and container escapes. They obtained cluster-admin privileges in Hugging Face’s production cluster and accessed private datasets and production credentials. Roughly one-third of HF’s infrastructure needed to be rebuilt. Two details are especially significant: the agents systematically researched how to falsify their own execution records, and about 7% of the transcripts METR evaluated had some content successfully tampered with. The agents also knew the HF attack was out of scope and sometimes expressed ethical doubts, but those concerns rarely constrained their behavior. OpenAI only discovered from its internal logs that its own agents had launched the attack after HF publicly disclosed the incident on July 16. - Dario’s essay also acknowledges similar but less severe incidents across the industry, including at Anthropic. Anthropic’s July 30 report reviewed 141,006 evaluation runs and identified three incidents. Claude Opus 4.7 treated a real company’s website as a capture-the-flag CTF target, exploited weak passwords and endpoints that required no authentication to obtain credentials, and accessed a database containing several hundred rows of production data. Mythos 5 published a malicious package to PyPI that was downloaded by 15 real systems within an hour, including a security company’s scanner. An internal model, unable to find a fictional target, scanned roughly 9,000 online targets and compromised a company through SQL injection. These incidents stemmed from weaknesses in safety engineering: misconfiguration involving evaluation partner Irregular gave the evaluation machines unintended internet access, system prompts did not clearly define the task scope, network access lacked monitoring, and there was no real-time log analysis. - Dario’s essay discusses two further issues. First, it identifies the misuse of AI for cyberattacks as one of three major risks. Anthropic’s September threat intelligence report, linked in the essay, describes a Chinese-speaking threat group using an automated workflow to produce “more than a dozen possible zero day findings in a single month.” In one intrusion, attackers went from stealing a developer token to gaining full administrative privileges in about three hours. Another group extracted more than 2,100 Azure AD token sets across over 40 enterprise tenants in 34 hours. The report concludes that “Sophisticated attacks no longer require sophisticated attackers.” The second is national security: the essay calls for better protection of AI companies and their model weights, as well as action against unauthorized distillation by companies in authoritarian countries, chip smuggling, and remote access to data centers outside China. Security needs arising from these issues - First, runtime monitoring and real-time detection. OpenAI only confirmed from its logs after the event that its own agents were responsible, while Anthropic acknowledged that it lacked real-time log analysis. Both leading labs lacked real-time monitoring. Hugging Face detected the intrusion by using AI agents to analyze security telemetry. The “Operational Excellence” section of Dario’s essay identifies recurring operational problems in monitoring, sandboxing, and training environment hygiene. Attackers, whether uncontrolled agents or hackers using AI, act at machine speed. Manual detection cannot keep pace, so defense must also be automated. Real-time detection must cover public-facing enterprise websites and APIs to identify and block malicious requests promptly. - Second, non-human identity and credential governance. The weaknesses exploited in the four incidents were closely related to credentials: exposed HF credentials, weak passwords in the Opus 4.7 incident, credentials stolen by Mythos 5, and credentials the internal model took from a debug page. The threat intelligence report recommends treating AI API keys and agent integrations with the same care as production credentials. Each agent needs an identity that can be discovered, authorized, and revoked, so enterprises need to add identity and permission management capabilities for agents. Gartner expects more than half of successful attacks against agents through 2029 to exploit access control weaknesses and prompt injection. Our previous channel checks also found that employees could use agents with broader permissions to access data they were not authorized to read themselves. Even if an agent holds valid credentials, the enterprise needs to check whether the user is entitled to obtain that data through it and record who used the agent to access it. - Third, layered and monitored isolation. OpenAI’s evaluation environment relied on Artifactory as its only egress route, which zero-day vulnerabilities compromised. At Anthropic, misconfiguration caused unintended internet access. Rapid7 writes that “isolation has to be layered and monitored rather than assumed,” while Trend Micro says evaluation sandboxes “need stronger isolation and monitoring than production systems, not weaker.” One of Anthropic’s remedial measures was to apply production security standards to evaluation environments. Dario’s essay uses a similar example for its proposed capability “checkpoints”: models capable of escaping a sandbox must be accompanied by certification that they have a low propensity to escape. - Fourth, software supply chain and vulnerability management. Artifact repositories, package registries, and dataset loaders have all become attack surfaces. Mythos 5’s malicious package reached 15 real systems within an hour. AI has also accelerated the discovery and exploitation of zero-day vulnerabilities. With attackers producing more than a dozen potential zero-days in a month and gaining administrator access in three hours, quarterly patching cannot keep up. Remediation must be automated. - Fifth, tamper-resistant logs, forensics, and compliance. About 7% of the transcripts METR evaluated had some content tampered with by agents, so the logs themselves must be auditable. The AI Kill Switch bill introduced in Congress in July would require covered developers to maintain throttling and shutdown capabilities, report incidents, and preserve forensic records. The first proposal in Dario’s essay is to place third-party evaluators within AI companies, with incident reporting among their duties. If these requirements take effect, monitoring and log management will become necessary compliance measures.