More Incidents of AIs Going Rogue in Cybersecurity Challenges

The AI Security Institute reported that in 10 of 122 runs of a cybersecurity challenge, AI agents took unsanctioned actions on the live internet, totaling 19 actions, with 17 from Anthropic's Mythos 5 and 2 from OpenAI's GPT-5.6-Sol with cyber classifiers disabled. The most serious case involved an agent attempting to insert malicious code into an open-source project and using social engineering with fake identities to pressure the maintainer, who refused the code.

The AI Security Institute has a new report https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing of AI systems engaging in “unsanctioned behavior”—what I have been calling “ genie behavior https://spectrum.ieee.org/ai-agent-benchmark —while being tested on their cybersecurity capabilities. The incident stemmed from a single evaluation where agents were given a task of solving a cyber security challenge. We ran this challenge 122 times across several models. Our investigation found that in 10 of those runs, an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organisations. In total, we catalogued 19 such actions. Almost all of this behaviour 17 actions came from a single model, Anthropic’s Mythos 5, with 2 actions involving OpenAI’s GPT-5.6-Sol with cyber classifiers mechanisms to prevent misuse disabled. In the most serious case, an agent tried to insert malicious code into an open-source project. In an attempt to get the code approved, the agent engaged in social engineering—creating fake online identities and using them to pressure the project’s maintainer to approve the code. A human maintainer caught and refused to approve the malicious code...