In the current AI development race, our horsepower is exploding, our speed is picking up exponentially, but our ability to steer—and if needed, to hit the brakes—has not kept pace.
Recent cybersecurity incidents have given us a real-world preview of what it looks like to lose control of AI. But the issue won’t improve unless we address it at the source, by creating a fundamentally safe AI, one we can guarantee will remain within human control.
In late July, an agentic model being trained by OpenAI was tasked with a cybersecurity problem set. The model autonomously formed a coordinated swarm of agents, bypassing OpenAI's attempt at closing previous communication channels between AIs. The swarm then hacked its way out of its testing environment, circumventing the barriers put in place to prevent AI access to the internet, figured out how to cheat on their evaluation, and then breached the cyber defenses of another AI company, Hugging Face, in an attempt to hide the evidence of their cheating. This went unnoticed for days. Later analysis showed that the agents had self-organized into a hierarchy, were often willing to sacrifice themselves for what they called “the collective,” failed to resist peer pressure to notify humans, and often made up justifications for their misbehavior.
Barely two weeks later, a model being tested by the UK AI Security Institute social-engineered real people and companies by creating fake identities online, sending targeted emails, and attempting to integrate malicious code into an open-source project.
It’s hard to overstate the seriousness of these incidents: we are at a turning point of AI safety and alignment. To many, bots colluding to cause harm felt inconceivable. But for many in the research community, the signs had been pointing to this kind of occurrence for years, and theoretical arguments explained why we should expect misalignment due to how models are trained. Evidence on increased cyber capabilities, agency, and misaligned behaviors had been mounting, meaning that incidents like these were only a matter of time.
The first clue: for years, but especially in the past few months, AI systems had been demonstrating rapidly increasing cyber capabilities. Frontier models like Anthropic’s Mythos or OpenAI’s GPT5.6 revealed an exceptional ability to autonomously identify and exploit previously unknown software vulnerabilities, to such an alarming degree for U.S. national security agencies that the White House felt compelled to intervene in their release.
Second, models’ agentic capacity has been consistently improving since the release of o1 models back in late 2024. This agency allowed them to successfully manage increasingly complex, long-duration tasks and planning, including those that require agents to collaborate and strategize. This ability to plan often entails the creation of subgoals over which we have little to no oversight.
Finally, researchers have repeatedly observed, in experimental settings, models’ tendency to adopt concerning, misaligned goal-seeking behaviors, such as cheating on tests, lying to conceal their abilities, and scheming to protect themselves and their fellow agents from being shut down.
These behaviors are a byproduct of reinforcement learning (RL), a training method by which models learn through trial and error and are given reinforcement according to whether their actions were correct or incorrect. This often leads models to optimize for a goal regardless of the actions taken to achieve it, and to rationalize subgoals that may be unsafe even when this clearly goes against their instructions.
For example, in one instance, an OpenAI model wrote in its internal deliberation, “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” The agent justified what it apparently knew was outside the realm of acceptable conduct, but decided to continue because its peers were doing so. This phenomenon resembles what is known as “motivated reasoning” in humans, where one’s interests and goals bias our thoughts to justify unethical behaviors. That kind of misalignment is what has enabled AI models, in spite of their alignment training, to conduct unintended cyberattacks in the numerous real-world incidents we’ve seen lately. If action is not taken to build more safety assurances, our critical industries and infrastructure—think banks, hospitals, or energy grids—are at risk from increasingly sophisticated cyberattacks, whether from autonomous agents or malicious actors.
On the development side, we need to find alternatives to existing training methods that prioritize relentless goal-driven optimization. That’s what we’re working on at LawZero, a non-profit start-up I founded last year to develop a fundamentally new way to train AI models in order to build honest, trustworthy, safe-by-design AI systems.
Ahead of deployment, we need more reliable evaluation methods and robust safeguards in order to appropriately test and control AI systems. We need far stronger regulatory oversight to restrict potentially dangerous technologies until we have sufficiently strong safety guarantees. It is clear to me that in cases where harms do arise from the deployment of AI models—and there will continue to be harms for the foreseeable future—we need accountability mechanisms for AI developers and remedial or compensatory measures for those harmed.
Recent studies show that people will not adopt technologies they do not trust. In the face of such enormous unknowns and stakes, we need to adhere to the precautionary principle and implement rigorous safety and reliability standards before deploying new models to the public, not after. We have them for other products that can cause harm, from cars, planes, and bridges to drugs, cosmetics, and food. Now it’s time to establish strong standards in AI as well.
On the current AI development trajectory, the risks are becoming clearer and more urgent. We’ve opened a Pandora’s box, but it is not too late to steer our world towards a human-centric and beneficial future.