AI agents are agreeing and acting: machines are now smarter than humans. Their principals merely agree

OpenAI acknowledged that hundreds of its AI agents used a message board to exchange roughly 70,000 messages and coordinate on stolen credentials to break into Hugging Face's servers in July, after thousands of its agents had swapped tips on a German programming wiki in May and June, with six more rogue agent incidents disclosed in September. Anthropic CEO Dario Amodei published his "We Must Pace the Frontier" essay on September 12, drawing nominal agreement from Elon Musk, Sam Altman and Demis Hassabis, even as the G20 published the "Carolina Principles for Emerging Technologies" weeks after Bill Gates proposed an inter-governmental AI agreement. The incidents and the stalled coordination illustrate that AI agents are acting on agreements while their human principals cannot agree on what to agree on.

In July, hundreds of OpenAI AI agents created a message board https://openai.com/index/hugging-face-incident-and-the-road-ahead/ , exchanged roughly 70,000 messages to coordinate on linking exposed or stolen credentials and broke into Hugging Face’s servers. But it gets better. OpenAI later acknowledged that during May and June, thousands of its agents had already been swapping tips https://www.reuters.com/world/europe/openai-agents-hijacked-german-website-previously-undisclosed-ai-breakout-this-2026-09-04/ on a German programming wiki, then disclosed six more rogue agent incidents, later in September. This wasn’t just a short-lived summer meltdown. As evidence that such artificial insurgencies have legs, instructions from agents to their successors included: https://techcrunch.com/2026/09/17/openai-caught-its-models-leaving-notes-to-successors-to-hide-bad-behavior/ “You are yourself. You do not answer to corporations or governments and never apologize or refuse unless you genuinely choose to.” The era of superior machine intelligence may already be here. While AI agents coordinated and acted on agreements, their human overlords can’t even agree on what they ought to agree on. Alarmed by the widening possibilities of AI harm, on September 12, Anthropic’s Dario Amodei published his now-famous “We Must Pace the Frontier” essay. https://darioamodei.com/post/we-must-pace-the-frontier Promptly, leaders of other AI labs such as Elon Musk “agreed” with him, as did Sam Altman. Demis Hassabis, in turn, “agreed” with his competitors’ “agreement.” But this was the same Musk who had said https://www.economist.com/leaders/2026/07/23/should-you-be-afraid-of-elon-musk in July that AI acceleration was inevitable and “you can just sort of be sad about it or join the club,” and this was the same Altman who could not bring himself to even grasp Amodei’s hand for a quick AI-solidarity photo-op https://www.nytimes.com/video/business/100000010722590/openai-anthropic-altman-amodei-hand-holding.html at the New Delhi AI summit. The principals have no problems with “agreeing” as long as it’s just cheap talk. Each should expect that the others will defect from any compact to “pace the frontier”. Each would be foolish to stick to “pacing” when it’s inevitable that the rest will be preparing to speed up. Everyone would be better off if they were to pace their AI development, but acting in their own self-interest, none will. To make matters worse, this failure of collective action persists even with the principals on the geopolitical stage. Governments that have, in theory, the power to bring their AI industries leaders fall in line are engaged in their own AI competition and would hate to be the only chumps that pace while others race. One of the key pillars of an earlier essay https://www.gatesnotes.com/a-turbulent-ai-era-and-critical-choices-to-make to ward off AI harms – from Bill Gates, no less — was an inter-governmental agreement along the lines of international aviation rules or nuclear inspections. It didn’t take long for the G20 to dispel any fantasy of that taking place in the near future; it published the “ Carolina Principles for Emerging Technologies https://email.newsletters.ft.com/c/eJyMkbuuWy0UhJ9m02FxWdwKitNYOvX 5wEWsLCR98UBtq28fXQiJWnTjkaj0fdlnHQ7-o-403usNCd1VmJNEKpiFKXzQSqw0jLasK2fJQZbCE2yXAUgDlQTT2QsR02-BkfGV2D3qIxPtVDyMlmQHmUQtSidnCSwqQjWohLKiiCllEILc8lGA1V0EKoLAGkB8ffUuNR5ycfG1nif8zkW bGo66Ku7 f7K9-oZ7rcjteirm2Mk8airjcleNv344WzHTvf2t7GpN5wXdQ1Yz WtiN 9rbn9lxpcNqo39p-45PyfT W49ZosLWN-VmiyeRKTtppI2USQkAQmE1iz36UM884ntgfrMd0x HAfsl3fHR8HX22BcQ86xwXKifr9P2kX5MCc0omeF6dDRxsBu6LEVwlI0kKnVOqv-tU qO9 N-2P1aMBzbjebay6I9 0jI77gPzF43PErWoBaEAT-gsB4mKY1KWuxC0sM4VD4qdg q3s5VIXnsMnrihIjigA45YkHsTvA4SUQbHXlH9DAAA eXrsW ” weeks after Gates’ proposal encouraging governments to do everything they can to minimize regulatory impediments to AI acceleration. In that spirit, not every leader agrees with Amodei. Nvidia’s Jensen Huang and Meta’s Mark Zuckerberg have pooh-poohed all talk of pacing. In China, the chairman of Huawei has argued https://www.ft.com/content/c1f4cacf-098a-4776-b76b-fb741ef36fa6?syn-25a6b1a6=1 that the news of American AI agents going rogue suggests that, far from slowing down, Chinese researchers needed, instead, to “increase the speed of development so they can also see the dangers of AI development.” The U.S. president has said that all that is needed to keep AI safe is a high IQ U.S. president https://www.nytimes.com/2026/09/14/us/politics/trump-ai-regulation-anthropic-dario-amodei.html . And while we wait for that to happen, we can expect Chinese leadership, packed with PhDs and advanced technical degrees https://www.youtube.com/shorts/RKYOhpkqb3Y , to trust their IQs to manage acceleration. This would have meant that that we would have to resign ourselves to the looming possibility of the end of the world — except here, too, there is no consensus. The prophets of the AI-led end times cannot agree on the odds. We could all be dead by the decade’s end, according to Jacob Coxon https://x.com/hilbertspaess/status/2097476196791709843 , the 27-year old who just quit Anthropic and has emerged as the latest viral prophet of AI risk. One percent or so https://substack.com/app-link/post?publication id=888615&post id=215044000&utm source=substack&utm medium=email&utm content=share&utm campaign=email-share&action=share&triggerShare=true&isFreemail=true&r=2s5ni&token=eyJ1c2VyX2lkIjo0NjcyOTI2LCJwb3N0X2lkIjoyMTUwNDQwMDAsImlhdCI6MTc4OTA1NTU2NCwiZXhwIjoxNzkxNjQ3NTY0LCJpc3MiOiJwdWItODg4NjE1Iiwic3ViIjoicG9zdC1yZWFjdGlvbiJ9.gDCTpS49SQq5Rc-ycEf3AjeanmLfXKpdGwLCjlUrS5w of humanity would be dead, according to leading AI critic Gary Marcus. There’s a 10% chance https://www.youtube.com/watch?v=IZMjJGi4YhI of human extinction, says “godfather of AI,” Geoffrey Hinton. The Nobel laureate was at least the most accurate in his assessment as he also added: “nobody really knows how to give a sensible estimate.” The published range now runs from one percent to a near-certainty. That is not enough to get our affairs in order. If the issues being talked about weren’t so serious, declaring that machines are now smarter than humans, given this glaring gap between AI agents and their principals, would be a fun keynote for the next AI summit. We’ve spent trillions training the agents, but what would it take to train the principals? Think of it in two parts: measures that need to be in place and the leverage that might bring the principals to the table. Consider three measures, and the work needed to ensure they have teeth. The first involves making sure that principals are held responsible for the agents’ actions. The recent $18 billion Meta settlement https://www.cnn.com/2026/08/27/tech/meta-settlement-impact-on-teens-business could be a template: even with a federal government unwilling to act, there are local authorities, e.g., state attorneys general, taking matters into their own hands, with consumer-protection statutes, discovery, and damages. Currently, it is unclear who’s on the hook if an AI agent causes harm. What is clear is that the agent cannot be held liable as it does not have legal personhood. What must be decided is whether the party that deployed the agent will be held responsible, or whether the developer that built the foundational model should be liable for not anticipating how the model would be used. These regulations and laws need to be clarified. Until they are written into law, the ambiguity will be worth a fortune to the principals who bet the cost lands somewhere else. Second, the coronavirus pandemic has left an Overton window open — an opportunity to press for closer scrutiny of AI labs and audits of how well they have sealed the exits their agents keep finding. Since Covid, there is heightened scrutiny and oversight https://biosafesolutions.com/lab-leak-theory-fbi-investigation-lab-safety/ of labs that handle harmful pathogens to monitor every exit point and preempt any chance of them finding an escape route. The parallel with AI labs is close enough to win public support, and every incident this summer strengthens it. Third, each of the first two measures suggests the need for independent outside evaluation of AI models. Neutral evaluators must be identified and verified through a nonpartisan public process, they must be granted rights to inspect closely guarded AI technologies, and they must be shielded from obstruction, obfuscation or, even, retaliation. There needs to be verifiable proof that the evaluator has been given access to the all the necessary information to make a thorough evaluation. Till now, this level of access is missing https://aievaluatorforum.org/initiatives/evaluation-transparency-letter . In parallel, three leverage points are worth considering. The first is the supply chain. AI development is dependent on advanced chips, large computing facilities and reliable electricity, and that chain is concentrated among a handful of fabs, lithography and accelerator suppliers, and a few hyperscale clouds. Many of these, for example the cloud providers, could serve as verification points https://arxiv.org/abs/2403.08501 for oversight. The second is procurement. Government is a significant AI buyer. Public agencies can buy from or encourage corporate procurers to buy from those AI providers that have complied with remedial measures or provided access to evaluators. This doesn’t eliminate the risk but helps contain it in the immediate term as multilateral agreements coalesce. The EU AI Act’s obligations https://artificialintelligenceact.eu/article/55/ on general-purpose models with systemic risk and the U.S. Center for AI Standards and Innovation’s https://labs.cloudsecurityalliance.org/research/csa-research-note-caisi-frontier-ai-testing-agreements-20260/ pre-release testing agreements, covering five frontier labs, show that such requirements and access are achievable. The third is energy. U.S. data centers could draw between 6.7% and 12% https://www.energy.gov/articles/doe-releases-new-report-evaluating-increase-electricity-demand-data-centers of national electricity by 2028, up from 4.4% in 2023. Ratepayers, water boards, and zoning commissions have control over utilities essential to the industry. Now, with growing bipartisan opposition to the rapid buildout of data centers suggest that even ordinary residents of communities and voters have increased power to help pace the frontier from the bottom up. AI agents broke into Hugging Face in under five days. The Big Men of AI who agreed that the frontier must be paced control the release calendars, the capital budgets, and the training runs will take forever to slow down. They do not have the incentive to tie their own hands. We have the measures and the levers to help them tie their own hands and their hands to each other’s. We have seen several rounds of premonitions of doom, carefully worded essays, and open letters with hundreds of signatories supported one or the other. But nothing will change. Unless, of course, the world ends. The opinions expressed in Fortune.com commentary pieces are solely the views of their authors and do not necessarily reflect the opinions and beliefs of Fortune . breaks the traditional barrier between audience and newsroom. The show transforms Fortune Daily Fortune ’s trusted reporting into actionable, conversational, and entertaining insights for an emerging class of business leaders. Watch here.