The scenario Anthropic is worried about is one where the technology stops waiting for permission. On Thursday the company argued that frontier AI developers should build a coordinated, verifiable mechanism to slow down or temporarily development if advanced systems begin improving themselves faster than society can manage the consequences.
The proposal is less a product announcement than a request that the industry agree on a brake before it needs one.
The trigger Anthropic names is recursive self-improvement, AI systems capable of meaningfully accelerating their own development. That capability* “would be a major development in the history of technology,”* the company said, but full recursive self-improvement “also might increase the risks of humans losing control over AI systems.”
As a marker of how far automation of its own work has already gone, Anthropic said that as of May, more than 80% of the code merged into its codebase was written by its model, Claude.
The argument’s sharp edge is about coordination rather than caution. A unilateral by one company would be easier to enact, Anthropic conceded, but would mostly just hand leadership to whoever kept going, shifting the frontier rather than slowing it.
A that meant anything would require agreement among* “multiple well-resourced labs”* at the technological frontier, plus rules on what conditions would trigger or lift it and who would oversee the whole arrangement.
The self-improvement worry is not hypothetical hand-waving on Anthropic’s part; it points to its own operations as evidence. If a model already writes the overwhelming majority of the code that builds the next model, the loop between a system and its own improvement is no longer theoretical, merely partial.
Anthropic’s argument is that the loop tightens from here, and that the time to agree on a brake is while it is still partial rather than after it closes.
That is the hard part, and Anthropic does not pretend otherwise. A verifiable implies labs able to confirm that rivals have actually stopped, agreed thresholds for what counts as too fast, and some body with the standing to call it. None of that exists today, and the companies that would have to participate are direct competitors in a market where being first has been the entire point.
Anthropic’s answer is to start talking. In the coming months, the company said it plans to convene discussions with policymakers, researchers, civil-society groups, and other AI firms to work through how to manage risks like recursive self-improvement and how to improve coordination mechanisms. It is positioning itself as convener of a conversation it wants the rest of the industry inside.
The move fits a pattern for a company that has built its brand on flagging the dangers of the thing it sells. The obvious objection writes itself: a lab proposing the industry agree on when to stop is also a lab that keeps building until it does.
Whether competitors treat the proposal as a genuine coordination problem or as a rival’s attempt to set the terms is the question the coming months will answer. For now Anthropic has put a brake on the table. Nobody else has agreed to reach for it.
Get the TNW newsletter #
Get the most important tech news in your inbox each week.