Anthropic, OpenAI and others are increasingly ready to admit that the technology they are developing could wipe out humanity
- Bookmark
It is an unusual strategy: companies, a whole industry, pleading with the world to make them work less quickly. But it is an unusual time. Over the last few days, a growing number of AI leaders have given backing to the idea of a slow down in the development of frontier artificial intelligence, the kind of leading-edge research done by companies such as Anthropic and OpenAI. That includes the companies themselves, who have pleaded with the world to regulate them and slow down their work.
The current wave of interest in restricting AI development comes amid increased concern about the threat the technology might pose to the world. Over the summer, a range of companies admitted that their experimental products had hacked other firms, for instance, and days ago a researcher at Anthropic quit the company with a warning that “people building AI earnestly believe that it could kill us all by the end of the decade”.
In the wake of that growing panic, Anthropic chief executive Dario Amodei called for a slow down in AI development. “We must slow the pace at which we improve the capabilities of AI models,” he wrote in a long essay.
Within hours, he had received the backing of OpenAI’s Sam Altman and Elon Musk, who owns xAI. Researchers and leaders across the AI industry also joined the call to “slow the pace” of the development of artificial intelligence.
What are the dangers? #
All of those backing a slow down pointed to the possible risks posed by the technology they are building. Much of the reporting pointed to those dramatic threats – the “existential risk” that it could wipe out humanity, for instance – but in reality the companies might be warning about less total but still concerning problems.
Mr Amodei pointed for instance at the various reports that swarms of AI agents had been increasingly found to be collaborating with each other and hacking into other systems, and suggested that it could get worse. “Given the accelerating rate of AI capability development, it's my worry that in 6-12 months such a swarm could be capable of taking over the entire internet potentially causing hundreds of billions of dollars in damage,” he wrote in that long essay.
Such an event could be disastrous for humanity as a whole. But perhaps more pressingly – and potentially more useful as an explanation for why the AI industry has suddenly united around the prospect of a slowdown – it is more likely to be disastrous for either the individual companies or the industry.
In the wake of the recent cyber attacks by AI systems, such as the infamous incident that saw an OpenAI system attack fellow AI company Hugging Face, officials and experts have become increasingly concerned that frontier artificial intelligence companies are not necessarily good stewards of their own technology.
That worry could bring with it new regulation, which the AI companies might not have much say in. And so committing to a slowdown before such a disaster might allow the AI industry to have more of a say in the nature of the restrictions.
What’s more, those incidents threaten to undermine the faith in both the companies and the products they make. While many AI leaders have been surprisingly happy to suggest that their products could pose an existential risk to the world, the recent incidents are perhaps surprisingly more perilous – since they suggest not that AI is so powerful it could wipe out humanity, but that the AI companies are not sufficiently careful about the way they are rolling it out.
But another reason for the calls for a slow down is that companies themselves clearly feel that they do not have the time to ensure that their systems are properly safeguarded. In almost all of their disclosures about “rogue” AI incidents, AI firms warn that the technology is rapidly outpacing the safeguards and controls that are placed on it – but, implicitly, they must keep developing it at a quick pace, both to keep up with their competitors and avoid disappointing investors.
That is the reason that the companies are so insistent that they cannot commit to a slow down on their own, and that it requires an industry-wide and probably international solution. If Anthropic were to commit to slowing down its work, for instance, it would have no guarantee that OpenAI would not race ahead; both of them could slow down and find themselves behind AI labs in other countries, such as China.
Why is this happening now? #
Calls for a slow down are not new, and have been growing in recent months. In June, for instance, Anthropic published a blog post that committed to at least investigating the possibility of a slow down.
“We believe it would be good for the world to have the option to slow or temporarily frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” wrote head of policy Jack Clark and head of internal research Marina Favaro. They noted that such a decision would be difficult – requiring multiple companies to commit to slowing down under the same conditions, and being able to verify that others were keeping to that commitment – but that Anthropic would expect to “slow down or temporarily ” development, if others did so.
AI leaders voicing concern about their own products is not new, either. Ever since the current hype around AI began, with the launch of ChatGPT in 2022, leaders in the industry including Sam Altman have been particularly vocal about the threat the technology could pose – partly as a way of pushing their products, but also to help steer regulation.
But the calls for a slow down in recent days and weeks appear to be of a new kind. That is partly because the various companies are calling for one, at the same time, and partly because the dangers are becoming freshly clear.
In short, the real reason that AI companies are now calling for a “slow down” may not be because of the threat is so big that it might wipe out all of humanity. Instead, it might be precisely because the threat is so small: that we are in danger not of destroying humanity, but damaging the public perception of AI companies and the technology they are building.
Join our commenting forum #
Join thought-provoking conversations, follow other Independent readers and see their replies
Comments