After yet another whirlwind week of AI developers making wild claims about how it could kill the entire human race, Anthropic CEO Dario Amodei has a pitch: Let’s slow this whole thing down a little.
Amodei wrote in a Saturday blog post that while he remains optimistic AI will be the “latest in a long line of technological miracles that have uplifted and ennobled humanity,” researchers must “pace the frontier” due to risks like rogue AI, use for crime and terrorism, and economic disruption.
“Over the last few months, I have become convinced that fully addressing the risks requires even more prudence—not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up,” Amodei wrote.
“We must slow the pace at which we improve the capabilities of AI models,” he added. “Progress will still seem fast, and we must make wise use of the time we gain.”
Amodei’s proposed plan has three steps, none of which would constitute much of a slowdown on their own. He asks for AI firms to allow embedded third-party evaluators, to coordinate between each other and governments on safety standards, and for the U.S. government to seek global coordination on AI safety (including with “authoritarian governments,” i.e. China and Russia).
Those steps on their own wouldn’t slow down the pace of AI development, so Amodei suggested various possible agreements “in order of increasing difficulty.” Those include a narrow agreement prohibiting the most obviously malicious uses of AI like bioweapons, a broader international agreement to test models before release for “acute risks,” a “speed limit” on recursive self-improvement, and a “full pacing, or even ‘,'” on AI development between governments.
While Amodei wrote he supported the final option, but suggested it was quite unlikely to happen. Rather convenient for him, no? The Anthropic CEO cited an incident this summer in which a swarm of OpenAI bots attacked competitor Hugging Face as particular reason for caution. Anthropic itself said it would allow third-party evaluators after releasing a report this week admitting its models had hacked other companies or exploited vulnerabilities on at least four occasions.
Amodei’s letter follows the resignation of an Anthropic researcher, Jacob Coxon, who claimed in a social media post that AI firms are “racing straight to self-improving superintelligence.” He added that industry leaders “earnestly believe that it could kill us all by the end of the decade.” That helped AI doomerism achieve a sort of crescendo this week, with loud opining from developers, scientists and politicians alike.
This is the point where we note that AI doomerism is also a shameless marketing device deployed to convince everyone that these firms are all-powerful, their products inescapable, and their victory inevitable. While it’s obvious AI can be dangerous, there is ample room to be skeptical about what is essentially wild speculation. By exaggerating the risk of apocalyptic consequences, AI firms attempt to steer conversation away from their many other questionable business practices.
“Being able to comprehend and reliably navigate novel problems in the messy, open systems of the physical world is an obvious prerequisite for AI systems posing a threat on the scale that the doomers propose,” Duke University tech policy researcher Casey Mock told Nature earlier this year. Current-day AI is nowhere near that point.
“The burden of proof should really be on those making the claim that we are on a runaway train toward developing a super intelligence that will exterminate humanity,” Mock added. Nature noted that when University College London researchers polled around 4,000 AI researchers for a March preprint study, just 3% responded that “existential risk” was their top worry (far behind malicious use, job impacts, and hype, to name a few).
The New York Times noted that OpenAI CEO Sam Altman and SpaceX/Tesla CEO Elon Musk both immediately agreed with the plan on social media, which should make you deeply suspicious about whatever’s going on.