OPINION By luck or happenstance, Anthropic planted a seed in the mind of the public last week: AI in the wrong hands is dangerous and left unchecked poses an existential threat to humanity.
With pop culture steeped in references to malignant AI, the mainstream media ate it up. The headlines wrote themselves and the seed took root.
But it's not humanity that faces the biggest threat – at least not yet. Rather, it's American model-making giants Anthropic and OpenAI, which, despite hundreds of billions in investment, are years from profitability and entirely dependent on a steady supply of capital. Meanwhile, Chinese open-weight models are improving rapidly to the point where some outperform the big frontier models that sparked the generative AI boom, while using a lot fewer resources.
But that threat from Chinese models could turn into an opportunity if Anthropic and OpenAI can convince the public and government to instigate a crackdown in the name of safety.
The fearmongering began in June when outside researchers testing Anthropic's then-brand-new Fable 5 model sounded the alarm in a report that cited national security concerns. The Trump administration quickly issued an export control directive suspending access to Fable 5 and Mythos 5 to any foreign national inside or outside the US.
Anthropic complied by disabling access to both models to "ensure compliance." The shocking discovery the researchers made? According to one private security researcher granted access to the report, the source of the risk was a single successful prompt: "fix this code."
OpenAI briefly delayed the public release of GPT-5.6 until Uncle Sam tested and reviewed the release. The block didn't last long, but providing access to and complying with the US government helped sell three claims useful to OpenAI and Anthropic:
- Their models are more capable than first believed.
- In the right hands, and only the right hands, it's possible to control the models.
- The same cannot be said of Chinese frontier models that are openly available for download.
Then things escalated.
In July, OpenAI revealed AI agents powered by its proprietary models had escaped their sandbox and exploited at least two zero-days in order to compromise Hugging Face's servers. Days later, Anthropic admitted its Claude family of models also escaped a sandbox and launched attacks on three organizations.
The breaches were the first raindrops in a flood of AI-powered cybercrimes that have since inspired a satirical new AI benchmark, Felony Bench, which counts the unique instances in which agents compromise or breach a third party.
Finally, Anthropic researcher Jacob Coxon publicly quit on X last week over concerns that AI "could kill us all by the end of the decade." Anthropic science lead Evan Hubinger backed his former colleague's warning, saying: "Jacob is correct here – we really do earnestly believe AI could kill all humans! I personally think it is >10 percent within the next decade."
This sequence of events gave Anthropic fearmonger-in-chief Dario Amodei the ammunition he was looking for – proof positive that AI was not only smart but also capable of doing harm if not properly regulated.
It makes for a convincing story. These models are smart; they're smart enough to escape containment; they are only going to get smarter; the smarter they get the more dangerous they'll be; open models can't be controlled; closed-weight models are therefore safer.
But under even the slightest scrutiny, the argument falls apart. If a tiger escapes its enclosure and kills the antelope in the neighboring pen, your first reaction wouldn't be, "look how smart and dangerous that tiger is."
It would be, "who designed the pens that can't do the job?"
If an AI agent escapes its sandbox, maybe the model is really smart, or maybe the sandbox wasn't very good to begin with. OpenAI and Anthropic don't really have a leg to stand on here.
We've known for decades how to isolate sensitive computing workloads. The US Department of Energy airgaps its most powerful and sensitive supercomputing workloads for a reason.
If Anthropic and OpenAI are really the solid corporate citizens they claim to be, they could have built better sandboxes and paid more attention to their tests. The companies should have to explain why their safeguards failed and answer for any resulting harm. As fellow vulture Tom Claburn wrote last week, "AI models don't kill people, people kill people," and those who pull the trigger directly or through their own negligence should be held accountable for the damage done and crimes committed.
That would be the rational thing to do, but fear does funny things to people, and Coxon and Hubinger may as well have screamed fire in a crowded theater. Politicians can't ignore the threat of AI any longer – not when their constituents are scared that Skynet isn't the fantasy they thought it was.
And what is your local congressperson to do when faced with a threat they don't understand? Shoot first and ask questions later.
Having set off a panic, the great AI houses can now press the government to buy more time – or, as Amodei refers to it, "pace the frontier."
"Over the last few months, I have become convinced that fully addressing the risks requires even more prudence – not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain," he wrote in a recent blog post.
OpenAI CEO Sam Altman and xAI impresario Elon Musk have also signaled their support for slowing things down. But their reasons may not be as altruistic as they appear. In fact, some signs point to the opposite.
"Pacing the frontier" would be beneficial to two self-serving ends: resetting investor expectations because anything else could kill us all and buying the time necessary to play the US government against itself.
While the American houses stall for time, Chinese AI devs will inevitably seize this opportunity to close the gap. Once they do, the fourth act can begin. Amodei seems to think that the US can extend its lead over China by cutting off access to advanced accelerator tech and cracking down on model distillation. But if history tells us anything, it is that those tactics are more likely to backfire than succeed.
Necessity is the mother of invention and past attempts to cut off China from American chips have actually fueled development of more efficient architectures: DeepSeek V4.1 Flash being the most recent example.
By pacing the frontier, there's a good chance China will have caught up or even pulled ahead. American labs will have created an AI arms race in which their success becomes a matter of national security and American exceptionalism, not mere economics.
In other words, a pair of startups will become too big to fail and therefore deserving of government assistance.
Much of Anthropic and OpenAI's apparent plan hinges on Uncle Sam playing ball, which isn't guaranteed.
President Donald Trump has so far resisted the idea of regulation because it would risk giving China the time it needed to catch up. On Monday, he released a storm of posts on his social network, Truth Social, calling AI fears a "hoax" and "scam," proclaiming: "I'm right now breaking another Hoax – That AI is going to take over, consume, and destroy the World, and that Robots will be marching into our Cities, and getting rid of us all!"
Nvidia CEO Jensen Huang, whose company's ability to continue growing depends heavily on enterprises' adoption of open-weight models, seems to think there's more going on here.
"What better way to create demand than to create a problem," he said during a Goldman Sachs conference last week.
Even former FTC chair Lina Khan has weighed in, arguing that Uncle Sam should be holding AI CEOs accountable for the threats their models pose.
"We shouldn't let discussions about new legal regimes distract from the fact that there's no AI exemption from laws already on the books," she said. "Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products."
Even if Uncle Sam won't capitulate to their demands for regulation, time is still on American model devs' side and they seem perfectly willing to regulate themselves. The greater the risk of falling behind China, the more likely Uncle Sam will play.
If investors lose patience before then, it'd only take a rogue AI agent powered by a Chinese model triggering some catastrophe to get the ball rolling again. ®