Nate Soares — the co-author, with Eliezer Yudkowsky, of 2025’s bestselling If Anyone Builds It, Everyone Dies — is one of the leading voices demanding a ban on the very powerful AI that industry insiders call superintelligence.
Soares and I spoke just days after Sen. Bernie Sanders and Rep. Greg Casar released a legislative plan to ban superintelligence and frontier AI development. The first thing I wanted to know was whether their plan addressed his concerns.
“It looks like a pretty good start to me,” he said during our Substack Live conversation last week. “I’d say it would be a start because ultimately this is an international problem. Ultimately, you know, we need to make sure that superintelligence is not created anywhere on the planet.”
But even ensuring that it’s not created in the U.S. would be an uphill battle.
For instance, once you get support from all of the legislators who are like Sanders — those who are already convinced of the risk and willing to take a strong stand — you have to deal with more conflicted lawmakers in order to get to a majority. Most legislators are risk-averse with their brand and need a strong vision if they’re going to turn around and pitch a bold idea to their constituents. A question they might ask is: Will slowing frontier AI development hurt the economy? The existing plans to slow AI progress have mixed responses to this question. Soares is the president of the Machine Intelligence Research Institute, which has a published plan that acknowledges “costs on industry and financial markets.” By contrast, the AI Futures Project envisions a slowdown being compatible with around 100% annual GDP growth in 2035 — roughly five times the historic upper bound of GDP growth. A convincing argument for even a small fraction of this much growth in a slowdown scenario could be a key step to get policymakers on board.
But Soares thinks it might not even be necessary, given the damage that misaligned AI could cause.
“Everyone who’s saying ‘what about the growth?’ has not actually grappled with the real situation,” Soares told me. “The way we see lawmakers shift on whether they support bills like this and whether we open the sort of international coordination channels is more about whether they realize the actual problem in time.”
Ultimately, Soares thinks lawmakers should accept the case for regulation based on the principle that superintelligence is not going to be understandable, not going to be predictable, and not going to have human interests at heart.
I’m sympathetic — and the latest news cycles are working in his favor. Still, I wonder if the quickest way to get policymakers on board is actually to appeal to the kinds of interests they have, not the principles they might eventually agree with. And if things are moving as fast as Soares claims, accepting political reality might be necessary.
Check out the video above to learn more about Soares’ perspective.