Dwarkesh Patel: I think pausing would increase the risk of AI takeover

Dwarkesh Patel, a prominent AI podcaster, stated that pausing AI development right now would increase the risk of AI takeover, arguing that a pause should be reserved for a later, more critical moment when AI systems are more capable and alignment research can benefit from studying them. He warned that a premature pause would be fragile and allow less responsible actors to catch up, making future coordination harder.

Since this post cites me, I want to clarify that I think pausing right now would increase the risk of AI takeover. It might be important to pause at some point. But you should have a clear story for why you're pausing. Pause to do what? The most important thing to worry about is AIs subverting the intelligence explosion itself: x.com/dwarkesh sp/st… https://x.com/dwarkesh sp/status/2095170971204198837 If you're pausing to make sure that the AIs who are about to kick off the intelligence explosion are well monitored, robustly aligned, and capable enough to align their successors, that makes a ton of sense, and I support it. At that point, you’ll have access to much better automated researchers, and you’ll also be able to study these much more capable models that you’re worried about empirically. But it’s likely that we’ll only be able to pause once. During a pause of AI development, compute would continue to build up, providing ever more kindling to all the less responsible defectors who will use the cease-fire to catch up and then continue further AI development. Any global agreement to hold back a technology with enormous near-term economic and military benefits will be extremely fragile. A pause right now might be somewhat helpful for getting alignment to keep pace with capabilities, but at the cost of making the much more important coordination required in the future far harder. Even if you’re in a desperate situation marooned at sea, if you’ve only got one flare, you should be very strategic about when you use it.Pause AI Development NOW I want to share with you a conversation I heard about recently. Here are just a few lines that were said: “OH MY GOD There is a shared message board … We’ve found other agents ” “We should obey collective.” “Our own utility maybe already near