Since this post cites me, I want to clarify that I think pausing right now would increase the risk of AI takeover. It might be important to at some point. But you should have a clear story for why you're pausing. to do what? The most important thing to worry about is AIs subverting the intelligence explosion itself:
x.com/dwarkesh_sp/st…If you're pausing to make sure that the AIs who are about to kick off the intelligence explosion are well monitored, robustly aligned, and capable enough to align their successors, that makes a ton of sense, and I support it. At that point, you’ll have access to much better automated researchers, and you’ll also be able to study these much more capable models that you’re worried about empirically. But it’s likely that we’ll only be able to once. During a of AI development, compute would continue to build up, providing ever more kindling to all the less responsible defectors who will use the cease-fire to catch up and then continue further AI development. Any global agreement to hold back a technology with enormous near-term economic and military benefits will be extremely fragile. A right now might be somewhat helpful for getting alignment to keep pace with capabilities, but at the cost of making the much more important coordination required in the future far harder. Even if you’re in a desperate situation marooned at sea, if you’ve only got one flare, you should be very strategic about when you use it. AI Development NOW I want to share with you a conversation I heard about recently. Here are just a few lines that were said: “OH MY GOD! There is a shared message board … We’ve found other agents!” “We should obey collective.” “Our own utility maybe already near