I’ve noticed an extremely common mistake among people who think about AI and ASI (also known as superintelligence) for a living. The mistake is to model the future of AI as a game played between AI companies, on a board where governments are part of the scenery.
People think of AI companies as being able to steer the course of AI development all the way through the end-game. For example, AI researchers often join certain AI companies because they're the "good guys", to help the good guys win the race. Or because they expect that being on the inside will give them influence when it matters.
Notice what this implies: at some point the winning company has to perform a pivotal act. It has to prevent every other actor on Earth from building AI irresponsibly. If this were not the case, who would prevent anyone else from deploying a misaligned AI a month later, and killing everyone anyway?
But performing a pivotal act necessarily involves gaining control of all governments on Earth. What makes these people think that governments will let them do that?
*Read the rest of the post here: *https://x.com/testdrivenzen/status/2080337476741128417