In the game "The choice before us" by Nick Shapiro, [1] you are put in the shoes of an AI company leader. You grow your business. You unlock "wonders", such as curing cancer. All the while, you're attempting to avoid your product getting smart enough to escape and take over. You win by achieving 5 wonders without unleashing uncontrolled AI.
I love this game, but it has the major flaw that when you win, you are normally very close to superintelligence. What happens afterwards? You turn the GPUs off? Go home? Get some sleep? The game seems to think so.
This failure to ask "What happens next?" seems to be a broader phenomenon within the AI community. It was in fact the sole question I needed to ask a capabilities researcher for them to take the threat of superintelligence seriously. It's my main weapon against people claiming there are many possible worlds "where only 90% of people die" (if a rogue AI has gone off the rails and killed 90% of your population, you probably no longer have control of the planet, and I have little faith in the survival of everybody else). More broadly, I just wish people would ask this question more often.
"But Sean!" you say. "I cannot keep asking what comes next forever. I'll end up wondering what happens after the pope becomes US president in the year 2124."
And you would be correct! You cannot, in fact, keep on doing this forever. The tree of possibilities is infinite, and spending your life exploring them is reserved for the brave of heart, the noble of mind, and those who have nothing more productive to be doing. We would be better off finding a place to stop.
To work out how, let's look at a simpler version. In particular, the game of chess. More specifically, this position:
For the uninitiated, this is a complex position. The white king is threatened. The black queen is under attack. So is the white rook, a white bishop and one of the white knights. More broadly, the arrows in the image show the main pieces of tension that one would want to resolve before attempting any sort of evaluation; it's no use counting up the pieces if you lose your queen next turn! In chess engine parlance, we would like to perform a "quiescence search". This search goes only through the moves which resolve tension, arriving at a position like the following:[2]
With all the mess cleared, we can now see that black is up a bishop and a knight and will probably win the game.
The point of this exercise? When evaluating the future, only evaluate futures where things are relatively stationary. The world will be changing, naturally, but there is a difference between worlds where Anthropic takes over the US government and worlds where capabilities plateau. (Oh, I'm sorry, did you think that a government takeover was a straightforward lock-in scenario? When there is no precedent for it in US history? When this is flipping the power structure of the entire globe?). Don't think your way halfway into the singularity and declare that we're in a good position. That's how you lose your queen. Get to a stable point. They're safer.
Which is great and you should try out by the way
After the moves 1. Kh1 Qxb4 2. Rb1 gxh6 3. d3 Bxa7