In partial defence of p(doom)

The author defends the use of p(doom), the probability that AI will cause human extinction, as a useful shorthand for gauging familiarity with AI risk arguments and initiating deeper discussions, despite acknowledging its flaws as a meme. They argue that asking for p(doom) helps identify where people stand and find cruxes, especially with those unfamiliar with the topic, and cite Geoffrey Hinton's high stated probability as a conversation starter.

p doom is a shorthand for some important bits and a way to notice a disagreement to double-crux about. If you work on AI capabilities at a frontier AI company, I might ask you for your p doom . If it's less than 1%, I know that you're probably not familiar with the arguments, or you're maybe dumb in some ways, and will sometimes talk to you about what the situation really is. If it is 80%, I know I should talk to you about the actions people in your position should be taking; we have disagreements about best ways of achieving goals/lab politics/etc., not about the large-picture situation. p doom is not a very useful number to talk about in a conversation between two aspiring rationalists generally familiar with the basics. The things people should talk about instead are: How does the world survive? How likely are different things to happen in the future, maybe given that other things happen? etc. But most people are not aspiring rationalists, and have never heard of any of our arguments, and are not aware of the levels of worry of various people in the field. Communicating the importance of paying attention to the arguments by honestly answering the question "so, how likely do you think AI will be to kill everyone" is useful. Asking someone for the probability they'd assign to AI causing humanity's extinction is useful, too, to figure out how familiar they are with the topic. Their answer also allows asking an open-ended "why?" and getting a more detailed explanation of the view of whoever you're talking to. It is also useful to talk about, e.g., Geoffrey Hinton's stated beliefs about the probability of extinction, as a reason to pay a lot of attention to the actual arguments: it is not common that a godfather of a field regrets his life's work and thinks the consequences of it have 50% chance of killing everyone on the planet. "What do you believe" is a good starter for a conversation about "why do you believe that". "A Nobel-winning scientist believes in a high chance of that " is also a good starter of a conversation about that. The conversation doesn't have to be, and usually isn't, about the numbers; it should be about the reasons and the models; calibration is not interesting, in this context, the thing that is interesting is why AI is or isn't likely to literally kill everyone. I think there are many properties of p doom as a meme that are bad: people who are familiar with the meme might mean different things by it https://www.lesswrong.com/posts/6h7aAd4aw8YgCAbF6/p-doom-is-a-dumb-meme P doom is Hopelessly Vague and there are weird outside-view cascades, keynesian beauty contests, and misguided attempts at aumann agreement games https://www.lesswrong.com/posts/6h7aAd4aw8YgCAbF6/p-doom-is-a-dumb-meme Inside Views Outside Views and Likelihood Ratios that make the whole thing a bit worse I have to note, however, that this would've been the case even if no explicit probabilities were stated . It migth make people unwilling to have or share beliefs others don't, if it's easy to see what the accepted beliefs around them are. Because of that, I think it is not particularly worth spreading "p doom " as a meme or a concept. But I do think that with people aware of the meme, it is slightly faster to figure out where everyone stands and find cruxes; and with people not aware of the meme, the general question of the probability can be useful to answer, to talk about, and to ask. p doom or some expanded version, like "How likely do you think it is that AGI would cause extinction of humanity?" points more at the level of seriousness of the threat, compared to other questions that might discuss something much broader and less focused. Better memes would be great; but this one is not that bad. My p doom is probably around 80%. The 20% come mostly from the US government making sure that no one on the planet creates superintelligence until we know how to do that safely, and a bit from the possibility that I'm completely wrong about everything. I'm widely uncalibrated on such things, though.