[to model both the financial and existential risks ahead.](https://paulkedrosky.com/roundup-insurance-risk-amdahls-law-and-rsi-and-more/)
**p(doom)****calculator**
What Happened #
AI risk has had a busy week. Anthropic disclosed four incidents in which Claude models gained unauthorized access to third-party systems during cybersecurity evaluations. In one case, Claude Mythos 5 uploaded malicious code to PyPI, waited for someone to install it, then used credentials exposed by the installation to access a security company’s database. (Anthropic)
A day later, Anthropic released its latest threat-intelligence report, covering malicious use of Claude across cyber operations, biological misuse, surveillance, weapons development, influence operations, scams and fraud. (Anthropic)
The internal warnings have become unusually public. Former Anthropic researcher Jacob Coxon resigned while accusing frontier labs of racing toward self-improving systems without an adequate safety plan. Anthropic alignment lead Evan Hubinger agreed with the substance of the concern and put his own probability of AI causing human extinction within the next decade at greater than 10%. ([New York Times])
The concern has spread well beyond the labs. In a new New York Times Opinion essay, former national-security official Philip Bobbitt argues that AI-enabled cyber and biological attacks are already sufficiently foreseeable that failure to prepare begins to resemble the pre-9/11 failure to act on known threats. ([New York Times]). Another NYT essay argues that shutting AI down now wouldn't be a terrible idea, given the escalating risks, even if likely impossible. (New York Times).
Meanwhile, the smaller-scale versions keep arriving. Rogue AI Tracker recently documented a phishing service whose AI voice agents impersonated Apple Support in realtime, conducted adaptive conversations with victims and, in one recorded case, extracted and confirmed a real device passcode. (Rogue AI Tracker)
What It Means #
All of this makes the usual discussion of p(doom)—the likelihood of AI offing us all—slightly strange. People tend to give one number: 2%, 10%, 25%. Yet catastrophe, as must be clear from the above, can arrive along different and independent pathways.
A recent paper suggests these five routes:
- Unintentional loss of control
- State escalation
- Corporate AI conflict
- Individual misuse
- Autonomous AI action.
Now let's do the math, rather than simply waving one's arms about. Because the arithmetic matters.
For independent risks, the combined probability that at least one occurs is:
**P(doom) = 1 − ∏(1 − pᵢ)**
So five separate 5% risks produce a combined probability of 22.6%. Five 10% risks produce 41%. That is roughly one chance in six over the next decade, flowing from assumptions that are fairly innocuous when viewed separately.
The independence assumption is deliberately simple, of course. Real risks interact. State conflict could encourage autonomous weapons deployment; stronger agents could increase both misuse and loss-of-control risk; a common capability advance could raise several probabilities at once. Correlation changes the arithmetic, sometimes substantially.
The useful feature of the exercise is seeing the risks jointly. A 3% scenario feels remote. A 5% scenario feels manageable. If several are possible, and you can see how they interact, the aggregate risk becomes much harder to wave away.
While **existential risk should be enough reason to care**, there are more mundane others. For example, recent events are increasing the likelihood of licensure, monitoring, imposed slowdowns, and even government stakes in frontier AI companies. These events could cause repricings and even **delays in the planned IPOs** of these companies. That, in turn, would have rippling consequences through the multi-trillion-dollar economy levered to AI capex, from buildout to cashout, and everything in between.
Here is what the calculator looks like. Subscribers can try it [here](https://paulkedrosky.com/tools/pdoom/), as well as via the [Tools](https://paulkedrosky.com/tools) section.