An Epistemic Audit for Existential Risks from AI

This post introduces a tool: an Epistemic Audit for Existential Risks from AI. It is a structured way to map, organize and track your beliefs across the key domains and questions that determine how likely existential risks from AI are. It follows the causal chain from "capable systems get built" to "existential risk." It is designed for repeated self-assessment, creating a longitudinal record while enabling standardized comparisons and aggregation across respondents for (potential) future elicitation studies. It has three main functions:

Audit yourself. For every question, note your epistemic uncertainty: not what you believe, but how much you trust what you believe and where more information and evidence can move or counter your beliefs. You will find out where your overall view on existential risk rests on things you have actually thought through, where it rests on takes you have absorbed from others, or places you have spent little time on in general.Route your effort. The questions where your uncertainty is high and which would move your overall view are exactly the ones you should study next. Where the entire field's uncertainty is high, these questions can then turn into a direction for a research project, literature review, or a blog post. Framing your beliefs as measurable and falsifiable helps clarify what evidence is most compelling, and what would change your mind, making learning more rigorous, transparent and cumulative.**Structure discussion. **By giving everyone the same numbered framework and the same three-point uncertainty scale, the tool creates a shared language for discussing cruxes rather than conclusions (e.g., "I seriously doubt 1.4, what is the most compelling evidence for or against it?"). This focuses discussion on the key assumptions, highlights where uncertainty and disagreement lie, promotes evidence-sharing and makes views easier to compare and aggregate across individuals.

I am not the first to decompose existential risks from AI. Joe Carlsmith's report decomposes the power-seeking threat model into six conditional premises and assigns credences to each. Zvi's Crux List is a (large) collection of questions on which people disagree. AGI Ruin: A List of Lethalities is the (in)famous list of reasons the problem is hard. I have not yet seen a protocol that you can run on yourself, with a defined scale, repeat over time to see whether your uncertainty is actually decreasing, and that makes it easier for you to guide your effort towards the questions that matter the most.

That is what I attempt in this post. I created this Epistemic Audit tool using standardized categories (low/medium/high uncertainty and falling/stable/rising confidence) to produce simple, comparable assessments across respondents. Hopefully this reduces reporting ambiguity while enabling aggregation and longitudinal analysis.

One note on scope: a question makes this list only if resolving it would noticeably shift an estimate of existential risk from AI. There are many interesting questions that don't do this, such as "what is intelligence?"; I have attempted to exclude these.

The domains below follow the causal chain that I think of for AI to become an existential risk: capable systems get built, they are agentic, we fail to align them, they get deployed anyway, and the failure leads to existential risks. I also discuss routes that bypass misalignment entirely, the mitigations, and finally the question of how to reason about any of this at all.

How to run the audit #

Open the sheets and make a copy ** https://docs.google.com/spreadsheets/d/1e5-52-bjyOtsKnMmQDwkyyI8bFyOZF5AQxyGO1N3Ook/edit?usp=sharing Begin with your Quick Summary (~5 minutes). **Rate your uncertainty as low, medium, or high. This will produce a radar chart, drawing your uncertainty shape (see mine in the comments, I encourage people to share it there as well). So that your ratings mean the same thing at your next audit (and comparable to everyone else's), use these anchors:Low: your view is resilient. You would be surprised if a quarter of a year of new evidence meaningfully moved it.Medium: a single strong paper, result, or event could noticeably move you.High: you don't trust your current view. One good argument could flip it.

Note the direction: has your uncertainty been rising, stable, or falling over the past year? (On your first run this is retrospective memory, which is rather unreliable, so the direction column only becomes trustworthy from your second measurement onward. That is fine; the first run is your baseline.)**Deep dive. Go through each domain, and mark your cruxes: identify the questions where a changed belief would substantially move your overall view on existential risks from AI. These are the ones worth your attention; the rest you can rate quickly and move on. Schedule your next audit. **The value of this audit is meant to increase over time. So, commit to a date to come back to your google sheets template. The first measurement will tell you where you stand. Two or more will formalise how your uncertainty is changing, and inform where you can make progress. Personally, I’ll (try to) re-run this audit every three months and compare.Make your uncertainty productive! Your high-uncertainty cruxes point to your greatest opportunities for learning. The questions in this audit are meant to steer you toward unresolved links in the causal chain leading to existential risk. I believe that further specifications in these questions can uncover promising research directions.

The questions are numbered so it's easier to refer to them personally or in the comments. My personal scoreboard with reflections is in the comments (and in the Google Sheets as an example).

1. Capabilities and timelines #

The elephant in the room in any conversation about AI is what these systems can do and how fast that changes. What matters here is capabilities, so what systems can actually do, economically and strategically, and the trajectory they are on.

• Are there capabilities that cannot emerge with scale alone (more compute/data/params), regardless of how far scaling continues?
• Which capabilities are mostly discontinuous (sharp jumps) rather than smooth and predictable?
• How far above the human range do ceilings on AI capabilities in various domains sit, if there even are ceilings? Is human-level capability a natural plateau, or just an arbitrary point that the curve passes through?
• Does the intelligence of current AI systems differ from human intelligence in ways that matter for forecasting? One concrete example: do models genuinely extrapolate beyond their training distribution, or only interpolate within it?
• Current AI systems cannot yet learn as continually and efficiently as humans. How hard is adding this—persistent memory, no catastrophic forgetting, higher sample efficiency—to current systems?
• If the current paradigm plateaus: what is missing, how long will it take, and does it require a (radically) different approach rather than an LLM successor?
• If it does not plateau: how long until AI systems are as economically and strategically capable as an average human across all domains?
• And how long until they are far beyond any human across all (relevant) domains?
• Will capability keep following well-known scaling laws?
• How will compute grow over the coming years (chips, energy, capital willing to be spent)?
• How will data grow, both in quantity and quality?
• How will algorithmic efficiency improve over the coming years?
• Do benchmark gains translate into real-world economic and strategic capability, or is there a persistent gap?

2. Agency and goals #

If we grant capable systems, we need those systems to be coherent pursuers of goals, i.e., agents.

• Do trained systems become coherent goal-pursuers at all, or do they remain tool-like and very context-dependent even at high capability?
• Are intelligence and final goals orthogonal axes on which agents can vary, both in principle and in practice for systems produced by current deep learning methods?
• Under what conditions do agents optimizing for final goals develop instrumental subgoals dangerous to humanity (examples: self-preservation, resource acquisition, power-seeking)?
• Which goals do trained agents actually end up with? How close are these to the training objective and how alien are they when they diverge?
• When do systems become situationally aware, i.e., knowing they are models being trained and evaluated, and what behavior follows from that?
• Do multi-agent settings lead to safer or less safe worlds than single-agent settings? How quickly, and in which environments, is defection/cooperation the default? (For whether many AIs make takeover harder or easier, see 5.5.) - When can models coordinate effectively between many instances of themselves and what do capabilities look like at that point?

3. The difficulty of alignment #

Conditional on capable, agentic systems: do we fail to align them despite trying?

• Is alignment hard in the strong sense of having deep theoretical obstacles or is it an engineering problem that yields to iteration, the way most engineering problems do?
• Do current alignment techniques such as RLHF and its descendants actually instill values or rather shape surface behavior while leaving the underlying system untouched?
• Do learned values generalize out of distribution as capability grows?
• How likely is deceptive alignment by default, i.e., training processes producing models that act aligned while observed and pursue other goals when not, and why does it happen?
• Will interpretability help us understand truly important internals of frontier models before the point where it's too late?
• Can weaker overseers reliably supervise stronger systems in the limit to superintelligence?
• Can we build corrigible systems, i.e., ones that allow correction and shutdown without an incentive to resist?
• Suppose we can align a system to some target: to what, and to whose values, should it be aligned? Is intent alignment, i.e., the system doing what its developer means, enough or does it need to be value specification? How different are these two?
• Can AI automate alignment research fast enough, i.e., sufficiently solve the alignment problem before the point where it's too late?
• How many warning shots do we get? (How society responds to them is 6.12.)
• Do alignment failures show up in small models or emerge only seriously at capability levels where it's too late?

4. Takeoff and self-improvement #

How fast the transition happens between different capabilities matters a lot. A slow takeoff gives society time to react and iterate whereas a fast one does not (though the former may bring different risks). This is my most cruxy domain.

• Can AI meaningfully accelerate AI research itself, and by how much?
• What will be the main bottleneck to recursive self-improvement? Compute, physical-world experiments, something else, everything roughly uniformly?
• How much do physical constraints such as datacenter construction, energy, chip manufacturing, slow the conversion of capability gains?
• Should we expect capability growth to be exponential, sigmoid, superexponential, something else?
• Does the transition to decisively superhuman systems play out over minutes, hours, days, years, or decades?
• Is takeoff concentrated in one lab or distributed across many comparable actors?
• How large is the leading actor's lead time, and is that gap widening or shrinking? How will this change with recursive self-improvement?

5. From misalignment to catastrophe #

A misaligned agentic superintelligence doesn't have to be a catastrophe. There need to be concrete mechanisms by which this leads to existential risks for humanity.

• What are the concrete pathways from misaligned capable AI to existential risks, e.g., sufficiently strong engineered pandemics, nanotechnology, cyberattacks on infrastructure?
• How much does the robotics bottleneck constrain a system without (current) physical actuators? Is it necessary at all?
• How dependent will critical infrastructure and high-stakes decision-making be on AI systems by the time it really matters, and how well-defended will that infrastructure be (e.g., by national AI safety and security institutes working together)?
• Can a misaligned system acquire resources and power gradually without being detected and stopped?
• Does a world of many AIs make takeover harder (through checking each other) or easier (because there is far more noise)?

6. Deployment, incentives, and coordination #

How current systems are deployed, what kind of incentives are present in the AI race, and how countries and frontier AI companies will (not) coordinate, matter immensely.

• How difficult is coordination between AI companies? What is needed for this to happen?
• How difficult is coordination between the relevant countries in the AI race?
• How likely is it that AI companies will become nationalized, e.g., a Manhattan project for AI?
• Conditional on the nationalization, how does this affect the race and overall dynamics?
• Conditional on there being agreement, how effective can verification mechanisms be?
• How much do competitive pressures mess up safety commitments or coordination efforts? How have frontier companies acted previously in this regard and what does that mean for the future?
• How fast will AI adoption be and in what form?
• What economic effects will AI cause and how do those feedback into development, e.g., investment cycles, automation-driven demand, political pressure?
• What will the dominant public sentiment toward AI be and what causal role does this sentiment play?
• How large is the alignment tax, i.e., how much capability or speed do safety measures cost? What is needed for frontier companies and countries to take this tax?
• How much legislation will happen, in what form, and how much binds to frontier development?
• If we see warning shots, i.e., small catastrophes but not existential, how will society respond to them?
• How does the open-sourcing of increasingly capable model weights change every answer above?

7. Misuse #

I personally think most about the above threat model, so acutely losing control of ASI, but existential risk does not require this. We can also have sufficiently capable (aligned) AI systems in the wrong human hands.

• Can AI-enabled biological weapons be existential rather than "merely" catastrophic?
• Can AI-enabled cyberattacks on critical infrastructure cascade to civilizational scale and how does this affect misuse or misalignment scenarios?
• What role will AI play in autonomous weapons and in nuclear command and control?
• How will we decide who gets hold of these sufficiently capable (aligned) AI systems?
• Does AI advantage attackers or defenders more, especially in bio and cyber?
• Which arrives first, existential risks through misuse or misalignment? How does preparing for one help or hurt preparing for the other?

8. Structural risks #

There are some other scenarios which I would count as existential risks from AI that don't quite map neatly onto the above two routes. I'd describe them as "we lose but it wasn't because we built ASI that was misaligned or someone majorly misused it."

• How plausible is gradual disempowerment, i.e., can humanity lose effective control through incremental delegation of decisions to AI with no discrete takeover moment ever occurring (and this actually being very bad, instead of a future where we choose to do so and it being good)?
• Does AI enable stable, permanent concentration of power, i.e., a global authoritarianism that can never fall?
• Does AI-driven persuasion and epistemic pollution degrade our collective decision/sense-making enough to break our ability to respond to every other risk on this list?
• Does the AI race itself raise the likelihood of a nuclear war?
• Could early AI actors lock in their values permanently?
• Do AI systems themselves acquire moral status, and if so, does a future that mistreats them at scale count as a catastrophe, e.g., similar to the mass animal farming we do now?
• If aligned ASI manages to "solve everything," are there ways in which the complete abundance and lack of need to do anything lead to a complete loss of meaning, in turn posing an existential risk to humanity?
• Are there plausible outcomes involving s-risks, i.e., astronomical suffering, that are worse than extinction or other existential risks and how do these play out?

9. Mitigations #

There are a number of people already attempting to mitigate existential risks from AI, myself included.

• What have existing mitigation efforts changed so far?
• Is technical safety research progressing faster or slower than capabilities? How could this be changed?
• What is the conversion rate from governance research to real, binding policy? How could this be changed?
• Is compute a viable control point for AI governance? For how long does this window stay open?
• How much worse or better would the world be if frontier model weights were stolen or leaked? How good is security today?
• How far can AI control go, i.e., extracting useful and safe work from possibly-misaligned systems through monitoring, containment, and restricted permissions as a complement to aligning them?
• What have been (un)successful attempts at tying evaluations on dangerous capabilities to if-then commitments? What is needed to trigger real action instead of companies dissolving/changing them if they get reached?
• What is needed for pausing or slowing frontier development? Is it net-positive once you account for compute overhang and which players keep racing?
• Which audiences matter most for education and field-building, e.g., general public, policymakers, researchers?
• How much does strengthening society at large help, e.g., hardening biosecurity and cyber-defense independent of any AI control, or strengthening democracies?

10. How to reason about any of this #

The main question here is how we form beliefs about an event that has never happened before. When thinking about existential risks, I think it's most valuable when thoroughly grounded in empirics, and since AI-driven existential risks have never played out, our empirics need to come from reference classes that can guide us through this.

• Which reference classes are informative in our situation? Humans driving other species extinct, past technological transitions, invasive species, corporations and other principal-agent problems, the long track record of failed doom predictions from humanity? What does each really mean?

• How often have humans been an existential risk to other species, and was it mostly negligence or intent?

• On what timescale did human-caused extinctions play out? Centuries, decades, years?

• How much was raw intelligence the determining factor, versus culture and coordination?

• How often are species-level extinctions driven by another species at all, compared to environmental causes?

• What is the track record of long-range technology forecasting? Specifically interesting is doing a meta-analysis on the predictions from the "AI safety community."

• How should the enormous spread in "expert estimates", which can range from well under 1% to above 99%, inform us?

• When inside-view models and outside-view base rates conflict, how much weight should each get?

What to do next #

If you ran the audit: your high-uncertainty cruxes are your curriculum. Pick the top three and go deep: read the strongest thing written on each side, or write the post that maps the question if none exists. If you dig into a question and find the field's uncertainty is as high as yours, you have found a research project, and those are exactly the projects the field needs. If you are in the Netherlands and want help turning one of these questions into an actual project, Safe AI Netherlands exists for precisely this: reach out. I will (hopefully) fully run this audit in the coming month, and then re-run it three months later and publish the difference. I’ll also start posting my object-level beliefs on various questions, which are missing from this first measurement. I encourage you to do the same.

Acknowledgements #

Many thanks to Ana Paula Castillo Rodriguez for fully designing and making the accompanying Google Sheets, besides providing extensive feedback. Thanks to Lucas Hogendoorn, Ilija Lichkovski, and Atakan Tekparmak for providing feedback and comments.

The first draft of this version contained roughly 50 questions in 6 domains. With Fable 5, I then iterated around 3 versions of this post to come to 10 domains, shuffled some questions to the domain they actually belonged to, came up with other questions in various domains, and improved the new ones I came up with. Roughly 60% of these questions are ones I came up with during the first draft (that haven't been changed through editing); roughly 20% are questions from Fable 5 that I edited substantially for them to make more sense; another 10% are questions I came up with based on the new questions that Fable 5 came up; the last 10% are questions that Fable 5 came up with that I copied verbatim.

Discuss