TL;DR: If you have an economics background, an interest in doing research and want to help ensure a future with transformative AI goes well, this post aims to provide a guide for you. This post covers the (sub-)cause area selection, an introduction to economics research in some of these areas, advice to identify bottlenecks and a path to impact, and concrete starting points for your own research.
Acknowledgements: We are incredibly thankful for many helpful comments by Esben Kran, Jaime Raldúa Veuthey, Joseph Levine, Jian Xin Lim, Christine Tan, Milena Canzler, Phil Trammell and Donghyun Suh. All errors are our own.
We used various large language models for the summary of gradual disempowerment, some open questions and a few other places. Probably less than 10% of the content is AI. Some open questions are direct quotes without quotation marks to improve readability.
Economists are increasingly interested in the potential effects of AI on the economy and society (Korinek & Stiglitz, 2018). However, the policy relevance of research in this field varies. And the stakes are high: AI might bring great benefits, but also risks, potentially as bad as human extinction (Bengio et al., 2026). As AI capabilities are progressing quickly, a core question for any researcher in the field should be “Which risk is my research addressing and how?”
In this essay, we discuss some classic risks of AI and how you, as an economist, could help solve them.
This is a living document. We are looking for collaborators to improve and update this document. Please reach out if you want to get involved (matt_eaw@proton.me).
Several sections of this essay touch upon areas in economics where we are not experts. We welcome feedback from anyone. In particular, we would appreciate pointers to areas we may have overlooked. We would particularly be thankful for your thoughts on our suggested “open questions” section, which aims to inspire new research in the respective field (especially since some of them are AI-generated or copied from Brynjolfsson et al. (2025), Korinek & Suh (2024), Korinek (2024), UK AI Security Institute (2025) and Fudenberg & Liang (2025). We would be surprised if we did not change any view expressed here in the coming months and years.
The object of study of all social sciences, including economics, is extremely challenging. Economic forecasts have a mixed record (Faust, 2008), and policy advice in similar situations differs strongly across space and time. The first virtue of an economist aiming for impact should be epistemic humility. There is very little we can say about the social world, and it is easy to draw the wrong conclusions. Existing and emerging work needs constant revision and correction. We recommend entering the field, proceeding carefully, and providing a critical view of existing work.
First, you should reflect on whether transformative AI is actually the area where you expect to have the largest impact. As some authors advise that one’s impact can differ by many orders of magnitude depending on which cause area one chooses (MacAskill, 2015), it seems central to develop one's own framework for evaluating cause areas.
As a first step, it is important to clarify your values and moral views. Having a look at an ethics textbook or taking an intro class to moral philosophy can be a great start to get an overview. If you are inclined to some version of consequentialism, an intro to EA program could be valuable.
Clarifying one’s moral views is important, as different moral views might lead to a focus on different problems within the broad space of the economics of transformative AI. There are two classic disagreements in philosophy and welfare economics which are worth highlighting. First, the question of which pure discount rate to choose (Fleurbaey & Zuber, 2021). Second, and related, is the degree of inequality aversion a society should endorse (Roemer, 1996).
A zero pure discount rate captures the idea that future welfare matters as much as present welfare. This idea could lead you to endorse strong long-termism, i.e. “the thesis that impact on the far future is the most important feature of our actions today” (Greaves & MacAskill, 2025). In that case, working on x-risks [1] or s-risks
Now consider inequality aversion, the second dimension mentioned. If you endorse a utilitarian welfare function, i.e. the unweighted sum of individual utilities, the number of beings your action might affect positively might dominate the magnitude.
On the other hand, if you endorse a Rawlsian welfare function, you probably want to focus on harm to the worst off that might evolve in a world with advanced AI.
It is important not to stop once you find your favourite theory or view. Meta-ethicists have urged that following one's ‘favourite theory’ has important shortcomings and argue for using some kind of aggregation, a method to deal with moral uncertainty, for example, by using expected utility theory with uncertainty over moral views (Dietrich & Jabarian, 2022; MacAskill et al., 2020). For example, if you are very uncertain how to weigh the well-being of future generations against current generations, you could look for areas that might affect both. For example, unemployment leads to suffering today but might also destabilise the political system and lead to a stable totalitarian regime.[3]
After deciding how you want to weigh different moral theories, you should be ready to determine which is the most important problem to work on from that perspective.
There seem to be two schools of thought for prioritising risks of AI. Some focus on the risk that bad choices related to AI development and deployment might reduce the value of the long-run future (Ord, 2020), guided by a moral view of longtermism (MacAskill, 2022). Others seem to be focused on risks affecting people living today (Bender et al., 2021), which could be grounded in a person-affecting view or by endorsing the procreation asymmetry (Roberts, 2024). We do not structure potential cause areas along those lines, since there might be significant overlaps. For example, current generations' lives might be at risk from risks that threaten the future of humanity as a whole. Moreover, biases that are harmful for marginalised groups today might crystallise and persist into the far future.
In the following, we discuss a few AI-related problems.
Equipped with a clear meta-ethical foundation, you can now go about choosing a cause area. For general cause prioritisation, you can find guidance here. In brief, organisations like 80,000 Hours recommend looking at the cause where you expect to be able to have the largest marginal impact. This typically depends on how important the cause is (given your meta-ethical view) and how easy it is to make progress on the cause. There might be more low-hanging fruit for causes which received little attention so far (neglectedness). Your cause prioritisation may or may not lead you to a range of causes related to AI, but given the limited scope of this post, we focus on risks around AI safety and ethics here. [4] Much of the economics-of-AI literature frames the issue as labour economics (focusing on the displacement of labour by AI). We treat it as broader, spanning political economy, growth, welfare economics, and finance.
The MIT AI Risk repository (Slattery et al., 2024) categorises AI risks in 7 categories: (1) Discrimination & Toxicity, (2) Privacy & Security, (3) Misinformation, (4) Malicious Actors, (5) Human-Computer Interaction, (6) Socioeconomic & Environmental, (7) AI System Safety, Failures & Limitations. Those are further broken down into 24 sub-categories. Table 1 reproduces Table 6 of Slattery et al. (Slattery et al., 2024), defining those sub-categories.
Economists are likely able to help mitigate risks in any of those categories. So, in order to narrow down your options, you can rank the options based on your values, which you identified before. For example, for many consequentialist views a first step could be to write down for each risk category (i) how many people are affected by the risks, (ii) how strongly they are affected (iii) how many other people are working on mitigating the risks (iv) if the mitigation work needs to get past thresholds (e.g. get legislation passed) (v) how tractable the risks are in general. This process often leads to studying worst-case scenarios, which are unlikely but affect many (potential) beings (MacAskill, 2022). In the following, we consider four scenarios which would count as catastrophic under consequentialist views and endorse some form of long-termism (Greaves & MacAskill, 2025; MacAskill, 2022). You should take these as examples and find scenarios which are best to work on according to your own moral values and views.
| Figure 1: How the Economics of Transformative AI could lead to impact |
We touch briefly on four risk categories, with no claim to comprehensiveness: Avoidable human-generated harm, value lock-in, Gradual disempowerment and a catastrophe due to misaligned AI.
Figure 1 shows that multiple areas of Economics relate to these risks. The core takeaway is that core research areas in Economics (dark blue) touch on a number of important risks from AI. Covering all potential links goes beyond the scope of this post. To give one example: Starting on the left. If you understand how AI affects economic growth, you can better understand AI’s implications on inequality. This, in turn, affects voter behaviour and, therefore, changes the incentives of democratic leaders, who design the regulations of AI. This shapes the race between firms, which may lead to a catastrophe due to misaligned AI.
Many AI researchers worry that when AI becomes more and more intelligent, it might be difficult or impossible for the principal (the operator) to control the AI (the agent). This is called the alignment problem. This agent might have developed intermediate goals which are inconsistent with the operator's preferences. Hence, the preferences of the principal and the agent are misaligned. If the agent becomes much more intelligent than the principal, the information asymmetry grows so large that the principal cannot extract any rents to meet their preferences. The world will be shaped by (potentially alien) AI preferences. This would constitute an existential catastrophe. A growing literature is interested in AI misalignment in an information design context. For example, Dworczak and Smolin (2026) study under which conditions using an AI advisor can be useful even if this advisor might be misaligned (Chen et al., 2024; Fudenberg & Liang, 2025). As economists like to study trade-offs, studying corner solutions, such as existential catastrophes in micro-economic settings, might be difficult to publish. This means those questions might be more neglected and useful for a non-academic audience.
The UK AI Security Institute provides a great set of starting points on how microeconomic theory could help mitigate the alignment problem.
Learn more:
Existing work in economics:
Potential starting point:
What liability and incentive structures should govern AI labs when their models demonstrably conceal misaligned reasoning from overseers — and how should these scale with model capability? (AI)[5]
The previous section was concerned with strategic interaction between a human operator (principal) and an AI (agent). Strategic interaction between AI agents also poses potential risks. Hammond et al. (2025) categorise potential failure modes in miscoordination, conflict and collusion. Miscoordination might occur when there is not enough time for communication, such that agents have to solve coordination without learning. Risks from conflict might materialise due to social dilemmas, such as the tragedy of the commons (Hardin, 1968), military applications of AI or AI engaging in coercion and extortion. Hammond et al. (2025) also map out relevant risk factors, such as information asymmetries (see section 3.1) and commitment and trust (section 3.5). All three failure modes and many of the risk factors are linked to a rich economic literature. The UK AI Security Institute provides a great set of starting points for how microeconomic theory can help address multi-agent risk as well.
Learn more
Potential starting points:
Source: direct quote from UK AI Security Institute (2025), to learn more see Bhatt et al. (2025), Tennenholtz (2004) and Hammond et al. (2025).
“Gradual disempowerment” (Kulveit et al., 2025) refers to the systemic existential risk posed by incremental AI advancements that could gradually erode human influence over crucial societal systems like the economy, culture, and nation-states. Unlike scenarios featuring sudden AI takeovers, this concept describes how even incremental improvements in AI capabilities can undermine both explicit human control mechanisms (like voting and consumer choice) and the implicit alignments that arise from societal systems' reliance on human participation. As AI increasingly replaces human labour and cognition across these domains, it weakens the feedback mechanisms that have historically ensured these systems remain aligned with human interests. These effects can be mutually reinforcing across different domains, potentially leading to an effectively irreversible loss of human influence over society, culminating in an existential catastrophe through humanity's permanent disempowerment [6]. There are multiple conditions which make gradual disempowerment more likely and which can be addressed by economists. First, understanding and modelling the phenomenon is essential to identify potential policies that mitigate this risk. Second, once the incentives that lead to this catastrophic scenario are understood, economists could help analyse potential policies and regulations that help mitigate the risk.
Learn more:
Existing work in economics:
Potential starting point:
Scholars have distinguished AI as labour-replacing or labour-augmenting technology (Brynjolfsson, 2023). If AI is predominantly labour-replacing, the capital share of GDP might approach 100% (Korinek & Suh, 2024). Hence, existing inequalities due to the highly unequal distribution of wealth (Chancel et al., 2022) might be exacerbated (Trammell & Patel, 2025).
Moreover, if AI technology turns out to be mostly labour-replacing, many societies might transform from being dependent on workers offering their labour and demanding political participation in return, to societies which depend on the extraction of a resource. AI could be exploited by a small minority, much like the revenues from resource extraction, which benefit only a small elite in some countries. This might undermine democratic institutions (Drago & Laine, 2025).
Race dynamics between countries might lead to nationalisation of AI companies. This, in combination with AI-enabled democratic backsliding, AI-enhanced surveillance, military dominance and automation in the security sector, might empower one or a few persons more persistently than ever before. History has seen many leaders who aimed at implementing a persistent value system that seems appalling from today's perspective. If decision makers in the near future manage to lock in their values, those values might be highly disagreeable as well. For example, if the elite in power scapegoats or simply dislikes a minority, empowering the elite might be catastrophic for this minority. Another concern would be that the well-being of non-human animals might not be taken into account, such that animal suffering might persist in the future. Thirdly, there is a risk of blackmail and extortion using suffering in multi-agent systems (Hammond et al., 2025; Sotala & Gloor, 2017). If you give some probability weight to moral views which emphasise rights, capabilities, equality, or simply weight the well-being of the worse off more, this might be a reason to prioritise this area.
There are many ways economists could contribute to reducing the risks of power concentration. First, a large literature in political economy is related to the question of how to preserve democratic institutions (Acemoglu & Robinson, 2013; Egorov & Sonin, 2024; Grillo et al., 2024; Guriev & Papaioannou, 2022; Rodrik, 2021; Zhuravskaya et al., 2020). Second, it could be valuable to explore in more depth how economic concentration (e.g. market concentration) links to power concentration. For example, a small but growing body of literature documents how market concentration (De Loecker et al., 2020) translates into political power (Bombardini & Trebbi, 2020; Zingales, 2017). If larger corporations can affect political decisions, market concentration means concentration of political power. This erodes democracies and can lead to value lock-in (Zingales, 2017). AI-related market concentration is a rarely studied phenomenon in this context that might behave differently than other sectors, for example, it might outpace regulatory responses.
Between-country inequality might also increase due to AI (Korinek & Stiglitz, 2021). For example, if AI is a substitute for labour, countries with an abundance of labour and scarce capital are disadvantaged in the global economy (Alonso et al., 2022). Hence, AI might block typical channels of catch-up growth. While a comprehensive literature review on AI and macro-development economics goes beyond this article, the macroeconomic effects of AI on low-income countries seem particularly neglected and therefore warrant further research.
From the perspective of welfare economics, both fundamental theorems of welfare economics may be violated due to transformative AI. Market concentration might lead to a violation of the first theorem (that competitive equilibria are Pareto efficient) and render the second welfare theorem (that any Pareto efficient allocation can be reached by redistributing endowments) irrelevant, as actors who are empowered by AI might use this power to resist redistribution. Learn more:
Related work in Economics
Potential starting point:
A last category of AI risks is related to scenarios where humans remain in control of AI but fail to avoid harm nonetheless. Many of the most worrisome risks mentioned in the last section might apply in a weaker form in a world with humans in control. For example, humans might fail to adequately attribute moral patienthood to animals (Singer & Tse, 2023) or digital beings (also see 1 and 2), or injustices faced by minorities due to biases might persist. Even if we fix some of these issues, we can still face catastrophic outcomes from the way humans choose to use TAI, also referred to as “Misuse Risks” (Martin, 2024). These apply to both intentional use and reckless deployment of AI. There are a number of subcategories of this risk (e.g. War, Totalitarian Scenarios, Societal Decay (i.e. Disinformation, Cyberattacks, or small bio-weapons)) (Martin, 2024). Many of them are now near-term risks discussed in popular media. Recent examples include debates around Anthropic's relationship with the US Department of Defence and controversies over Claude's new model, Mythos. Disinformation is an especially prevalent concern in this category, as both state and non-state actors may soon be able to target individuals with AI-enabled precision propaganda (CSET, 2021). Work on AI disinformation could build on the larger economic literature on disinformation (e.g. Allcott & Gentzkow, 2017).
Learn more:
Potential Starting point:
Having outlined the importance of having a theory of change in the previous chapter, we attempt such a theory of change in Figure 1 for various economic subfields in this chapter. This theory of change should be understood not as a final product, but as a first draft that should be extended, corrected and revised. Figure 1 depicts how work in various areas of economics could reduce risks: Dark blue areas are common areas of economic research. Light blue areas are predominantly researched by political science. However, some economists are working in those areas. Light purple blocks are studied by finance, and dark purple blocks are studied by legal scholars. On the right-hand side, we outline some risks from AI: S-Risks, Value Lock-In, Gradual Disempowerment and a Catastrophe due to misaligned AI.[7]
To backchain from there: war might lead to suffering risks, extreme power concentration (the winning country controlling more people), and a race between states (trying to improve AI for geostrategic reasons). War might be made more likely by a race between states or incentives of oligarchic leaders (power-seeking behaviour) or democratic leaders (Burgfriedenspolitik - the historical tendency of political elites to suspend domestic opposition and rally behind national causes during perceived external threats). Democratic leaders’ incentives are affected by voters’ behaviour, which in turn is affected by growth and inequality. Oligarchic leaders’ incentives might be affected by growth as well, as it increases the pie which can be seized. Understanding how fast AI accelerates growth and increases or decreases inequality could help to study potential solutions, e.g. redistribution of gains from capital/AI.
| Figure 1: How the Economics of Transformative AI could lead to impact |
Extreme power concentration could follow directly from inequality in an explosive growth scenario: capital owners of relevant assets become extremely rich very fast, while people who have not (yet) accumulated capital lose the ability to gain income by supplying labour and have hence no income.
Extreme power concentration might also be a result of war, in a scenario where core industry branches are nationalised, and one country gains a strong military advantage due to new technology.
The profit-seeking imperative of the legal form of the corporation could also lead to extreme power concentration, where one corporation becomes a monopolist for a large fraction of goods and services, and everyone having shares of that company becomes rich and powerful, while others become disempowered (Mayer, 2018).
Another scenario which we have not seen discussed much yet is using up the resources needed by the future [8]. Potentially, explosive growth in the next few years will use up a lot of resources for meaningless projects. This scenario is relevant if (human) life is bound to Earth
Finally, the economics of transformative AI could increase or decrease the risk of a catastrophe due to misaligned AI. Races between states and firms, profit-maximising motives might increase the likelihood of such a catastrophe. The tighter the race, the higher the incentives to cut corners.
To sum up, there is a range of different risks that economists might be able to mitigate. It seems likely that some economists will help mitigate risks while others are aggravating them. Hence, the choice of research topic or job in the area should be made with great care.
If you have read until here, you are probably thinking: “Ok, let's go! What can I do next?” In this section, we have attempted to provide you with an (incomplete) list that you can use to advance your knowledge in the field of the Economics of TAI, get in contact with potential mentors, find research topics and get active yourself. We propose you skim this list and find one thing you think is useful to you right now, and start there! Set yourself a goal for the next week, and come back to this list once you reach that goal - either to keep going at the same action, or to pick up the next thing. EA Forum Posts are a great and approachable starter for any EA-related quest. Here are some you might find helpful:
A short reading list of nice starting points in this field:
General Economics of (T)AI Resources Growth theory and economics of Transformative AI
Micro-theory/Mechanism Design
Ideal for people with some background in Game Theory, Contract Theory or Mechanism Design
Learning can happen in different forms depending on what type of learner you are. If you are looking for more structured approaches with a clear list of assignments and readings, consider doing one or two of these courses to learn more about your topic of choice:
They are an extremely convenient way of getting more grip within a specific field. A graduate degree in a relevant area is typically recommended, though not a strict requirement. But depending on your profile, it can be hard to get into one of them - don't be discouraged! I (Matt) was also rejected many times. Keep trying!
Building a theory of change is enormously hard and depends a lot on information which is extremely difficult to obtain. The easiest and most recommended way to fill these knowledge gaps is to ask people who are actively working on mitigating a risk you care about what they need. What is the kind of economic research which would make their work easier? Hence, any research project should start with stakeholder outreach, asking experts in policy, governance and technical AI what they need. For example, if you consider working on game-theoretic questions of enabling a global AI treaty, reach out to people who are actively working on this (e.g. red lines).
I (Max) first read this in the 80k Guide, and I very much agree: It can be very helpful to get a mentor in your field. Most of the time, people are very happy to help, especially if it's in a field they are interested in/care about. Here are some resources on that topic and ways to go about it:
Some other opportunities you can consider while planning your next steps:
Author contributions: ME conceived the project and led its execution developing the overall framework and theory of change (Figure 1), writing the first draft, and coordinating feedback from reviewers and collaborators. MP contributed to the writing throughout, with primary responsibility for developing and structuring the "Open Questions" and "Recommendations and Resources" sections.
[Chen, E. O., Ghersengorin, A., & Petersen, S. (2024). Imperfect Recall and AI Delegation.](https://www.zotero.org/google-docs/?lr1MBG)
[Drago, L., & Laine, R. (2025). The Intelligence Curse. https://intelligence-curse.ai/](https://www.zotero.org/google-docs/?lr1MBG)
[Hilton, B. (2022). ‘S-risks’. 80,000 Hours. https://80000hours.org/problem-profiles/s-risks/](https://www.zotero.org/google-docs/?lr1MBG)
[Mayer, C. (2018). Prosperity: Better Business Makes the Greater Good. Oxford University Press.](https://www.zotero.org/google-docs/?lr1MBG)
Table 1. Domain Taxonomy of AI Risks | | #
| Domain/Subdomain | Description | | 1 Discrimination & toxicity | | | 1.1 Unfair discrimination and misrepresentation | Unequal treatment of individuals or groups by Al, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups. | | 1.2 Exposure to toxic content | Al that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography. | | 1.3 Unequal performance across groups | Accuracy and effectiveness of Al decisions and actions is dependent on group membership, where decisions in Al system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users. | | 2 Privacy & security | | | 2.1 Compromise of privacy by obtaining, leaking, or correctly inferring sensitive information | Al systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property. | | 2.2 Al system security vulnerabilities and attacks | Vulnerabilities that can be exploited in Al systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior. | | 3 Misinformation | | | 3.1 False or misleading information | Al systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional, or material harms. | | 3.2 Pollution of information ecosystem and loss of consensus reality | Highly personalized Al-generated misinformation that creates "filter bubbles" where individuals only see what matches their existing beliefs, undermining shared reality and weakening social cohesion and political processes. | | 4 Malicious actors & misuse | | | 4.1 Disinformation, surveillance, and influence at scale | Using Al systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior. | | 4.2 Cyberattacks, weapon development or use, and mass harm | Using Al systems to develop cyber weapons (e.g. by, coding cheaper, more effective malware), develop new or enhance existing weapons (e.g., Lethal Autonomous Weapons or chemical, biological, radiological, nuclear, and high-yield explosives), or use weapons to cause mass harm. | | 4.3 Fraud, scams, and targeted manipulation | Using Al systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail, or targeted manipulation of beliefs or behavior. Examples include Al-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery. | | 5 Human-computer interaction | | | 5.1 Overreliance and unsafe use | Anthropomorphizing, trusting, or relying on Al systems by users, leading to emotional or material dependence and to inappropriate relationships with or expectations of Al systems. Trust can be exploited by malicious actors (e.g., to harvest information or enable manipulation), or result in harm from inappropriate use of Al in critical situations (e.g., medical emergency). Over reliance on Al systems can compromise autonomy and weaken social ties. | | 5.2 Loss of human agency and autonomy | Delegating by humans of key decisions to Al systems, or Al systems that make decisions that diminish human control and autonomy, potentially leading to humans feeling disempowered, losing the ability to shape a fulfilling life trajectory, or becoming cognitively enfeebled. | | 6 Socioeconomic & environmental harms | | | 6.1 Power centralization and unfair distribution of benefits | AI-driven concentration of power and resources within certain entities or groups, especially those with access to or ownership of powerful Al systems, leading to inequitable distribution of benefits and increased societal inequality. | | 6.2 Increased inequality and decline in employment quality | Social and economic inequalities caused by widespread use of Al, such as by automating jobs, reducing the quality of employment, or producing exploitative dependencies between workers and their employers. | | 6.3 Economic and cultural devaluation of human effort | Al systems capable of creating economic or cultural value, including through reproduction of human innovation or creativity (e.g., art, music, writing, coding, invention), destabilizing economic and social systems that rely on human effort. The ubiquity of Al-generated content may lead to reduced appreciation for human skills, disruption of creative and knowledge-based industries, and homogenization of cultural experiences. | | 6.4 Competitive dynamics | Competition by Al developers or state-like actors in an Al "race" by rapidly developing, deploying, and applying Al systems to maximize strategic or economic advantage, increasing the risk they release unsafe and error-prone systems. | | 6.5 Governance failure | Inadequate regulatory frameworks and oversight mechanisms that fail to keep pace with Al development, leading to ineffective governance and the inability to manage Al risks appropriately. | | 6.6 Environmental harm | The development and operation of Al systems that cause environmental harm, such as through energy consumption of data centers or the materials and carbon footprints associated with Al hardware. | | 7 Al system safety, failures & limitations | | | 7.1 Al pursuing its own goals in conflict with human goals or values | Al systems that act in conflict with ethical standards or human goals or values, especially the goals of designers or users. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, and may result in Al using dangerous capabilities such as manipulation, deception, or situational awareness to seek power, self-proliferate, or achieve other goals. | | 7.2 Al possessing dangerous capabilities | Al systems that develop, access, or are provided with capabilities that increase their potential to cause mass harm through deception, weapons development and acquisition, persuasion and manipulation, political strategy, cyber-offense, Al development, situational awareness, and self-proliferation. These capabilities may cause mass harm due to malicious human actors, misaligned Al systems, or failure in the Al system. | | 7.3 Lack of capability or robustness | Al systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning. | | 7.4 Lack of transparency or interpretability | Challenges in understanding or explaining the decision-making processes of Al systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors. | | 7.5 Al welfare and rights | Ethical considerations regarding the treatment of potentially sentient Al entities, including discussions around their potential rights and welfare, particularly as Al systems become more advanced and autonomous. | | 7.6 Multi-agent risks | Risks from multi-agent interactions due to incentives (which can lead to conflict or collusion) and/or the structure of multi-agent systems, which can create cascading failures, selection pressures, new security vulnerabilities, and a lack of shared information and trust. | | Source: Table 6 in Slattery et al. 2024, Licensed under |
Existential risk – One where an adverse outcome would either annihilate Earth−originating intelligent life or permanently and drastically curtail its potential (Bostrom, 2002).
People working on suffering risks or s-risks attempt to reduce the risk of something causing vastly more suffering than has existed on Earth so far (Hilton, 2022).
We are thankful to Donghyun Suh for that point.
As we are not experts on ethics the list of causes we outline here is tentative at best.
We did not keep track of AI usage for open questions. We indicate the open questions where AI was probably used. It is possible that AI was used for some open questions where it is not mentioned.
Claude’s summary of Kulveit et al. (2025), edited by ME This list should be seen as a set of illustrative examples. It does not claim to be comprehensive.
We assume someone is working on this, we just missed it so far.
One can assume that either space travel is very hard, or if spreading artificial life in the universe turns out not to be a valuable project and spreading carbon life in the universe is prohibitively difficult.
Agglomeration effects may be offset by congestion effects or weakened in highly automated futures.