AI is increasingly capable. Suppose we're lucky and the alignment problem gets solved, so AI reliably does what its operators want. But we’re not done yet: it lends immense power to those controlling it by progressively replacing the need for humans, disempowering many. [1] AI could enable extreme power concentration, producing states unaccountable to their citizenry, and more stable and totalitarian than any previous ones have been.
While LessWrongers have somewhat discussed this (e.g. in Gradual Disempowerment, Intelligence Curse, AI-enabled coups and the like), I still think there are far too few people working on this! As far as I can tell, it’s largely Forethought (who do macrostrategy), Formation Research and EuroSafeAI (shameless plug), as well as some distributed work here and there. Most people seem to agree it’s a large concern, and that few are working on it, but many aren’t sure what to do. [2] I hope this post deconfuses the space and helps technical people contribute, hopefully increasing tractability. In this post, I want to
In summary, I’ll explain this graph, and give recommendations.
I think this is an important and neglected problem; I'm uncertain about magnitude; this post maps the problem rather than forecasting it. It’s also preliminary, hopefully contributing to discussion on the topic, and omits details (with some relegated to footnotes). Different opinions and additions would be much appreciated! Furthermore, I will generally assume that operator-alignment is more or less solved (a tall ask, I know).
Most stories of extreme power concentration are pretty specific or concern one pathway (such as AI-enabled coups). For the sake of this post, I’ll focus on the general case.
The two core drivers are fairly simple: AI automation of labour, and runaway AI R&D giving a decisive lead. The first is the main engine: if you don't need humans to do the work, they lose leverage, and you don't need to listen to them. The same goes for holding power itself: historically that required many cooperating humans (soldiers who could refuse, officials who could slow-walk), whereas AI agents could be far more loyal. [4] Put in economic terms, capital becomes a substitute for labour, and because capital is much more concentrated than labour, control over AI labour concentrates in the hands of capital holders.
But how should you think about power? It can be a bit confusing to think about, especially since human society is more multi-part than an AI system. Breaking down sources of power, I have grown to like the well-established IEMP model by Michael Mann*. *The IEMP model divides sources of power into *ideological, economic, military *and political dimensions. These can be converted into each other (a businessman can support a politician, or Taylor Swift can convert appeal into money), but usually, they tend to be surprisingly independent: A powerful FBI bureaucrat (political power) may struggle to get an absurdly high income or become famous (economic and ideological power).
So, how does AI automation of labour concentrate power? A few arguments:
Those are the main factors, but much was omitted for brevity. A summary can be found in this graph, with some plausible leaf nodes as well.
Below I have linked to RFPs and other people’s suggestions. What follows is my (opinionated) additional ideas of what works, take it with the usual grains of salt. I suspect policy work will be the biggest lever, but I am generally more technical. If you are coming up with ideas, I recommend checking out previous work, or starting de novo from the IEMP model.
Of the four factors, economic power seems hardest to solve for technical people. I don’t think this is an accident: The economy is fastest to adapt based on capability, and less regulated than others. (I suspect, compared to innovation, that policy will make a big difference here: The public response to AI will probably largely be driven by economic effects rather than e.g. catastrophic risks. But note that this simply routes through political power rather than economic power. This is why I’m generally more interested in keeping the state accountable). Perhaps we can create jobs where people will be glad to pay a human premium, [12] ensure humans stay usefully in the loop
That leaves ideological, military and political power. I’m unsure about the ideological side (I suspect it has similar dynamics to economic power), but things like proof of personhood and the like are probably good. My sense is the least work has been done in military and political scenarios: I think we can greatly increase the ‘usefulness/alignment’ tradeoff in these areas, and suggest good alternatives. Alternatively, if that fails, we can document risks posed by AI here. Promising ideas have been expanded upon by others (see below) so I will not reiterate them here.
For the military and bureaucracy: AI removes the friction that has always made power seizure hard. A leader used to need many people to go along (soldiers who could refuse, officials who could slow-walk), which is why coups are modelled as coordination games (see Singh’s Seizing Power). [14] Loyal AI agents bypass that coordination problem, so the goal is to keep the cost of capture high. Notably, democracies prevent coups less by counterbalancing (as autocrats do) than by legitimacy as a focal point, distributed veto points, and professional norms, each of which assumes the captor needs many humans to cooperate. The coup-proofing literature is a direct resource if we invert it: the same mechanisms protecting institutions rather than rulers (Quinlivan; De Bruin). Concrete directions: multi-party authorization for consequential actions, law-following rather than narrowly operator-following models, and making capture observable through audits and attestation. Perhaps, as Madison noted, we can also have AI risk, at its core, is about what happens when enormous capability outpaces our mechanisms for keeping it accountable. Misalignment is one branch of that story. Extreme power concentration is another, and right now it has perhaps a dozen people working on it full-time. The central claim of this post is that AI erodes what historically kept power contestable: the need for many cooperating humans, and labour's bargaining leverage. Those frictions were never designed; we inherited them. And they can erode across each of Mann's four sources of power, ideological, economic, military and political, with political power as the natural endpoint where the others converge. If that's right, the work ahead is to rebuild contestability deliberately, dimension by dimension: keep the cost of capture high, make capture observable, keep humans load-bearing in the institutions that matter most, and keep rival actors empowered to push back.
This map is preliminary, and I expect parts of it are wrong. I'd gladly invite discussion in the comments; disagreements, additions, and better framings are all very welcome. And if you're a technical person looking for a way in, the RFPs and project lists above are a good place to start, or reach out to us at EuroSafeAI, or to me directly.
For more details on the threat landscape, I like these two articles:
[https://www.formationresearch.com/power-concentration-survey.pdf](https://www.formationresearch.com/power-concentration-survey.pdf)
[https://80000hours.org/problem-profiles/extreme-power-concentration/](https://80000hours.org/problem-profiles/extreme-power-concentration/)
For ideas specifically, these are great:
[https://www.longview.org/request-for-proposals-on-extreme-power-concentration/](https://www.longview.org/request-for-proposals-on-extreme-power-concentration/)
https://www.lesswrong.com/posts/GAv4DRGyDHe2orvwB/gradual-disempowerment-concrete-research-projects
[https://gradual-disempowerment.ai/mitigating-the-risk](https://gradual-disempowerment.ai/mitigating-the-risk)
[https://intelligence-curse.ai/breaking/](https://intelligence-curse.ai/breaking/)
[https://www.lesswrong.com/posts/ugBoeexGYvNLxZKA7/a-research-agenda-for-secret-loyalties](https://www.lesswrong.com/posts/ugBoeexGYvNLxZKA7/a-research-agenda-for-secret-loyalties)
[https://zhijing-jin.com/d/2026-ai-risk.pdf](https://zhijing-jin.com/d/2026-ai-risk.pdf)
There are two rough ways: Power-to, e.g. the ability to produce an effect, and power-over, e.g. the ability to make Bob do something Bob doesn’t want to. In practice, ideological, military and political power seem more zero-sum to me, so power-to and power-over somewhat converge there. For economic power these diverge somewhat, but it’s not too relevant for the argument. Next to Michael Mann’s IEMP model, I also like Lukes’ Three Faces of political power.
I expect power concentration to happen on a continuum as AI capabilities increase, so it’s intentionally left somewhat vague. I think the economic definition that ‘capital becomes a gross substitute for labour’ is pretty good, though.
I was somewhat surprised by this (given my background is fairly technical, and so I didn’t see similarities), but Tom Davidson put this well:
In fact, AI-enabled coups and AI takeover have pretty similar threat models. To see this, here’s a very basic threat model for AI takeover:
- Humanity develops superhuman AI
- Superhuman AI is misaligned and power-seeking- Superhuman AI seizes power
for itselfAnd now here’s a closely analogous threat model for AI-enabled coups:
- Humanity develops superhuman AI
- Superhuman AI is controlled by a small group- Superhuman AI seizes power
for the small group Put more broadly, the general story of AI risks is about AI power/capabilities — leading into managing misuse, misalignment and power concentration.
Thanks to Zhijing Jin and EuroSafeAI members for valuable feedback. All errors and opinions are mine.
This isn’t a particularly new point or anything: All of AGI's issues are due to its capability, including misalignment, or also misuse for e.g. CBRNs. A misaligned ant hurts nobody.
It is now regularly mentioned at Alignment workshops, and both OpenAI’s charter and subsequently Anthropic mention it strongly, the former with near equal importance to traditional alignment.
E.g. if you’ve read Intelligence Curse or 80k’s excellent https://80000hours.org/problem-profiles/extreme-power-concentration/ This is a meaningful crux, both for misalignment concerns of course, but also whether we can have AI systems themselves advocate for humanity’s interests.
E.g. global Palma ratios for wealth are around 2-3x larger than for income. Similarly, gini coefficients are often around 0.2 higher.
The possibility of 30% YoY GDP growth is often discussed, e.g. by Epoch AI or Philip Trammell. It is not entirely unreasonable, since GDP is heavily bottlenecked by population size. Assuming you are no longer bottle-necked on labour, it does make sense to presume much faster growth, but as usual the economic picture is hard to predict.
Specifically, this depends on how displacement and reinstatement effects relate to each other. There is evidence suggesting we have had more displacement over the last 30 years.
For more details of this argument, see Philip Trammell’s* *https://philiptrammell.com/static/economic_growth_under_transformative_ai.pdf. It is unclear how far this will go (and, as all of the above arguments, this again depends on the regulatory response). Assuming robotics is solved (something I do assume in this scenario), there seem to me 3 big categories of tasks left: Politically protected ones (e.g. licensure, legally-mandated ratios etc., such as potentially lawyers, doctors and other professions which will defend their jobs vigorously), those where there is demand for human input (such as care work, service work, pastors, and perhaps influencers), and rents due to scarcity (e.g. house/energy rents). Open-weight models decentralize the model layer, but running AI at scale still routes through a concentrated hardware stack (NVIDIA, TSMC, ASML, hyperscaler compute).
Many papers now put LLM persuasiveness higher than human persuasiveness in chatting (see https://arxiv.org/pdf/2606.16475v1).
Przeworski has famously noted that no democracy wealthier than Argentina in 1975 has ever fallen, i.e. roughly USD 14k today. I now think this somewhat doubtful with Turkey, but the general trend holds.
In the EU, around 22% of jobs are already regulated in some sense. (https://www.pubaffairsbruxelles.eu/eu-institution-news/commission-calls-on-18-member-states-to-strengthen-the-eu-single-market-for-regulated-professions/)
See Jan Kulveit or Erik Brynjolfsson’s recent work
Although coups are the less likely way democracies die today, see https://www.journalofdemocracy.org/articles/on-democratic-backsliding/ for a good review.