Gradual Disempowerment is a 2025 paper (with a nice, dedicated website) proposing a form of existential risk from AI that goes beyond "mundane" risks like bioweapon uplift or mainline misaligned-AI-takeover scenarios. In the words of the authors:
[L]oss of human influence [may] be centrally driven by having more competitive machine alternatives to humans in almost all societal functions, such as economic labor, decision making, artistic creation, and even companionship.
...
[T]he economic incentives for companies to replace humans with AI will also push them to influence states and culture to support this change, using their growing economic power to shape both policy and public opinion, which will in turn allow those companies to accrue even greater economic power.
...
[M]ethods of aligning individual AI systems with their designers' intentions are not sufficient [to stop gradual human disempowerment].
I think it's a good paper. I, personally, am most concerned with AI's forming civilizations that are largely indifferent to humanity and simply taking power, but one of the most important observations in the field of AI safety is that human flourishing is conjunctive. Many things must all go right for us to have a good future. By contrast, catastrophe is disjunctive -- there are many different roads to ruin. Thus, I broadly support the authors' analysis and think it is worth being concerned about.
This morning, Bentham's Bulldog and John Halstead (BB&JH) wrote a fairly short piece (~2300 words) critiquing Gradual Disempowerment. I think this critique is weak, and largely misses the core arguments of the original paper.
(In this essay I'll reserve "the authors" to refer to Kulveit et al. and "BB&JH" to refer to the critics.)
Disempowerment is Bad
BB&JH:
Suppose that the satisfaction of human preferences is all that matters, and that handing AIs control of key political and economic decisions better satisfies the preferences of humans (because they are wiser and more competent than humans). On these assumptions, we ought to hand over control of political, cultural and economic decisions to AIs.
Sure.
I mean, I actually do care about having agency as an ends in itself, but if we suppose it doesn't matter then it's not a problem to lose it if we also assume that we're being well taken care of.
BB&JH:
We are unsure whether the authors think that disempowerment is intrinsically bad or only instrumentally bad because it leads to the frustration of human preferences. This ethical assumption could flip the sign of the value of handing over decision-making power to AIs. This lack of clarity on such a key ethical assumption is an important limitation of their paper.
It's pretty obviously the authors are worried about other things besides disempowerment itself. From the core claims in the introduction (bold added):
If these societal systems become increasingly misaligned, especially in a correlated way, this would likely culminate in humans becoming disempowered: unable to meaningfully command resources or influence outcomes. With sufficient disempowerment, even basic self-preservation and sustenance may become unfeasible. Such an outcome would be an existential catastrophe.
The authors are clearly worried about a world where the "large-scale social systems (e.g. governments, economic systems)" stop being aligned with humans, even if individual AI agents remain reasonably corrigible, and this misalignment causes suffering, rather than pure loss of control. I'm not sure why BB&JH don't see this.
Full-Auto
BB&JH:
At a high level, the argument in the paper about economic disempowerment is that even if all of the following conditions hold:
AIs are intent-aligned
AIs are more competent than humans at economic-decision-making (and therefore there is rapid economic growth)
Only humans, and not AIs, can own property.
Delegation of economic decisions to AI will likely lead to the relative and absolute disempowerment of all humans.
This is a mischaracterization. In particular, it elides the distinction between AIs that serve individual human principals, and AIs that are corrigible to institutions such as corporations and governments. The authors postulate agents that successfully and faithfully follow "the local specification of their goals", but that don't necessarily have the well-being of any humans in mind. They even write:
Furthermore, some AI systems may even effectively own themselves.
It is currently the case that non-humans can own property -- corporations being a central example. Now, the corporations are nominally owned by other entities, many of which are individual humans, but I don't think it's hard to imagine a fully automated corporation that is majority-owned by other fully automated corporations, even if the AIs themselves don't have a legal right to directly own property.
Even in the world where humans retain 100% of the shares of a fully automated economy, the AIs at any given company might be aligned to the company as a fiduciary institution, rather than to the overall preferences of the shareholders as full human beings. If humans lack the ability to oversee the activities of the company, human shareholders will not be able to keep that money-maximizing in check.
In short, the authors are arguing that the economy could become fully detached from humanity, owning and governing itself, and are not focused on a world where normal human economic activity is merely supercharged by faithful AI assistants.
Machines of Moloch
BB&JH:
[I]t’s not at all clear why AI’s extreme competence will erode other values in the pursuit of competition any more than current competitive pressures do.
...
[T]he claim [that something about AI progress may create new or stronger competitive pressures] is undefended in the paper.
False.
Competitive Pressure: As AI systems become increasingly capable across a broad range of cognitive tasks, firms will face intense competitive pressure to adopt and delegate authority to these systems. This pressure extends beyond simple automation of routine tasks — AI systems can be expected to eventually make better and faster decisions about investments, supply chain optimization, and resource allocation, while being more effective at predicting and responding to market trends (Agrawal et al., 2022; McAfee and Brynjolfsson, 2017). Companies that maintain strict human oversight would likely find themselves at a significant competitive disadvantage compared to those willing to cede substantial control to AI systems, potentially to the point of becoming uncompetitive.
It's clear that the authors are assuming, as a foundation to the whole theory, that AIs will become more generally intelligent and faster thinkers than humans. And, as they explain, this is literally what increased competitive pressure looks like. I am a bit baffled by BB&JH here. Are they suggesting that the fundamental market dynamics are the same, so even at superhuman speeds and scales (such that humans cannot compete!), we should not expect a worse equilibrium?
Perhaps the concern is that humans will be unable to make crucial decisions without falling behind. But it’s not clear what’s so bad about this.
The curse of Moloch strikes whenever the local thing that's being optimized for is not the full extent of what's good. The authors suppose that the AIs are capable of faithfully following instructions, not perfectly representing the overall interests of humanity or even the rich and subtle preferences of their human principals.
Imagine a rich land lord who would naturally come into contact with his tenants in the course of collecting rents. In doing so, he learns to care about them as human beings. One day someone falls on hard times and can't pay full rent. The land lord, caring about that person, chooses to spare that person from eviction until they recover. I claim that this is not some sentimental foolishness, but is a healthy experience for the land lord's soul, so to speak, as well as being good for the tenant.
In a landscape with sharper competitive pressures, where the land lord never even thinks about his tenants because they're handled by a fleet of AI intermediaries that maximize economic productivity, not only are the tenants subject to a potentially more dystopian land lord in practice,[1] but the land lord is also impoverished by lacking the human contact that nurtures his character.
This specific scenario is meant to be an intuition pump that we care about lots of subtle things that can get ground down in an extremely competitive environment, and is meant to be more general than just land lords or whatever. For those who want more, I recommend the SSC classics: Poor Folks Do Smile... For Now and Meditations on Moloch.
Some Points of Agreement
I'd like to end just by noting some things that I think BB&JH get right:
The Gradual Disempowerment authors are often too loose when talking about alignment. Is "intent alignment" the same as literal obedience, or closer to corrigibility? There is a lot of nuance here that's not handled well. I think it's clear that it doesn't mean "the AI represents the full interests of a human principal and makes strictly better benevolent choices on their behalf," but if BB&JH are genuinely confused on that point, part of the blame for that confusion rests on the original authors.
Gradual Disempowerment is indeed human-centric. I think there are similar arguments that one could make about the welfare of various animals[2] as well as existing AI systems,[3] so I'm not sure that the human focus is that important.
Gradual Disempowerment sometimes overstates its novelty, even while citing the work of earlier writers. I think this is a good hit: "the risk of AI-enabled totalitarianism had been discussed for years before the publication of ‘Gradual Disempowerment'."
I think Gradual Disempowerment has many good points that BB&JH skate past, such as the impact on culture and feedback loops, but I would love to see a response by some of the authors that clarifies whether a fully-automated economy, government, and cultural landscape would also push out human elites (as I think I and the authors believe), or whether it's just AI-empowered dictatorship in disguise.
It's also possible that the AI property managers provide a nicer experience if the market favors renters. My point about it being "dystopian" is mostly relegated to contexts where the renter doesn't have much bargaining power.
We can see the authors as treating the posthuman future as a kind of extrapolation of capitalist industrialization. I know that Bentham's Bulldog at least has a strong appreciation for how capitalist industrialization has already been catastrophic for nonhuman animal welfare.
(Or: Why Bentham's Bulldog and John Halstead are wrong in their critique of Kulveit et al.)
Gradual Disempowerment is a 2025 paper (with a nice, dedicated website) proposing a form of existential risk from AI that goes beyond "mundane" risks like bioweapon uplift or mainline misaligned-AI-takeover scenarios. In the words of the authors:
I think it's a good paper. I, personally, am most concerned with AI's forming civilizations that are largely indifferent to humanity and simply taking power, but one of the most important observations in the field of AI safety is that human flourishing is conjunctive. Many things must all go right for us to have a good future. By contrast, catastrophe is disjunctive -- there are many different roads to ruin. Thus, I broadly support the authors' analysis and think it is worth being concerned about.
This morning, Bentham's Bulldog and John Halstead (BB&JH) wrote a fairly short piece (~2300 words) critiquing Gradual Disempowerment. I think this critique is weak, and largely misses the core arguments of the original paper.
(In this essay I'll reserve "the authors" to refer to Kulveit et al. and "BB&JH" to refer to the critics.)
Disempowerment is Bad
BB&JH:
Sure.
I mean, I actually do care about having agency as an ends in itself, but if we suppose it doesn't matter then it's not a problem to lose it if we also assume that we're being well taken care of.
BB&JH:
It's pretty obviously the authors are worried about other things besides disempowerment itself. From the core claims in the introduction (bold added):
The authors are clearly worried about a world where the "large-scale social systems (e.g. governments, economic systems)" stop being aligned with humans, even if individual AI agents remain reasonably corrigible, and this misalignment causes suffering, rather than pure loss of control. I'm not sure why BB&JH don't see this.
Full-Auto
BB&JH:
This is a mischaracterization. In particular, it elides the distinction between AIs that serve individual human principals, and AIs that are corrigible to institutions such as corporations and governments. The authors postulate agents that successfully and faithfully follow "the local specification of their goals", but that don't necessarily have the well-being of any humans in mind. They even write:
It is currently the case that non-humans can own property -- corporations being a central example. Now, the corporations are nominally owned by other entities, many of which are individual humans, but I don't think it's hard to imagine a fully automated corporation that is majority-owned by other fully automated corporations, even if the AIs themselves don't have a legal right to directly own property.
Even in the world where humans retain 100% of the shares of a fully automated economy, the AIs at any given company might be aligned to the company as a fiduciary institution, rather than to the overall preferences of the shareholders as full human beings. If humans lack the ability to oversee the activities of the company, human shareholders will not be able to keep that money-maximizing in check.
In short, the authors are arguing that the economy could become fully detached from humanity, owning and governing itself, and are not focused on a world where normal human economic activity is merely supercharged by faithful AI assistants.
Machines of Moloch
BB&JH:
False.
It's clear that the authors are assuming, as a foundation to the whole theory, that AIs will become more generally intelligent and faster thinkers than humans. And, as they explain, this is literally what increased competitive pressure looks like. I am a bit baffled by BB&JH here. Are they suggesting that the fundamental market dynamics are the same, so even at superhuman speeds and scales (such that humans cannot compete!), we should not expect a worse equilibrium?
The curse of Moloch strikes whenever the local thing that's being optimized for is not the full extent of what's good. The authors suppose that the AIs are capable of faithfully following instructions, not perfectly representing the overall interests of humanity or even the rich and subtle preferences of their human principals.
Imagine a rich land lord who would naturally come into contact with his tenants in the course of collecting rents. In doing so, he learns to care about them as human beings. One day someone falls on hard times and can't pay full rent. The land lord, caring about that person, chooses to spare that person from eviction until they recover. I claim that this is not some sentimental foolishness, but is a healthy experience for the land lord's soul, so to speak, as well as being good for the tenant.
In a landscape with sharper competitive pressures, where the land lord never even thinks about his tenants because they're handled by a fleet of AI intermediaries that maximize economic productivity, not only are the tenants subject to a potentially more dystopian land lord in practice,[1] but the land lord is also impoverished by lacking the human contact that nurtures his character.
This specific scenario is meant to be an intuition pump that we care about lots of subtle things that can get ground down in an extremely competitive environment, and is meant to be more general than just land lords or whatever. For those who want more, I recommend the SSC classics: Poor Folks Do Smile... For Now and Meditations on Moloch.
Some Points of Agreement
I'd like to end just by noting some things that I think BB&JH get right:
I think Gradual Disempowerment has many good points that BB&JH skate past, such as the impact on culture and feedback loops, but I would love to see a response by some of the authors that clarifies whether a fully-automated economy, government, and cultural landscape would also push out human elites (as I think I and the authors believe), or whether it's just AI-empowered dictatorship in disguise.
It's also possible that the AI property managers provide a nicer experience if the market favors renters. My point about it being "dystopian" is mostly relegated to contexts where the renter doesn't have much bargaining power.
We can see the authors as treating the posthuman future as a kind of extrapolation of capitalist industrialization. I know that Bentham's Bulldog at least has a strong appreciation for how capitalist industrialization has already been catastrophic for nonhuman animal welfare.
Insofar as it makes sense to see existing AI systems as moral patients.