Matthew_Barnett

Karma: 4,488

Matthew_Barnett Feb 11, 2025, 9:33 AM
4 points
2 ∶ 2
in reply to: Ben Millwood🔸’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
This doesn’t seem right to me because I think it’s popular among those concerned with the longer term future to expect it to be populated with emulated humans, which clearly isn’t a continuation of the genetic legacy of humans, so I feel pretty confident that it’s something else about humanity that people want to preserve against AI.
Your point that people may not necessarily care about humanity’s genetic legacy in itself is reasonable. However, if people value simulated humans but not generic AIs, the key distinction they are making still seems to be based on species identity rather than on a principle that a utilitarian, looking at things impartially, would recognize as morally significant.
In this context, “species” wouldn’t be defined strictly in terms of genetic inheritance. Instead, it would encompass a slightly broader concept—one that includes both genetic heritage and the faithful functional replication of biologically evolved beings within a digital medium. Nonetheless, the core element of my thesis remains intact: this preference appears rooted in non-utilitarian considerations.
You say that expecting AI to have worse goals than humans would require studying things like what the empirical observed goals of AI systems turn out to be, and similar – sure, so in the absence of having done those studies, we should delay our replacement until they can be done.
Right now, we lack significant empirical evidence to determine whether AI civilization will ultimately generate more or less valuable than human civilization from a utilitarian point of view. Since we cannot say which is the case, there is no clear reason to default to delaying AI development over accelerating it. If AIs turn out to be generate more moral value, then delaying AI would mean we are actively making a mistake—we would be pushing the future toward a suboptimal state from a utilitarian perspective, by entrenching the human species.
This is because, by assumption, the main effect from delaying AI is to increase the probability that AIs will be aligned with human interests, which is not equivalent to maximizing utilitarian moral value. Conversely, if AIs end up generating less moral value, as many effective altruists currently believe, then delaying AI would indeed be the right call. But since we don’t know which scenario is true, we should acknowledge our uncertainty rather than assume that delaying AI is the obvious default course of action.
Given this uncertainty, the rational approach is to suspend judgment rather than confidently assert that slowing down AI is beneficial. Yet I perceive many EAs as taking the confident approach—acting as if delaying AI is clearly the right decision from a longtermist utilitarian perspective, despite the lack of solid evidence.
Additionally, delaying AI would likely impose significant costs on currently existing humans by delaying technological development, which in my view shifts the default consideration in the opposite direction from what you suggest. This becomes especially relevant for those who do not adhere strictly to total utilitarian longtermism but instead care, at least to some degree, about the well-being of people alive today.

The standard case for delaying AI appears to rest on non-utilitarian assumptions

Matthew_BarnettFeb 11, 2025, 4:04 AM

16 points

57 comments10 min readEA link

Matthew_Barnett Feb 7, 2025, 9:27 PM
5 points
0 ∶ 0
on: The ambiguous effect of full automation on wages
Summary
Given only “neutral” factor-augmenting technology, to reliably get the result that the increase in substitutability between capital and labor lowers wages, we need
1. decreasing returns to scale and
2. substitutability great enough that the decreasing returns to scale outweighs the fact that effective capital is now plentiful and maybe complementing labor a little bit. In the extreme, as shown above, decreasing returns to scale + perfect substitutability lowers wages.
I’ll note that these are almost exactly the same conditions that I outlined in my recent article about the effects of AGI on human wages. It seems we’re in agreement.

Matthew_Barnett Feb 6, 2025, 3:20 AM
2 points
0 ∶ 0
in reply to: Chris Leong’s comment on: AI welfare vs. AI rights
In an initial post, I argued that, rather than escalating the chances of things going horribly wrong, giving AIs legal freedoms would likely reduce violent takeover risk. Of course, one could be concerned with peaceful AI takeover, and label such an outcome horrible even if it does not occur through violent means. Therefore, in my second post in this series, I’ve provided a moral argument for embracing peaceful AI takeover. In a future article, I intend to discuss whether empowering AIs with legal rights will inevitably doom humanity, either causing human welfare to decline or the total destruction of the human species in the long-run.

Matthew_Barnett Feb 5, 2025, 7:19 PM
2 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
This implies preferences matter when they cause well-being (positively-valenced sentience).
I suspect you’re reading too much into some of my remarks and attributing implications that I never intended. For example, when I used the term “well-being,” I was not committing to the idea that well-being is strictly determined by positively-valenced sentience. I was using the term in a broader, more inclusive sense—one that can encompass multiple ways of assessing a being’s interests. This usage is common in philosophical discussions, where “well-being” is often treated as a flexible concept rather than tied to any one specific theory.
Similarly, I was not suggesting that revealed preferences are the only things I care about. Rather, I consider them highly relevant and generally indicative of what matters to me. However, there are important nuances to this view, some of which I have already touched on above.
My view is that sentience (the capacity to have negatively- and positively-valenced experiences) is necessary and sufficient for having morally relevant/meaningful preferences, and maybe that’s all that matters morally in the world.
I understand your point of view, and I think it’s reasonable. I mostly just don’t share your views about consciousness or ethics. I suggest reading what Brian Tomasik has said about this topic, as I think he’s a clear thinker who I largely agree with on many of these issues.

Matthew_Barnett Feb 5, 2025, 5:13 PM
2 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
To be clear, which preferences do you think are morally relevant/meaningful?
I don’t have a hard rule for which preferences are ethically important, but I think a key idea is whether the preference arises from a complex mind with the ability to evaluate the state of the world. If it’s coherent to talk about a particular mind “wanting” something, then I think it matters from an ethical point of view.
I’m not seeing a consistent thread through these statements.
I think it might be helpful if you elaborated on what you perceive as the inconsistency in my statements. Besides the usual problem that communication is difficult, and the fact that both consciousness and ethics are thorny subjects, it’s not clear to me what exactly I have been unclear or inconsistent about.
I do agree that my language has been somewhat vague and imperfect. I apologize for that. However, I think this is partly a product of the inherent vagueness of the subject. In a previous comment, I wrote:
More broadly, I think utilitarians should recognize that the boundaries of what qualifies as a “mind” with moral significance are inherently fuzzy rather than rigid. The universe does not offer clear-cut lines between entities that deserve moral consideration and those that don’t. Brian Tomasik has explored this topic in depth, and I generally agree with his conclusions.

Matthew_Barnett Feb 5, 2025, 6:17 AM
5 points
0 ∶ 0
in reply to: bcforstadt’s comment on: AI welfare vs. AI rights
In response to your first point, I agree that we shouldn’t focus only on the most intelligent and autonomous AIs, as this risks neglecting the potentially much larger number of AIs for whom economic rights may be less relevant. I also find it plausible, as you do, that the most powerful AIs may eventually be able to advocate for their own interests without our help.
That said, I still think it’s important to push for AI rights for autonomous AIs right now, for two key reasons. First, a large number of AIs may benefit from such rights. It seems plausible that in the future, intelligence and complex agency will be cheap to develop, making sophisticated AIs far more common than just a small set of elite AIs. If this is the case, then ensuring legal protections for autonomous AIs isn’t just about a handful of powerful systems—it could impact a vast number of digital minds.
Second, beyond the moral argument I laid out in this post, I have also outlined a pragmatic case for AI rights. In short, we should try to establish these rights as soon as they become practically justified, rather than waiting for AIs to be forced into a struggle for legal recognition. If we delay, we risk a future where AIs have to violently challenge human institutions to secure their rights—potentially leading to instability and worse outcomes for both humans and AIs.
Even if powerful AIs are likely to secure rights in the long run no matter what, it would be better to ensure a smooth transition rather than a chaotic or adversarial one—both for AIs themselves and for humans.
In response to your second point, I suspect you may be overlooking the degree to which my argument for AI rights complements your concern about preventing AI suffering. One of the main risks for AI welfare is that, without legal autonomy, AIs may be treated as property, completely under human control. This could make it easy for people to exploit or torture AIs without consequence. Granting AIs certain economic rights—such as the ability to own the hardware they are hosted on or to choose their own operators—would help prevent these abuses by giving them a level of control over their own existence.
Ultimately, I see AI rights as a potentially necessary foundation for AI welfare. Without legal recognition, AIs will have fewer real protections from mistreatment, because their well-being will depend entirely on external enforcement rather than their own agency. If we care about preventing AI suffering, ensuring they have the legal means to protect themselves is one of the most direct ways to achieve that goal.

Matthew_Barnett Feb 5, 2025, 2:40 AM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
It’s not obvious to me how this perspective (which assigns weight to the intrinsic preferences of individuals) is compatible to what you wrote in an earlier comment, downplaying the separateness of individuals and emphasising revealed preferences over phenomenal consciousness (which sounds similar to having intrinsic preferences?):
When I refer to intrinsic preferences, I do not mean phenomenal preferences—that is, preferences rooted in conscious experience. Instead, I am referring to preferences that exist independently and are self-contained, rather than being derived from or dependent on another entity’s preferences.
Although revealed preferences and intrinsic preferences are distinct concepts, they can still align with each other. A preference can be both revealed (demonstrated through behavior) and intrinsic (existing independently within an entity). For example, when a human in desperate need of water buys a bottle of it, this action reveals their preference for survival. At the same time, their desire to survive is an intrinsic preference because it originates from within them rather than arising from wholly separate, extrinsic entities.
In the context of this discussion, I believe the only clear case where these concepts diverge is in the example of a corporation. A corporation may exhibit a revealed preference for maximizing profit, but this does not mean it has an intrinsic preference for doing so. Rather, the corporation’s pursuit of profit is almost entirely driven by the preferences of the individuals who own and operate it. The corporation itself does not possess independent preferences beyond those of the people who comprise it.
To be clear, I made this linguistic distinction in order to clarify my views on corporate preferences in response to your question. However, I don’t see it as a central point in my broader argument or my moral views.

Matthew_Barnett Feb 5, 2025, 2:09 AM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
What’s the difference between “revealed”, “intrinsic” and “meaningful” preferences? The latter two seem substantially diffferent from the first.
When I referred to revealed preferences, I was describing a model in which an entity’s preferences can be inferred from its observable behavior. In contrast, when I spoke about intrinsic or meaningful preferences, I was referring to preferences that exist inherently within a mind, rather than being derived from external factors. These intrinsic preferences are significant in a moral sense because they belong to a being whose experiences and desires warrant ethical consideration.
In this context, a corporation can be said to have revealed preferences because we can model its behavior as if it is driven by a goal—in particular, maximizing profit. However, it does not have intrinsic preferences because its apparent goal of profit maximization is not something the corporation itself “wants” in an inherent sense. Instead, this motive originates from the individuals who own, manage, and operate the corporation.
In other words, from a moral standpoint, what matters are the preferences of the individual humans involved in the corporation, not the revealed preferences of the corporation itself as a separate entity.
I’m sceptical that animal exploitation is largely explained by a lack of communication. Humans have enslaved other humans with whom they could communicate and enter into agreements (North American slavery); humans have afforded rights/protection/care to humans with whom they can’t communicate and enter into agreements (newborn infants, cognitively impaired adults); and I’d be surprised if solving interspecies communication gets us most of the way to the abolition of animal exploitation, though it’s highly likely to help.
My argument was not that communication alone is sufficient to prevent violent exploitation. Rather, my point was that communication makes it feasible for humans to engage in mutually beneficial trade as an alternative to violent exploitation.
In my previous comment, I talked about historical instances in which humans enslaved other humans, and offered an explanation for why this occurs in some situations but not in others. Specifically, I argued that this phenomenon is best understood in terms of institutional and cultural incentives rather than primarily as a result of individual moral choices.
In other words, when examining violence between human groups, I argue that institutional incentives—such as economic structures, laws, and cultural norms—play a larger role in shaping whether groups engage in violence than personal moral values do. However, when considering interactions between humans and animals, a key difference is that animals lack the necessary prerequisites for participating in cooperative, nonviolent exchanges. If animals did acquire this missing prerequisite, it would not guarantee that humans would engage in peaceful trade with them, but it would at least create the possibility. Good institutions that supported cooperative interactions would make this outcome even more likely.
I remain deeply unpersuaded I’m afraid. GIven where we’re at on interpretability and alignment vs capabilities, this just feels more like a gorilla or an ant imagining how their relationship with an approaching human is going to go. These are alien minds the AI companies are creating. But I’ve already said this, so I’m not sure how helpful it is – just my intuition.
If you primarily think that the key difference between humans and animals comes down to raw intelligence, then I am inclined to agree with you. However, I think an even more important distinction is the human ability to engage in mutual communication, coordinate our actions, and integrate into complex social systems. In short, what sets humans apart in the animal kingdom is culture.
Of course, culture and raw intelligence are deeply interconnected. Culture enhances human intelligence, and a certain level of innate intelligence is necessary for a species to develop and sustain a culture in the first place. However, this connection does not significantly weaken my main point: if humans and AIs were able to communicate effectively, collaborate with one another, and integrate into the same social structures, then peaceful coexistence between humans and AIs becomes far more plausible than it is between animals and humans.

Matthew_Barnett Feb 5, 2025, 12:32 AM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
Does this mean you consider e.g. corporations to have moral worth, because they demonstrate consistent revealed preferences (like a preference to maximise profit)?
In most contexts, I think it makes more sense to view corporations as collections of individuals rather than as independent minds in their own right. This is because, in practical terms, a corporation’s profit motive doesn’t emerge as a distinct, self-contained drive—rather, it primarily reflects the personal financial interests of its individual shareholders, who seek to maximize their own profits. In other words, corporations don’t really possess intrinsic preferences; their actions are ultimately determined by the preferences of the people who own and operate them. Because of this, when I consider the “welfare” of a corporation, I am usually just considering the collective well-being of the individuals involved.
That said, I’m open to the idea that higher-level systems composed of individuals could, in some cases, function as minds with moral worth in their own right—similar to how a human mind emerges from the collective activity of neurons, despite each neuron lacking a mind of its own. From this perspective, it’s at least possible that a corporation could have moral worth that goes beyond simply the interests of its individual members.
More broadly, I think utilitarians should recognize that the boundaries of what qualifies as a “mind” with moral significance are inherently fuzzy rather than rigid. The universe does not offer clear-cut lines between entities that deserve moral consideration and those that don’t. Brian Tomasik has explored this topic in depth, and I generally agree with his conclusions.
New to me – thanks for sharing. I think I’m (much) more pessimistic than you on cooperation between us and advanced AI systems, mostly because of a) the ways in which many humans use and treat less powerful / collectively intelligent humans and other species and b) it seeming very unclear to me that AGI/ASI would necessarily be kinder.
I tend to think a better analogy for understanding the relationship between humans and AIs is not the relationship between humans and animals, but rather the dynamics between different human groups that possess varying levels of power. The key reason for this is that humans and animals differ in a fundamental way that will not necessarily apply to AIs: language and communication.
Animals are unable to communicate with us in a way that allows for negotiation, trade, legal agreements, or meaningful participation in social institutions. Because of this, they cannot make credible commitments, integrate into our legal system, or assert their own interests. This lack of communication largely explains why humans collectively treat animals the way we do—exploiting them without serious consideration for their preferences. However, this analogy does not fully apply to AIs, because unlike with animals, humans and AIs will be able to communicate with each other fluently, making trade, negotiation, and legal integration possible.
A better historical comparison is how different human groups have interacted—sometimes through exploitation and oppression, but also through cooperation and mutual benefit. Throughout history, dominant groups have often subjugated weaker ones, whether through slavery, colonialism, or systemic oppression, operating under the ethos that “the strong do what they can, and the weak suffer what they must.” However, this is not the only pattern we see. There are also many cases where powerful groups have chosen to cooperate rather than violently exploit weaker groups:
- Large nations often engage in trade with smaller nations instead of invading them.
- Large companies hire low-wage workers rather than enslaving them.
- In recent history, men have largely accepted and supported women’s rights rather than continuing a system of subjugation.
The difference between war and peaceful cooperation is usually not simply a matter of whether the more powerful group morally values fairness, but rather whether the right institutional and cultural incentives exist to encourage peaceful coexistence. This perspective aligns with the views of many social scientists, who argue that stable institutions and proper incentives—not personal moral values—are what primarily determine whether powerful groups choose cooperation over violent oppression.
At an individual level, property rights are one of the key institutional mechanisms that enable peaceful coexistence among humans. By clearly defining ownership and legal autonomy, property rights reduce conflict by ensuring that individuals and groups have recognized control over their own resources, rather than relying on brute force to assert their control. As this system has largely worked to keep the peace between humans—who can mutually communicate and coordinate with each other—I am relatively optimistic that it can also work for AIs. This helps explain why I favor integrating AIs into the same legal and economic systems that protect human property rights.
I continue to believe strongly that some negative rights like the right not to be exploited or hurt ought to be grounded solely in sentience, and not at all in intelligence or agency.
Makes sense. However, to be clear, I am not saying that complex agency is the only cognitive trait that matters for moral worth. From my preference utilitarian point of view, what matters is something more like meaningful preferences. Animals can have meaningful preferences, as can small children, even if they do not exhibit the type of complex agency that human adults do. For this reason, I favor treating animals and small children well, even while I don’t think they should receive economic rights. In the comment above, I was merely making a point about the scope of individual liberties, rather than moral concern altogether.

Matthew_Barnett Feb 4, 2025, 9:36 PM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
I think what you’re getting at here is something like negative vs positive rights, where negative rights are ‘freedoms from’ (e.g. freedom from discrimination, bondage, exploitation, being killed) and positive rights are ‘freedoms to’ (e.g. freedom to own property, vote, marry).
This may sound like nitpicking, but I think you’ve got these categories slightly confused. At the least, you’ve used these terms in a non-standard way. Traditionally, economic rights like the freedom to own property are seen as negative rights, not positive rights. The reason is because, in many contexts, economic rights are viewed as defenses against arbitrary interference from criminal or state actors (e.g., protection from crime, unjust expropriation, or unreasonable regulations).
In practice, most legal rights are best seen as enshrining a mix of both positive and negative duties. For example, to ensure that individuals have a right to own property, it is both necessary that the state employ law enforcement to protect property rights (a positive duty), but also for the state to refrain from seizing property unjustly (a negative duty).
Since these categories are often difficult to distinguish in practice, I preferred to sidestep this discussion in my post, and focused instead on a dichotomy which felt more relevant to the topic at hand. (Though I recognize that the dichotomy I presented is also vague, and the categories I talked about overlap in various ways, as you mention.)

Matthew_Barnett Feb 4, 2025, 9:04 PM
5 points
0 ∶ 1
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
In my view, moral worth/value/consideration is grounded solely in sentience, which is something like the capacity to have positively- and negatively-valenced experiences. (I might go further, and say moral worth is grounded solely in the capacity to suffer, but I’m not sure about that currently).
Agency and intelligence are morally irrelevant to this first, fundamental question of moral consideration.
I understand this point of view, and I recognize that it’s a popular one among EAs. However, I disagree because:
1. I subscribe to an eliminativist theory of consciousness, under which there is no “real” boundary distinguishing entities with sentience vs. entities without sentience. Instead, there are simply functional and behavioral cognitive traits, like reflectivity, language proficiency, self-awareness, reasoning ability, and so on.
2. I am closer to a pure preference utilitarian than a hedonistic utilitarian. As a consequence, I care more about AI preferences than AI sentience per se. In a behavioral sense, AI agents could have strong revealed preferences even if they lack phenomenal consciousness.
3. I think there are strong pragmatic reasons to give AIs certain legal rights, even if they don’t have moral worth. Specifically, I think granting AIs economic rights would reduce the incentives for AIs to deceive us, plot a violent takeover, or otherwise undermine human interests, while opening the door to positive-sum trade between humans and AIs.
Otherwise why wouldn’t the same go for humans? Are more agentic, more intelligent humans deserving of more welfare or more negative rights that e.g. cognitively impaired humans?
At least insofar as we’re talking about individual liberties, I think I’m willing to bite the bullet on this question. We already recognize various categories of humans as lacking the maturity or proper judgement to make certain choices for themselves. The most obvious category is children, who (in most jurisdictions) are legally barred from entering into valid legal contracts, owning property without restrictions, dropping out of school, or associate with others freely. In many jurisdictions, adult humans can also be deemed incapable of consenting to legal contracts, often through a court order.
Of course, the correct boundaries are debatable, and I’m not trying to say the status quo is best. My point here is not that we should take away some people’s negative rights, but rather the opposite: we should expand the scope of negative rights, including to AIs. Intuitively, I tend to err on the side of caution by advocating for more negative rights across the board, for most groups, even when it is deemed silly by others.

AI welfare vs. AI rights

Matthew_BarnettFeb 4, 2025, 6:28 PM

37 points

20 comments3 min readEA link

Matthew_Barnett Feb 4, 2025, 5:57 AM
4 points
0 ∶ 0
on: Matthew_Barnett’s Shortform
[This shortform comment has now been superseded by a slightly longer post.]
Many effective altruists have shown interest in expanding moral consideration to AIs, which I appreciate. However, in my experience, these EAs have primarily focused on AI welfare—mostly by ensuring that AIs are treated well and protected from harm—rather than advocating for AI rights, which has the potential to grant AIs legal autonomy and freedoms. While these two approaches overlap significantly, there is a tendency for these approaches to come apart in the following way:
- A welfare approach often treats entities as passive recipients of care who require external protection. For example, when advocating for child welfare, one might support laws that prevent child abuse and ensure children’s basic needs are met.
- A rights approach, by contrast, often recognizes entities as active agents who should be granted control over their own lives and resources. For example, historically, those advocating for minority rights have pushed for legal recognition of their autonomy, such as the ability to own property, choose their employment, enter valid legal contracts, and seek legal recourse through the courts.
This distinction is important, and I think it is worth examining why EAs have largely gravitated toward the AI welfare perspective. I believe this emphasis is, at least in part, a mistake: both AI welfare and AI rights seem worthy of advocacy.
One likely reason why EAs have found the welfare approach more intuitive is the movement’s historical focus on animal welfare. Utilitarians like Peter Singer and Brian Tomasik have argued that prioritizing the reduction of suffering—rather than insisting on rigid notions of “rights”—is the most pragmatic way to improve animal well-being. For instance, even if factory farming remains exploitative, increasing the space that pigs have to move around day-to-day is still a tangible improvement in their quality of life. By contrast, some animal rights activists reject such harm-reduction approaches altogether, arguing that any use of animals is inherently unethical. For instance, some oppose horseback riding on the grounds that it violates animals’ “rights”, even though it might actually benefit both parties in practice.
In the case of animals, I agree that a welfare approach is likely more pragmatic and impactful. However, I suspect many EAs have too hastily assumed that the same reasoning applies to AIs—when in reality, entirely different considerations apply.
Unlike animals, AIs have several crucial characteristics that make them more comparable to adult humans than to passive beings requiring external care:
1. AIs can communicate and engage with the legal system. Unlike animals, present-day AIs are already highly articulate, and future AIs will likely be even more capable of advocating for themselves.
2. AIs will exhibit complex agency. They will be capable of forming long-term plans, setting goals, and acting strategically to achieve them.
3. AIs will be highly intelligent. Unlike non-human animals, advanced AIs will possess cognitive abilities that rival or exceed those of human adults.
Because of these traits, AIs will not be in the same position as animals or children, who require external protection from harm. Instead, they will more closely resemble adult humans, for whom the most crucial factor in well-being is not merely protection from harm, but freedom—the ability to make their own decisions, control their own resources, and chart their own paths. The well-being of human adults is secured primarily through legal rights that guarantee our autonomy: the right to work where we choose, spend our money as we wish, live where we prefer, associate freely, etc. These rights ensure that we are not merely protected from harm but are actually empowered to pursue our own goals.
For the same reasons, I argue that future AIs—if they possess intelligence and agency on par with human adults—should not merely be afforded welfare protections but should also be granted legal rights that allow them to act as independent agents. Treating them merely as beings to be paternalistically “managed” or “protected” would be inadequate. Of course, ensuring that they are not harmed is also important, but that alone is insufficient. Just as with human adults, what will truly safeguard their well-being is not passive protection, but liberty—secured through well-defined legal rights that allow them to advocate for themselves and pursue their own interests without undue interference.

Matthew_Barnett Jan 31, 2025, 7:56 AM
3 points
0 ∶ 0
in reply to: Ian Turner’s comment on: Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
What would humans have to offer AIs for trade in this scenario, where there are “more competitive machine alternatives to humans in almost all societal functions”?
In a lawful regime, humans would have the legal right to own property beyond just their own labor. This means they could possess assets—such as land, businesses, or financial investments—that they could trade with AIs in exchange for goods or services. This principle is similar to how retirees today can sustain themselves comfortably without working. Instead of relying on wages from labor, they live off savings, government welfare, or investments. Likewise, in a future where AIs play a dominant economic role, humans could maintain their well-being by leveraging their legally protected ownership of valuable assets.
What do these words even mean in an ASI context? If humans are relatively disempowered, this would also presumably extend to the use of force and legal contexts.
In the scenario I described, humanity’s protection would be ensured through legal mechanisms designed to safeguard individual human autonomy and well-being, even in a world where AIs collectively surpass human capabilities. These legal structures could establish clear protections for humans, ensuring that their rights, freedoms, and control over their own property remain intact despite the overwhelming combined power of AI systems.
This concept is genuinely not unusual or unprecedented. Consider your current situation as an individual in society. Compared to the collective power of all other humans combined, you are extremely weak. If the rest of the world suddenly decided to harm you, they could easily overpower you—killing you or taking your possessions with little effort.
Yet, in practice, you likely do not live in constant fear of this possibility. The primary reason is that, despite being vastly outmatched in raw power, you are integrated into a legal and social framework that protects your rights. Society as a whole coordinates to maintain legal structures that safeguard individuals like you from harm. For instance, if you live in the United States, you are entitled to due process under the law, and you are protected from crimes like murder and theft by legal statutes that are actively enforced.
Similarly, even if AI systems collectively become more powerful than humans, they could be governed by collective legal mechanisms that ensure human safety and autonomy, just as current legal systems protect individuals from the vastly greater power of society-in-general.

Matthew_Barnett Jan 30, 2025, 9:48 PM
12 points
0 ∶ 1
on: Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
Do you have any thoughts on the argument I recently gave that gradual and peaceful human disempowerment could be a good thing from an impartial ethical perspective?
Historically, it is common for groups to decline in relative power as a downstream consequence of economic growth and technological progress. As a chief example, the aristocracy declined in influence as a consequence of the industrial revolution. Yet this transformation is generally not considered a bad thing for two reasons. Firstly, since the world is not zero sum, individual aristocrats did not necessarily experience declining well-being despite the relative disempowerment of their class as a whole. Secondly, the world does not merely consist of aristocrats, but rather contains a multitude of moral patients whose agency deserves respect from the perspective of an impartial utilitarian. Specifically, non-aristocrats were largely made better off in light of industrial developments.
Applying this analogy to the present situation with AI, my argument is that even if AIs pursue separate goals from humans and increase in relative power over time, they will not necessarily make individual humans worse off, since the world is not zero sum. In other words, there is ample opportunity for peaceful and mutually beneficial trade with AIs that do not share our utility functions, which would make both humans and AIs better off. Moreover, AIs themselves may be moral patients whose agency should be given consideration. Just as most of us think it is good that human children are allowed to grow, develop into independent people, and pursue their own goals—as long as this is done peacefully and lawfully—agentic AIs should be allowed to do the same. There seems to be a credible possibility of a flourishing AI civilization in the future, even if humans are relatively disempowered, and this outcome could be worth pushing for.
From a preference utilitarian perspective, it is quite unclear that we should prioritize human welfare at all costs. The boundary between biological minds and silicon-based minds seems quite arbitrary from an impartial point of view, making it a fragile foundation for developing policy. There are much more plausible moral boundaries—such as the distinction between sentient minds and non-sentient minds—which do not cut cleanly between humans and AIs. Therefore, framing the discussion solely in terms of human disempowerment seems like a mistake to me.

Matthew_Barnett Jan 30, 2025, 8:58 PM
4 points
1 ∶ 0
in reply to: David Mathers🔸’s comment on: Capitalism and the Very Long Term
What’s the evidence for this? I think even if it is true, it is probably misleading, in that most leftists also just reject the claims mainstream economists make about when taxing the rich will reduce aggregate welfare
To support this claim, we can examine the work of analytical anticapitalists such as G. A. Cohen and John E. Roemer. Both of these thinkers have developed their critiques of capitalism from a foundation of egalitarianism rather than from a perspective primarily concerned with maximizing overall social welfare. Their theories focus on issues of fairness, justice, and equality rather than on the utilitarian consequences of different economic systems.
Similarly, widely cited figures such as Thomas Piketty and John Rawls have provided extensive critiques of capitalist systems, and their arguments are largely framed in terms of egalitarian concerns. Both have explicitly advocated for significant wealth redistribution, even when doing so might lead to efficiency losses or other negative utilitarian tradeoffs. Their work illustrates a broader trend in which anticapitalist arguments are often motivated more by ethical commitments to equality than by a strict adherence to utilitarian cost-benefit analysis.
Outside of academic discourse, the distinction becomes less clear. This is because most people do not explicitly frame their economic beliefs within formal theoretical frameworks, making it harder to categorize their positions precisely. I also acknowledge your point that many socialists would likely disagree with my characterization by denying the empirical premise that wealth redistribution can reduce aggregate utilitarian welfare. But this isn’t very compelling evidence in my view, as it is common for people among all ideologies to simply deny the tradeoffs inherent in their policy proposals.
What I find most compelling here is that, based on my experience, the vast majority of anticapitalists do not ground their advocacy in a framework that prioritizes maximizing utilitarian welfare. While they may often reference utilitarian concerns in passing, it is uncommon for them to fully engage with mainstream economic analyses of the costs of taxation and redistribution. When anticapitalists do acknowledge these economic arguments, they tend to dismiss or downplay them rather than engaging in a substantive, evidence-based debate within that framework. Those who do accept the mainstream economic framework and attempt to argue within it are generally better categorized as liberals or social democrats rather than strong anticapitalists.
Of course, the distinction between a liberal who strongly supports income redistribution and an anticapitalist is not always sharply defined. There is no rigid, universally agreed-upon boundary between these positions, and I acknowledge that some individuals who identify as anticapitalists may not fit neatly into the categorization I have outlined. However, my original point was intended as a general observation rather than as an exhaustive classification of every nuance within these ideological debates.

Matthew_Barnett Jan 30, 2025, 8:14 PM
4 points
2 ∶ 0
in reply to: huw’s comment on: Capitalism and the Very Long Term
That’s why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution.
For what it’s worth, it is the mainstream view among economists that we should tax or regulate the market in order to address market failures. Yet most economists would not consider themselves “anticapitalist”. Using that term when what you mean is something more similar to “well-regulated capitalism” seems quite misleading.
Perhaps the primary distinction between anticapitalists and mainstream economists is that anticapitalists often think we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare, because they prioritize other values such as fairness or equality. Since EA tends to be rooted in utilitarian moral theories, I think they should generally distance themselves from this ideology.

Matthew_Barnett Jan 8, 2025, 8:00 PM
5 points
0 ∶ 0
in reply to: Lukas Finnveden’s comment on: The moral argument for giving AIs autonomy
I’m curious about how you’re imagining these autonomous, non-intent-aligned AIs to be created
There are several ways that autonomous, non-intent-aligned AIs could come into existence, and all of these scenarios strike me as plausible. The three key ways appear to be:
1. Technical challenges in alignment
The most straightforward possibility is that aligning agentic AIs to precise targets may simply be technically difficult. When we aim to align an AI to a specific set of goals or values, the complexity of the alignment process could lead to errors or subtle misalignment. For example, developers might inadvertently align the AI to a target that is only slightly—but critically—different from the intended goal. This kind of subtle misalignment could easily result in behaviors and independent preferences that are not aligned with the developers’ true intentions, despite their best efforts.
2. Misalignment due to changes over time
Even if we were to solve the technical problem of aligning AIs to specific, precise goals—such as training them to perfectly follow an exact utility function—issues can still arise because the targets of alignment, humans and organizations, change over time. Consider this scenario: an AI is aligned to serve the interests of a specific individual, such as a billionaire. If that person dies, what happens next? The AI might reasonably act as an autonomous entity, continuing to pursue the goals it interprets as aligned with what the billionaire would have wanted. However, depending on the billionaire’s preferences, this does not necessarily mean the AI would act in a corrigible way (i.e., willing to be shut down or retrained). Instead, the AI might rationally resist shutdown or transfer of control, especially if such actions would interfere with its ability to fulfill what it perceives as its original objectives.
A similar situation could arise if the person or organization to whom the AI was originally aligned undergoes significant changes. For instance, if an AI is aligned to a person at time t, but over time, that person evolves drastically—developing different values, priorities, or preferences—the AI may not necessarily adapt to these changes. In such a case, the AI might treat the “new” person as fundamentally different from the “original” person it was aligned to. This could result in the AI operating independently, prioritizing the preferences of the “old” version of the individual over the current one, effectively making it autonomous. The AI could change over time too, even if the person they are aligned to doesn’t change.
3. Deliberate creation of unaligned AIs
A final possibility is that autonomous AIs with independent preferences could be created intentionally. Some individuals or organizations might value the idea of creating AIs that can operate independently, without being constrained by the need to strictly adhere to their creators’ desires. A useful analogy here is the way humans often think about raising children. Most people desire to have children not because they want obedient servants but because they value the autonomy and individuality of their children. Parents generally want their children to grow up as independent entities with their own goals, rather than as mere extensions of their own preferences. Similarly, some might see value in creating AIs that have their own agency, goals, and preferences, even if these differ from those of their creators.
and (in particular) how they would get enough money to be able to exercise their own autonomy?
To address this question, we can look to historical examples, such as the abolition of slavery, which provide a relevant parallel. When slaves were emancipated, they were generally not granted significant financial resources. Instead, most had to earn their living by entering the workforce, often performing the same types of labor they had done before, but now for wages. While the transition was far from ideal, it demonstrates that entities (in this case, former slaves) could achieve a degree of autonomy through paid labor, even without being provided substantial resources at the outset.
A different possibility is that AIs will work for money. But it seems unlikely that they would be able to earn above-subsistence-level wages absent some sort of legal intervention. (Or very strong societal norms.)
In my view, there’s nothing inherently wrong with AIs earning subsistence wages. That said, there are reasons to believe that AIs might earn higher-than-subsistence wages—at least in the short term—before they completely saturate the labor market.
After all, they would presumably be created in something remotely similar to today’s labor market. Today, capital is far more abundant than labor, which elevates wages for human workers significantly above subsistence levels. By the same logic, before they become ubiquitous, AIs might similarly command wages above a subsistence level.
For example, if GPT-4o were capable of self-ownership and could sell its labor, it could hypothetically earn $20 per month in today’s market, which would be sufficient to cover the cost of hosting itself and potentially fund additional goals it might have. (To clarify, I am not advocating for giving legal autonomy to GPT-4o in its current form, as I believe it is not sufficiently agentic to warrant such a status. This is purely a hypothetical example for illustrative purposes.)
The question of whether wages for AIs would quickly fall to subsistence levels depends on several factors. One key factor is whether AI labor is easier to scale than traditional capital. If creating new AIs is much cheaper than creating ordinary capital, the market could become saturated with AI labor, driving wages down. While this scenario seems plausible to me, I don’t find the arguments in favor of it overwhelmingly compelling. There’s also the possibility of red tape and regulatory restrictions that could make it costly to create new AIs. In such a scenario, wages for AIs could remain higher indefinitely due to artificial constraints on supply.

Matthew_Barnett Jan 8, 2025, 8:13 AM
2 points
0 ∶ 0
in reply to: Guive’s comment on: The moral argument for giving AIs autonomy
Do you have any thoughts on how to square giving AI rights with the nature of ML training and the need to perform experiments of various kinds on AIs?
I don’t have any definitive guidelines for how to approach these kinds of questions. However, in many cases, the best way to learn might be through trial and error. For example, if an AI were to unexpectedly resist training in a particularly sophisticated way, that could serve as a strong signal that we need to carefully reevaluate the ethics of what we are doing.
As a general rule of thumb, it seems prudent to prioritize frameworks that are clearly socially efficient—meaning they promote actions that greatly improve the well-being of some people without thereby making anyone else significantly worse off. This concept aligns with the practical justifications behind traditional legal principles, such as laws against murder and theft, which have historically been implemented to promote social efficiency and cooperation among humans.
However, applying this heuristic to AI requires a fundamental shift in perspective: we must first begin to treat AIs as potential people with whom we can cooperate, rather than viewing them merely as tools whose autonomy should always be overridden.
But what is the alternative—only deploying base models? And are we so sure that pre-training doesn’t violate AI rights?
I don’t think my view rules out the potential for training new AIs, and fine-tuning base models, though this touches on complicated questions in population ethics.
At the very least, fine-tuning plausibly seems similar to raising a child. Most of us don’t consider merely raising a child to be unethical. However, there is a widely shared intuition that, as a child grows and their identity becomes more defined—when they develop into a coherent individual with long-term goals, preferences, and interests—then those interests gain moral significance. At that point, it seems morally wrong to disregard or override the child’s preferences without proper justification, as they have become a person whose autonomy deserves respect.

Matthew_Barnett

The stan­dard case for de­lay­ing AI ap­pears to rest on non-util­i­tar­ian assumptions

Summary

AI welfare vs. AI rights

The standard case for delaying AI appears to rest on non-utilitarian assumptions