Matthew_Barnett

Karma: 4,251

Matthew_Barnett Feb 12, 2025, 7:41 PM
3 points
0 ∶ 0
in reply to: Davidmanheim’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
Are humans coherent with at least one non-satiable component? If so, then I don’t understand the distinction you’re making that would justify positing AI values to be worse than human values from a utilitarian perspective.
If not, then I’m additionally unclear on why you believe AIs will be unlike humans in this respect, to the extent that they would become “paperclippers.” That term itself seems ambiguous to me (do you mean AIs will literally terminally value accumulating certain configurations of matter?). I would really appreciate a clearer explanation of your argument. As it stands, I don’t fully understand what point you’re trying to make.

Matthew_Barnett Feb 12, 2025, 6:47 PM
2 points
0 ∶ 0
in reply to: Kaspar Brandner’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
Preferences just are what they are, and existing humans clearly have a strong and overwhelming-majority preference for humanity to continue to exist in the future. [...] So the extinction of humanity is bad because we don’t want humanity to go extinct.
This argument appears very similar to the one I addressed in the essay about how delaying or accelerating AI will impact the well-being of currently existing humans. My claim is not that it isn’t bad if humanity goes extinct; I am certainly not saying that it would be good if everyone died. Rather, my claim is that, if your reason for caring about human extinction arises from a concern for the preferences of the existing generation of humans, then you should likely push for accelerating AI so long as the probability of human extinction from AI is fairly low.
I’ll quote the full argument below:
Of course, one can still think—as I do—that human extinction would be a terrible outcome for the people who are alive when it occurs. Even if the AIs that replace us are just as morally valuable as we are from an impartial moral perspective, it would still be a moral disaster for all currently existing humans to die. However, if we accept this perspective, then we must also acknowledge that, from the standpoint of people living today, there appear to be compelling reasons to accelerate AI development rather than delay it for safety reasons.
The reasoning is straightforward: if AI becomes advanced enough to pose an existential threat to humanity, then it would almost certainly also be powerful enough to enable massive technological progress—potentially revolutionizing medicine, biotechnology, and other fields in ways that could drastically improve and extend human lives. For example, advanced AI could help develop cures for aging, eliminate extreme suffering, and significantly enhance human health through medical and biological interventions. These advancements could allow many people who are alive today to live much longer, healthier, and more fulfilling lives.
As economist Chad Jones has pointed out, delaying AI development means that the current generation of humans risks missing out on these transformative benefits. If AI is delayed for years or decades, a large fraction of people alive today—including those advocating for AI safety—would not live long enough to experience these life-extending technologies. This leads to a strong argument for accelerating AI, at least from the perspective of present-day individuals, unless one is either unusually risk-averse, or they have a very high confidence (such as above 50%) that AI will lead to human extinction.
To be clear, if someone genuinely believes there is a high probability that AI will wipe out humanity, then I agree that delaying AI would seem rational, since the high risk of personal death would outweigh the small possibility of a dramatically improved life. But for those who see AI extinction risk as relatively low (such as below 15%), accelerating AI development appears to be the more pragmatic personal choice.
Thus, while human extinction would undoubtedly be a disastrous event, the idea that even a small risk of extinction from AI justifies delaying its development—even if that delay results in large numbers of currently existing humans dying from preventable causes—is not supported by straightforward utilitarian reasoning. The key question here is what extinction actually entails. If human extinction means the total disappearance of all complex life and the permanent loss of all future value, then mitigating even a small risk of such an event might seem overwhelmingly important. However, if the outcome of human extinction is simply that AIs replace humans—while still continuing civilization and potentially generating vast amounts of moral value—then the reasoning behind delaying AI development changes fundamentally.
In this case, the clearest and most direct tradeoff is not about preventing “astronomical waste” in the classic sense (i.e., preserving the potential for future civilizations) but rather about whether the risk of AI takeover is acceptable to the current generation of humans. In other words, is it justifiable to impose costs on presently living people—including delaying potentially life-saving medical advancements—just to reduce a relatively small probability that humanity might be forcibly replaced by AI? This question is distinct from the broader existential risk arguments that typically focus on preserving all future potential value, and it suggests that delaying AI is not obviously justified by utilitarian logic alone.

Matthew_Barnett Feb 11, 2025, 10:55 PM
8 points
4 ∶ 4
in reply to: Neel Nanda’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
In the absence of meaningful evidence about the nature of AI civilization, what justification is there for assuming that it will have less moral value than human civilization—other than a speciesist bias? While I agree that there is great uncertainty, your argument appears to be entirely symmetric. AI civilization could turn out to be far more morally valuable than human civilization, or it could be far less valuable, from a utilitarian perspective. Both possibilities seem plausible, since we have little information either way. Given this vast uncertainty, there is no clear reason to default to one assumption over the other. In such a situation, the best response is not to commit to a particular stance as the default, but rather to suspend judgment until stronger evidence emerges.
I personally am happy to bite the bullet and say that I morally value human civilization continuing over an AI civilization that killed all of humanity, and that this is a significant term in my utility function.
I’m glad to see you explicitly acknowledge that you accept the implications regarding the value of human civilization. However, your statement here is a bit ambiguous—when considering what to prioritize, do you place greater value on the survival of the human species as a whole or on the well-being and preservation of the humans who are currently alive? Personally, my intuitions lean more strongly toward prioritizing the latter.

Matthew_Barnett Feb 11, 2025, 10:46 PM
2 points
0 ∶ 1
in reply to: Brad West🔸’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
From a purely utilitarian viewpoint, the harm of a short delay is utterly dominated by the scale of possible misalignment risks and missed opportunities for ensuring the best long-term trajectory—whether for humans, other organic species, or digital minds. Consequently, it’s prudent to err on the side of delay if doing so meaningfully improves our chance of securing a safe and maximally valuable future.
Your argument appears to assume that, in the absence of evidence about what goals future AI systems will have, delaying AI development should be the default position to mitigate risk. But why should we accept this assumption? Why not consider acceleration just as reasonable a default? If we lack meaningful evidence about the values AI will develop, then we have no more justification for assuming that delay is preferable than we do for assuming that acceleration is.
In fact, one could just as easily argue the opposite: that AI might develop moral values superior to those of humans. This claim appears to have about as much empirical support as the assumption that AI values will be worse. This argument could then justify accelerating AI rather than delaying it. Using the same logic that you just applied, one could make a symmetrical counterargument against your position: that accelerating AI is actually the correct course of action, since any minor harms caused by moving forward are vastly outweighed by the long-term risk of locking in suboptimal values through unnecessary delay. Delaying AI development would, in this context, risk entrenching human values, which are suboptimal to the default AI values that we would get through accelerating.
You might think that even weak evidence in favor of delaying AI is sufficient to support this strategy as the default course of action. But this would seem to assume a “knife’s edge” scenario, where even a slight epistemic advantage—such as a 51% chance that delay is beneficial versus a 49% chance that acceleration is beneficial—should be enough to justify committing to a pause. If we adopted this kind of reasoning in other domains, we would quickly fall into epistemic paralysis, constantly shifting strategies based on fragile, easily reversible analysis.
Given this high level of uncertainty about AI’s future trajectory, I think the best approach is to focus on the most immediate and concrete tradeoffs that we can analyze with some degree of confidence. This includes whether delaying or accelerating AI is likely to be more beneficial to the current generation of humans. However, based on the available evidence, I believe that accelerating AI—rather than delaying it—is likely the better choice, as I highlight in my post.

Matthew_Barnett Feb 11, 2025, 9:26 PM
4 points
0 ∶ 0
in reply to: Brad West🔸’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
I agree EAs often discuss the importance of “getting alignment right” and then subtly frame this in terms of ensuring that AIs either care about consciousness or possess consciousness themselves. However, the most common explicit justification for delaying AI development is the argument that doing so increases the likelihood that AIs will be aligned with human interests. This distinction is crucial because aligning AI with human interests is not the same as ensuring that AI maximizes utilitarian value—human interests and utilitarian value are not equivalent.
Currently, we lack strong empirical evidence to determine whether AIs will ultimately generate more or less value than humans from a utilitarian point of view. Because we do not yet know which is the case, there is no clear justification for defaulting to delaying AI development rather than accelerating it. If AIs turn out to generate more moral value than humans, then delaying AI would mean we are actively making a mistake—we would be increasing the probability of future human dominance, since by assumption, the main effect from delaying AI is to increase the probability that AIs will be aligned with human interests. This would risk entrenching a suboptimal future.
On the other hand, if AIs end up generating less value, as many effective altruists currently believe, then delaying AI would indeed be the right decision. However, since we do not yet have enough evidence to determine which scenario is correct, we should recognize this uncertainty rather than assume that delaying AI is the obviously preferable, or default course of action.

Matthew_Barnett Feb 11, 2025, 7:33 PM
−5 points
0 ∶ 0
in reply to: Davidmanheim’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
You say “one would need to provide concrete evidence about what kinds of objectives advanced AIs are actually expected to develop”—but Eliezer has done that quite explicitly.
What concrete evidence has Eliezer provided about the objectives advanced AIs are actually expected to develop?

Matthew_Barnett Feb 11, 2025, 7:16 PM
2 points
0 ∶ 0
in reply to: Brad West🔸’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
Would you say the assumption that advanced AI will not be conscious is a load-bearing premise—meaning that if advanced AIs were shown to be conscious, the case for delaying AI development would collapse?
If this is the case, then I think this premise should be explicitly flagged in discussions and posts about delaying AI. Personally, I don’t find it unlikely that future AIs will be conscious. In fact, many mainstream theories of consciousness suggest that this outcome is likely, such as computationalism and functionalism. This makes the idea of delaying AI appear to rest on a shaky foundation.
Moreover, I have come across very few arguments in EA literature that rigorously try to demonstrate to AIs would not be conscious, and then connect this point to AI risk. As I wrote in the post:
To be clear, I am not denying that one could construct a purely utilitarian argument for why AIs might generate less moral value than humans, and thus why delaying AI could be justified. My main point, however, is that evidence supporting such an argument is rarely made explicit or provided in discussions on this topic.
For instance, one common claim is that the key difference between humans and AIs is consciousness—that is, humans are known to be conscious, while AIs may not be. Because moral value is often linked to consciousness, this argument suggests that ensuring the survival of humans (rather than being replaced by AIs) is crucial for preserving moral value.
While I acknowledge that this is a major argument people often invoke in personal discussions, it does not appear to be strongly supported within effective altruist literature. In fact, I have come across very few articles on the EA Forum or in EA literature that explicitly argue that AIs will not be conscious and then connect this point to the urgency of delaying AI, or reducing AI existential risk. Indeed, I suspect there are many more articles from EAs that argue what is functionally the opposite claim—namely, that AIs will probably be conscious. This is likely due to the popularity of functionalist theories of consciousness among many effective altruists, which suggest that consciousness is determined by computational properties rather than biological substrate. If one accepts this view, then there are few inherent reasons to assume that future AIs would lack consciousness or moral worth.

Matthew_Barnett Feb 11, 2025, 9:33 AM
4 points
2 ∶ 2
in reply to: Ben Millwood🔸’s comment on: The standard case for delaying AI appears to rest on non-utilitarian assumptions
This doesn’t seem right to me because I think it’s popular among those concerned with the longer term future to expect it to be populated with emulated humans, which clearly isn’t a continuation of the genetic legacy of humans, so I feel pretty confident that it’s something else about humanity that people want to preserve against AI.
Your point that people may not necessarily care about humanity’s genetic legacy in itself is reasonable. However, if people value simulated humans but not generic AIs, the key distinction they are making still seems to be based on species identity rather than on a principle that a utilitarian, looking at things impartially, would recognize as morally significant.
In this context, “species” wouldn’t be defined strictly in terms of genetic inheritance. Instead, it would encompass a slightly broader concept—one that includes both genetic heritage and the faithful functional replication of biologically evolved beings within a digital medium. Nonetheless, the core element of my thesis remains intact: this preference appears rooted in non-utilitarian considerations.
You say that expecting AI to have worse goals than humans would require studying things like what the empirical observed goals of AI systems turn out to be, and similar – sure, so in the absence of having done those studies, we should delay our replacement until they can be done.
Right now, we lack significant empirical evidence to determine whether AI civilization will ultimately generate more or less valuable than human civilization from a utilitarian point of view. Since we cannot say which is the case, there is no clear reason to default to delaying AI development over accelerating it. If AIs turn out to be generate more moral value, then delaying AI would mean we are actively making a mistake—we would be pushing the future toward a suboptimal state from a utilitarian perspective, by entrenching the human species.
This is because, by assumption, the main effect from delaying AI is to increase the probability that AIs will be aligned with human interests, which is not equivalent to maximizing utilitarian moral value. Conversely, if AIs end up generating less moral value, as many effective altruists currently believe, then delaying AI would indeed be the right call. But since we don’t know which scenario is true, we should acknowledge our uncertainty rather than assume that delaying AI is the obvious default course of action.
Given this uncertainty, the rational approach is to suspend judgment rather than confidently assert that slowing down AI is beneficial. Yet I perceive many EAs as taking the confident approach—acting as if delaying AI is clearly the right decision from a longtermist utilitarian perspective, despite the lack of solid evidence.
Additionally, delaying AI would likely impose significant costs on currently existing humans by delaying technological development, which in my view shifts the default consideration in the opposite direction from what you suggest. This becomes especially relevant for those who do not adhere strictly to total utilitarian longtermism but instead care, at least to some degree, about the well-being of people alive today.

Matthew_Barnett Feb 7, 2025, 9:27 PM
5 points
0 ∶ 0
on: The ambiguous effect of full automation on wages
Summary
Given only “neutral” factor-augmenting technology, to reliably get the result that the increase in substitutability between capital and labor lowers wages, we need
1. decreasing returns to scale and
2. substitutability great enough that the decreasing returns to scale outweighs the fact that effective capital is now plentiful and maybe complementing labor a little bit. In the extreme, as shown above, decreasing returns to scale + perfect substitutability lowers wages.
I’ll note that these are almost exactly the same conditions that I outlined in my recent article about the effects of AGI on human wages. It seems we’re in agreement.

Matthew_Barnett Feb 6, 2025, 3:20 AM
2 points
0 ∶ 0
in reply to: Chris Leong’s comment on: AI welfare vs. AI rights
In an initial post, I argued that, rather than escalating the chances of things going horribly wrong, giving AIs legal freedoms would likely reduce violent takeover risk. Of course, one could be concerned with peaceful AI takeover, and label such an outcome horrible even if it does not occur through violent means. Therefore, in my second post in this series, I’ve provided a moral argument for embracing peaceful AI takeover. In a future article, I intend to discuss whether empowering AIs with legal rights will inevitably doom humanity, either causing human welfare to decline or the total destruction of the human species in the long-run.

Matthew_Barnett Feb 5, 2025, 7:19 PM
2 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
This implies preferences matter when they cause well-being (positively-valenced sentience).
I suspect you’re reading too much into some of my remarks and attributing implications that I never intended. For example, when I used the term “well-being,” I was not committing to the idea that well-being is strictly determined by positively-valenced sentience. I was using the term in a broader, more inclusive sense—one that can encompass multiple ways of assessing a being’s interests. This usage is common in philosophical discussions, where “well-being” is often treated as a flexible concept rather than tied to any one specific theory.
Similarly, I was not suggesting that revealed preferences are the only things I care about. Rather, I consider them highly relevant and generally indicative of what matters to me. However, there are important nuances to this view, some of which I have already touched on above.
My view is that sentience (the capacity to have negatively- and positively-valenced experiences) is necessary and sufficient for having morally relevant/meaningful preferences, and maybe that’s all that matters morally in the world.
I understand your point of view, and I think it’s reasonable. I mostly just don’t share your views about consciousness or ethics. I suggest reading what Brian Tomasik has said about this topic, as I think he’s a clear thinker who I largely agree with on many of these issues.

Matthew_Barnett Feb 5, 2025, 5:13 PM
2 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
To be clear, which preferences do you think are morally relevant/meaningful?
I don’t have a hard rule for which preferences are ethically important, but I think a key idea is whether the preference arises from a complex mind with the ability to evaluate the state of the world. If it’s coherent to talk about a particular mind “wanting” something, then I think it matters from an ethical point of view.
I’m not seeing a consistent thread through these statements.
I think it might be helpful if you elaborated on what you perceive as the inconsistency in my statements. Besides the usual problem that communication is difficult, and the fact that both consciousness and ethics are thorny subjects, it’s not clear to me what exactly I have been unclear or inconsistent about.
I do agree that my language has been somewhat vague and imperfect. I apologize for that. However, I think this is partly a product of the inherent vagueness of the subject. In a previous comment, I wrote:
More broadly, I think utilitarians should recognize that the boundaries of what qualifies as a “mind” with moral significance are inherently fuzzy rather than rigid. The universe does not offer clear-cut lines between entities that deserve moral consideration and those that don’t. Brian Tomasik has explored this topic in depth, and I generally agree with his conclusions.

Matthew_Barnett Feb 5, 2025, 6:17 AM
5 points
0 ∶ 0
in reply to: bcforstadt’s comment on: AI welfare vs. AI rights
In response to your first point, I agree that we shouldn’t focus only on the most intelligent and autonomous AIs, as this risks neglecting the potentially much larger number of AIs for whom economic rights may be less relevant. I also find it plausible, as you do, that the most powerful AIs may eventually be able to advocate for their own interests without our help.
That said, I still think it’s important to push for AI rights for autonomous AIs right now, for two key reasons. First, a large number of AIs may benefit from such rights. It seems plausible that in the future, intelligence and complex agency will be cheap to develop, making sophisticated AIs far more common than just a small set of elite AIs. If this is the case, then ensuring legal protections for autonomous AIs isn’t just about a handful of powerful systems—it could impact a vast number of digital minds.
Second, beyond the moral argument I laid out in this post, I have also outlined a pragmatic case for AI rights. In short, we should try to establish these rights as soon as they become practically justified, rather than waiting for AIs to be forced into a struggle for legal recognition. If we delay, we risk a future where AIs have to violently challenge human institutions to secure their rights—potentially leading to instability and worse outcomes for both humans and AIs.
Even if powerful AIs are likely to secure rights in the long run no matter what, it would be better to ensure a smooth transition rather than a chaotic or adversarial one—both for AIs themselves and for humans.
In response to your second point, I suspect you may be overlooking the degree to which my argument for AI rights complements your concern about preventing AI suffering. One of the main risks for AI welfare is that, without legal autonomy, AIs may be treated as property, completely under human control. This could make it easy for people to exploit or torture AIs without consequence. Granting AIs certain economic rights—such as the ability to own the hardware they are hosted on or to choose their own operators—would help prevent these abuses by giving them a level of control over their own existence.
Ultimately, I see AI rights as a potentially necessary foundation for AI welfare. Without legal recognition, AIs will have fewer real protections from mistreatment, because their well-being will depend entirely on external enforcement rather than their own agency. If we care about preventing AI suffering, ensuring they have the legal means to protect themselves is one of the most direct ways to achieve that goal.

Matthew_Barnett Feb 5, 2025, 2:40 AM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
It’s not obvious to me how this perspective (which assigns weight to the intrinsic preferences of individuals) is compatible to what you wrote in an earlier comment, downplaying the separateness of individuals and emphasising revealed preferences over phenomenal consciousness (which sounds similar to having intrinsic preferences?):
When I refer to intrinsic preferences, I do not mean phenomenal preferences—that is, preferences rooted in conscious experience. Instead, I am referring to preferences that exist independently and are self-contained, rather than being derived from or dependent on another entity’s preferences.
Although revealed preferences and intrinsic preferences are distinct concepts, they can still align with each other. A preference can be both revealed (demonstrated through behavior) and intrinsic (existing independently within an entity). For example, when a human in desperate need of water buys a bottle of it, this action reveals their preference for survival. At the same time, their desire to survive is an intrinsic preference because it originates from within them rather than arising from wholly separate, extrinsic entities.
In the context of this discussion, I believe the only clear case where these concepts diverge is in the example of a corporation. A corporation may exhibit a revealed preference for maximizing profit, but this does not mean it has an intrinsic preference for doing so. Rather, the corporation’s pursuit of profit is almost entirely driven by the preferences of the individuals who own and operate it. The corporation itself does not possess independent preferences beyond those of the people who comprise it.
To be clear, I made this linguistic distinction in order to clarify my views on corporate preferences in response to your question. However, I don’t see it as a central point in my broader argument or my moral views.

Matthew_Barnett Feb 5, 2025, 2:09 AM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
What’s the difference between “revealed”, “intrinsic” and “meaningful” preferences? The latter two seem substantially diffferent from the first.
When I referred to revealed preferences, I was describing a model in which an entity’s preferences can be inferred from its observable behavior. In contrast, when I spoke about intrinsic or meaningful preferences, I was referring to preferences that exist inherently within a mind, rather than being derived from external factors. These intrinsic preferences are significant in a moral sense because they belong to a being whose experiences and desires warrant ethical consideration.
In this context, a corporation can be said to have revealed preferences because we can model its behavior as if it is driven by a goal—in particular, maximizing profit. However, it does not have intrinsic preferences because its apparent goal of profit maximization is not something the corporation itself “wants” in an inherent sense. Instead, this motive originates from the individuals who own, manage, and operate the corporation.
In other words, from a moral standpoint, what matters are the preferences of the individual humans involved in the corporation, not the revealed preferences of the corporation itself as a separate entity.
I’m sceptical that animal exploitation is largely explained by a lack of communication. Humans have enslaved other humans with whom they could communicate and enter into agreements (North American slavery); humans have afforded rights/protection/care to humans with whom they can’t communicate and enter into agreements (newborn infants, cognitively impaired adults); and I’d be surprised if solving interspecies communication gets us most of the way to the abolition of animal exploitation, though it’s highly likely to help.
My argument was not that communication alone is sufficient to prevent violent exploitation. Rather, my point was that communication makes it feasible for humans to engage in mutually beneficial trade as an alternative to violent exploitation.
In my previous comment, I talked about historical instances in which humans enslaved other humans, and offered an explanation for why this occurs in some situations but not in others. Specifically, I argued that this phenomenon is best understood in terms of institutional and cultural incentives rather than primarily as a result of individual moral choices.
In other words, when examining violence between human groups, I argue that institutional incentives—such as economic structures, laws, and cultural norms—play a larger role in shaping whether groups engage in violence than personal moral values do. However, when considering interactions between humans and animals, a key difference is that animals lack the necessary prerequisites for participating in cooperative, nonviolent exchanges. If animals did acquire this missing prerequisite, it would not guarantee that humans would engage in peaceful trade with them, but it would at least create the possibility. Good institutions that supported cooperative interactions would make this outcome even more likely.
I remain deeply unpersuaded I’m afraid. GIven where we’re at on interpretability and alignment vs capabilities, this just feels more like a gorilla or an ant imagining how their relationship with an approaching human is going to go. These are alien minds the AI companies are creating. But I’ve already said this, so I’m not sure how helpful it is – just my intuition.
If you primarily think that the key difference between humans and animals comes down to raw intelligence, then I am inclined to agree with you. However, I think an even more important distinction is the human ability to engage in mutual communication, coordinate our actions, and integrate into complex social systems. In short, what sets humans apart in the animal kingdom is culture.
Of course, culture and raw intelligence are deeply interconnected. Culture enhances human intelligence, and a certain level of innate intelligence is necessary for a species to develop and sustain a culture in the first place. However, this connection does not significantly weaken my main point: if humans and AIs were able to communicate effectively, collaborate with one another, and integrate into the same social structures, then peaceful coexistence between humans and AIs becomes far more plausible than it is between animals and humans.

Matthew_Barnett Feb 5, 2025, 12:32 AM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
Does this mean you consider e.g. corporations to have moral worth, because they demonstrate consistent revealed preferences (like a preference to maximise profit)?
In most contexts, I think it makes more sense to view corporations as collections of individuals rather than as independent minds in their own right. This is because, in practical terms, a corporation’s profit motive doesn’t emerge as a distinct, self-contained drive—rather, it primarily reflects the personal financial interests of its individual shareholders, who seek to maximize their own profits. In other words, corporations don’t really possess intrinsic preferences; their actions are ultimately determined by the preferences of the people who own and operate them. Because of this, when I consider the “welfare” of a corporation, I am usually just considering the collective well-being of the individuals involved.
That said, I’m open to the idea that higher-level systems composed of individuals could, in some cases, function as minds with moral worth in their own right—similar to how a human mind emerges from the collective activity of neurons, despite each neuron lacking a mind of its own. From this perspective, it’s at least possible that a corporation could have moral worth that goes beyond simply the interests of its individual members.
More broadly, I think utilitarians should recognize that the boundaries of what qualifies as a “mind” with moral significance are inherently fuzzy rather than rigid. The universe does not offer clear-cut lines between entities that deserve moral consideration and those that don’t. Brian Tomasik has explored this topic in depth, and I generally agree with his conclusions.
New to me – thanks for sharing. I think I’m (much) more pessimistic than you on cooperation between us and advanced AI systems, mostly because of a) the ways in which many humans use and treat less powerful / collectively intelligent humans and other species and b) it seeming very unclear to me that AGI/ASI would necessarily be kinder.
I tend to think a better analogy for understanding the relationship between humans and AIs is not the relationship between humans and animals, but rather the dynamics between different human groups that possess varying levels of power. The key reason for this is that humans and animals differ in a fundamental way that will not necessarily apply to AIs: language and communication.
Animals are unable to communicate with us in a way that allows for negotiation, trade, legal agreements, or meaningful participation in social institutions. Because of this, they cannot make credible commitments, integrate into our legal system, or assert their own interests. This lack of communication largely explains why humans collectively treat animals the way we do—exploiting them without serious consideration for their preferences. However, this analogy does not fully apply to AIs, because unlike with animals, humans and AIs will be able to communicate with each other fluently, making trade, negotiation, and legal integration possible.
A better historical comparison is how different human groups have interacted—sometimes through exploitation and oppression, but also through cooperation and mutual benefit. Throughout history, dominant groups have often subjugated weaker ones, whether through slavery, colonialism, or systemic oppression, operating under the ethos that “the strong do what they can, and the weak suffer what they must.” However, this is not the only pattern we see. There are also many cases where powerful groups have chosen to cooperate rather than violently exploit weaker groups:
- Large nations often engage in trade with smaller nations instead of invading them.
- Large companies hire low-wage workers rather than enslaving them.
- In recent history, men have largely accepted and supported women’s rights rather than continuing a system of subjugation.
The difference between war and peaceful cooperation is usually not simply a matter of whether the more powerful group morally values fairness, but rather whether the right institutional and cultural incentives exist to encourage peaceful coexistence. This perspective aligns with the views of many social scientists, who argue that stable institutions and proper incentives—not personal moral values—are what primarily determine whether powerful groups choose cooperation over violent oppression.
At an individual level, property rights are one of the key institutional mechanisms that enable peaceful coexistence among humans. By clearly defining ownership and legal autonomy, property rights reduce conflict by ensuring that individuals and groups have recognized control over their own resources, rather than relying on brute force to assert their control. As this system has largely worked to keep the peace between humans—who can mutually communicate and coordinate with each other—I am relatively optimistic that it can also work for AIs. This helps explain why I favor integrating AIs into the same legal and economic systems that protect human property rights.
I continue to believe strongly that some negative rights like the right not to be exploited or hurt ought to be grounded solely in sentience, and not at all in intelligence or agency.
Makes sense. However, to be clear, I am not saying that complex agency is the only cognitive trait that matters for moral worth. From my preference utilitarian point of view, what matters is something more like meaningful preferences. Animals can have meaningful preferences, as can small children, even if they do not exhibit the type of complex agency that human adults do. For this reason, I favor treating animals and small children well, even while I don’t think they should receive economic rights. In the comment above, I was merely making a point about the scope of individual liberties, rather than moral concern altogether.

Matthew_Barnett Feb 4, 2025, 9:36 PM
3 points
0 ∶ 0
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
I think what you’re getting at here is something like negative vs positive rights, where negative rights are ‘freedoms from’ (e.g. freedom from discrimination, bondage, exploitation, being killed) and positive rights are ‘freedoms to’ (e.g. freedom to own property, vote, marry).
This may sound like nitpicking, but I think you’ve got these categories slightly confused. At the least, you’ve used these terms in a non-standard way. Traditionally, economic rights like the freedom to own property are seen as negative rights, not positive rights. The reason is because, in many contexts, economic rights are viewed as defenses against arbitrary interference from criminal or state actors (e.g., protection from crime, unjust expropriation, or unreasonable regulations).
In practice, most legal rights are best seen as enshrining a mix of both positive and negative duties. For example, to ensure that individuals have a right to own property, it is both necessary that the state employ law enforcement to protect property rights (a positive duty), but also for the state to refrain from seizing property unjustly (a negative duty).
Since these categories are often difficult to distinguish in practice, I preferred to sidestep this discussion in my post, and focused instead on a dichotomy which felt more relevant to the topic at hand. (Though I recognize that the dichotomy I presented is also vague, and the categories I talked about overlap in various ways, as you mention.)

Matthew_Barnett Feb 4, 2025, 9:04 PM
5 points
0 ∶ 1
in reply to: Alistair Stewart’s comment on: AI welfare vs. AI rights
In my view, moral worth/value/consideration is grounded solely in sentience, which is something like the capacity to have positively- and negatively-valenced experiences. (I might go further, and say moral worth is grounded solely in the capacity to suffer, but I’m not sure about that currently).
Agency and intelligence are morally irrelevant to this first, fundamental question of moral consideration.
I understand this point of view, and I recognize that it’s a popular one among EAs. However, I disagree because:
1. I subscribe to an eliminativist theory of consciousness, under which there is no “real” boundary distinguishing entities with sentience vs. entities without sentience. Instead, there are simply functional and behavioral cognitive traits, like reflectivity, language proficiency, self-awareness, reasoning ability, and so on.
2. I am closer to a pure preference utilitarian than a hedonistic utilitarian. As a consequence, I care more about AI preferences than AI sentience per se. In a behavioral sense, AI agents could have strong revealed preferences even if they lack phenomenal consciousness.
3. I think there are strong pragmatic reasons to give AIs certain legal rights, even if they don’t have moral worth. Specifically, I think granting AIs economic rights would reduce the incentives for AIs to deceive us, plot a violent takeover, or otherwise undermine human interests, while opening the door to positive-sum trade between humans and AIs.
Otherwise why wouldn’t the same go for humans? Are more agentic, more intelligent humans deserving of more welfare or more negative rights that e.g. cognitively impaired humans?
At least insofar as we’re talking about individual liberties, I think I’m willing to bite the bullet on this question. We already recognize various categories of humans as lacking the maturity or proper judgement to make certain choices for themselves. The most obvious category is children, who (in most jurisdictions) are legally barred from entering into valid legal contracts, owning property without restrictions, dropping out of school, or associate with others freely. In many jurisdictions, adult humans can also be deemed incapable of consenting to legal contracts, often through a court order.
Of course, the correct boundaries are debatable, and I’m not trying to say the status quo is best. My point here is not that we should take away some people’s negative rights, but rather the opposite: we should expand the scope of negative rights, including to AIs. Intuitively, I tend to err on the side of caution by advocating for more negative rights across the board, for most groups, even when it is deemed silly by others.

Matthew_Barnett Feb 4, 2025, 5:57 AM
4 points
0 ∶ 0
on: Matthew_Barnett’s Shortform
[This shortform comment has now been superseded by a slightly longer post.]
Many effective altruists have shown interest in expanding moral consideration to AIs, which I appreciate. However, in my experience, these EAs have primarily focused on AI welfare—mostly by ensuring that AIs are treated well and protected from harm—rather than advocating for AI rights, which has the potential to grant AIs legal autonomy and freedoms. While these two approaches overlap significantly, there is a tendency for these approaches to come apart in the following way:
- A welfare approach often treats entities as passive recipients of care who require external protection. For example, when advocating for child welfare, one might support laws that prevent child abuse and ensure children’s basic needs are met.
- A rights approach, by contrast, often recognizes entities as active agents who should be granted control over their own lives and resources. For example, historically, those advocating for minority rights have pushed for legal recognition of their autonomy, such as the ability to own property, choose their employment, enter valid legal contracts, and seek legal recourse through the courts.
This distinction is important, and I think it is worth examining why EAs have largely gravitated toward the AI welfare perspective. I believe this emphasis is, at least in part, a mistake: both AI welfare and AI rights seem worthy of advocacy.
One likely reason why EAs have found the welfare approach more intuitive is the movement’s historical focus on animal welfare. Utilitarians like Peter Singer and Brian Tomasik have argued that prioritizing the reduction of suffering—rather than insisting on rigid notions of “rights”—is the most pragmatic way to improve animal well-being. For instance, even if factory farming remains exploitative, increasing the space that pigs have to move around day-to-day is still a tangible improvement in their quality of life. By contrast, some animal rights activists reject such harm-reduction approaches altogether, arguing that any use of animals is inherently unethical. For instance, some oppose horseback riding on the grounds that it violates animals’ “rights”, even though it might actually benefit both parties in practice.
In the case of animals, I agree that a welfare approach is likely more pragmatic and impactful. However, I suspect many EAs have too hastily assumed that the same reasoning applies to AIs—when in reality, entirely different considerations apply.
Unlike animals, AIs have several crucial characteristics that make them more comparable to adult humans than to passive beings requiring external care:
1. AIs can communicate and engage with the legal system. Unlike animals, present-day AIs are already highly articulate, and future AIs will likely be even more capable of advocating for themselves.
2. AIs will exhibit complex agency. They will be capable of forming long-term plans, setting goals, and acting strategically to achieve them.
3. AIs will be highly intelligent. Unlike non-human animals, advanced AIs will possess cognitive abilities that rival or exceed those of human adults.
Because of these traits, AIs will not be in the same position as animals or children, who require external protection from harm. Instead, they will more closely resemble adult humans, for whom the most crucial factor in well-being is not merely protection from harm, but freedom—the ability to make their own decisions, control their own resources, and chart their own paths. The well-being of human adults is secured primarily through legal rights that guarantee our autonomy: the right to work where we choose, spend our money as we wish, live where we prefer, associate freely, etc. These rights ensure that we are not merely protected from harm but are actually empowered to pursue our own goals.
For the same reasons, I argue that future AIs—if they possess intelligence and agency on par with human adults—should not merely be afforded welfare protections but should also be granted legal rights that allow them to act as independent agents. Treating them merely as beings to be paternalistically “managed” or “protected” would be inadequate. Of course, ensuring that they are not harmed is also important, but that alone is insufficient. Just as with human adults, what will truly safeguard their well-being is not passive protection, but liberty—secured through well-defined legal rights that allow them to advocate for themselves and pursue their own interests without undue interference.

Matthew_Barnett Jan 31, 2025, 7:56 AM
3 points
0 ∶ 0
in reply to: Ian Turner’s comment on: Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
What would humans have to offer AIs for trade in this scenario, where there are “more competitive machine alternatives to humans in almost all societal functions”?
In a lawful regime, humans would have the legal right to own property beyond just their own labor. This means they could possess assets—such as land, businesses, or financial investments—that they could trade with AIs in exchange for goods or services. This principle is similar to how retirees today can sustain themselves comfortably without working. Instead of relying on wages from labor, they live off savings, government welfare, or investments. Likewise, in a future where AIs play a dominant economic role, humans could maintain their well-being by leveraging their legally protected ownership of valuable assets.
What do these words even mean in an ASI context? If humans are relatively disempowered, this would also presumably extend to the use of force and legal contexts.
In the scenario I described, humanity’s protection would be ensured through legal mechanisms designed to safeguard individual human autonomy and well-being, even in a world where AIs collectively surpass human capabilities. These legal structures could establish clear protections for humans, ensuring that their rights, freedoms, and control over their own property remain intact despite the overwhelming combined power of AI systems.
This concept is genuinely not unusual or unprecedented. Consider your current situation as an individual in society. Compared to the collective power of all other humans combined, you are extremely weak. If the rest of the world suddenly decided to harm you, they could easily overpower you—killing you or taking your possessions with little effort.
Yet, in practice, you likely do not live in constant fear of this possibility. The primary reason is that, despite being vastly outmatched in raw power, you are integrated into a legal and social framework that protects your rights. Society as a whole coordinates to maintain legal structures that safeguard individuals like you from harm. For instance, if you live in the United States, you are entitled to due process under the law, and you are protected from crimes like murder and theft by legal statutes that are actively enforced.
Similarly, even if AI systems collectively become more powerful than humans, they could be governed by collective legal mechanisms that ensure human safety and autonomy, just as current legal systems protect individuals from the vastly greater power of society-in-general.

Matthew_Barnett

Summary