Error

Thank you for writing this post. An evergreen difficulty that applies to discussing topics of such a broad scope is the large number of matters that are relevant, difficult to judge, and where one’s judgement (whatever it may be) can be reasonably challenged. I hope to offer a crisper summary of why I am not persuaded.

I understand from this the primary motivation of MCE is avoiding AI-based dystopias, with the implied causal chain being along the lines of, “If we ensure the humans generating the AI have a broader circle of moral concern, the resulting post-human civilization is less likely to include dystopic scenarios involving great multitudes of suffering sentiences.”

There are two considerations that speak against this being a greater priority than AI alignment research: 1) Back-chaining from AI dystopias leaves relatively few occasions where MCE would make a crucial difference. 2) The current portfolio of ‘EA-based’ MCE is poorly addressed to averting AI-based dystopias.

Re. 1): MCE may prove neither necessary nor sufficient for ensuring AI goes well. On one hand, AI designers, even if speciesist themselves, might nonetheless provide the right apparatus for value learning such that resulting AI will not propagate the moral mistakes of its creators. On the other, even if the AI-designers have the desired broad moral circle, they may have other crucial moral faults (maybe parochial in other respects, maybe selfish, maybe insufficiently reflective, maybe some mistaken particular moral judgements, maybe naive approaches to cooperation or population ethics, and so on) - even if they do not, there are manifold ways in the wider environment (e.g. arms races), or in terms of technical implementation, that may incur disaster.

It seems clear to me that, pro tanto, the less speciesist the AI-designer, the better the AI. Yet for this issue to be of such fundamental importance to be comparable to AI safety research generally, the implication is of an implausible doctrine of ‘AI immaculate conception’: only by ensuring we ourselves are free from sin can we conceive an AI which will not err in a morally important way.

Re 2): As Plant notes, MCE does not arise from animal causes alone: global poverty, climate change also act to extend moral circles, as well as propagating other valuable moral norms. Looking at things the other way, one should expect the animal causes found most valuable from the perspective of avoiding AI-based dystopia to diverge considerably from those picked on face-value animal welfare. Companion animal causes are far inferior from the latter perspective, but unclear on the former if this a good way of fostering concern for animals; if the crucial thing is for AI-creators not to be speciest over the general population, targeted interventions like ‘Start a petting zoo at Deepmind’ look better than broader ones, like the abolition of factory farming.

The upshot is that, even if there are some particularly high yield interventions in animal welfare from the far future perspective, this should be fairly far removed from typical EAA activity directed towards having the greatest near-term impact on animals. If this post heralds a pivot of Sentience Institute to directions pretty orthogonal to the principal component of effective animal advocacy, this would be welcome indeed.

Notwithstanding the above, the approach outlined above has a role to play in some ideal ‘far future portfolio’, and it may be reasonable for some people to prioritise work on this area, if only for reasons of comparative advantage. Yet I aver it should remain a fairly junior member of this portfolio compared to AI-safety work.

I thought this piece was good. I agree that MCE work is likely quite high impact—perhaps around the same level as X-risk work—and that it has been generally ignored by EAs. I also agree that it would be good for there to be more MCE work going forward. Here’s my 2 cents:

You seem to be saying that AIA is a technical problem and MCE is a social problem. While I think there is something to this, I think there are very important technical and social sides to both of these. Much of the work related to AIA so far has been about raising awareness about the problem (eg the book Superintelligence), and this is more a social solution than a technical one. Also, avoiding a technological race for AGI seems important for AIA, and this also is more a social problem than a technical one.

For MCE, the 2 best things I can imagine (that I think are plausible) are both technical in nature. First, I expect clean meat will lead to the moral circle expanding more to animals. I really don’t see any vegan social movement succeeding in ending factory farming anywhere near as much as I expect clean meat to. Second, I’d imagine that a mature science of consciousness would increase MCE significantly. Many people don’t think animals are conscious, and almost no one thinks anything besides animals can be conscious. How would we even know if an AI was conscious, and if so, if it was experiencing joy or suffering? The only way would be if we develop theories of consciousness that we have high confidence in. But right now we’re very limited in studying consciousness, because our tools at interfacing with the brain are crude. Advanced neurotechnologies could change that—they could allow us to potentially test hypotheses about consciousness. Again, developing these technologies would be a technical problem.

Of course, these are just the first ideas that come into my mind, and there very well may be social solutions that could do more than the technical solutions I mentioned, but I don’t think we should rule out the potential role of technical solutions, either.

Thanks for writing this, I thought it was a good article. And thanks to Greg for funding it.

My pushback would be on the cooperation and coordination point. It seems that a lot of other people, with other moral values, could make a very similar argument: that they need to promote their values now, as the stakes as very high with possible upcoming value lock-in. To people with those values, these arguments should seem roughly as important as the above argument is to you.

• Christians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with sinners who will go to hell.

• Egalitarians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with wider and wider diversities of wealth.

• Libertarians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with property rights violations.

• Naturalists could argue that, if the singularity is approaching, it is vitally important that we ensure the beauty of nature won’t be bespoiled all over the universe.

• Nationalists could argue that, if the singularity is approaching, it is vitally important that we ensure the universe will be filled with people who respect the flag.

But it seems that it would be very bad if everyone took this advice literally. We would all end up spending a lot of time and effort on propaganda, which would probably be great for advertising companies but not much else, as so much of it is zero sum. Even though it might make sense, by their values, for expanding-moral-circle people and pro-abortion people to have a big propaganda war over whether foetuses deserve moral consideration, it seems plausible we’d be better off if they both decided to spend the money on anti-malaria bednets.

In contrast, preventing the extinction of humanity seems to occupy a privileged position—not exactly comparable with the above agendas, though I can’t exactly cache out why it seems this way to me. Perhaps to devout Confucians a pre-occupation with preventing extinction seems to be just another distraction from the important task of expressing filial piety – though I doubt this.

(Moral Realists, of course, could argue that the situation is not really symmetric, because promoting the true values is distinctly different from promoting any other values.)

Thanks for this post. Some scattered thoughts:

The main risk for AIA seems to be that the technical research done to better understand how to build an aligned AI will increase AI capabilities generally, meaning it’s also easier for humanity to produce an unaligned AI.

This doesn’t seem like a big consideration to me. Even if unfriendly AI comes sooner by an entire decade, this matters little on a cosmic timescale. An argument I find more compelling: If we plot the expected utility of an AGI as a function of the amount of effort put into aligning it, there might be a “valley of bad alignment” that is worse than no attempt at alignment at all. (A paperclip maximizer will quickly kill us and not generate much long-term suffering, whereas an AI that understands the importance of human survival but doesn’t understand any other values will imprison us for all eternity. Something like that.)

I’d like to know more about why people think that our moral circles have expanded. I suspect activism plays a smaller role than you think. Steven Pinker talks about possible reasons for declining violence in his book The Better Angels of Our Nature. I’m guessing this is highly related to moral circle expansion.

One theory I haven’t seen elsewhere is that self-interest plays a big role in moral circle expansion. Consider the example of slavery. The BBC writes:

It becomes clear that humanitarianism and imperial muscling were able bedfellows...

One can be certain that the high ideals of abolition and the promotion of legitimate trade were equally matched by economic and territorial ambitions, impulses which brought forward partition and colonial rule in Africa in the late 19th century.

You’ll note that the villains of the slave story are the slavers—people with an interest in slavery. The heroes seem to have been Britons who would not lose much if slavery was outlawed (though I guess boycotting sugar would go against their self-interest?) Similarly, I think I remember reading that poor northern whites were motivated to fight in the US Civil War because they were worried their labor would be displaced by slave labor.

According to this story, the expanding circle is a side effect of the world growing wealthier. As lower levels of Maslow’s hierarchy are met, people care more about humanitarian issues. (I’m assuming that genetic relatedness predicts where on the hierarchy another being falls.) Conquest is less common now because it’s more profitable to control a multinational company than control lots of territory. Slavery is less common because unskilled laborers are less of an asset & more of a liability, and it’s hard to coerce skilled labor. Violence has declined because sub-replacement fertility means we’re no longer in a zero-sum competition for resources. (Note that the bloodiest war in recent memory happened in the Democratic Republic of Congo, a country where women average six children each—source. Congo has a lot of mineral wealth, which seems to incentivize conflict. Probably this wealth doesn’t diminish as much in the presence of conflict as much as e.g. manufacturing wealth would.)

I suppose a quick test for the Maslow’s hierarchy story is to check whether wealthy people are more likely to be vegan (controlling for meat calories being as expensive as non-meat calories).

I don’t think everyone is completely self-interested all the time, but I think people are self-interested enough that it makes sense for activists to apply leverage strategically.

Re: a computer program used to mine asteroids, I’d expect certain AI alignment work to be useful here. If we understand AI algorithms more deeply, an asteroid miner can be simpler and less likely sentient. Contrast with the scenario where AI progress is slow, brain emulations come before AGI, and the asteroid miner is piloted using an emulation of someone’s brain.

I’m not comfortable relying on innate human goodness to deal with moral dilemmas. I’d rather eliminate incentives for immoral behavior. In the presence of bad incentives, I worry about activism backfiring as people come up with rationalizations for their immoral behavior. See e.g. biblical justifications for slavery in the antebellum south. Instead of seeing the EA movement as something that will sweep the globe and make everyone altruistic, I’m more inclined to see it as a team of special forces working to adjust the incentives that everyone else operates under in order to create good outcomes as a side effect of everyone else working towards their incentives.

Another moral circle expansion story involves improved hygiene. See also.

Singer and Pinker talk a lot about the importance of reason and empathy to the expanding moral circle. This might be achieved through better online discussion platforms, widespread adoption of meditation, etc.

Anyway, I think that if we take a broad view of moral circle expansion, the best way to achieve it might be some unexpected thing: improving the happiness of voters who control nuclear weapons, helping workers deal with technological job displacement, and so on. IMO, more EAs should work on world peace.

I come back to this post quite frequently when considering whether to prioritize MCE (via animal advocacy) or AI safety. It seems that these two cause areas often attract quite different people with quite different objectives, so this post is unique in its attempt to compare the two based on the same long-term considerations.

I especially like the discussion of bias. Although some might find the whole discussion a bit ad hominem, I think people in EA should take seriously the worry that certain features common in the EA community (e.g., an attraction towards abstract puzzles) might bias us towards particular cause areas.

I recommend this post for anyone interested in thinking more broadly about longtermism.

I think there’s a significant[8] chance that the moral circle will fail to expand to reach all sentient beings, such as artificial/​small/​weird minds (e.g. a sophisticated computer program used to mine asteroids, but one that doesn’t have the normal features of sentient minds like facial expressions). In other words, I think there’s a significant chance that powerful beings in the far future will have low willingness to pay for the welfare of many of the small/​weird minds in the future.[9]

I think it’s likely that the powerful beings in the far future (analogous to humans as the powerful beings on Earth in 2018) will use large numbers of less powerful sentient beings

So I’m curious for your thoughts. I see this concern about “incidental suffering of worker-agents” stated frequently, which may be likely in many future scenarios. However, it doesn’t seem to be a crucial consideration, specifically because I care about small/​weird minds with non-complex experiences (your first consideration).

Caring about small minds seems to imply that “Opportunity Cost/​Lost Risks” are the dominate consideration—if small minds have moral value comparable to large minds, then the largest-EV risk is not optimizing for small minds and wasting resources thrown at large minds with complex/​expensive experiences (or thrown at something even less efficient, like biological beings, any non-total-consequentialist view, etc). This would you lose you many orders of magnitude of optimized happiness, and this loss would be worse than the other scenarios’ aggregate incidental suffering. Even if this inefficient moral position merely reduced optimized happiness by 10% - far less than an order of magnitude—this would dominate incidental suffering, even if the incidental suffering scenarios were significantly more probable. And even if you very heavily weight suffering compared to happiness, my math still suggests this conclusion survives by a significant margin).

Also note that Moral Circle Expansion is relevant conditional on solving the alignment problem, so we’re in the set of worlds where the alignment problem was actually solved in some way (humanity’s values are somewhat intact). So, the risk is that whatever-we’re-optimizing-the-future-for is far less efficient than ideal hedonium could have been, because we’re wasting it on complex minds, experiences that require lots of material input, or other not-efficiently-value-creating things. “Oh, what might have been”, etc. Note this still says values spreading might be very important, but I think this version has a slightly different flavor that implies somewhat different actions. Thoughts?

Next to the counterpoints mentioned by Gregory Lewis, I think there is an additional reason why MCE seems less effective than more targeted interventions to improve the quality of the long-term future: Gains from trade between humans with different values become easier to implement as the reach of technology increases. As long as a non-trivial fraction of humans end up caring about animal wellbeing or digital minds, it seems likely it would be cheap for other coalitions to offer trades. So whether 10% of future people end up with an expanded moral circle or 100% may not make much of a difference to the outcome: It will be reasonably good either way if people reap the gains from trade.

One might object that it is unlikely that humans would be able to cooperate efficiently, given that we don’t see this type of cooperation happening today. However, I think it’s reasonable to assume that staying in control of technological progress beyond the AGI transition requires a degree of wisdom and foresight that is very far away from where most societal groups are at today. And if humans do stay in control, then finding a good solution for value disagreements may be the easier problem, or at worst similarly hard. So it feels to me that most likely, we either we get a future that goes badly for reasons related to lack of coordination and sophistication in the pre-AGI stage, or we get a future where humans set things up wisely enough to actually design an outcome that is nice (or at least not amongst the 10% of worst outcomes) by the lights of nearly everyone.

Brian Tomasik made the point that conditional on human values staying in control, we may be very unlikely to get something like broad moral reflection. Instead, values could be determined by a very small group of individuals who happened to be in power by the time AGI arrives (as opposed to individuals ending up there because they were unusually foresighted and also morally motivated). This feels possible too, but it seems to not be the likely default to me because I suspect that you’d need to necessarily increase your philosophical sophistication in order to stay in control of AGI, and that probably gives you more pleasant outcomes (correlational claim). Iterated amplification for instance, as an approach to AI alignment, has several uses for humans: Humans are not only where the resulting values come from, but they’re also in charge of keeping the bootstrapping process on track and corrigible. And as this post on factored cognition illustrates, this requires sophistication to set up. So if that’s the bar that AGI creators need to pass before they can determine how “human values” are to be extrapolated, maybe we shouldn’t be too pessimistic about the outcome. It seems kind of unlikely that someone would go through all of that only to be like “I’m going to implement my personal best guess about what matters to me, with little further reflection, and no other humans get a say here.” Similarly, it also feels unlikely that people would go through with all that and not find a way to make subparts of the population reasonably content about how sentient subroutines are going to be used.

Now, I feel a bit confused about the feasibility of AI alignment if you were to do it somewhat sloppily and with lower standards. I think that there’s a spectrum from “it just wouldn’t work at all and not be competitive” (and then people would have to try some other approach) to “it would produce a capable AGI but it would be vulnerable to failure modes like adversarial exploits or optimization daemons, and so it would end up with not human values”. These failure modes, to the very small degree I currently understand them, sound like they would not be sensitive to whether the human whose approval you tried to approximate had an expanded moral circle or not. I might be wrong about that. If people mostly want sophisticated alignment procedures because they care about preserving the option for philosophical reflection, rather than because they also think that you simply run into large failure modes otherwise, then it seems like (conditional on some kind of value alignment) whether we get an outcome with broad moral reflection is not so clear. If it’s technically easier to build value-aligned AI with very parochial values, then MCE could make a relevant difference to these non-reflection outcomes.

But all in all my argument is that it’s somewhat strange to assume that a group of people could succeed at building an AGI optimized for its creators’ values, without having to put in so much thinking about how to get this outcome right that they’d almost can’t help but become reasonably philosophically sophisticated in the process. And sure, philosophically sophisticated people can still have fairly strange values by your own lights, but it seems like there’s more convergence. Plus I’d at least be optimistic about their propensity to strive towards positive-sum outcomes, given how little scarcity you’d have if the transition does go well.

Of course, maybe value-alignment is going to work very differently from what people currently think. The main way I’d criticize my above points is that they’re based on heavy-handed inside-view thinking about how difficult I (and others I’m updating towards) expect the AGI transition to be. If AGI will be more like the Industrial Revolution rather than something that is even more difficult to stay remotely in control of, or if some other technology proves to be more consequential than AGI, then my argument has less force. I mainly see this as yet another reason to caveat that the ex ante plausible-seeming position that MCE can have a strong impact on AGI outcomes starts to feel more and more conjunctive the more you zoom in and try to identify concrete pathways.

This post is extremely valuable—thank you! You have caused me to reexamine my views about the expected value of the far future.

What do you think are the best levers for expanding the moral circle, besides donating to SI? Is there anything else outside of conventional EAA?

I thought this was very interesting, thanks for writing up. Two comments

1. It was useful to have a list of reasons why you think the EV of the future could be around zero, but it still found it quite vague/​hard to imagine—why exactly would more powerful minds be mistreating less powerful minds? etc. - so I’d would have liked to see that sketched in slightly more depth.

2. It’s not obvious to me it’s correct/​charitable to draw the neglectedness of MCE so narrowly. Can’t we conceive of a huge ammount of moral philosophy, and well as social activism, both new and old, as MCE? Isn’t all EA outreach an indirect form of MCE?

A very interesting and engaging article indeed.

I agree that people often underestimate the value of strategic value spreading. Oftentimes, proposed moral models that AI agents will follow have some lingering narrowness to them, even when they attempt to apply the broadest of moral principles. For instance, in Chapter 14 of Superintelligence, Bostrom highlights his common good principle:

Superintelligence should be developed only for the benefit of all of humanity and in the service of widely shared ethical ideals.

Clearly, even something as broad as that can be controversial. Specifically, it doesn’t speak at all about any non-human interests except insofar as humans express widely held beliefs to protect them.

I think one thing to add is that AIA researchers who hold more traditional moral beliefs (as opposed to wide moral circles and transhumanist beliefs) are probably less likely to believe that moral value spreading is worth much. The reason for this is obvious: if everyone around you holds, more or less, the same values that you do, then why change anyone’s mind? This may explain why many people dismiss the activity you proposed.

On this topic, I similarly do still believe there’s a higher likelihood of creating hedonium; I just have more skepticism about it than I think is often assumed by EAs.

This is the main reason I think the far future is high EV. I think we should be focusing on p(Hedonium) and p(Delorium) more than anything else. I’m skeptical that, from a hedonistic utilitarian perspective, byproducts of civilization could come close to matching the expected value from deliberately tiling the universe (potentially multiverse) with consciousness optimized for pleasure or pain. If p(H)>p(D), the future of humanity is very likely positive EV.

You say

I think a given amount of dolorium/​dystopia (say, the amount that can be created with 100 joules of energy) is far larger in absolute moral expected value than hedonium/​utopia made with the same resources

Could you elaborate more on why this is the case? I would tend to think that a prior would be that they’re equal, and then you update on the fact that they seem to be asymmetrical, and try to work out why that is the case, and whether those factors will apply in future. They could be fundamentally asymmetrical, or evolutionary pressures may tend to create minds with these asymmetries. The arguments I’ve heard for why are:

• The worst thing that can happen to an animal, in terms of genetic success, is much worse than the best thing.

This isn’t entirely clear to me: I can imagine a large genetic win such as securing a large harem could be comparable to the genetic loss of dying, and many animals will in fact risk death for this. This seems particularly true considering that dying leaving no offspring doesn’t make your contribution to the gene pool zero, just that it’s only via your relatives.

• There is selection against strong positive experiences in a way that there isn’t against strong negative experiences.

The argument here is, I think, that strong positive experiences will likely result in the animal sticking in the blissful state, and neglecting to feed, sleep, etc, whereas strong negative experiences will just result in the animal avoiding a particular state, which is less maladaptive. This argument seems stronger to me but still not entirely satisfying—it seems to be quite sensitive to how you define states.

My current position is that the amount of pleasure/​suffering that conscious entities will experience in a far-future technological civilization will not be well-defined. Some arguments for this:

1. Generally utility functions or reward functions are invariant under affine transformations (with suitable rescaling for the learning rate for reward functions). Therefore they cannot be compared between different intelligent agents as a measure of pleasure.

2. The clean separation of our civilization into many different individuals is an artifact of how evolution operates. I don’t expect far future civilization to have a similar division of its internal processes into agents. Therefore the method of counting conscious entities with different levels of pleasure is inapplicable.

3. Theoretical computer science gives many ways to embed one computational process within another so that it is unclear whether or how many times the inner process “occurs”, such as running identical copies of the same program, using a quantum computer to run the same program with many inputs in superposition, and homomorphic encryption. Similar methods we don’t know about will likely be discovered in the future.

4. Our notions of pleasure and suffering are mostly defined extensionally with examples from the present and the past. I see no reason that such an extensionally-derived concept to have a natural definition that applies to extremely different situations. Uncharitably, it seems like the main reason people assume this is a sort of wishful thinking due to their normal moral reasoning breaking down if they allow pleasure/​suffering to be undefined.

I’m currently uncertain about how to make decisions relating to the far future in light of the above arguments. My current favorite position is to try to understand the far future well enough until I find something I have strong moral intuitions about.

Thanks very much for writing this, and thanks to Greg for funding it! I think this is a really important discussion. Some slightly rambling thoughts below.

We can think about 3 ways of improving the EV of the far future:

1: Changing incentive structures experienced by powerful agents in the future (e.g. avoiding arms races, power struggles, selection pressures)

2: a) Changing the moral compass of powerful agents in the future in specific directions (e.g. MCE).

b) Indirect ways to improve the moral compass of powerful agents in the future (e.g. philosophy research, education, intelligence/​empathy enhancement)

All of these are influenced both by strategies such as activism, improving institutions, and improving education, as well as by AIA. I am inclined to think of AIA as a particularly high-leverage point at which we can have influence on these.

However, these are issues are widely encountered. Consider 2b: we have to decide how to educate the next generation of humans, and they may well end up with ethical beliefs that are different from ours, so we must judge how much to try and influence or constrain them, and how much to accept that the changes are actually progress. This is similar to the problem of defining CEV: we have some vague idea of the direction in which better values lie (more empathy, more wisdom, more knowledge), but we can’t say exactly what the values should be. For this intervention, working on AIA may be more important than activism because it has more leverage—it is likely to be more tractable and have greater influence on the future than the more diffuse ways in we can push on education and intergenerational moral progress.

This framework also suggests that MCE is just one example of a collection of similar interventions. MCE involves pushing for a fairly specific belief and behaviour change on a principle that’s fairly uncontroversial. You could also imagine similar interventions—for instance, helping people reduce unwanted aggressive or sadistic behaviour. We could call this something like ‘uncontroversial moral progress’: helping individuals and civilisation to live by their values more. (on a side note: sometimes I think of this as the minimal core of EA: trying to live according to your best guess of what’s right)

The choice between working on 2a and 2b depends, among other things, on your level of moral uncertainty.

I am inclined to think that AIA is the best way to work on 1 and 2b, as it is a particularly high-leverage intervention point to shape the power structures and moral beliefs that exist in the future. It gives us more of a clean slate to design a good system, rather than having to work within a faulty system.

I would really like to see more work on MCE and other examples of ‘uncontroversial moral progress’. Historical case studies of value changes seem like a good starting point, as well as actually testing the tractability of changing people’s behaviour.

I also really appreciated your perspective on different transformative AI scenarios, as I’m worried I’m thinking about it in an overly narrow way.

Thank you for providing an abstract for your article. I found it very helpful.

(and I wish more authors here would do so as well)

Random thought: (factory farm) animal welfare issues will likely eventually be solved by cultured (lab grown) meat when it becomes cheaper than growing actual animals. This may take a few decades, but social change might take even longer. The article even suggests technical issues may be easier to solve, so why not focus more on that (rather than on MCE)?

Thank you for this piece. I enjoyed reading it and I’m glad that we’re seeing more people being explicit about their cause-prioritization decisions and opening up discussion on this crucially important issue.

I know that it’s a weak consideration, but I hadn’t, before I read this, considered the argument for the scale of values spreading being larger than the scale of AI alignment (perhaps because, as you pointed out, the numbers involved in both are huge) so thanks for bringing that up.

I’m in agreement with Michael_S that hedonium and delorium should be the most important considerations when we’re estimating the value of the far-future, and from my perspective the higher probability of hedonium likely does make the far-future robustly positive, despite the valid points you bring up. This doesn’t necessarily mean that we should focus on AIA over MCE (I don’t), but it does make it more likely that we should.

Another useful contribution, though others may disagree, was the biases section: the biases that could potentially favour AIA did resonate with me, and they are useful to keep in mind.

While the central thesis to expand one’s moral circles can be well-enjoyed by the community, this post is not selling it well. This is exemplified by the “One might be biased towards AIA if…” section, which makes assumptions about individuals who focus on AI alignment. Further, while the post includes a section on cooperation, it discourages it. [Edit: Prima facie,] the post does not invite critical discussion. Thus, I would not recommend this post to any readers interested in moral circles expansion, AI alignment, or cooperation. Thus, I would recommend this post to readers interested in moral circles expansion, AI alignment, and cooperation, as long as they are interested in a vibrant discourse.

Impressive article—I especially liked the biases section. I would recommend doing a quantitative model of cost effectiveness comparing to AIA, as I have done for global agricultural catastrophes, especially because neglectedness is hard to define in your case.

Thanks for writing it.

Here are my reasons for the belief wild animal/​small minds/​… suffering agenda is based mostly on errors and uncertainties. Some of the uncertainties should warrant research effort, but I do not believe the current state of knowledge justifies prioritization ofany kind of advocacy or value spreading.

1] The endeavour seems to be based on extrapolating intuitive models far outside the scope for which we have data. The whole suffering calculus is based on extrapolating the concept of suffering far away from the domain for which we have data from human experience.

2] Big part of it seems arbitrary. When expanding the moral circle toward small computational processes and simple systems, why not expand it toward large computational processes and complex systems? E.g. we can think about the DNA based evolution as about large computational/​optimization process—suddenly “wild animal suffering” has a purpose and traditional environmnet and biodiversity protection efforts make sense.

(Similarly we could argue much “human utility” is in the larger system structure above individual humans)

3] We do not know how to measure and aggregate utility of mind states. Like, really don’t know. E.g. it sems to me completely plausible the utility of 10 people reaching some highly joyful mindstates is the dominanat contribution over all human and animal minds.

4] Part of the reasoning usually seems contradictory. If the human cognitive processes are in the priviledged position of creating meaning in this universe … well, then they are in the priviledged postion, and there _is_ a categorical difference between humans and other minds. If they are not in the priviledged positions, how it comes humans should impose their ideas about meaning on other agents?

5] MCE efforts directed toward AI researchers with the intent of influencing values of some powerful AI may increase x-risk. E.g. if the AI is not “speciist” and gives the same weight to satysfing prefrences of all humans and all chicken, the chicken would outnumber humans.

@Matthew_Barnett As a senior electrical engineering student, proficient in a variety of programming languages, I do think and believe that AI is important to think about and discuss. The theoretical threat of a malevolent strong AI would be immense. But that does not mean one has cause or a valid reason to support CS grad students financially.

A large, significant, asteroid collision with Earth would also be quite devastating. Yet, to fund and support aerospace grads does not follow. Perhaps I really mean this: AI safety is an Earning to Give non sequitur.

Lastly, again, there is no evidence or results. Effective Altruism is about being beneficent instead of merely benevolent (meaning well). In other words, making decisions off well researched initiatives (e.g., bed nets). Since strong AI does not exist, it does not make sense to support though E2G. (I’m not saying it will never exist; that is unknown.) Of course, there are medium-term (systematic change) with results that more or less rely on historical-type empiricism—but that’s still some type of evidence. For poverty we have RCTs and developmental economics. For AI safety [something?]. For animal suffering we have proof that less miserable conditions can become a reality.

The lists were interesting on how they allude to the different psychology and motivations each EA has between the two camps. I hope someday I can have a civil discussion with someone not directly benefiting from AIA (such as being involved in the research). Aside, I have a friend who’s crazy about futurism, 2045 Initiative/​propaganda, and in love with everything Musk says on Twitter.

“The idea of ‘helping the worst off’ is appealing.” Why wouldn’t it be? Copenhagen Consensus.

“Their reaction when they look about extinction risk or AI safety is nonsensical”, imaginary and completely unknown—zero tractability. No evidence to go off of since such technology does not exist. Why give to CS grad students? It’s like trying to fund a mission to Mars, not priority. It’s like funding time travel safety research, non sequitur.

“They are generally an unhappy person.” I just had to laugh and compare how one interested in AI safety matched up. A neo-Freudian Jung/​MBTI type of deal. Almost like zodiac signs. Although, the Minnesota Multiphasic Personality Inventory (MMPI) is rigorous—so who am I to judge this informal inventory.

Anyway, I simply do not see that individual action or donation to AIA research has measurable outcomes. We’re talking about Strong AI here—it doesn’t even exist! Not that it couldn’t though. In the future, even the medium-term future, general standards of living could be significantly improved. Synthetic meat on a production scale is a much more realistic research area (or even anti-malaria mosquitoes) instead of making a fuss about imaginary-theoretical events. We’re at a unique sliver in time where it is extremely practical to help lessen the suffering of humans and animals in the near and medium-term future. (I.e., we have rapid transportation and instant information transfer.)