Really appreciate the time it took you to write this and detailed analysis!
That said, I strongly disagree with this post. This tl;dr of the post is
“Assume total utilitarianism and longtermism is given. Then given several reasonable assumptions and some simple math, wild animal welfare will dominate human welfare in the foreseeable future, so total utilitarian longtermists should predominantly be focused on animal welfare today.”
I think this is wrong, or at least the conclusions don’t follow from the premises, mostly due to weird science-fictiony reasons.
The rest of my rebuttal will be speculative and science-fictiony, so if you prefer reading comments that sound reasonable, I encourage you to read elsewhere.
Like the post I’m critiquing, I will assume longtermism and total utilitarianism for the sake of the argument, and not defend them here. (Unlike the poster, I personally have a lot of sympathy towards both beliefs).
By longtermism, I mean a moral discount rate of epsilon (epsilon >=0, epsilon ~=0). By total utilitarianism, I posit the existence of moral goods (“utility”) that we seek to maximize, and the aggregation function is additive. I’m agnostic for most of the response about what the moral goods in question are (but will try to give plausible ones where it’s necessary to explain subpoints).
I have two core, mostly disjunctive arguments for things the post missed about where value in the long-term future are:
A. Heavy tailed distribution of engineered future experiences B. Cooperativeness with Future Agents
A1. Claim: Biological organisms today are mostly not optimized/designed for extreme experiences.
I think this is obviously true. Even within the same species (humans), there is a wide variance of reported, eg, happiness for people living in ~the same circumstances, and most people will agree that this represents wide variance in actual happiness (rather than entirely people being mistaken about their own experiences.
Evolutionarily, we’re replicator machines, not experience machines.
This goes for negative as well as positive experiences. Billions of animals are tortured in factory farms, but the telos of factory farms isn’t torture, it’s so that humans get access to meat. No individual animal is *deliberately* optimized by either evolution or selective breeding to suffer.
A2. Claim: It’s possible to design experiences that have much more utility than anything experienced today. I can outline two viable paths (disjunctive):
A2a. Simulation For this to hold, you have to believe: A2ai. Humans or human-like things can be represented digitally. I think there is philosophical debate, but most people who I trust think this is doable.
A2aii. Such a reproduction can be cheap
I think this is quite reasonable since again, existing animals (including human animals) are not strongly optimized for computation.
A2aiii. simulated beings are capable of morally relevant experiences or otherwise production of goods of intrinsic moral value.
Some examples may be lots of happy experiences, or (if you have a factor for complexity) lots of varied happy experiences, or other moral goods that you may wish to produce, like great works of art, deep meaningful relationships, justice, scientific advances, etc.
A2b. Genetic engineering I think this is quite viable. The current variance among human experiences is an existence proof. There are lots of seemingly simple ways that improve on current humans to suffer less, and be happier (eg, lots of unnecessary pain during childbirth just because we’ve evolved to be bipedal + have big heads).
A3. Claim: Our descendants may wish to optimize for positive moral goods.
I think this is a precondition for EAs and do-gooders in general “winning”, so I almost treat the possibility of this as a tautology.
A4. Claim: There is a distinct possibility that a high % of vast future resources will be spent on building valuable moral goods, or the resource costs of individual moral goods are cheap, or both.
A4ai. Proportions: This mostly falls from A3. If enough of our descendants care about optimizing for positive moral goods, then we would reasonably expect them to devote a lot of our resources to producing more of them. Eg, 1% of resources being spent on moral goods isn’t crazy.
A4aii. Absolute resources: Conservatively assuming that we never leave the solar system, right now ~1/10^9 of the Sun’s energy reaches Earth. Of the one-billionth of light that reaches Earth, less than 1% of that energy is used by plants for photosynthesis (~ all of our energy needs, with the exception of nuclear power and geothermal, comes from extracting energy that at one point came from photosynthesis—the extraction itself being a particularly wasteful process. Call it another 1-2 of magnitude discount?).
All of life on Earth uses <1/10^11 (less than one in one hundred billionth!) of the Sun’s energy. Humans use maybe 1/10^12 − 1/10^13 of that.
It’s not crazy that one day we’ll use multiple orders of magnitude of energy more for producing moral goods than we currently spend doing all of our other activities combined.
If, for example, you think the core intrinsic moral good we ought to optimize is “art”, right now Arts and Culture compose 4% of the US GDP (Side note: this is much larger than I would have guessed), and probably a similar or smaller number for world GDP.
A4b. This mostly falls from A2.
A4bi. Genetic engineering: In the spirit of doing things with made-up numbers, it sure seems likely that we can engineer humans to be 10x happier, suffer 10x less, etc. If you have weird moral goals (like art or scientific insight), it’s probably even more doable to genetically engineer humans 100x+ better at producing art, come up with novel mathematics, etc.
A4bii. It’s even more extreme with digital consciousness. The upper bound for cost is however much it costs to emulate (genetically enhanced) humans, which is probably at least 10x cheaper than the biological version, and quite possibly much less than that. But in theory, so many other advances can be made by not limiting ourselves to the human template, and abstractly consider what moral goods we want and how to get there.
A5. Conclusion: for total utilitarians, it seems likely that A1-A4 will lead us to believe that expected utility in the future will be dominated by scenarios of heavy-tails of extreme moral goods.
A5a. Thus, people now should work on some combination of preventing existential risks and steering our descendants (wherever possible) to those heavy-tailed scenarios of producing lots of positive moral goods.
A5b. The argument that people may wish to directly optimize for positive utility, but nobody actively optimizes for negative utility, is in my mind some (and actually quite strong) evidence that total or negative-leaning *hedonic* utilitarians should focus more on avoiding extinction + ensuring positive outcomes than on avoiding negative outcomes.
A5c. If you’re a negative or heavily negative-leaning hedonic utilitarian, then your priority should be to prevent the extreme tail of really really bad engineered negative outcomes. (Another term for this is “S-Risks”)
B. Short argument for Future Agent Cooperation:
B1. Claim: (Moral) Agents will have more power in the future than they do today.
This is in contrast to sub- or above- agent entities (eg, evolution), which held a lot of sway in the past.
B1a. As an aside, this isn’t central to my point that we may expect more computational resources in the future to be used by moral agents rather than just moral patients without agency (like factory farmed animals today).
B2. Claim: Most worlds we care about are ones where agents have a lot of power.
World ruled by non-moral processes rather than agents probably have approximately zero expected utility.
This is actually a disjunctive claim from B1. *Even* if we think B1 is wrong, we still want to care more about agent-ruled worlds, since the future with those are more important.
B3. Claim: Moral uncertainty may lead us to defer to future agents on questions of moral value.
For example, total utilitarians today may be confused about measurements of utility today. To the extent that either moral objectivity or “moral antirealism+get better results after long reflection” is true, our older and wiser descendants (who may well be continuous with us!) will have a better idea of what to do than we do.
B4. Conclusion: While this conclusion is weaker than the previous point, there is prima facie reason that we should be very cooperative to reasonable goals that future agents may have.
C. Either A or B should be substantial evidence that a lot of future moral value comes from thinking through (and getting right) weird futurism concerns. This is in some contrast to doing normal, “respectable” forecasting research and just following expected trendlines like population numbers.
D. What does this mean about animals? I think it’s unclear. A lot of animal work today may help with moral circle expansion.
This is especially true if you think (as I do, but eg, Will Macaskill mostly does not) that we live in a “hinge of history” moment where our values are likely to be locked in in the near future (next 100 years) AND you think that the future is mostly composed of moral patients that are themselves not moral agents (I do not) AND that our descendants are likely to be “wrong” in important ways relative to our current values (I’m fairly neutral on this question, and slightly lean against).
Whatever you believe, it seems hard to escape the conclusion that “weird, science-fictiony scenarios” have non-trivial probability, and that longtermist total utilitarians can’t ignore thinking about them.
A5b. The argument that people may wish to directly optimize for positive utility, but nobody actively optimizes for negative utility, is in my mind some (and actually quite strong) evidence that total or negative-leaning *hedonic* utilitarians should focus more on avoiding extinction + ensuring positive outcomes than on avoiding negative outcomes.
I’ve argued against this point here (although I don’t think my objection is very strong). Basically, we (or whoever) could be mistaken about which of our AI tools are sentient or matter, and end up putting them in conditions in which they suffer inadvertently or without concern for them, like factory farmed animals. If sentient tools are adapted to specific conditions (e.g. evolved), a random change in conditions is more likely to be detrimental than beneficial.
Also, individuals who are indifferent to or unaware of negative utility (generally or in certain things) may threaten you with creating a lot of negative utility to get what they want. EAF is doing research on this now.
If sentient tools are adapted to specific conditions (e.g. evolved), a random change in conditions is more likely to be detrimental than beneficial.
I don’t think it’s obvious that this is in expectation negative. I’m not at all confident that negative valence is easier to induce than positive valence today (though I think it’s probably true), but conditional upon that being true, I also think it’s a weird quirk of biology that negative valence may be more common than positive valence in evolved animals. Naively I would guess that the experiences of tool AI (that we may wrongly believe to not be sentient, or are otherwise callous towards) is in expectation zero. However, this may be enough for hedonic utilitarians with a moderate negative lean (3-10x, say) to believe that suffering overrides happiness in those cases.
I want to make a weaker claim however, which is that per unit of {experience, resource consumed}, I’d just expect intentional, optimized experience to be multiple orders of magnitude greater than incidental suffering or happiness (or other relevant moral goods).
If this is true, to believe that the *total* expected unintentional suffering (or happiness) of tool AIs to exceed that of intentional experiences of happiness (or suffering), you need to believe that the sheer amount of resources devoted to these tools are several orders of magnitude greater than the optimized resources.
This seems possible but not exceedingly likely.
If I was a negative utilitarian, I might think really hard about trying to prevent agents deliberately optimizing for suffering (which naively I would guess to be pretty unlikely but not vanishingly so).
Also, individuals who are indifferent to or unaware of negative utility (generally or in certain things) may threaten you with creating a lot of negative utility to get what they want. EAF is doing research on this now.
Yeah that’s a good example. I’m glad someone’s working on this!
I don’t think it’s obvious that this is in expectation negative. I’m not at all confident that negative valence is easier to induce than positive valence today (though I think it’s probablytrue), but conditional upon that being true, I also think it’s a weird quirk of biology that negative valence may be more common than positive valence in evolved animals. Naively I would guess that the experiences of tool AI (that we may wrongly believe to not be sentient, or are otherwise callous towards) is in expectation zero. However, this may be enough for hedonic utilitarians with a moderate negative lean (3-10x, say) to believe that suffering overrides happiness in those cases.
It might be 0 in expectation to a classical utilitarian in the conditions for which they are adapted, but I expect it to go negative if the tools are initially developed through evolution (or some other optimization algorithm for design) and RL (for learning and individual behaviour optimization), and then used in different conditions. Think of “sweet spots”: if you raise temperatures, that leads to more deaths by hyperthermia, but if you decrease temperatures, more deaths by hypothermia. Furry animals have been selected to have the right amount of fur for the temperatures they’re exposed to, and sentient tools may be similarly adapted. I think optimization algorithms will tend towards local maxima like this (although by local maxima here, I mean with respect to conditions, while the optimization algorithm is optimizing genes; I don’t have a rigorous proof connecting the two).
On the other hand, environmental conditions which are good to change in one direction and bad in the other should cancel in expectation when making a random change (with a uniform prior), and conditions that lead to improvement in each direction don’t seem stable (or maybe I just can’t even think of any), so are less likely than conditions which are bad to change in each direction. I.e. is there any kind of condition such that a change in each direction is positive? Like increasing the temperature and decreasing the temperature are both good?
This is also a (weak) theoretical argument that wild animal welfare is negative on average, because environmental conditions are constantly changing.
Thanks for laying out this response! It was really interesting, and I think probably a good reason to not take animals as seriously as I suggest you ought to, if you hold these beliefs.
I think something interesting is that this, and the other objections that have been presented to my piece have brought out is that to avoid focusing exclusively on animals in longtermist projects, you have to have some level of faith in these science-fiction scenarios happening. I don’t necessarily think that is a bad thing, but it isn’t something that’s been made explicit in past discussions of long-termism (at least, in the academic literature), and perhaps ought to be explicit?
A few comments on your two arguments:
Claim: Our descendants may wish to optimize for positive moral goods.
I think this is a precondition for EAs and do-gooders in general “winning”, so I almost treat the possibility of this as a tautology.
This isn’t usually assumed in the longtermist literature. It seems more like the argument is made on the basis of future human lives being net-positive, and therefore good that there will be many of them. I think the expected value of your argument (A) hinges on this claim, so it seems like accepting it as a tautology, or something similar, is actually really risky. If you think this is basically 100% likely to be true, of course your conclusion might be true. But if you don’t, it seems plausible that that, like you mention, priority possibly ought to be on s-risks.
In general, a way to summarize this argument, and others given here could be something like, “there is a non-zero chance that we can make loads and loads of digital welfare in the future (more than exists now), so we should focus on reducing existential risk in order to ensure that future can happen”. This raises a question—when will that claim not be true / the argument you’re making not be relevant? It seems plausible that this kind of argument is a justification to work on existential risk reduction until basically the end of the universe (unless we somehow solve it with 100% certainty, etc.), because we might always assume future people will be better at producing welfare than us.
I assume people have discussed the above, and I’m not well read in the area, but it strikes me as odd that the primary justification in these sci-fi scenarios for working on the future is just a claim that can always be made, instead of working directly on making lives with good welfare (but maybe this is a consideration with longtermism in general, and not just this argument).
I guess part of the issue here is you could have an incredibly tiny credence in a very specific number of things being true (the present being at the hinge of history, various things about future sci-fi scenarios), and having those credences would always justify deferral of action.
I’m not totally sure what to make of this, but I do think it gives me pause. But, I admit I haven’t really thought about any of the above much, and don’t read in this area at all.
tl;dr: My above comment relies on longtermism + total utilitarianism (but I attempted to be neutral on the exact moral goods that compose the abstraction of “utility”). With those two strong assumptions + a bunch of other more reasonable ones, I think you can’t escape thinking about science-fictiony scenarios. I think you may not need to care as much about science-fictiony scenarios with moderate probabilities (but extremely high payoffs in expected utility) if your views are primarily non-consequentialist, or if you’re a consequentialist but the aggregative function is not additive.
I also appreciate your thoughtful post and responses!
This isn’t usually assumed in the longtermist literature
Having read relatively little of it, my understanding is that the point of the academic literature (which do not usually assume total utilitarian views?) on longtermism is to show that longtermism is compatible (in some cases required) under a broad scope of moral views that are considered respectable within the academic literature.
So they don’t talk about science-fictiony stuff, since their claim is that longtermism is robustly true (or compatible) with reasonable academic views in moral philosophy.
The point of my comment is that longtermism + total utilitarianism must lead you to think about these science-fictiony scenarios that have significant probability, rather than that longtermism itself must lead you to consider them.
I guess part of the issue here is you could have an incredibly tiny credence in a very specific number of things being true (the present being at the hinge of history, various things about future sci-fi scenarios), and having those credences would always justify deferral of action.
I think if the credence is sufficiently low, either moral uncertainty (since most people aren’t total utilitarians with 100% probability) or model uncertainty will get you to do different actions.
At very low probabilities, you run into issues like Pascal’s Wager and Pascal’s Mugging, but right now the future is so hazy that I think it’s too hubristic to say anything super-concrete about the future. I’m reasonably confident that I’m willing to defend that all of the claims I’ve made above has percentage points of probability[1], which I think is well above the threshold for “not willing to get mugged by low probabilities.”
I suspect that longetermism + moral axiologies that are less demanding/less tail-driven than total utilitarianism will rely less on the speculative/weird/science-fictiony stuff. I hadn’t thought about them in detail.
To demonstrate what I roughly mean, I made up two imaginary axilogies (I think I can understand other nonhedonic total utilitarian views well enough to represent them faithfully, but I’m not well-read enough on non-total utilitarian views, so I made up fake ones rather than accidentally strawman existing views):
1. An example of an axiology that is long-termist but not utilitarian is that you want to maximize the *probability* that the long-term future will be a “just” world, where “justice” is a binary variable (rather than a moral good that can be maximized). If you have some naive prior that this will have 50-50 chance of happening by default, then you might want to care somewhat about extremely good outcomes for justice (eg, creating a world which can never backslide into an unjust one), and extremely bad outcomes (avoiding a dictatorial lock-in that will necessitate permanent unjust society).
But your decisions are by default going to focus more on the median outcomes rather than the tails. Depending on whether you think animals are treated justly right now, this may entail doing substantial work on farmed animals (and whether to work on wild animal welfare depends on positive vs negative conceptions of justice).
2. An example of an axiology that is long-termist and arguably utilitarian but not total utilitarian is if you have something like instead of doing Sum(Utility across all beings), you instead have Average(Log(Utility per being)). In such a world, the tails dominate if and only if you can have a plausible case for why the tails will have extremely large outcomes even on a log scale. I think this is still technically possible, but you need much stronger assumptions or better arguments than the ones I outlined above.
I’d actually be excited for you or someone else with non-total utilitarian views to look into what other moral philosophies (that people actually believe) + longtermism will entail.
[1] A way to rephrase “there’s less than a 1% probability that our descendants will wish to optimize for moral goods” is that “I’m over 99% confident that our descendants wouldn’t care about moral goods, or care very little about them.” And I just don’t think we know enough about the longterm future to be that confident about anything like that.
Having read relatively little of it, my understanding is that the point of the academic literature (which do not usually assume total utilitarian views?) on longtermism is to show that longtermism is compatible (in some cases required) under a broad scope of moral views that are considered respectable within the academic literature.
So they don’t talk about science-fictiony stuff, since their claim is that longtermism is robustly true (or compatible) with reasonable academic views in moral philosophy.
This is also my impression of some of Toby Ord’s work in The Precipice (particularly chapter 2) and some of the work of GPI, at least. I’m not sure how much it applies more widely to academic work that’s explicitly on longtermism, as I haven’t read a great deal of it yet.
On the other hand, many of Bostrom’s seminal works on existential risks very explicitly refer to such “science-fictiony” scenarios.. And these works effectively seem like seminal works for longtermism too, even if they didn’t yet use that term. E.g., Bostrom writes:
Another estimate, which assumes that future minds will be mainly implemented in computational hardware instead of biological neuronal wetware, produces a lower bound of 10^54 human-brain-emulation subjective life-years (or 10^71 basic computational operations) (Bostrom 2003). If we make the less conservative assumption that future civilizations could eventually press close to the absolute bounds of known physics (using some as yet unimagined technology), we get radically higher estimates of the amount of computation and memory storage that is achievable and thus of the number of years of subjective experience that could be realized.
Indeed, in the same paper, he even suggests that not ending up in such scenarios could count as an existential catastrophe in itself:
Permanent stagnation is instantiated if humanity survives but never reaches technological maturity — that is, the attainment of capabilities affording a level of economic productivity and control over nature that is close to the maximum that could feasibly be achieved (in the fullness of time and in the absence of catastrophic defeaters). For instance, a technologically mature civilization could (presumably) engage in large-scale space colonization through the use of automated self-replicating “von Neumann probes.” (Freitas 1980; Moravec 1988; Tipler 1980) It would also be able to modify and enhance human biology — say, through the use of advanced biotechnology or molecular nanotechnology (Freitas 1999 and 2003). Further, it could construct extremely powerful computational hardware and use it to create whole-brain emulations and entirely artificial types of sentient, superintelligent minds (Sandberg and Bostrom 2008). It might have many additional capabilities, some of which may not be fully imaginable from our current vantage point.
This is also relevant to some other claims of Abraham’s in the post or comments, such as “it seems worth noting that much the literature on longtermism, outside Foundation Research Institute, isn’t making claims explicitly about digital minds as the primary holders of future welfare, but just focuses on the future organic human populations (Greaves and MacAskill’s paper, for example), and similar sized populations to the present day human population at that.” I think this may well be true for the academic literature that’s explicitly about “longtermism”, but I’m less confident it’s true for the wider literature on “longtermism”, or the academic literature that seems effectively longtermist.
It also seems worth noting that, to the extent that a desire to appear respectable/conservative explains why academic work on longtermism shies away from discussing things like digital minds, it may also explain why such literature makes relatively little mention of nonhuman animals. I think a substantial concern for the suffering of wild animals would be seen as similarly “wacky” to many audiences, perhaps even more so than a belief that most “humans” in the future may be digital minds. So it may not be the case that, “behind closed doors”, people from e.g. GPI wouldn’t think about the relevance of animals to far future stuff.
(Personally, I’d prefer it if people could just state all such beliefs pretty openly, but I can understand strategic reasons to refrain from doing so in some settings, unfortunately.)
Also, interestingly, Bostrom does appear to note wild animal suffering in the same paper (though only in one footnote):
We might also have responsibilities to nonhuman beings, such as terrestrial (and possible extraterrestrial) animals. Although we are not currently doing much to help them, we have the opportunity to do so in the future. If rendering aid to suffering nonhuman animals in the natural environment is an important value, then achieving technological maturity in a manner that fails to produce such aid could count as flawed realization. Cf. McMahan 2010; Pearce 2004.
Thanks for this. I think for me the major lessons from comments / conversations here is that many longtermists have much stronger beliefs in the possibility of future digital minds than I thought, and I definitely see how that belief could lead one to think that future digital minds are of overwhelming importance. However, I do think that for utilitarian longtermists, animal considerations might dominate in possible futures where digital minds don’t happen or spread massively, so to some extent one’s credence in my argument / concern for future animals ought to be defined by how much you believe in or disbelieve in the possibility and importance of future digital minds.
As someone who is not particularly familiar with longtermist literature, outside a pretty light review done for this piece, and a general sense of this topic from having spent time in the EA community, I’d say I did not really have the impression that the longtermist community was concerned with future digital minds (outside EA Foundation, etc). Though that just may have been bad luck.
cross-posted from FB.
Really appreciate the time it took you to write this and detailed analysis!
That said, I strongly disagree with this post. This tl;dr of the post is
“Assume total utilitarianism and longtermism is given. Then given several reasonable assumptions and some simple math, wild animal welfare will dominate human welfare in the foreseeable future, so total utilitarian longtermists should predominantly be focused on animal welfare today.”
I think this is wrong, or at least the conclusions don’t follow from the premises, mostly due to weird science-fictiony reasons.
The rest of my rebuttal will be speculative and science-fictiony, so if you prefer reading comments that sound reasonable, I encourage you to read elsewhere.
Like the post I’m critiquing, I will assume longtermism and total utilitarianism for the sake of the argument, and not defend them here. (Unlike the poster, I personally have a lot of sympathy towards both beliefs).
By longtermism, I mean a moral discount rate of epsilon (epsilon >=0, epsilon ~=0). By total utilitarianism, I posit the existence of moral goods (“utility”) that we seek to maximize, and the aggregation function is additive. I’m agnostic for most of the response about what the moral goods in question are (but will try to give plausible ones where it’s necessary to explain subpoints).
I have two core, mostly disjunctive arguments for things the post missed about where value in the long-term future are:
A. Heavy tailed distribution of engineered future experiences
B. Cooperativeness with Future Agents
A1. Claim: Biological organisms today are mostly not optimized/designed for extreme experiences.
I think this is obviously true. Even within the same species (humans), there is a wide variance of reported, eg, happiness for people living in ~the same circumstances, and most people will agree that this represents wide variance in actual happiness (rather than entirely people being mistaken about their own experiences.
Evolutionarily, we’re replicator machines, not experience machines.
This goes for negative as well as positive experiences. Billions of animals are tortured in factory farms, but the telos of factory farms isn’t torture, it’s so that humans get access to meat. No individual animal is *deliberately* optimized by either evolution or selective breeding to suffer.
A2. Claim: It’s possible to design experiences that have much more utility than anything experienced today.
I can outline two viable paths (disjunctive):
A2a. Simulation
For this to hold, you have to believe:
A2ai. Humans or human-like things can be represented digitally.
I think there is philosophical debate, but most people who I trust think this is doable.
A2aii. Such a reproduction can be cheap
I think this is quite reasonable since again, existing animals
(including human animals) are not strongly optimized for computation.
A2aiii. simulated beings are capable of morally relevant experiences or otherwise production of goods of intrinsic moral value.
Some examples may be lots of happy experiences, or (if you have a factor for complexity) lots of varied happy experiences, or other moral goods that you may wish to produce, like great works of art, deep meaningful relationships, justice, scientific advances, etc.
A2b. Genetic engineering
I think this is quite viable. The current variance among human experiences is an existence proof. There are lots of seemingly simple ways that improve on current humans to suffer less, and be happier (eg, lots of unnecessary pain during childbirth just because we’ve evolved to be bipedal + have big heads).
A3. Claim: Our descendants may wish to optimize for positive moral goods.
I think this is a precondition for EAs and do-gooders in general “winning”, so I almost treat the possibility of this as a tautology.
A4. Claim: There is a distinct possibility that a high % of vast future resources will be spent on building valuable moral goods, or the resource costs of individual moral goods are cheap, or both.
A4ai. Proportions: This mostly falls from A3. If enough of our descendants care about optimizing for positive moral goods, then we would reasonably expect them to devote a lot of our resources to producing more of them. Eg, 1% of resources being spent on moral goods isn’t crazy.
A4aii. Absolute resources: Conservatively assuming that we never leave the solar system, right now ~1/10^9 of the Sun’s energy reaches Earth. Of the one-billionth of light that reaches Earth, less than 1% of that energy is used by plants for photosynthesis (~ all of our energy needs, with the exception of nuclear power and geothermal, comes from extracting energy that at one point came from photosynthesis—the extraction itself being a particularly wasteful process. Call it another 1-2 of magnitude discount?).
All of life on Earth uses <1/10^11 (less than one in one hundred billionth!) of the Sun’s energy. Humans use maybe 1/10^12 − 1/10^13 of that.
It’s not crazy that one day we’ll use multiple orders of magnitude of energy more for producing moral goods than we currently spend doing all of our other activities combined.
If, for example, you think the core intrinsic moral good we ought to optimize is “art”, right now Arts and Culture compose 4% of the US GDP (Side note: this is much larger than I would have guessed), and probably a similar or smaller number for world GDP.
A4b. This mostly falls from A2.
A4bi. Genetic engineering: In the spirit of doing things with made-up numbers, it sure seems likely that we can engineer humans to be 10x happier, suffer 10x less, etc. If you have weird moral goals (like art or scientific insight), it’s probably even more doable to genetically engineer humans 100x+ better at producing art, come up with novel mathematics, etc.
A4bii. It’s even more extreme with digital consciousness. The upper bound for cost is however much it costs to emulate (genetically enhanced) humans, which is probably at least 10x cheaper than the biological version, and quite possibly much less than that. But in theory, so many other advances can be made by not limiting ourselves to the human template, and abstractly consider what moral goods we want and how to get there.
A5. Conclusion: for total utilitarians, it seems likely that A1-A4 will lead us to believe that expected utility in the future will be dominated by scenarios of heavy-tails of extreme moral goods.
A5a. Thus, people now should work on some combination of preventing existential risks and steering our descendants (wherever possible) to those heavy-tailed scenarios of producing lots of positive moral goods.
A5b. The argument that people may wish to directly optimize for positive utility, but nobody actively optimizes for negative utility, is in my mind some (and actually quite strong) evidence that total or negative-leaning *hedonic* utilitarians should focus more on avoiding extinction + ensuring positive outcomes than on avoiding negative outcomes.
A5c. If you’re a negative or heavily negative-leaning hedonic utilitarian, then your priority should be to prevent the extreme tail of really really bad engineered negative outcomes. (Another term for this is “S-Risks”)
B. Short argument for Future Agent Cooperation:
B1. Claim: (Moral) Agents will have more power in the future than they do today.
This is in contrast to sub- or above- agent entities (eg, evolution), which held a lot of sway in the past.
B1a. As an aside, this isn’t central to my point that we may expect more computational resources in the future to be used by moral agents rather than just moral patients without agency (like factory farmed animals today).
B2. Claim: Most worlds we care about are ones where agents have a lot of power.
World ruled by non-moral processes rather than agents probably have approximately zero expected utility.
This is actually a disjunctive claim from B1. *Even* if we think B1 is wrong, we still want to care more about agent-ruled worlds, since the future with those are more important.
B3. Claim: Moral uncertainty may lead us to defer to future agents on questions of moral value.
For example, total utilitarians today may be confused about measurements of utility today. To the extent that either moral objectivity or “moral antirealism+get better results after long reflection” is true, our older and wiser descendants (who may well be continuous with us!) will have a better idea of what to do than we do.
B4. Conclusion: While this conclusion is weaker than the previous point, there is prima facie reason that we should be very cooperative to reasonable goals that future agents may have.
C. Either A or B should be substantial evidence that a lot of future moral value comes from thinking through (and getting right) weird futurism concerns. This is in some contrast to doing normal, “respectable” forecasting research and just following expected trendlines like population numbers.
D. What does this mean about animals? I think it’s unclear. A lot of animal work today may help with moral circle expansion.
This is especially true if you think (as I do, but eg, Will Macaskill mostly does not) that we live in a “hinge of history” moment where our values are likely to be locked in in the near future (next 100 years) AND you think that the future is mostly composed of moral patients that are themselves not moral agents (I do not) AND that our descendants are likely to be “wrong” in important ways relative to our current values (I’m fairly neutral on this question, and slightly lean against).
Whatever you believe, it seems hard to escape the conclusion that “weird, science-fictiony scenarios” have non-trivial probability, and that longtermist total utilitarians can’t ignore thinking about them.
I’ve argued against this point here (although I don’t think my objection is very strong). Basically, we (or whoever) could be mistaken about which of our AI tools are sentient or matter, and end up putting them in conditions in which they suffer inadvertently or without concern for them, like factory farmed animals. If sentient tools are adapted to specific conditions (e.g. evolved), a random change in conditions is more likely to be detrimental than beneficial.
Also, individuals who are indifferent to or unaware of negative utility (generally or in certain things) may threaten you with creating a lot of negative utility to get what they want. EAF is doing research on this now.
I don’t think it’s obvious that this is in expectation negative. I’m not at all confident that negative valence is easier to induce than positive valence today (though I think it’s probably true), but conditional upon that being true, I also think it’s a weird quirk of biology that negative valence may be more common than positive valence in evolved animals. Naively I would guess that the experiences of tool AI (that we may wrongly believe to not be sentient, or are otherwise callous towards) is in expectation zero. However, this may be enough for hedonic utilitarians with a moderate negative lean (3-10x, say) to believe that suffering overrides happiness in those cases.
I want to make a weaker claim however, which is that per unit of {experience, resource consumed}, I’d just expect intentional, optimized experience to be multiple orders of magnitude greater than incidental suffering or happiness (or other relevant moral goods).
If this is true, to believe that the *total* expected unintentional suffering (or happiness) of tool AIs to exceed that of intentional experiences of happiness (or suffering), you need to believe that the sheer amount of resources devoted to these tools are several orders of magnitude greater than the optimized resources.
This seems possible but not exceedingly likely.
If I was a negative utilitarian, I might think really hard about trying to prevent agents deliberately optimizing for suffering (which naively I would guess to be pretty unlikely but not vanishingly so).
Yeah that’s a good example. I’m glad someone’s working on this!
It might be 0 in expectation to a classical utilitarian in the conditions for which they are adapted, but I expect it to go negative if the tools are initially developed through evolution (or some other optimization algorithm for design) and RL (for learning and individual behaviour optimization), and then used in different conditions. Think of “sweet spots”: if you raise temperatures, that leads to more deaths by hyperthermia, but if you decrease temperatures, more deaths by hypothermia. Furry animals have been selected to have the right amount of fur for the temperatures they’re exposed to, and sentient tools may be similarly adapted. I think optimization algorithms will tend towards local maxima like this (although by local maxima here, I mean with respect to conditions, while the optimization algorithm is optimizing genes; I don’t have a rigorous proof connecting the two).
On the other hand, environmental conditions which are good to change in one direction and bad in the other should cancel in expectation when making a random change (with a uniform prior), and conditions that lead to improvement in each direction don’t seem stable (or maybe I just can’t even think of any), so are less likely than conditions which are bad to change in each direction. I.e. is there any kind of condition such that a change in each direction is positive? Like increasing the temperature and decreasing the temperature are both good?
This is also a (weak) theoretical argument that wild animal welfare is negative on average, because environmental conditions are constantly changing.
Fair enough on the rest.
Thanks for laying out this response! It was really interesting, and I think probably a good reason to not take animals as seriously as I suggest you ought to, if you hold these beliefs.
I think something interesting is that this, and the other objections that have been presented to my piece have brought out is that to avoid focusing exclusively on animals in longtermist projects, you have to have some level of faith in these science-fiction scenarios happening. I don’t necessarily think that is a bad thing, but it isn’t something that’s been made explicit in past discussions of long-termism (at least, in the academic literature), and perhaps ought to be explicit?
A few comments on your two arguments:
This isn’t usually assumed in the longtermist literature. It seems more like the argument is made on the basis of future human lives being net-positive, and therefore good that there will be many of them. I think the expected value of your argument (A) hinges on this claim, so it seems like accepting it as a tautology, or something similar, is actually really risky. If you think this is basically 100% likely to be true, of course your conclusion might be true. But if you don’t, it seems plausible that that, like you mention, priority possibly ought to be on s-risks.
In general, a way to summarize this argument, and others given here could be something like, “there is a non-zero chance that we can make loads and loads of digital welfare in the future (more than exists now), so we should focus on reducing existential risk in order to ensure that future can happen”. This raises a question—when will that claim not be true / the argument you’re making not be relevant? It seems plausible that this kind of argument is a justification to work on existential risk reduction until basically the end of the universe (unless we somehow solve it with 100% certainty, etc.), because we might always assume future people will be better at producing welfare than us.
I assume people have discussed the above, and I’m not well read in the area, but it strikes me as odd that the primary justification in these sci-fi scenarios for working on the future is just a claim that can always be made, instead of working directly on making lives with good welfare (but maybe this is a consideration with longtermism in general, and not just this argument).
I guess part of the issue here is you could have an incredibly tiny credence in a very specific number of things being true (the present being at the hinge of history, various things about future sci-fi scenarios), and having those credences would always justify deferral of action.
I’m not totally sure what to make of this, but I do think it gives me pause. But, I admit I haven’t really thought about any of the above much, and don’t read in this area at all.
Thanks again for the response!
tl;dr: My above comment relies on longtermism + total utilitarianism (but I attempted to be neutral on the exact moral goods that compose the abstraction of “utility”). With those two strong assumptions + a bunch of other more reasonable ones, I think you can’t escape thinking about science-fictiony scenarios. I think you may not need to care as much about science-fictiony scenarios with moderate probabilities (but extremely high payoffs in expected utility) if your views are primarily non-consequentialist, or if you’re a consequentialist but the aggregative function is not additive.
I also appreciate your thoughtful post and responses!
Having read relatively little of it, my understanding is that the point of the academic literature (which do not usually assume total utilitarian views?) on longtermism is to show that longtermism is compatible (in some cases required) under a broad scope of moral views that are considered respectable within the academic literature.
So they don’t talk about science-fictiony stuff, since their claim is that longtermism is robustly true (or compatible) with reasonable academic views in moral philosophy.
The point of my comment is that longtermism + total utilitarianism must lead you to think about these science-fictiony scenarios that have significant probability, rather than that longtermism itself must lead you to consider them.
I think if the credence is sufficiently low, either moral uncertainty (since most people aren’t total utilitarians with 100% probability) or model uncertainty will get you to do different actions.
At very low probabilities, you run into issues like Pascal’s Wager and Pascal’s Mugging, but right now the future is so hazy that I think it’s too hubristic to say anything super-concrete about the future. I’m reasonably confident that I’m willing to defend that all of the claims I’ve made above has percentage points of probability[1], which I think is well above the threshold for “not willing to get mugged by low probabilities.”
I suspect that longetermism + moral axiologies that are less demanding/less tail-driven than total utilitarianism will rely less on the speculative/weird/science-fictiony stuff. I hadn’t thought about them in detail.
To demonstrate what I roughly mean, I made up two imaginary axilogies (I think I can understand other nonhedonic total utilitarian views well enough to represent them faithfully, but I’m not well-read enough on non-total utilitarian views, so I made up fake ones rather than accidentally strawman existing views):
1. An example of an axiology that is long-termist but not utilitarian is that you want to maximize the *probability* that the long-term future will be a “just” world, where “justice” is a binary variable (rather than a moral good that can be maximized). If you have some naive prior that this will have 50-50 chance of happening by default, then you might want to care somewhat about extremely good outcomes for justice (eg, creating a world which can never backslide into an unjust one), and extremely bad outcomes (avoiding a dictatorial lock-in that will necessitate permanent unjust society).
But your decisions are by default going to focus more on the median outcomes rather than the tails. Depending on whether you think animals are treated justly right now, this may entail doing substantial work on farmed animals (and whether to work on wild animal welfare depends on positive vs negative conceptions of justice).
2. An example of an axiology that is long-termist and arguably utilitarian but not total utilitarian is if you have something like instead of doing Sum(Utility across all beings), you instead have Average(Log(Utility per being)). In such a world, the tails dominate if and only if you can have a plausible case for why the tails will have extremely large outcomes even on a log scale. I think this is still technically possible, but you need much stronger assumptions or better arguments than the ones I outlined above.
I’d actually be excited for you or someone else with non-total utilitarian views to look into what other moral philosophies (that people actually believe) + longtermism will entail.
[1] A way to rephrase “there’s less than a 1% probability that our descendants will wish to optimize for moral goods” is that “I’m over 99% confident that our descendants wouldn’t care about moral goods, or care very little about them.” And I just don’t think we know enough about the longterm future to be that confident about anything like that.
Yeah, the idea of looking into longtermism for nonutilitarians is interesting to me. Thanks for the suggestion!
I think regardless, this helped clarify a lot of things for me about particular beliefs longtermists might hold (to various degrees). Thanks!
Yeah I think that’d be useful to do.
I’m glad it was helpful!
This is also my impression of some of Toby Ord’s work in The Precipice (particularly chapter 2) and some of the work of GPI, at least. I’m not sure how much it applies more widely to academic work that’s explicitly on longtermism, as I haven’t read a great deal of it yet.
On the other hand, many of Bostrom’s seminal works on existential risks very explicitly refer to such “science-fictiony” scenarios.. And these works effectively seem like seminal works for longtermism too, even if they didn’t yet use that term. E.g., Bostrom writes:
Indeed, in the same paper, he even suggests that not ending up in such scenarios could count as an existential catastrophe in itself:
This is also relevant to some other claims of Abraham’s in the post or comments, such as “it seems worth noting that much the literature on longtermism, outside Foundation Research Institute, isn’t making claims explicitly about digital minds as the primary holders of future welfare, but just focuses on the future organic human populations (Greaves and MacAskill’s paper, for example), and similar sized populations to the present day human population at that.” I think this may well be true for the academic literature that’s explicitly about “longtermism”, but I’m less confident it’s true for the wider literature on “longtermism”, or the academic literature that seems effectively longtermist.
It also seems worth noting that, to the extent that a desire to appear respectable/conservative explains why academic work on longtermism shies away from discussing things like digital minds, it may also explain why such literature makes relatively little mention of nonhuman animals. I think a substantial concern for the suffering of wild animals would be seen as similarly “wacky” to many audiences, perhaps even more so than a belief that most “humans” in the future may be digital minds. So it may not be the case that, “behind closed doors”, people from e.g. GPI wouldn’t think about the relevance of animals to far future stuff.
(Personally, I’d prefer it if people could just state all such beliefs pretty openly, but I can understand strategic reasons to refrain from doing so in some settings, unfortunately.)
Also, interestingly, Bostrom does appear to note wild animal suffering in the same paper (though only in one footnote):
Thanks for this. I think for me the major lessons from comments / conversations here is that many longtermists have much stronger beliefs in the possibility of future digital minds than I thought, and I definitely see how that belief could lead one to think that future digital minds are of overwhelming importance. However, I do think that for utilitarian longtermists, animal considerations might dominate in possible futures where digital minds don’t happen or spread massively, so to some extent one’s credence in my argument / concern for future animals ought to be defined by how much you believe in or disbelieve in the possibility and importance of future digital minds.
As someone who is not particularly familiar with longtermist literature, outside a pretty light review done for this piece, and a general sense of this topic from having spent time in the EA community, I’d say I did not really have the impression that the longtermist community was concerned with future digital minds (outside EA Foundation, etc). Though that just may have been bad luck.
Did you mean not moral agents?
Yes, thanks for the catch!