Philosophy, global priorities and animal welfare research. My current specific interests include: philosophy of mind, moral weights, person-affecting views, preference-based views and subjectivism, moral uncertainty, decision theory, deep uncertainty/cluelessness and backfire risks, s-risks, and indirect effects on wild animals.
I’ve also done economic modelling for some animal welfare issues.
Couldn’t a country just opt out unilaterally, and then others follow suit? And should we trust their assessment of s-risks even if proceeding by global consensus?
You could defend the idea that extinction risk reduction is net negative or highly ambiguous in value, even just within EA and adjacent communities. Convincing people to not work on things that are net negative by your lights seems not to break good heuristics or norms.
When I edited this comment, it removed my vote percentage from it.
(Edited to elaborate and for clarity.)
Thomas (2019) calls these sorts of person-affecting views “wide”. I think “narrow” person-affecting views can be more liberal (due to incommensurability) about what kinds of beings are brought about.
And narrow asymmetric person-affecting views, as in Thomas, 2019 and Pummer, 2024, can still tell you to prevent “bad” lives or bads in lives, but, contrary to antinatalist views, “good” lives and goods in lives can still offset the bad. Pummer (2024) solves a special case of the Nonidentity problem this way, by looking at goods and bads in lives.
But these asymmetric views may be less liberal than strict/symmetric narrow person-affecting views, because they could be inclined to prevent the sorts of lives of which many are bad in favour of better average lives. Or more liberal, depending on how you think of liberalism. If someone would have a horrible life to which they would object, it seems illiberal to force them to have it.
I think these papers have made some pretty important progress in further developing person-affecting views.[1]
I think they need to be better adapted to choices between more than 2 options, in order to avoid the Repugnant Conclusion and replacement (St. Jules, 2024). I’ve been working on this and have a tentative solution, but I’m struggling to find anyone interested in reading my draft.
For a given individual, can they have a higher probability of averting extinction (i.e. making the difference) or for a different long-term trajectory change? If you discount small enough probabilities of making a difference or are otherwise difference-making risk averse (as an individual), would one come out ahead as a result?
Some thoughts: extinction is a binary event. But there’s a continuum of possible values that future agents could have, including under value lock-in. A small tweak in locked-in values seems more achievable counterfactually than being the difference for whether we go extinct. And a small tweak in locked-in values would still have astronomical impact if they persist into the far future. It seems like value change might depend less on very small probabilities of making a difference.
Since others have discussed the implications, I want to push a bit on the assumptions.
I worry that non-linear axiologies[1] end up endorsing egoism, helping only those whose moral patienthood you are most confident in or otherwise prioritizing them far too much over those of less certain moral patienthood. See Oesterheld, 2017 and Tarsney, 2023.
(I also think average utilitarianism in particular is pretty bad, because it would imply that if the average welfare is negative (even torturous), adding bad lives can be good, as long as they’re even slightly less bad than average.)
Maybe you can get around this with non-aggregative or partially aggregative views. EDIT: Or, if you’re worried about fanaticism, difference-making views.
Assuming completeness, transitivity and the independence of irrelevant alternatives and each marginal moral patient matters less.
I don’t think it’s valuable to ensure future moral patients exist for their own sake, and extinction risk reduction only really seems to expectably benefit humans who would otherwise die in an extinction event, who would be in the billions. An astronomical number of future moral patients could have welfare at stake if we don’t go extinct, so I’d prioritize them on the basis of their numbers.
I might go back and forth on whether “the good” exists, as my subjective order over each set of outcomes (or set of outcome distributions). This example seems pretty compelling against it.
However, I’m first concerned with “good/bad/better/worse to someone” or “good/bad/better/worse from a particular perspective”. Then, ethics is about doing better by and managing tradeoffs between these perspectives, including as they change (e.g. with additional perspectives created through additional moral patients). This is what my sequence is about. Whether “the good” exists doesn’t seem very important.
Hmm, interesting.
I think this bit from the footnote helped clarify, since I wasn’t sure what you meant in your comment:
Note, however, that there is no assumption that d—f are outcomes for anyone to choose, as opposed to outcomes that might arise naturally. Thus, it is not clear how the appeal to choice set dependent betterness can be used to block the argument that f is not worse than d, since there are no choice sets in play here.
I might be inclined to compare outcome distributions using the same person-affecting rules as I would for option sets, whether or not they’re being chosen by anyone. I think this can make sense on actualist person-affecting views, illustrated with my “Best in the outcome argument”s here, which is framed in terms of betterness (between two outcome distributions) and not choice. (The “Deliberation path argument” is framed in terms of choice.)
Then, I’d disagree with this:
And if, say, it makes the outcome better if an additional happy person happens to exist without anyone making it so
I’m a moral anti-realist (subjectivist), so I don’t think there’s an objective (stance-independent) fact of the matter. I’m just describing what I would expect to continue to endorse under (idealized) reflection, which depends on my own moral intuitions. The asymmetry is one of my strongest moral intuitions, so I expect not to give it up, and if it conflicts with other intuitions of mine, I’d sooner give those up instead.
How asymmetric do you think things are?
I think I’m ~100% on no non-instrumental benefit from creating moral patients. Also pretty high on no non-instrumental benefit from creating new desires, preferences, values, etc. within existing moral patients. (I try to develop and explain my views in this sequence.)
I haven’t thought a lot about tradeoffs between suffering and other things, including pleasure, within moral patients that would exist anyway. I could see these tradeoffs going like they would for a classical utilitarian, if we hold an individual’s dispositions fixed.
To be clear, I’m a moral anti-realist (subjectivist), so I don’t think there’s any stance-independent fact about how asymmetric things should be.
Also, I’m curious if we can explain why you react like this:
Maximising pleasure intuitively feels meh to me, but maximising suffering sounds pretty awful
Some ideas: Complexity of value but not disvalue or the urgency of suffering is explained by the intensity of desire, not unpleasantness? Do you have any ideas?
A pause would still give more groups more time catch up on existing research and to build infrastructure for AGI (energy, datacenters), right? Then when the pause is lifted, we could have more players at the research frontier and ready to train frontier models.
I think Thomas did not take a stance on whether it was axiological or deontic in the GPI working paper, and instead just described the structure of a possible view. Pummer described his as specifically deontic and not axiological.
I’m not sure what should be classified as axiological or how important the distinction is. I’m certainly giving up the independence of irrelevant alternatives, but I think I can still rank outcomes in a way that depends on the option set.
I’ll get back to you on this, since I think this will take me longer to answer and can get pretty technical.
That seems true, but I’m not convinced it’s the best way to reduce s-risks on the margin. See, for example, Vinding, 2024.
I’d also want to see a fuller analysis of ways it could backfire. For example, a pause might make multipolar scenarios more likely by giving more groups time to build AGI, which could increase the risks of conflict-based s-risks.
Yes, we probably aren’t benefitting future individuals in a strict and narrow person-affecting sense much or at all.
However, there are some other person-affecting views that are concerned with differences for future moral patients:
On wide person-affecting views, if Alice would have a better life than Bob, then it’s better for Alice to come to exist than for Bob to come to exist, all else equal (the Nonidentity problem). This doesn’t imply it’s better to ensure Alice or Bob exists than neither does, though. (See Thomas, 2019 or Meacham, 2012 for examples).
On asymmetric person-affecting views, it can still be good to prevent bad lives. (This needn’t imply antinatalism, because it could be that good lives can offset bad lives (Thomas, 2019, Pummer, 2024).)
I’m also pretty skeptical of convergence, largely because I’m a moral anti-realist. I don’t see why we would converge to any view in particular, except by coincidence or mechanisms that don’t track stance-independent moral truths (because there are none). It’s just people weighing their own particular moral intuitions. Humans differ in our most basic moral intuitions and leanings.
Barring value lock-in, I suspect there would be convergence towards the recognition that unnecessary suffering is bad and worth preventing (when cheap enough) because this seems pretty widely held and something societies move towards, but I’d guess there will still be disagreement on some of these:
population ethics
hedonism vs preference views vs others
whether non-sentient/non-conscious things matter terminally, e.g. preserving nature
whether groups of moral patients have special status beyond their aggregates, e.g. ethnic groups, species
deontology vs consequentialism vs virtue ethics
what counts as conscious/sentient (I think this is partly normative, not just empirical)
decision theory, attitudes towards risk and ambiguity, fanaticism
Starting my own discussion thread.
My biggest doubt for the value of extinction risk reduction is my (asymmetric) person-affecting intuitions: I don’t think it makes things better to ensure future people (or other moral patients) come to exist for their own sake or the sake of the value within their own lives. But if future people will exist, I want to make sure things go well for them. This is summarized by the slogan “Make people happy, not make happy people”.
If this holds, then extinction risk reduction saves the lives of people who would otherwise die in an extinction event, which is presumably good for them, but this is only billions of humans.[1] If we don’t go extinct, then the number of our descendant moral patients could be astronomical. It therefore seems better to prioritize our descendant moral patients conditional on our survival because there are far far more of them.
Aliens (including alien artificial intelligence) complicate the picture. We (our descendants, whether human, AI or otherwise) could
use the resources aliens would have otherwise for our purposes instead of theirs, i.e. replace them,
help them, or
harm them or be harmed by them, e.g. through conflict.
I’m interested in others’ takes on this.
And it’s not clear we want to save other animals, in case their lives are bad on average. It can also make a difference whether we’re talking about human extinction only or all animal extinction.
Since this comment gets some attention every now and then, I’m leaving another comment here to point to this sequence where I flesh out this kind of view more.
Also, for those interested in animal welfare specifically: https://www.animaladvocacycareers.org/
Seems fine to direct people to 80,000 Hours for AI/x-risk, Animal Advocacy Careers for animal welfare and Probably Good more generally.