I gave this a read through, and then asked claude to summarise. I’m curious how accurate you find the following summary:
Wild animal suffering is a more pressing cause area than existential risk prevention according to total utilitarianism and longtermism. This is a central thesis being argued.
Humans will likely be vastly outnumbered by wild animals, even if humans spread to space. This is based on the vast number of wildlife on Earth currently and the potential for them to also spread to space.
Any human or animal suffering before an existential catastrophe would be negligible compared to the potential future suffering prevented by reducing existential risk. This is a core longtermist assumption.
It may be possible to make “permanent” changes to wild animal welfare by developing interventions and spreading concern for wild animal suffering now. The author argues this could influence how animals are treated in the far future.
Both x-risk prevention and wild animal welfare are highly neglected areas compared to their importance. This informs the tractability estimates.
Personal fit should be a major factor in cause prioritization given the high uncertainty in these long-term forecasts. This is why the author recommends staying in your cause area if you already have a good fit.
The future could plausibly be net negative in expectation according to the author’s assumptions. This makes reducing existential risk potentially net negative.
The spread of wildlife to space is likely and difficult to reverse, making wild animal welfare interventions potentially high impact. This informs the tractability of permanently improving wild animal welfare.
Various assumptions about the probabilities and potential scales of different existential and suffering risks, which inform the expected value calculations.
To broadly summarise my thoughts here:
● I agree with others that it’s really great to be thinking through your own models of cause prio ● I am sceptical of S-Risk related arguments that point towards x-risk mitigation being negative. I believe the dominant consideration is option value. Option value could be defined as “keeping your options open in case something better comes up later”. Option value is pretty robustly good if your unsure about a lot of things (like whether the long-term future is filled with good or bad). I suspect X-risk mitigation preserves option value particularly well. For that reason I prefer the phrase “preventing lock-in” vs “preventing x-risk”, since we’re focusing our sights on option value. ● Why should somebody suffering focused want to preserve option value? Even if you assume the long-term future is filled with suffering, you should have uncertainty factored inside and outside your model—which leads you wanting to extend the amount of time and people working on the problem, and the options they have to choose from.
Also, I agree with @Isaac Dunn that it’s really great you’re thinking through your own cause prioritisation.
Thanks for reading the post and for the summary. I find the summary quite accurate, but here’s a few notes:
Point 1: I’m not trying to argue that WAW is a more pressing cause area, however that IF my estimations are somewhat correct, then WAW and x-risk prevention are roughly similar. Also I wanted to present a possible way (the “main formula”) that could universally be used to compare longtermist causes. (Sidenote: I actually considered writing the post as a comparison of “insuring a future” to “making the future better”)
Point 3: I agree, except for the use of the word “existential risk”. As I see it (for the most part), suffering-risks prevention are for reducing future suffering and existential risk prevention are for ensuring that the hopefully net positive future will happen. Even though some areas might be related to both (e.g. AI-alignment). So I think the word “existential risk” should be changed to “suffering-risks”.
The priority of what’s most important might be a bit off (which I guess is to be expected with AI). E.g. I think “recommendations on how to use this information” should take up a bigger part of a summary that size, because of its higher importance. But I guess that’s debatable.
IDK if this is relevant but here we go: I don’t think the summary fits the target audience super well. E.g. “Both x-risk prevention and wild animal welfare are highly neglected areas compared to their importance” is probably not that new information for most people reading this. But this might be fixable with giving more precise instructions to the AI.
Thanks alot for your thoughts on my theory. I never heard the term option value before, I’m gonna read more about it and see if it changes my beliefs. Here’s my thoughts on your thoughts:
- I didn’t directly consider option value in my theory/calculation, but I think there is a strong overlap since my calculation only considers “permanent” changes (similar to the “lock-in” you’re referring to).
- To clarify, I don’t believe that the mean future will be net negative. However the possibility of a “net-negative lock-in scenario” makes the expected value of working on preventing x-risk lower.
- I’m somewhat skeptical of the value of option value, because it assumes that humans will do the right thing. It’s important to remember that the world is not made up of EAs or philosophers and an AGI will likely not have much better values than the people who created it or controls it. And because of humans’ natural limited moral circle, I think it’s likely that the majority of sentient experiences (most likely from (wild) animals or artificial sentience) will be mostly ignored. Which could mean that the future overall will be net negative, even if we don’t end up in any “lock-in” scenario.
- With all that said, I still think the mean value of working on x-risk mitigation is extremely valuable. Maybe even the most or second most impactful cause area based on total utilitarianism and longtermism. But I do think that the likelihood and scale of certain “lock-in” net negative futures, could potentially make working on s-risk directly or indirectly more impactful.
- I’m somewhat skeptical of the value of option value, because it assumes that humans will do the right thing.
I’d argue there’s a much lower bar for an option value preference. To have a strong preference for option value, you need only assume that you’re not the most informed, most capable person to make that decision.
To intuition pump, this is a (good imo) reason why doctors recommend young people wait before getting a vasectomy. That person is able to use other forms of contraception whilst they hand that decision to someone that might be better informed (i.e. themselves in 10 years time).
Because of physical ageing, we don’t often encounter option value in our personal lives. It’s pretty common for choices available to us to be close to equal on option value. But this isn’t the case when we are taking decisions that have implications on the long term future.
But I do think that the likelihood and scale of certain “lock-in” net negative futures, could potentially make working on s-risk directly or indirectly more impactful.
To what extent do you think approaches like AI-alignment will protect against S-risks? Or phrased another way, how often will unaligned super-intelligence result in a S-risk scenario.
______________
I want to try explore some of the assumptions that are building your world model. Why do you think that the world, in our current moment, contains more suffering than pleasure? What forces do you think resulted in this equilibrium?
I’d argue there’s a much lower bar for an option value preference. To have a strong preference for option value, you need only assume that you’re not the most informed, most capable person to make that decision.
I do agree that there are more capable people to make that decision than me and there will be even better in the future. But I don’t believe this to be the right assessment for the desirability of option value. I think the more correct question is “whether the future person/people in power (which may be the opinion of the average human in case of a “singleton democracy”) would be more capable than me?”.
I feel unsure whether my morals will be better or worse than that future person or people because of the following:
The vast majority of moral patients currently, according to my knowledge, are invertebrates (excluding potential/unknown sentient beings like aliens, AI made by aliens, AI sentient humans already made unknowingly, microorganisms etc.). My impression is that the mean moral circle is wider than it was 10 years ago and that most people’s moral circle increases with the decrease in poverty, the decrease in personal problems and the increase in free time. However, whether or not the majority will ever care about “ant-suffering” and the belief that interventions should be done is unclear to me. (So this argument can go both ways)
A similar argument can be used for future AI sentients. My impression is that a lot of humans care somewhat about AI sentients and that this will most likely increase in the future. However, I’m unsure how much people will care if AI sentients mainly come from non-communicating computers that have next to nothing in common with humans.
To what extent do you think approaches like AI-alignment will protect against S-risks? Or phrased another way, how often will unaligned super-intelligence result in a S-risk scenario.
Well, I think working on AI-alignment could significantly decrease the likelihood of s-risks where humans are the main ones suffering. So if that’s your main concern, then working on AI-alignment is the best option (both with your and my beliefs).
While I don’t think that the probability of “AGI-caused S-risk” is high. I also don’t think the AGI will prevent or care largely for invertebrates or artificial sentience. E.g. I don’t think the AGI will stop a person from doing directed panspermia or prevent the development of artificial sentience. I think the AGI will most likely have similar values to the people who created it or control it (which might again be (partly) the whole human adult population).
I’m also worried that if WAW concerns are not spread, nature conservation (or less likely but even worse, the spread of nature) will be the enforced value. Which could prevent our attempts to make nature better and ensure that the natural suffering will continue.
And since you asked for beliefs of the likelihood, here you go (partly copied from my explanation in Appendix 4):
I put the “probability” for an “AI misalignment caused s-risk” as being pretty low (1 %), because most scenarios of AI misalignment, will according to my previous statements, be negligible (talking about s-risk, not x-risk). It would in this case only be relevant if AI keeps us and/or animals alive “permanently” to have net negative lives (which most likely would require traveling outside of the solar system). I also put “how bad the scenario would be” pretty low (0,5) because I think most likely (but not guaranteed) the impact will be minimal to animals (which technically might mean that it would not be considered a s-risk).
I want to try explore some of the assumptions that are building your world model. Why do you think that the world, in our current moment, contains more suffering than pleasure? What forces do you think resulted in this equilibrium?
I would argue that whether or not the current world is net positive or net negative depends on the experience of invertebrates since they make up the majority of moral patients. Most people caring about WAW believe one of the following:
That invertebrates most likely suffer more than they experience pleasure.
It is unclear whether invertebrates suffer or experience pleasure more.
I’m actually leaning more towards the latter. My guess is there’s a 60 % probability that they suffer more and a 40 % probability that they feel pleasure more.
So the cause for my belief that the current world is slightly more likely to be net negative is simply: evolution did not take ethics into account. (So the current situation is unrelated to my faith in humanity).
With all that said, I still think the future is more likely to be net positive than net negative.
I gave this a read through, and then asked claude to summarise. I’m curious how accurate you find the following summary:
To broadly summarise my thoughts here:
● I agree with others that it’s really great to be thinking through your own models of cause prio
● I am sceptical of S-Risk related arguments that point towards x-risk mitigation being negative. I believe the dominant consideration is option value. Option value could be defined as “keeping your options open in case something better comes up later”. Option value is pretty robustly good if your unsure about a lot of things (like whether the long-term future is filled with good or bad). I suspect X-risk mitigation preserves option value particularly well. For that reason I prefer the phrase “preventing lock-in” vs “preventing x-risk”, since we’re focusing our sights on option value.
● Why should somebody suffering focused want to preserve option value? Even if you assume the long-term future is filled with suffering, you should have uncertainty factored inside and outside your model—which leads you wanting to extend the amount of time and people working on the problem, and the options they have to choose from.
Also, I agree with @Isaac Dunn that it’s really great you’re thinking through your own cause prioritisation.
Thanks for reading the post and for the summary. I find the summary quite accurate, but here’s a few notes:
Point 1: I’m not trying to argue that WAW is a more pressing cause area, however that IF my estimations are somewhat correct, then WAW and x-risk prevention are roughly similar. Also I wanted to present a possible way (the “main formula”) that could universally be used to compare longtermist causes. (Sidenote: I actually considered writing the post as a comparison of “insuring a future” to “making the future better”)
Point 3: I agree, except for the use of the word “existential risk”. As I see it (for the most part), suffering-risks prevention are for reducing future suffering and existential risk prevention are for ensuring that the hopefully net positive future will happen. Even though some areas might be related to both (e.g. AI-alignment). So I think the word “existential risk” should be changed to “suffering-risks”.
The priority of what’s most important might be a bit off (which I guess is to be expected with AI). E.g. I think “recommendations on how to use this information” should take up a bigger part of a summary that size, because of its higher importance. But I guess that’s debatable.
IDK if this is relevant but here we go: I don’t think the summary fits the target audience super well. E.g. “Both x-risk prevention and wild animal welfare are highly neglected areas compared to their importance” is probably not that new information for most people reading this. But this might be fixable with giving more precise instructions to the AI.
Thanks alot for your thoughts on my theory. I never heard the term option value before, I’m gonna read more about it and see if it changes my beliefs. Here’s my thoughts on your thoughts:
- I didn’t directly consider option value in my theory/calculation, but I think there is a strong overlap since my calculation only considers “permanent” changes (similar to the “lock-in” you’re referring to).
- To clarify, I don’t believe that the mean future will be net negative. However the possibility of a “net-negative lock-in scenario” makes the expected value of working on preventing x-risk lower.
- I’m somewhat skeptical of the value of option value, because it assumes that humans will do the right thing. It’s important to remember that the world is not made up of EAs or philosophers and an AGI will likely not have much better values than the people who created it or controls it. And because of humans’ natural limited moral circle, I think it’s likely that the majority of sentient experiences (most likely from (wild) animals or artificial sentience) will be mostly ignored. Which could mean that the future overall will be net negative, even if we don’t end up in any “lock-in” scenario.
- With all that said, I still think the mean value of working on x-risk mitigation is extremely valuable. Maybe even the most or second most impactful cause area based on total utilitarianism and longtermism. But I do think that the likelihood and scale of certain “lock-in” net negative futures, could potentially make working on s-risk directly or indirectly more impactful.
Feel free to change my mind on any of this.
I’d argue there’s a much lower bar for an option value preference. To have a strong preference for option value, you need only assume that you’re not the most informed, most capable person to make that decision.
To intuition pump, this is a (good imo) reason why doctors recommend young people wait before getting a vasectomy. That person is able to use other forms of contraception whilst they hand that decision to someone that might be better informed (i.e. themselves in 10 years time).
Because of physical ageing, we don’t often encounter option value in our personal lives. It’s pretty common for choices available to us to be close to equal on option value. But this isn’t the case when we are taking decisions that have implications on the long term future.
To what extent do you think approaches like AI-alignment will protect against S-risks? Or phrased another way, how often will unaligned super-intelligence result in a S-risk scenario.
______________
I want to try explore some of the assumptions that are building your world model. Why do you think that the world, in our current moment, contains more suffering than pleasure? What forces do you think resulted in this equilibrium?
I do agree that there are more capable people to make that decision than me and there will be even better in the future. But I don’t believe this to be the right assessment for the desirability of option value. I think the more correct question is “whether the future person/people in power (which may be the opinion of the average human in case of a “singleton democracy”) would be more capable than me?”.
I feel unsure whether my morals will be better or worse than that future person or people because of the following:
The vast majority of moral patients currently, according to my knowledge, are invertebrates (excluding potential/unknown sentient beings like aliens, AI made by aliens, AI sentient humans already made unknowingly, microorganisms etc.). My impression is that the mean moral circle is wider than it was 10 years ago and that most people’s moral circle increases with the decrease in poverty, the decrease in personal problems and the increase in free time. However, whether or not the majority will ever care about “ant-suffering” and the belief that interventions should be done is unclear to me. (So this argument can go both ways)
A similar argument can be used for future AI sentients. My impression is that a lot of humans care somewhat about AI sentients and that this will most likely increase in the future. However, I’m unsure how much people will care if AI sentients mainly come from non-communicating computers that have next to nothing in common with humans.
Well, I think working on AI-alignment could significantly decrease the likelihood of s-risks where humans are the main ones suffering. So if that’s your main concern, then working on AI-alignment is the best option (both with your and my beliefs).
While I don’t think that the probability of “AGI-caused S-risk” is high. I also don’t think the AGI will prevent or care largely for invertebrates or artificial sentience. E.g. I don’t think the AGI will stop a person from doing directed panspermia or prevent the development of artificial sentience. I think the AGI will most likely have similar values to the people who created it or control it (which might again be (partly) the whole human adult population).
I’m also worried that if WAW concerns are not spread, nature conservation (or less likely but even worse, the spread of nature) will be the enforced value. Which could prevent our attempts to make nature better and ensure that the natural suffering will continue.
And since you asked for beliefs of the likelihood, here you go (partly copied from my explanation in Appendix 4):
I put the “probability” for an “AI misalignment caused s-risk” as being pretty low (1 %), because most scenarios of AI misalignment, will according to my previous statements, be negligible (talking about s-risk, not x-risk). It would in this case only be relevant if AI keeps us and/or animals alive “permanently” to have net negative lives (which most likely would require traveling outside of the solar system). I also put “how bad the scenario would be” pretty low (0,5) because I think most likely (but not guaranteed) the impact will be minimal to animals (which technically might mean that it would not be considered a s-risk).
I would argue that whether or not the current world is net positive or net negative depends on the experience of invertebrates since they make up the majority of moral patients. Most people caring about WAW believe one of the following:
That invertebrates most likely suffer more than they experience pleasure.
It is unclear whether invertebrates suffer or experience pleasure more.
I’m actually leaning more towards the latter. My guess is there’s a 60 % probability that they suffer more and a 40 % probability that they feel pleasure more.
So the cause for my belief that the current world is slightly more likely to be net negative is simply: evolution did not take ethics into account. (So the current situation is unrelated to my faith in humanity).
With all that said, I still think the future is more likely to be net positive than net negative.