Some additional thoughts on existential and extinction risk
1.
Michael Aird (2020), âDatabase of existential risk estimatesâ (an EA Forum post accompanying the above-linked spreadsheet), addresses the fact that we only have extremely rough estimates of the extinction probability. He reviews some of the implications of this fact, and ultimately concludes that attempting to construct such estimates is still worthwhile. I think he explains the relevant issues pretty well, so I wonât address this problem other than to say that I basically endorse Airdâs analysis.
Iâm very glad you seem to have found this database useful as one input into this valuable-seeming project!
If any readers are interested in my arguments/âanalysis on that matter, Iâd actually recommend instead my EAGxVirtual Unconference talk. Itâs basically a better structured version of my post (though lacking useful links), as by then Iâd had a couple extra months to organise my thoughts on the topic.
2.
If we use a moderately high estimate for the current probability of extinction (say, 0.2% per year), it seems implausible that this probability could remain at a similar level for thousands of years. A 0.2% annual extinction probability translates into a 1 in 500 million chance that humanity lasts longer than 10,000 years. Humanity has already survived for about 200,000 years, so on priors, this tiny probability seems extremely suspect.
Iâm not sure I see the reasoning in that last sentence. It seems like youâre saying that something which has a 1 in 500 million chance of happening is unlikelyâwhich is basically true âby definitionââand that we know this because humanity already survived about 200,000 yearsâwhich seems somewhat irrelevant, and in any case unnecessary to point out? Wouldnât it be sufficiently merely to note that an annual 0.2% chance of A happening (whatever A is) means that, over a long enough time, itâs extremely likely that either A has happened already or the annual chance actually went down?
Relatedly, you write:
The third claim allows us to use a nontrivial long-term discount due to existential risk. I find it the least plausible of the threeânot because of particularly any good inside-view argument, but because it seems unlikely on priors.
Canât we just say it is unlikelyâit logically must involve extremely low probabilities, even if we abstract away all the specificsârather than than it seems unlikely on priors, or based on some reference class forecasting, or the like?
(Maybe Iâm totally misunderstanding what youâre getting at here.)
3.
One of these three claims must be true:
1. The annual probability of extinction is quite low, on the order of 0.001% per year or less.
2. Currently, we have a relatively high probability of extinction, but if we survive through the current crucial period, then this probability will dramatically decrease.
[...] The second claim seems to represent the most common view among long-term-focused effective altruists.
Personally, I see it as something like âThereâs a 5-90% chance that people like Toby Ord are basically right, and thus that 2 is true. Iâm not very confident about that, and 1 is also very plausible. But this is enough to make the expected value of existential risk reduction very high (as long as there are tractable reduction strategies which wouldnât be adopted âby defaultâ).â
I suspect that something like that perspectiveâin which view 2 is not given far more credence than view 1, but ends up seeming especially decision-relevantâis quite common among longtermist EAs. (Though Iâm sure there are also many with more certainty in view 2.)
(This isnât really a key pointâjust sharing how I see things.)
4. When you say âlong-run discount rateâ, do you mean âdiscount-rate that applies after the short-runâ, or âdiscount rate that applies from now till a very long time from nowâ? Iâm guessing you mean the former?
I ask because you say:
If we accept the first or second claim, this implies existential risk has nearly zero impact on the long-run discount rate
But it seems like the second claimâa high extinction risk now, which declines laterâcould still imply a non-trivial total existential risk across all time (e.g., 25%), just with this mostly concentrated over the coming decades or centuries.
My point was that we know humanity is capable of lasting 200,000 years, because it already did that. So on priors, we should expect humanity to last about another 200,000 years. We might update this prior downward based on facts like âwe have nukes nowâ or âwe might develop unfriendly AI soonâ. But if we assume a 0.2% annual probability of extinction, that gives a 1 in 10^174 chance of surviving 200,000 years, which requires an absurdly strong update away from the prior.
Canât we just say it is unlikelyâit logically must involve extremely low probabilities
I find it really implausible that 10^-174 is the true probability that humanity survives 200,000 years. I donât think we are 10^-174 confident about anything ever.
Personally, I see it as something like âThereâs a 5-90% chance that people like Toby Ord are basically right, and thus that 2 is true. Iâm not very confident about that, and 1 is also very plausible. But this is enough to make the expected value of existential risk reduction very high (as long as there are tractable reduction strategies which wouldnât be adopted âby defaultâ).â
The conclusion does not follow, for two reasons. The value of reducing x-risk might actually be lower if x-risk is higher. For an explanation, see the appendix of this paper: https://ââonlinelibrary.wiley.com/ââdoi/ââfull/ââ10.1111/ââ1758-5899.12318 (I think you need an account to download, but you can also get the paper on sci-hub.) But there are good arguments that decreasing the discount rate is more important than increasing consumption, which is also discussed in that paper.
âLong-runâ means âdiscount rate that applies after the short-runâ.
2. I think I now have a better sense of what you mean.
2a. It sounds like, when you wrote:
The current relatively high probability of extinction will maintain indefinitely.
...youâd include âThe high probability maintains for a while, and then we do go extinctâ as a case where the high probability maintains indefinitely?
This seems an odd way of phrasing things to me, given that, if we go extinct, the probability that we go extinct at any time after that is 0, and the probability that we are extinct at any time after that is 1. So whatever the current probability is, it would change after that point. (Though I guess we could talk about the probability that we will be extinct at the end of a time period, which would be high â 1 - post-extinction, so if that probability is currently high it could then stay high indefinitely, even if the actual probability changes.)
I thought you were instead talking about a case where the probability stays relatively high for a very long time, without us going extinct. (That seemed to me like the most intuitive interpretation of the current probability maintaining indefinitely.) Thatâs why I was saying that thatâs just unlikely âby definitionâ, basically.
Relatedly, when you wrote:
Currently, we have a relatively high probability of extinction, but if we survive through the current crucial period, then this probability will dramatically decrease.
Would that hypothesis include cases where we donât survive through the current period?
My view would basically be that the probability might be low now or might be relatively high. And if it is relatively high, then it must be either that itâll go down before a long time passes or that weâll become extinct. Iâm not currently sure whether that means I split my credence over the 1st and 2nd views you outline only, or over all 3.
2b. It also sounds like you were actually focusing on an argument like that the ânaturalâ extinction rate must be low, given how long humanity has survived thus far. This would be similar to an argument Ord gives in The Precipice, and thatâs also given in this paper I havenât actually read, which says in the abstract:
Using only the information that Homo sapiens has existed at least 200,000 years, we conclude that the probability that humanity goes extinct from natural causes in any given year is almost guaranteed to be less than one in 14,000, and likely to be less than one in 87,000.
Thatâs an argument I agree with. I also see it as a reason to believe that, if we handle all the anthropogenic extinction risks, the extinction risk level from then on would be much lower than it might now be.
Though Iâm not sure Iâd draw from it the implication you draw: it seems totally plausible we could enter a state with a new, higher âbackgroundâ extinction rate, which is also driven by our activities. And it seems to me that the only obvious reasons to believe this state wouldnât last a long time are (a) the idea that humanity will likely strive to get out of this state, and (b) the simple fact that, if the rate is high enough and lasts for long enough, extinction happening at some point becomes very likely. (One can also argue against believing that weâd enter such a state in the first place, or that weâve done so thus farâIâm just talking about why we might not believe the state would last a long time, if we did enter it.)
So when you say:
if we assume a 0.2% annual probability of extinction, that gives a 1 in 10^174 chance of surviving 200,000 years, which requires an absurdly strong update away from the prior.
Wouldnât it make more sense to instead say something like: âThe non-anthropogenic annual human extinction rate seems likely to be less than 1 in 87,000. To say the current total annual human extinction rate is 1 in 500 (0.2%) requires updating away from priors by a factor of 174 (87,000/â500).â (Perhaps this should instead be phrased as â...requires thinking that humans have caused the total rate to increase by a factor of 174.â)
Updating by a factor of 174 seems far more reasonable than the sort of update you referred to.
And then lasting 200,000 years at such an annual rate is indeed extremely implausible, but I donât think anyoneâs really arguing against that idea. The implication of a 0.2% annual rate, which isnât reduced, would just be that extinction becomes very likely in much less than 200,000 years.
3.
The conclusion does not follow, for two reasons. The value of reducing x-risk might actually be lower if x-risk is higher.
I havenât read that paper, but Ord makes what I think is a similar point in The Precipice. But, if I recall correctly, that was in a simple model, and he thought that in a more realistic model it does seem important how high the risk is now.
Essentially, I think x-risk work may be most valuable if the âbackgroundâ x-risk level is quite low, but currently the risk levels are unusually high, such that (a) the work is urgent (we canât just punt to the future, or thereâd be a decent chance that future wouldnât materialise), and (b) if we do succeed in that work, humanity is likely to last for a long time.
If instead the risk is high now but this is because there are new and large risks that emerge in each period, and what we do to fix them doesnât help with the later risks, then that indeed doesnât necessarily suggest x-risk work is worth prioritising.
And if instead the risk is pretty low across all time, that canstill suggest x-risk work is worth prioritising, because we have a lower chance of succumbing to a risk in any given period but would lose more in expectation if we do. (And thatâs definitely an interesting and counterintuitive implication of that argument that Ord mentions.) But I think being in that situation would push somewhat more in favour of things like investing, movement-building, etc., rather than working on x-risks âdirectlyâ âright nowâ.
So if weâre talking about the view that âCurrently, we have a relatively high probability of extinction, but if we survive through the current crucial period, then this probability will dramatically decreaseâ, I think more belief in that view does push more in favour of work on x-risks now.
Some additional thoughts on existential and extinction risk
1.
Iâm very glad you seem to have found this database useful as one input into this valuable-seeming project!
If any readers are interested in my arguments/âanalysis on that matter, Iâd actually recommend instead my EAGxVirtual Unconference talk. Itâs basically a better structured version of my post (though lacking useful links), as by then Iâd had a couple extra months to organise my thoughts on the topic.
2.
Iâm not sure I see the reasoning in that last sentence. It seems like youâre saying that something which has a 1 in 500 million chance of happening is unlikelyâwhich is basically true âby definitionââand that we know this because humanity already survived about 200,000 yearsâwhich seems somewhat irrelevant, and in any case unnecessary to point out? Wouldnât it be sufficiently merely to note that an annual 0.2% chance of A happening (whatever A is) means that, over a long enough time, itâs extremely likely that either A has happened already or the annual chance actually went down?
Relatedly, you write:
Canât we just say it is unlikelyâit logically must involve extremely low probabilities, even if we abstract away all the specificsârather than than it seems unlikely on priors, or based on some reference class forecasting, or the like?
(Maybe Iâm totally misunderstanding what youâre getting at here.)
3.
Personally, I see it as something like âThereâs a 5-90% chance that people like Toby Ord are basically right, and thus that 2 is true. Iâm not very confident about that, and 1 is also very plausible. But this is enough to make the expected value of existential risk reduction very high (as long as there are tractable reduction strategies which wouldnât be adopted âby defaultâ).â
I suspect that something like that perspectiveâin which view 2 is not given far more credence than view 1, but ends up seeming especially decision-relevantâis quite common among longtermist EAs. (Though Iâm sure there are also many with more certainty in view 2.)
(This isnât really a key pointâjust sharing how I see things.)
4. When you say âlong-run discount rateâ, do you mean âdiscount-rate that applies after the short-runâ, or âdiscount rate that applies from now till a very long time from nowâ? Iâm guessing you mean the former?
I ask because you say:
But it seems like the second claimâa high extinction risk now, which declines laterâcould still imply a non-trivial total existential risk across all time (e.g., 25%), just with this mostly concentrated over the coming decades or centuries.
Do you have a transcript of your EAGx talk?
Yes, but I hadnât put it up anywhere. Thanks for making me realise that might be useful to people!
You can find the transcript here. And Iâve also now linked to it from the YouTube video description.
My point was that we know humanity is capable of lasting 200,000 years, because it already did that. So on priors, we should expect humanity to last about another 200,000 years. We might update this prior downward based on facts like âwe have nukes nowâ or âwe might develop unfriendly AI soonâ. But if we assume a 0.2% annual probability of extinction, that gives a 1 in 10^174 chance of surviving 200,000 years, which requires an absurdly strong update away from the prior.
I find it really implausible that 10^-174 is the true probability that humanity survives 200,000 years. I donât think we are 10^-174 confident about anything ever.
The conclusion does not follow, for two reasons. The value of reducing x-risk might actually be lower if x-risk is higher. For an explanation, see the appendix of this paper: https://ââonlinelibrary.wiley.com/ââdoi/ââfull/ââ10.1111/ââ1758-5899.12318 (I think you need an account to download, but you can also get the paper on sci-hub.) But there are good arguments that decreasing the discount rate is more important than increasing consumption, which is also discussed in that paper.
âLong-runâ means âdiscount rate that applies after the short-runâ.
(Possibly somewhat rambly, sorry)
2. I think I now have a better sense of what you mean.
2a. It sounds like, when you wrote:
...youâd include âThe high probability maintains for a while, and then we do go extinctâ as a case where the high probability maintains indefinitely?
This seems an odd way of phrasing things to me, given that, if we go extinct, the probability that we go extinct at any time after that is 0, and the probability that we are extinct at any time after that is 1. So whatever the current probability is, it would change after that point. (Though I guess we could talk about the probability that we will be extinct at the end of a time period, which would be high â 1 - post-extinction, so if that probability is currently high it could then stay high indefinitely, even if the actual probability changes.)
I thought you were instead talking about a case where the probability stays relatively high for a very long time, without us going extinct. (That seemed to me like the most intuitive interpretation of the current probability maintaining indefinitely.) Thatâs why I was saying that thatâs just unlikely âby definitionâ, basically.
Relatedly, when you wrote:
Would that hypothesis include cases where we donât survive through the current period?
My view would basically be that the probability might be low now or might be relatively high. And if it is relatively high, then it must be either that itâll go down before a long time passes or that weâll become extinct. Iâm not currently sure whether that means I split my credence over the 1st and 2nd views you outline only, or over all 3.
2b. It also sounds like you were actually focusing on an argument like that the ânaturalâ extinction rate must be low, given how long humanity has survived thus far. This would be similar to an argument Ord gives in The Precipice, and thatâs also given in this paper I havenât actually read, which says in the abstract:
Thatâs an argument I agree with. I also see it as a reason to believe that, if we handle all the anthropogenic extinction risks, the extinction risk level from then on would be much lower than it might now be.
Though Iâm not sure Iâd draw from it the implication you draw: it seems totally plausible we could enter a state with a new, higher âbackgroundâ extinction rate, which is also driven by our activities. And it seems to me that the only obvious reasons to believe this state wouldnât last a long time are (a) the idea that humanity will likely strive to get out of this state, and (b) the simple fact that, if the rate is high enough and lasts for long enough, extinction happening at some point becomes very likely. (One can also argue against believing that weâd enter such a state in the first place, or that weâve done so thus farâIâm just talking about why we might not believe the state would last a long time, if we did enter it.)
So when you say:
Wouldnât it make more sense to instead say something like: âThe non-anthropogenic annual human extinction rate seems likely to be less than 1 in 87,000. To say the current total annual human extinction rate is 1 in 500 (0.2%) requires updating away from priors by a factor of 174 (87,000/â500).â (Perhaps this should instead be phrased as â...requires thinking that humans have caused the total rate to increase by a factor of 174.â)
Updating by a factor of 174 seems far more reasonable than the sort of update you referred to.
And then lasting 200,000 years at such an annual rate is indeed extremely implausible, but I donât think anyoneâs really arguing against that idea. The implication of a 0.2% annual rate, which isnât reduced, would just be that extinction becomes very likely in much less than 200,000 years.
3.
I havenât read that paper, but Ord makes what I think is a similar point in The Precipice. But, if I recall correctly, that was in a simple model, and he thought that in a more realistic model it does seem important how high the risk is now.
Essentially, I think x-risk work may be most valuable if the âbackgroundâ x-risk level is quite low, but currently the risk levels are unusually high, such that (a) the work is urgent (we canât just punt to the future, or thereâd be a decent chance that future wouldnât materialise), and (b) if we do succeed in that work, humanity is likely to last for a long time.
If instead the risk is high now but this is because there are new and large risks that emerge in each period, and what we do to fix them doesnât help with the later risks, then that indeed doesnât necessarily suggest x-risk work is worth prioritising.
And if instead the risk is pretty low across all time, that can still suggest x-risk work is worth prioritising, because we have a lower chance of succumbing to a risk in any given period but would lose more in expectation if we do. (And thatâs definitely an interesting and counterintuitive implication of that argument that Ord mentions.) But I think being in that situation would push somewhat more in favour of things like investing, movement-building, etc., rather than working on x-risks âdirectlyâ âright nowâ.
So if weâre talking about the view that âCurrently, we have a relatively high probability of extinction, but if we survive through the current crucial period, then this probability will dramatically decreaseâ, I think more belief in that view does push more in favour of work on x-risks now.
(I could be wrong about that, though.)
4. Thanks for the clarification!