2. I think I now have a better sense of what you mean.
2a. It sounds like, when you wrote:
The current relatively high probability of extinction will maintain indefinitely.
...youâd include âThe high probability maintains for a while, and then we do go extinctâ as a case where the high probability maintains indefinitely?
This seems an odd way of phrasing things to me, given that, if we go extinct, the probability that we go extinct at any time after that is 0, and the probability that we are extinct at any time after that is 1. So whatever the current probability is, it would change after that point. (Though I guess we could talk about the probability that we will be extinct at the end of a time period, which would be high â 1 - post-extinction, so if that probability is currently high it could then stay high indefinitely, even if the actual probability changes.)
I thought you were instead talking about a case where the probability stays relatively high for a very long time, without us going extinct. (That seemed to me like the most intuitive interpretation of the current probability maintaining indefinitely.) Thatâs why I was saying that thatâs just unlikely âby definitionâ, basically.
Relatedly, when you wrote:
Currently, we have a relatively high probability of extinction, but if we survive through the current crucial period, then this probability will dramatically decrease.
Would that hypothesis include cases where we donât survive through the current period?
My view would basically be that the probability might be low now or might be relatively high. And if it is relatively high, then it must be either that itâll go down before a long time passes or that weâll become extinct. Iâm not currently sure whether that means I split my credence over the 1st and 2nd views you outline only, or over all 3.
2b. It also sounds like you were actually focusing on an argument like that the ânaturalâ extinction rate must be low, given how long humanity has survived thus far. This would be similar to an argument Ord gives in The Precipice, and thatâs also given in this paper I havenât actually read, which says in the abstract:
Using only the information that Homo sapiens has existed at least 200,000 years, we conclude that the probability that humanity goes extinct from natural causes in any given year is almost guaranteed to be less than one in 14,000, and likely to be less than one in 87,000.
Thatâs an argument I agree with. I also see it as a reason to believe that, if we handle all the anthropogenic extinction risks, the extinction risk level from then on would be much lower than it might now be.
Though Iâm not sure Iâd draw from it the implication you draw: it seems totally plausible we could enter a state with a new, higher âbackgroundâ extinction rate, which is also driven by our activities. And it seems to me that the only obvious reasons to believe this state wouldnât last a long time are (a) the idea that humanity will likely strive to get out of this state, and (b) the simple fact that, if the rate is high enough and lasts for long enough, extinction happening at some point becomes very likely. (One can also argue against believing that weâd enter such a state in the first place, or that weâve done so thus farâIâm just talking about why we might not believe the state would last a long time, if we did enter it.)
So when you say:
if we assume a 0.2% annual probability of extinction, that gives a 1 in 10^174 chance of surviving 200,000 years, which requires an absurdly strong update away from the prior.
Wouldnât it make more sense to instead say something like: âThe non-anthropogenic annual human extinction rate seems likely to be less than 1 in 87,000. To say the current total annual human extinction rate is 1 in 500 (0.2%) requires updating away from priors by a factor of 174 (87,000/â500).â (Perhaps this should instead be phrased as â...requires thinking that humans have caused the total rate to increase by a factor of 174.â)
Updating by a factor of 174 seems far more reasonable than the sort of update you referred to.
And then lasting 200,000 years at such an annual rate is indeed extremely implausible, but I donât think anyoneâs really arguing against that idea. The implication of a 0.2% annual rate, which isnât reduced, would just be that extinction becomes very likely in much less than 200,000 years.
3.
The conclusion does not follow, for two reasons. The value of reducing x-risk might actually be lower if x-risk is higher.
I havenât read that paper, but Ord makes what I think is a similar point in The Precipice. But, if I recall correctly, that was in a simple model, and he thought that in a more realistic model it does seem important how high the risk is now.
Essentially, I think x-risk work may be most valuable if the âbackgroundâ x-risk level is quite low, but currently the risk levels are unusually high, such that (a) the work is urgent (we canât just punt to the future, or thereâd be a decent chance that future wouldnât materialise), and (b) if we do succeed in that work, humanity is likely to last for a long time.
If instead the risk is high now but this is because there are new and large risks that emerge in each period, and what we do to fix them doesnât help with the later risks, then that indeed doesnât necessarily suggest x-risk work is worth prioritising.
And if instead the risk is pretty low across all time, that canstill suggest x-risk work is worth prioritising, because we have a lower chance of succumbing to a risk in any given period but would lose more in expectation if we do. (And thatâs definitely an interesting and counterintuitive implication of that argument that Ord mentions.) But I think being in that situation would push somewhat more in favour of things like investing, movement-building, etc., rather than working on x-risks âdirectlyâ âright nowâ.
So if weâre talking about the view that âCurrently, we have a relatively high probability of extinction, but if we survive through the current crucial period, then this probability will dramatically decreaseâ, I think more belief in that view does push more in favour of work on x-risks now.
(Possibly somewhat rambly, sorry)
2. I think I now have a better sense of what you mean.
2a. It sounds like, when you wrote:
...youâd include âThe high probability maintains for a while, and then we do go extinctâ as a case where the high probability maintains indefinitely?
This seems an odd way of phrasing things to me, given that, if we go extinct, the probability that we go extinct at any time after that is 0, and the probability that we are extinct at any time after that is 1. So whatever the current probability is, it would change after that point. (Though I guess we could talk about the probability that we will be extinct at the end of a time period, which would be high â 1 - post-extinction, so if that probability is currently high it could then stay high indefinitely, even if the actual probability changes.)
I thought you were instead talking about a case where the probability stays relatively high for a very long time, without us going extinct. (That seemed to me like the most intuitive interpretation of the current probability maintaining indefinitely.) Thatâs why I was saying that thatâs just unlikely âby definitionâ, basically.
Relatedly, when you wrote:
Would that hypothesis include cases where we donât survive through the current period?
My view would basically be that the probability might be low now or might be relatively high. And if it is relatively high, then it must be either that itâll go down before a long time passes or that weâll become extinct. Iâm not currently sure whether that means I split my credence over the 1st and 2nd views you outline only, or over all 3.
2b. It also sounds like you were actually focusing on an argument like that the ânaturalâ extinction rate must be low, given how long humanity has survived thus far. This would be similar to an argument Ord gives in The Precipice, and thatâs also given in this paper I havenât actually read, which says in the abstract:
Thatâs an argument I agree with. I also see it as a reason to believe that, if we handle all the anthropogenic extinction risks, the extinction risk level from then on would be much lower than it might now be.
Though Iâm not sure Iâd draw from it the implication you draw: it seems totally plausible we could enter a state with a new, higher âbackgroundâ extinction rate, which is also driven by our activities. And it seems to me that the only obvious reasons to believe this state wouldnât last a long time are (a) the idea that humanity will likely strive to get out of this state, and (b) the simple fact that, if the rate is high enough and lasts for long enough, extinction happening at some point becomes very likely. (One can also argue against believing that weâd enter such a state in the first place, or that weâve done so thus farâIâm just talking about why we might not believe the state would last a long time, if we did enter it.)
So when you say:
Wouldnât it make more sense to instead say something like: âThe non-anthropogenic annual human extinction rate seems likely to be less than 1 in 87,000. To say the current total annual human extinction rate is 1 in 500 (0.2%) requires updating away from priors by a factor of 174 (87,000/â500).â (Perhaps this should instead be phrased as â...requires thinking that humans have caused the total rate to increase by a factor of 174.â)
Updating by a factor of 174 seems far more reasonable than the sort of update you referred to.
And then lasting 200,000 years at such an annual rate is indeed extremely implausible, but I donât think anyoneâs really arguing against that idea. The implication of a 0.2% annual rate, which isnât reduced, would just be that extinction becomes very likely in much less than 200,000 years.
3.
I havenât read that paper, but Ord makes what I think is a similar point in The Precipice. But, if I recall correctly, that was in a simple model, and he thought that in a more realistic model it does seem important how high the risk is now.
Essentially, I think x-risk work may be most valuable if the âbackgroundâ x-risk level is quite low, but currently the risk levels are unusually high, such that (a) the work is urgent (we canât just punt to the future, or thereâd be a decent chance that future wouldnât materialise), and (b) if we do succeed in that work, humanity is likely to last for a long time.
If instead the risk is high now but this is because there are new and large risks that emerge in each period, and what we do to fix them doesnât help with the later risks, then that indeed doesnât necessarily suggest x-risk work is worth prioritising.
And if instead the risk is pretty low across all time, that can still suggest x-risk work is worth prioritising, because we have a lower chance of succumbing to a risk in any given period but would lose more in expectation if we do. (And thatâs definitely an interesting and counterintuitive implication of that argument that Ord mentions.) But I think being in that situation would push somewhat more in favour of things like investing, movement-building, etc., rather than working on x-risks âdirectlyâ âright nowâ.
So if weâre talking about the view that âCurrently, we have a relatively high probability of extinction, but if we survive through the current crucial period, then this probability will dramatically decreaseâ, I think more belief in that view does push more in favour of work on x-risks now.
(I could be wrong about that, though.)
4. Thanks for the clarification!