Thanks for the write-up. I’m broadly sympathetic to a lot of these criticisms tbh, despite not being very left-leaning. A couple of points you relate I think are importantly false:
(Thorstad’s claim that) there’s no empirical basis for believing existential risk will drop to near-zero after our current, uniquely dangerous period before achieving long-term stability.
I don’t know about ‘empirical’, but there’s a simple mathematical basis for imagining it dropping to near zero in a sufficiently advanced future where we have multiple self-sustaining and hermetically independent settlements e.g. (though not necessarily) on different planets. Then even if you assume disasters befalling one aren’t independent, you have to believe they’re extremely correlated for this not to net out to extremely high civilisational resilience as you get to double digit settlements. That level of correlation is possible if it turns out to be possible e.g. to trigger a false vacuum decay—in which case Thorstad is right—or if a hostile AGI could wipe out everything before it—though that probability will surely either be realised or drop close to 0 within a few centuries.
If you accept the concept of Existential Risk and give them any credence, it logically follows that any such risk is much worse than any other horrible, terrible, undesirable one that does not lead to human extinction.
It doesn’t, and I wish the EA movement would move away from this unestablished claim. Specifically, one must have some difference in credence between achieving whatever longterm future one desires given no ‘minor’ catastrophe and achieving it given at least one. That credence differential is, to a first approximation, the fraction representing how much of ‘1 extinction’ your minor catastrophe is. Assuming we’re reasonably ambitious in our long term goals (e.g., per above, developing a multiplanetary or interstellar civilisation), it seems crazy to me to suppose that fraction should be less than 1⁄10. I suspect it should be substantially higher, since on restart we would have to survive a high risk in time-of-perils-2 while proceeding to the safe end state much slower, given the depletion of fossil fuels and other key resources.
If we think a restart is >= 1/10x as bad as extinction then we have to ask serious questions about whether it’s >= 10x as likely. I think it’s at least defensible to claim that e.g. extreme climate change is 10x as likely as an AI destroying literally all humanity.
Hello, and thanks for engaging with it. A couple of notes about the points you mention:
I have only read Thorstad’s arguments as they appear summarized in the book (he does have a blog in which one of his series, which I haven’t read yet, goes into detail on this: https://reflectivealtruism.com/category/my-papers/existential-risk-pessimism ). I have gone back to the chapter, and his thesis, in a bit more detail, would be that Ord’s argument is predicated on a lot of questionable assumptions, i.e.,the time of perils will be short, that the current moment is very dangerous, the future time will be much less so and it will stay so for a long time. He questions the evidence for all those assumptions, but particularly the last: “For humans to survive for a billion years, the annual average risk of our extinction needs to be no higher than one in a billion. That just doesn’t seem plausible—and it seems even less plausible that we could know something like that this far in advance.” He also goes on to expand it citing extreme uncertainty of events far in time, that it is unlikely that treaties or world government could keep risk low, that ‘becoming more intelligent’ is too vague and AGI absurdly implausible (“The claim that humanity will soon develop superhuman artificial agents is controversial enough,” he writes. “The follow-up claim that superintelligent artificial systems will be so insightful that they can foresee and prevent nearly every future risk is, to most outside observers, gag-inducingly counterintuitive.”).
As for the second statement, my point wasn’t that extinction always trumps everything else in expected value calculations, but that if you grant the concept of existential risk any credence, then -ceteris paribus-, the sheer scale of what’s at stake (e.g., billions of future lives across time and space) makes extinction risks of overriding importance in principle. That doesn’t mean that catastrophic-but-non-extinction events are negligible, just that their moral gravity derives from how they affect longterm survival and flourishing. I think you make a very good argument that massive, non-extinction catastrophes might be nearly as bad as extinction if they severely damage humanity’s trajectory but I feel it is highly speculative on the difficulties of making comebacks and on the likelihood of extreme climate change, and I still find the difference between existential risk and catastrophe(s) significant.
Fwiw I commented on Thorstad’s linkpost for the paper when he first posted about it here. My impression is that he’s broadly sympathetic to my claim about multiplanetary resilience, but either doesn’t believe we’ll get that far or thinks that the AI counterconsideration dominates it.
In this light, I think that the claim that annual x-risk being lower than 1/(10^-9) being ‘implausible’ is much too strong if it’s being used to undermine EV reasoning. Like I said—if we become interstellar and no universe-ending doomsday technologies exist, then multiplicativity of risk gets you there pretty fast. If each planet has, say 1/(10^5) annual chance of extinction, then n planets have 1/(10^(5^n)) chance of all independently going extinct in a given year. For n=2 that’s already one in ten billion.
Obviously there’s a) a much higher chance that they could go extinct in different years and b) that they could go all extinct in any given period from non-independent events such as war. But even so, it’s hard to believe that increasing k, say to double digits, doesn’t rapidly outweigh such considerations, especially given that an advanced civilisation could probably create new self-sustaining settlements in a matter of years.
I feel it is highly speculative on the difficulties of making comebacks and on the likelihood of extreme climate change
I don’t understand how you think climate change is more speculative than AI risk. I think it’s reasonable to have higher credence in human extinction from the latter, but those scenarios are entirely speculative. Extreme climate change is possible if a couple of parameters turn out to have been mismeasured.
As for the probability of making comebacks, I’d like to write a post about this, but the narrative goes something like this:
to ‘flourish’ (in an Ordian sense), we need to reach a state of sufficiently low x-risk
per above, by far the mathematically plausible way of doing this is just increasing our number of self-sustaining settlements
you could theoretically do it with an exceptionally stable political/social system, but I’m with Thorstad that the level of political stability this requires seems implausible
to reach that state, we have to develop advanced technologies—well beyond what we have now. So the question about ‘comebacks’ is misplaced—the question is about our prospect of getting from the beginning to (a good) end of at least one time of perils without a catastrophe
Dating our current time of perils to 1945, it looks like we’re on course, barring global catastrophes, to develop a self-sustaining civilisation in maybe 120-200 years
Suppose there’s a constant k probability annual risk of a catastrophe that regresses us to pre-time-of perils technology. Then our outlook in 1945 was, approximately, (1-k)^160 chance of getting to a multiplanetary state. Since we’ve made it 80 years in, we have a substantially better ~(1-k)^80.
If we restart from pre-1945 levels of technology, we will do so with max 10% of the fossil fuel energy we had, as well as many other depleted resources (fertiliser, uranium, etc). This will slow any kind of reboot substantially. See e.g. comparisons of coal to photovoltaics here.
There’s huge uncertainty here, but when you multiply out the friction from all the different depleted resources, I think progress the second time around will be optimistically 1⁄2 the speed, and pessimistically 1/10x or worse. (Based on above link, if photovoltaics were to entirely substitute fossil fuels, that drag alone would be around a ~30/5.5 multiplier on the cost of generating energy, which seems like it could easily slow economic development by a comparable amount)
That means in a reboot, we have optimistically (1-k)^320 chance of getting to a good outcome, pessimistically (1-k)^1600
During that reboot, we can expect the new civilisation to preferentially use up the most efficient resources just as we do (it doesn’t have to destroy them, just move them to much higher entropy states, such as our current practice of flushing fertiliser into the ocean) - but they have 2x, 10x or however much longer doing so.
That means civilisation 3 would have as much a disadvantage over civilisation 2 as civilisation 2 would over us, giving it optimistically a (1-k)^640 chance of a good outcome, pessimistically a (1-k)^16000 chance.
If we plug in k=0.001, which seems to be a vaguely representative estimate among x-risk experts, then in 1945 we would have had an 85% chance, today we would have a 92% chance, after one backslide we would have optimistically 73%, pessimistically 20%, and after Backslide Two we would have optimistically 53%, pessimistically basically 0.
We can roughly convert these to units of ‘extinction’ by dividing the loss of probability by our current prospects. So going to probability 53%, would be losing 32% of our current prospects, which is 32%/85% as bad in the long term as extinction.
This is missing a lot of nuance, obviously, which I’ve written about in this sequence, so we certainly shouldn’t take these numbers very seriously. But I think they overall paint a pretty reasonable picture of a ‘minor’ catastrophe being, in long-run expectation and aside from any short-term suffering or change in human morality, perhaps in the range of 15-75% as bad as extinction. Lots of room for discussing particulars, but not something we should dismiss as extinction being ‘much worse’ than—and in particular, not sufficiently lower that we can in practice afford to ignore the relative probabilities of extinction vs lesser global catastrophe.
Thanks for the write-up. I’m broadly sympathetic to a lot of these criticisms tbh, despite not being very left-leaning. A couple of points you relate I think are importantly false:
I don’t know about ‘empirical’, but there’s a simple mathematical basis for imagining it dropping to near zero in a sufficiently advanced future where we have multiple self-sustaining and hermetically independent settlements e.g. (though not necessarily) on different planets. Then even if you assume disasters befalling one aren’t independent, you have to believe they’re extremely correlated for this not to net out to extremely high civilisational resilience as you get to double digit settlements. That level of correlation is possible if it turns out to be possible e.g. to trigger a false vacuum decay—in which case Thorstad is right—or if a hostile AGI could wipe out everything before it—though that probability will surely either be realised or drop close to 0 within a few centuries.
It doesn’t, and I wish the EA movement would move away from this unestablished claim. Specifically, one must have some difference in credence between achieving whatever longterm future one desires given no ‘minor’ catastrophe and achieving it given at least one. That credence differential is, to a first approximation, the fraction representing how much of ‘1 extinction’ your minor catastrophe is. Assuming we’re reasonably ambitious in our long term goals (e.g., per above, developing a multiplanetary or interstellar civilisation), it seems crazy to me to suppose that fraction should be less than 1⁄10. I suspect it should be substantially higher, since on restart we would have to survive a high risk in time-of-perils-2 while proceeding to the safe end state much slower, given the depletion of fossil fuels and other key resources.
If we think a restart is >= 1/10x as bad as extinction then we have to ask serious questions about whether it’s >= 10x as likely. I think it’s at least defensible to claim that e.g. extreme climate change is 10x as likely as an AI destroying literally all humanity.
Hello, and thanks for engaging with it. A couple of notes about the points you mention:
I have only read Thorstad’s arguments as they appear summarized in the book (he does have a blog in which one of his series, which I haven’t read yet, goes into detail on this: https://reflectivealtruism.com/category/my-papers/existential-risk-pessimism ). I have gone back to the chapter, and his thesis, in a bit more detail, would be that Ord’s argument is predicated on a lot of questionable assumptions, i.e.,the time of perils will be short, that the current moment is very dangerous, the future time will be much less so and it will stay so for a long time. He questions the evidence for all those assumptions, but particularly the last: “For humans to survive for a billion years, the annual average risk of our extinction needs to be no higher than one in a billion. That just doesn’t seem plausible—and it seems even less plausible that we could know something like that this far in advance.” He also goes on to expand it citing extreme uncertainty of events far in time, that it is unlikely that treaties or world government could keep risk low, that ‘becoming more intelligent’ is too vague and AGI absurdly implausible (“The claim that humanity will soon develop superhuman artificial agents is controversial enough,” he writes. “The follow-up claim that superintelligent artificial systems will be so insightful that they can foresee and prevent nearly every future risk is, to most outside observers, gag-inducingly counterintuitive.”).
As for the second statement, my point wasn’t that extinction always trumps everything else in expected value calculations, but that if you grant the concept of existential risk any credence, then -ceteris paribus-, the sheer scale of what’s at stake (e.g., billions of future lives across time and space) makes extinction risks of overriding importance in principle. That doesn’t mean that catastrophic-but-non-extinction events are negligible, just that their moral gravity derives from how they affect longterm survival and flourishing. I think you make a very good argument that massive, non-extinction catastrophes might be nearly as bad as extinction if they severely damage humanity’s trajectory but I feel it is highly speculative on the difficulties of making comebacks and on the likelihood of extreme climate change, and I still find the difference between existential risk and catastrophe(s) significant.
Fwiw I commented on Thorstad’s linkpost for the paper when he first posted about it here. My impression is that he’s broadly sympathetic to my claim about multiplanetary resilience, but either doesn’t believe we’ll get that far or thinks that the AI counterconsideration dominates it.
In this light, I think that the claim that annual x-risk being lower than 1/(10^-9) being ‘implausible’ is much too strong if it’s being used to undermine EV reasoning. Like I said—if we become interstellar and no universe-ending doomsday technologies exist, then multiplicativity of risk gets you there pretty fast. If each planet has, say 1/(10^5) annual chance of extinction, then n planets have 1/(10^(5^n)) chance of all independently going extinct in a given year. For n=2 that’s already one in ten billion.
Obviously there’s a) a much higher chance that they could go extinct in different years and b) that they could go all extinct in any given period from non-independent events such as war. But even so, it’s hard to believe that increasing k, say to double digits, doesn’t rapidly outweigh such considerations, especially given that an advanced civilisation could probably create new self-sustaining settlements in a matter of years.
I don’t understand how you think climate change is more speculative than AI risk. I think it’s reasonable to have higher credence in human extinction from the latter, but those scenarios are entirely speculative. Extreme climate change is possible if a couple of parameters turn out to have been mismeasured.
As for the probability of making comebacks, I’d like to write a post about this, but the narrative goes something like this:
to ‘flourish’ (in an Ordian sense), we need to reach a state of sufficiently low x-risk
per above, by far the mathematically plausible way of doing this is just increasing our number of self-sustaining settlements
you could theoretically do it with an exceptionally stable political/social system, but I’m with Thorstad that the level of political stability this requires seems implausible
to reach that state, we have to develop advanced technologies—well beyond what we have now. So the question about ‘comebacks’ is misplaced—the question is about our prospect of getting from the beginning to (a good) end of at least one time of perils without a catastrophe
Dating our current time of perils to 1945, it looks like we’re on course, barring global catastrophes, to develop a self-sustaining civilisation in maybe 120-200 years
Suppose there’s a constant k probability annual risk of a catastrophe that regresses us to pre-time-of perils technology. Then our outlook in 1945 was, approximately, (1-k)^160 chance of getting to a multiplanetary state. Since we’ve made it 80 years in, we have a substantially better ~(1-k)^80.
If we restart from pre-1945 levels of technology, we will do so with max 10% of the fossil fuel energy we had, as well as many other depleted resources (fertiliser, uranium, etc). This will slow any kind of reboot substantially. See e.g. comparisons of coal to photovoltaics here.
There’s huge uncertainty here, but when you multiply out the friction from all the different depleted resources, I think progress the second time around will be optimistically 1⁄2 the speed, and pessimistically 1/10x or worse. (Based on above link, if photovoltaics were to entirely substitute fossil fuels, that drag alone would be around a ~30/5.5 multiplier on the cost of generating energy, which seems like it could easily slow economic development by a comparable amount)
That means in a reboot, we have optimistically (1-k)^320 chance of getting to a good outcome, pessimistically (1-k)^1600
During that reboot, we can expect the new civilisation to preferentially use up the most efficient resources just as we do (it doesn’t have to destroy them, just move them to much higher entropy states, such as our current practice of flushing fertiliser into the ocean) - but they have 2x, 10x or however much longer doing so.
That means civilisation 3 would have as much a disadvantage over civilisation 2 as civilisation 2 would over us, giving it optimistically a (1-k)^640 chance of a good outcome, pessimistically a (1-k)^16000 chance.
If we plug in k=0.001, which seems to be a vaguely representative estimate among x-risk experts, then in 1945 we would have had an 85% chance, today we would have a 92% chance, after one backslide we would have optimistically 73%, pessimistically 20%, and after Backslide Two we would have optimistically 53%, pessimistically basically 0.
We can roughly convert these to units of ‘extinction’ by dividing the loss of probability by our current prospects. So going to probability 53%, would be losing 32% of our current prospects, which is 32%/85% as bad in the long term as extinction.
This is missing a lot of nuance, obviously, which I’ve written about in this sequence, so we certainly shouldn’t take these numbers very seriously. But I think they overall paint a pretty reasonable picture of a ‘minor’ catastrophe being, in long-run expectation and aside from any short-term suffering or change in human morality, perhaps in the range of 15-75% as bad as extinction. Lots of room for discussing particulars, but not something we should dismiss as extinction being ‘much worse’ than—and in particular, not sufficiently lower that we can in practice afford to ignore the relative probabilities of extinction vs lesser global catastrophe.