extraordinary evidence would be required to move up sufficiently many orders of magnitude for an AI or bio terrorist attack to have a decent chance of causing human extinction.
How extraordinary does the evidence need to be? You can easily get many orders of magnitude changes in probabilities given some evidence. For example, as of 1900 on priors the probability that >1B people would experience powered flight in the year 2000 would have been extremely low, but someone paying attention to technological developments would have been right to give it a higher probability.
I’ve written something up on why I think this is likely: Out-of-distribution Bioattacks. Short version: I expect a technological change which expands which actors would try to cause harm.
(Thanks for sharing a draft with me in advance so I could post a full response at the same time instead of leaving “I disagree, and will say why soon!” comments while I waited for information hazard review!)
I have not thought about this in any significant detail, but it is a good question! I think David Thorstad’s series exagerating the risks has some relevant context.
You can easily get many orders of magnitude changes in probabilities given some evidence. For example, as of 1900 on priors the probability that >1B people would experience powered flight in the year 2000 would have been extremely low, but someone paying attention to technological developments would have been right to give it a higher probability.
Is that a fair comparison? I think the analogous comparison would involve replacing terrorist attack deaths per year by the number of different people travelling by plane per year. So we would have to assume, in the last 51.5 years, only:
9.63 k different people travelled by plane in a random single calendar year.
At most 44.6 k different people travelled by plane in a single calendar year.
Then, analogously to asking about a terrorist attack causig human extinction next year, we would ask about the probability that every single human (or close) would travel by plane next year, which would a priori be astronomically unlikely given the above.
I’ve written something up on why I think this is likely: Out-of-distribution Bioattacks. Short version: I expect a technological change which expands which actors would try to cause harm.
I am glad you did. It is a useful complement/follow-up to my post. I qualitatevely agree with the points you make, although it is still unclear to me how much higher the risk will become.
(Thanks for sharing a draft with me in advance so I could post a full response at the same time instead of leaving “I disagree, and will say why soon!” comments while I waited for information hazard review!)
You are welcome, and thanks for letting me know about that too!
I think the analogous comparison would involve replacing terrorist attack deaths per year by the number of different people travelling by plane per year. So we would have to assume, in the last 51.5 years, only:
9.63 k different people travelled by plane in a random single calendar year.
At most 44.6 k different people travelled by plane in a single calendar year.
I don’t really understand what you’re getting at here? Would you be able to spell it out more clearly?
(Or if someone else understands and I’m just missing it, feel free to jump in!)
Sorry for the lack of clarity. Basically, I was trying to point out that the structure of the data we had on people travelling by plane in 1900 (only in 1903) is different from that we have on terrorist attack deaths now. Then I described hypothetical data on travelling by plane with a similar structure to that we have on terrorist attack deaths now.
One could argue no people having travelled by plane until 1900 is analogous to no people having been killed in terrorist attacks, which would set an even lower prior probability of human extinction due to a terrorist attack (I would just be extrapolating based on e.g. 50 or so 0s, respecting 50 or so years of no terrorist attack deaths), whereas apparently 45 % of people in the US travelled by plane in 2015.
However, in the absence of data on people travelling by plane, it would make sense to use other reference class instead of extrapolating based on a bunch of 0s. Once one used an appropriate reference class, it is possible lots of people travelling now by plane does not seem so surprising. In addition, one may be falling prey to hindsight bias to some extent. Maybe so many people travelling by plane (e.g. instead of having more remote work) was not that likely ex ante.
How extraordinary does the evidence need to be? You can easily get many orders of magnitude changes in probabilities given some evidence. For example, as of 1900 on priors the probability that >1B people would experience powered flight in the year 2000 would have been extremely low, but someone paying attention to technological developments would have been right to give it a higher probability.
I’ve written something up on why I think this is likely: Out-of-distribution Bioattacks. Short version: I expect a technological change which expands which actors would try to cause harm.
(Thanks for sharing a draft with me in advance so I could post a full response at the same time instead of leaving “I disagree, and will say why soon!” comments while I waited for information hazard review!)
Thanks for the comment, Jeff!
I have not thought about this in any significant detail, but it is a good question! I think David Thorstad’s series exagerating the risks has some relevant context.
Is that a fair comparison? I think the analogous comparison would involve replacing terrorist attack deaths per year by the number of different people travelling by plane per year. So we would have to assume, in the last 51.5 years, only:
9.63 k different people travelled by plane in a random single calendar year.
At most 44.6 k different people travelled by plane in a single calendar year.
Then, analogously to asking about a terrorist attack causig human extinction next year, we would ask about the probability that every single human (or close) would travel by plane next year, which would a priori be astronomically unlikely given the above.
I am glad you did. It is a useful complement/follow-up to my post. I qualitatevely agree with the points you make, although it is still unclear to me how much higher the risk will become.
You are welcome, and thanks for letting me know about that too!
I don’t really understand what you’re getting at here? Would you be able to spell it out more clearly?
(Or if someone else understands and I’m just missing it, feel free to jump in!)
Sorry for the lack of clarity. Basically, I was trying to point out that the structure of the data we had on people travelling by plane in 1900 (only in 1903) is different from that we have on terrorist attack deaths now. Then I described hypothetical data on travelling by plane with a similar structure to that we have on terrorist attack deaths now.
One could argue no people having travelled by plane until 1900 is analogous to no people having been killed in terrorist attacks, which would set an even lower prior probability of human extinction due to a terrorist attack (I would just be extrapolating based on e.g. 50 or so 0s, respecting 50 or so years of no terrorist attack deaths), whereas apparently 45 % of people in the US travelled by plane in 2015.
However, in the absence of data on people travelling by plane, it would make sense to use other reference class instead of extrapolating based on a bunch of 0s. Once one used an appropriate reference class, it is possible lots of people travelling now by plane does not seem so surprising. In addition, one may be falling prey to hindsight bias to some extent. Maybe so many people travelling by plane (e.g. instead of having more remote work) was not that likely ex ante.