I think the mistake I’d be making here is lumping in AGI with cars, construction machinery, etc, into one single category. But then I imagine the people who worry about extinction from war are also imagining a kind of war which should belong in a different category to previous wars.
I agree that would be the mistake. I think one can use an analysis similar to mine to conclude, for example, that the probability of car accidents causing human extinction is astronomically low. However, I would say using past war deaths to estimate future war deaths is much more appropriate than using past car accident deaths to estimate the risk of human extinction due to advanced AI.
I assume increasing capability to cause damage is the main reason for people arguing that future wars would belong to a different category. Yet:
I think war capabilities have been decreasing or not changing much in the last few decades:
“Nuclear risk has been decreasing. The estimated destroyable area by nuclear weapons deliverable in a first strike has decreased 89.2 % (= 1 − 65.2/601) since its peak in 1962” (see 1st graph below).
Military expenditure as a fraction of global GDP has decreased from 1960 to 2000, and been fairly constant since then (see 2nd graph below).
Taking a broader view, war capabilities do have been increasing, but there is not a clear trend in the deaths in conflicts as a fraction of the global population since 1400 (see last figure in the post).
Increases in the capability to cause damage are usually associated with increases in the capability to prevent damage, which I guess explains what I said just above, so one should not forecast future risk based on just one side alone.
Something like deaths in car accidents does not capture the relevant factors which can lead to AI causing human extinction. I think the best reference class for this is looking into how species have gone extinct in the past. Jacob Steinhardt did an analysis which has some relevant insights:
Thus, in general most species extinctions are caused by:
A second species which the original species has not had a chance to adapt to. This second species must also not be reliant on the original species to propagate itself.
A catastrophic natural disaster or climate event. [As a side note, I believe we have pretty good reasons to think this point applies much more weakly to humans than animals.]
Habitat destruction or ecosystem disruption caused by one of the two sources above.
Advanced AI is much more analogous to a new species than e.g. cars, so per the 1st point it makes sense that extinction risk from advanced AI is much higher than from e.g. cars. I would still claim deaths in past wars and terrorist attacks provide a strong basis for arguing that humans will not go extinct via an AI war or terrorist attack. However, the 1st point alludes to what seems to me to be the greatest risk from AI, natural selection favoring AIs over humans.
I should note I do not consider humans being outcompeted by AI as necessarily bad. I endorse expected total hedonistic utilitarianism (ETHU), and humans are unlikely to be the most efficient way of increasing welfare longterm. At the same time, minimising nearterm extinction risk from AI is an arguably helpful heuristic to align AI with ETHU.
It seems to me like a pretty relevant comparison would be to the probabilty that an asteroid impact causes >1 billion deaths.
As in, run this exact methodology using deaths due to asteroid impacts per year over the past 200 years as the dataset (we could also use injuries instead of deaths).
My understanding is that this would result in predicting an astronomically, astronomically low probability of >1 billion deaths.
So either:
- Updating up massively on such astronomically low priors is common in cases where we have other arguments at hand (in the asteroid case it would be various other data sources on asteriod collisions, in the war case it would be arguments related to bioweapons or similar) - This methodology provides a very bad prior for asteroids.
Yes, asteroids are slightly cherry picked, but when talking about probabilities of 10^-13 this amount of cherry picking doesn’t matter.
In general, extrapolations based on a given dataset become less reliable as one tries to predict events which are increasingly far away from the region for which there is data. Therefore my methodology is way more applicable to estimate the annual risk of human extinction from wars than from asteroids:
My maximum annual war deaths as a fraction of the global population is 0.300 %[1] (= 1.50*10^-3/0.5), which is just 2.5 orders of magnitude away from extinction.
I guess the maximum annual deaths from asteroids in the past 200 years were less than 800, i.e. less than 10^-7 (= 800/(8*10^9)) of the global population, which is at least 7 orders of magnitude away from extinction.
In any case, annual extinction risk from asteroids and comets is astronomically low based on inside view models. The results of Table 1 of Salotti 2022 suggest it is 2.2*10^-14 (= 2.2*10^-12/100), which is around 1⁄3 of my best guess prior for wars of 6.36*10^-14.
Agreed. At the same time, my methodology resulting in an astronomically low extinction risk from asteroids would arguably still be qualitatively in agreement with the results of Salotti 2022.
Great reflection, Toby!
I agree that would be the mistake. I think one can use an analysis similar to mine to conclude, for example, that the probability of car accidents causing human extinction is astronomically low. However, I would say using past war deaths to estimate future war deaths is much more appropriate than using past car accident deaths to estimate the risk of human extinction due to advanced AI.
I assume increasing capability to cause damage is the main reason for people arguing that future wars would belong to a different category. Yet:
I think war capabilities have been decreasing or not changing much in the last few decades:
“Nuclear risk has been decreasing. The estimated destroyable area by nuclear weapons deliverable in a first strike has decreased 89.2 % (= 1 − 65.2/601) since its peak in 1962” (see 1st graph below).
Military expenditure as a fraction of global GDP has decreased from 1960 to 2000, and been fairly constant since then (see 2nd graph below).
Taking a broader view, war capabilities do have been increasing, but there is not a clear trend in the deaths in conflicts as a fraction of the global population since 1400 (see last figure in the post).
Increases in the capability to cause damage are usually associated with increases in the capability to prevent damage, which I guess explains what I said just above, so one should not forecast future risk based on just one side alone.
Something like deaths in car accidents does not capture the relevant factors which can lead to AI causing human extinction. I think the best reference class for this is looking into how species have gone extinct in the past. Jacob Steinhardt did an analysis which has some relevant insights:
Advanced AI is much more analogous to a new species than e.g. cars, so per the 1st point it makes sense that extinction risk from advanced AI is much higher than from e.g. cars. I would still claim deaths in past wars and terrorist attacks provide a strong basis for arguing that humans will not go extinct via an AI war or terrorist attack. However, the 1st point alludes to what seems to me to be the greatest risk from AI, natural selection favoring AIs over humans.
I should note I do not consider humans being outcompeted by AI as necessarily bad. I endorse expected total hedonistic utilitarianism (ETHU), and humans are unlikely to be the most efficient way of increasing welfare longterm. At the same time, minimising nearterm extinction risk from AI is an arguably helpful heuristic to align AI with ETHU.
It seems to me like a pretty relevant comparison would be to the probabilty that an asteroid impact causes >1 billion deaths.
As in, run this exact methodology using deaths due to asteroid impacts per year over the past 200 years as the dataset (we could also use injuries instead of deaths).
My understanding is that this would result in predicting an astronomically, astronomically low probability of >1 billion deaths.
So either:
- Updating up massively on such astronomically low priors is common in cases where we have other arguments at hand (in the asteroid case it would be various other data sources on asteriod collisions, in the war case it would be arguments related to bioweapons or similar)
- This methodology provides a very bad prior for asteroids.
Yes, asteroids are slightly cherry picked, but when talking about probabilities of 10^-13 this amount of cherry picking doesn’t matter.
Thanks for the suggestion, Ryan!
In general, extrapolations based on a given dataset become less reliable as one tries to predict events which are increasingly far away from the region for which there is data. Therefore my methodology is way more applicable to estimate the annual risk of human extinction from wars than from asteroids:
My maximum annual war deaths as a fraction of the global population is 0.300 %[1] (= 1.50*10^-3/0.5), which is just 2.5 orders of magnitude away from extinction.
I guess the maximum annual deaths from asteroids in the past 200 years were less than 800, i.e. less than 10^-7 (= 800/(8*10^9)) of the global population, which is at least 7 orders of magnitude away from extinction.
In any case, annual extinction risk from asteroids and comets is astronomically low based on inside view models. The results of Table 1 of Salotti 2022 suggest it is 2.2*10^-14 (= 2.2*10^-12/100), which is around 1⁄3 of my best guess prior for wars of 6.36*10^-14.
For my best guess of war deaths of combatants equal to 50 % of total deaths.
(Yeah, the asteroids comparison is perhaps more relevant to the post on terrorist attacks.)
Agreed. At the same time, my methodology resulting in an astronomically low extinction risk from asteroids would arguably still be qualitatively in agreement with the results of Salotti 2022.
Thanks for this response!