I’ll point out that I’m skeptical that most biorisks could be existential risks—I think they are plausible global catastrophic risks, and overlapping risks could lead to extinction, but I think people’s idea of a disease that kills over 99.9% of the global population is very unlikely even when accounting for near-term potential bioweapons.
This does help answer the question, but it conflates extinction risk with existential risk, which is I think a big mistake in general. This chapter in How Worlds Collapse does a nice job of explaining this:
“Currently, existential risk scholars tend to focus on events and processes for which a fairly direct, simple story can be told about how they could lead to extinction. [...] However, if there is a substantial probability that collapse could destroy humanity’s longterm potential [including by recovery with bad values], this should change one’s view of catastrophic risks...”
To be clear, I think there are other good reasons to weight biorisk more (as I do).
Okay but I just think that’s not that common a view. If you leave 1,000 − 10,000 humans alive, the longterm future is probably fine. So that’s the existential risk reduction down by 60 − 90%
If you leave 1,000 − 10,000 humans alive, the longterm future is probably fine
This is a very common claim that I think needs to be defended somewhat more robustly instead of simply assumed. If we have one strength as a community, is in not simply assuming things.
My read is that the evidence here is quite limited, the outside view suggests that losing 99.9999% of a species / having a very small population is a significant extinction risk, and that the uncertainty around the long-term viability of collapse scenarios is enough reason to want to avoid near-extinction events.
And again, even if this happens in some places — even if some groups fought each other until they literally ended up starving to death — it would be completely bizarre for it to happen to every group in the world. You just need one group of around 300 people to survive for them to be able to rebuild the species.
These people seem likely to be very incentivised towards survival—humans generally like surviving. It would be awful for them, sure, but the question is would they rebuild us as a species? And I think the answer is probably.
And let’s remember that this is the absolute worst case scenario. The human race has twice dropped nuclear bombs and then never again. It seems a big leap to imagine that not only will we do so but we will wipe ourselves out to the extent of only 1 such group.
Every successive group that could rebuild the human race is extra. I imagine that actually 100s of millions would survive an actual worldwide nuclear war, so the point we are litigating is a very small chance anwyay.
the outside view suggests that losing 99.9999% of a species / having a very small population is a significant extinction risk
I don’t really know what base rates I’d use here. Feels like you want natural disasters rather than predation. When the meteor hit do we know how population size affected repopulation? Even then, humans are just way more competent than any other animals. So as I said originally we might be looking at a 10 − 40% chance given the near worst case scenario, but I don’t buy your outside view.
I’d be curious what others outside views are here and if anyone has actual base rates on disaster driven animal populations and repopulation.
As an aside,
I think needs to be defended somewhat more robustly instead of simply assumed
I disagree. I’ve said what I think, you can push back on it if you want, but why is it bad to “simply assume” my view rather than yours?
My point is precisely that you should not assume any view. My position is that the uncertainties here are significant enough to warrant some attention to nuclear war as a potential extinction risk, rather than to simply bat away these concerns on first principles and questionable empirics.
Where extinction risk is concerned, it is potentially very costly to conclude on little evidence that something is not an extinction risk. We do need to prioritize, so I would not for instance propose treating bad zoning laws as an X-risk simply because we can’t demonstrate conclusively that they won’t lead to extinction. Luckily there are very few things that could kill very large numbers of people, and nuclear war is one of them.
I don’t think my argument says anything about how nuclear risk should be prioritized relative to other X-risks, I think the arguments for deprioritizing it relative to others are strong and reasonable people can disagree; YMMV.
My argument does say something about how nuclear risk shoud be prioritised. It is a lower priority if both existed. Maybe much lower.
The complicated thing is that nuclear risks do exist whereas biorisk and AI risk are much more speculative in terms of actually existing. In this sense I can believe nuclear should be funded more.
I think your arguments do suggest good reasons why nuclear risk might be prioritized lower; since we operate on the most effective margin, as you note, it is also possible at the same time for there to be significant funding margins in nuclear that are highly effective in expectation.
I’m not Matt, but I do work on nuclear risk. If we went down to 1000 to 10,000 people, recovery would take a long time, so there is significant chance of supervolcanic eruption or asteroid/comet impact causing extinction. People note that agriculture/cities developed independently, indicating it is high probability. However, it only happened when we had a stable moderate climate, which might not recur. Furthermore, the Industrial Revolution only happened once, so there is less confidence that it would happen again. In addition, it would be more difficult with depleted fossil fuels, phosphorus, etc. Even if we did recover industry, I think our current values are better than randomly chosen values (e.g. slavery might continue longer or democracy be less prevalent).
This feels too confident. A nuclear war into a supervolcano is just really unlikely. Plus if there were 1000 people then there would be so much human canned goods left over—just go to a major city and sit in a supermarket.
If a major city can support a million people for 3 days on its reserves it can support a 1000 people for 30 years.
Again, I’m not saying that I think it doesn’t matter, but I think my answers are good reasons why it’s less than AI
A nuclear war into a supervolcano is just really unlikely.
A nuclear war happening at the same time as a supervolcano is very unlikely. However, it could take a hundred thousand years to recover population, so if the frequency of supervolcanic eruptions is roughly every 30,000 years, it’s quite likely there would be one before we recover.
Plus if there were 1000 people then there would be so much human canned goods left over—just go to a major city and sit in a supermarket.
The scenario I’m talking about is one where the worsening climate and loss of technology means they would not be enough food, so the stored food would be consumed quickly. Furthermore, edible wild species including fish may be eaten to extinction.
Again, I’m not saying that I think it doesn’t matter, but I think my answers are good reasons why it’s less than AI
It seems very unlikely that a nuclear war will kill all of us, compared to biorisk where this seems more possible.
Not sure this should affect funding in general, but explicitly longtermist funders will therefore weight biorisk more.
I’ll point out that I’m skeptical that most biorisks could be existential risks—I think they are plausible global catastrophic risks, and overlapping risks could lead to extinction, but I think people’s idea of a disease that kills over 99.9% of the global population is very unlikely even when accounting for near-term potential bioweapons.
This does help answer the question, but it conflates extinction risk with existential risk, which is I think a big mistake in general. This chapter in How Worlds Collapse does a nice job of explaining this:
To be clear, I think there are other good reasons to weight biorisk more (as I do).
Okay but I just think that’s not that common a view. If you leave 1,000 − 10,000 humans alive, the longterm future is probably fine. So that’s the existential risk reduction down by 60 − 90%
This is a very common claim that I think needs to be defended somewhat more robustly instead of simply assumed. If we have one strength as a community, is in not simply assuming things.
My read is that the evidence here is quite limited, the outside view suggests that losing 99.9999% of a species / having a very small population is a significant extinction risk, and that the uncertainty around the long-term viability of collapse scenarios is enough reason to want to avoid near-extinction events.
Why do I think 1,000 −10,000 humans is probably (60 − 90%) fine?
According to Luisa Rodriguez, you need about 300 people to rebuild the human race.
These people seem likely to be very incentivised towards survival—humans generally like surviving. It would be awful for them, sure, but the question is would they rebuild us as a species? And I think the answer is probably.
And let’s remember that this is the absolute worst case scenario. The human race has twice dropped nuclear bombs and then never again. It seems a big leap to imagine that not only will we do so but we will wipe ourselves out to the extent of only 1 such group.
Every successive group that could rebuild the human race is extra. I imagine that actually 100s of millions would survive an actual worldwide nuclear war, so the point we are litigating is a very small chance anwyay.
I don’t really know what base rates I’d use here. Feels like you want natural disasters rather than predation. When the meteor hit do we know how population size affected repopulation? Even then, humans are just way more competent than any other animals. So as I said originally we might be looking at a 10 − 40% chance given the near worst case scenario, but I don’t buy your outside view.
I’d be curious what others outside views are here and if anyone has actual base rates on disaster driven animal populations and repopulation.
As an aside,
I disagree. I’ve said what I think, you can push back on it if you want, but why is it bad to “simply assume” my view rather than yours?
My point is precisely that you should not assume any view. My position is that the uncertainties here are significant enough to warrant some attention to nuclear war as a potential extinction risk, rather than to simply bat away these concerns on first principles and questionable empirics.
Where extinction risk is concerned, it is potentially very costly to conclude on little evidence that something is not an extinction risk. We do need to prioritize, so I would not for instance propose treating bad zoning laws as an X-risk simply because we can’t demonstrate conclusively that they won’t lead to extinction. Luckily there are very few things that could kill very large numbers of people, and nuclear war is one of them.
I don’t think my argument says anything about how nuclear risk should be prioritized relative to other X-risks, I think the arguments for deprioritizing it relative to others are strong and reasonable people can disagree; YMMV.
My argument does say something about how nuclear risk shoud be prioritised. It is a lower priority if both existed. Maybe much lower.
The complicated thing is that nuclear risks do exist whereas biorisk and AI risk are much more speculative in terms of actually existing. In this sense I can believe nuclear should be funded more.
I think your arguments do suggest good reasons why nuclear risk might be prioritized lower; since we operate on the most effective margin, as you note, it is also possible at the same time for there to be significant funding margins in nuclear that are highly effective in expectation.
Do you work on researching nuclear risk?
How do you think this disagreement could be more usefully delineated. It seems like there is some interesting disagreement here?
I’m not Matt, but I do work on nuclear risk. If we went down to 1000 to 10,000 people, recovery would take a long time, so there is significant chance of supervolcanic eruption or asteroid/comet impact causing extinction. People note that agriculture/cities developed independently, indicating it is high probability. However, it only happened when we had a stable moderate climate, which might not recur. Furthermore, the Industrial Revolution only happened once, so there is less confidence that it would happen again. In addition, it would be more difficult with depleted fossil fuels, phosphorus, etc. Even if we did recover industry, I think our current values are better than randomly chosen values (e.g. slavery might continue longer or democracy be less prevalent).
This feels too confident. A nuclear war into a supervolcano is just really unlikely. Plus if there were 1000 people then there would be so much human canned goods left over—just go to a major city and sit in a supermarket.
If a major city can support a million people for 3 days on its reserves it can support a 1000 people for 30 years.
Again, I’m not saying that I think it doesn’t matter, but I think my answers are good reasons why it’s less than AI
A nuclear war happening at the same time as a supervolcano is very unlikely. However, it could take a hundred thousand years to recover population, so if the frequency of supervolcanic eruptions is roughly every 30,000 years, it’s quite likely there would be one before we recover.
The scenario I’m talking about is one where the worsening climate and loss of technology means they would not be enough food, so the stored food would be consumed quickly. Furthermore, edible wild species including fish may be eaten to extinction.
I agree that more total money should be spent on AGI safety than nuclear issues. However, resilience to sunlight reduction is much more neglected than AGI safety. That’s why the Monte Carlo analyses found that the cost-effectiveness of resilience to loss of electricity (e.g. high-altitude detonations of nuclear weapons causing electromagnetic pulses) and resilience to nuclear winter are competitive with AGI safety.