I haven’t looked at this in much detail, but Ord’s estimates seem too high to me. It seems really hard for humanity to go extinct, considering that there are people in remote villages, people in submarines, people in mid-flight at the time a disaster strikes, and even people on the International Space Station. (And yes, there are women on the ISS, I looked that up.) I just don’t see how e.g. a pandemic would plausibly kill all those people.
Also, if engineered pandemics, or “unforeseen” and “other” anthropogenic risks have a chance of 3% each of causing extinction, wouldn’t you expect to see smaller versions of these risks (that kill, say, 10% of people, but don’t result in extinction) much more frequently? But we don’t observe that.
(I haven’t read Ord’s book so I don’t know if he addresses these points.)
What is the significance of the people on the ISS? Are you suggesting that six people could repopulate the human species? And what sort of disaster takes less time than a flight, and only kills people on the ground?
Also, I expect to see small engineered pandemics, but only after effective genetic engineering is widespread. So the fact that we haven’t seen any so far is not much evidence.
Also, I expect to see small engineered pandemics, but only after effective genetic engineering is widespread. So the fact that we haven’t seen any so far is not much evidence.
Yes, that was broadly the response I had in mind as well. Same goes for most of the “unforeseen”/”other” anthropogenic risks; those categories are in the chapter on “Future risks”, and are mostly things Ord appears to think either will or may get riskier as certain technologies are developed/advanced.
Sleepy reply to Tobias’ “Ord’s estimates seem too high to me”: An important idea in the book is that “the per-century extinction risks from “natural” causes must be very low, based in part on our long history of surviving such risks” (as I phrase it in this post). The flipside of that is roughly the argument that we haven’t got strong evidence of our ability to survive (uncollapsed and sans dystopia) a long period with various technologies that will be developed later, but haven’t been yet.
Of course, that doesn’t seem sufficient by itself as a reason for a high level of concern, as some version of that could’ve been said at every point in history when “things were changing”. But if you couple that general argument with specific reasons to believe upcoming technologies could be notably risky, you could perhaps reasonably arrive at Ord’s estimates. (And there are obviously a lot of specific details and arguments and caveats that I’m omitting here.)
Also, if engineered pandemics, or “unforeseen” and “other” anthropogenic risks have a chance of 3% each of causing extinction, wouldn’t you expect to see smaller versions of these risks (that kill, say, 10% of people, but don’t result in extinction) much more frequently? But we don’t observe that.
I don’t think so. I think a reasonable prior on this sort of thing would have killing 10% of people not much more likely than killing 100% of people, and actually IMO it should be substantially less likely. (Consider a distribution over asteroid impacts. The range of asteroids that kill around 10% of humans is a narrow band of asteroid sizes, whereas any asteroid above a certain size will kill 100%.)
Moreover, if you think 10% is much more likely than 100%, you should think 1% is much more likely than 10%, and so on. But we don’t in fact see lots of unknown risks killing even 0.1% of the population. So that means the probability of x-risk from unknown causes according to you must be really really really small. But that’s just implausible. You could have made the exact same argument in 1917, in 1944, etc. and you would have been wildly wrong.
Well, historically, there have been quite a few pandemics that killed more than 10% of people, e.g. the Black Death or Plague of Justinian. There’s been no pandemic that killed everyone.
Is your point that it’s different for anthropogenic risks? Then I guess we could look at wars for historic examples. Indeed, there have been wars that killed something on the order of 10% of people, at least in the warring nations, and IMO that is a good argument to take the risk of a major war quite seriously.
But there have been far more wars that killed fewer people, and none that caused extinction. The literature usually models the number of casualties as a Pareto distribution, which means that the probability density is monotonically decreasing in the number of deaths. (For a broader reference class of atrocities, genocides, civil wars etc., I think the picture is similar.)
But we don’t in fact see lots of unknown risks killing even 0.1% of the population.
Smoking, lack of exercise, and unhealthy diets each kill more than 0.1% of the population each year. Coronavirus may kill 0.1% in some countries. The advent of cars in the 20th century resulted in 60 million road deaths, which is maybe 0.5% of everyone alive over that time (I haven’t checked this in detail). That can be seen as an unknown from the perspective of someone in 1900. Granted, some of those are more gradual than the sort of catastrophe people have in mind—but actually I’m not sure why that matters.
Looking at individual nations, I’m sure you can find many examples of civil wars, famines, etc. killing 0.1% of the population of a certain country, but far fewer examples killing 10% (though there are some). I’m not claiming the latter is 100x less likely but it is clearly much less likely.
You could have made the exact same argument in 1917, in 1944, etc. and you would have been wildly wrong.
I don’t understand this. What do you think the exact same argument would have been, and why was that wildly wrong?
I think there are some risks which have “common mini-versions,” to coin a phrase, and others which don’t. Asteroids have mini-versions (10%-killer-versions), and depending on how common they are the 10%-killers might be more likely than the 100%-killers, or vice versa. I actually don’t know which is more likely in that case.
AI risk is the sort of thing that doesn’t have common mini-versions, I think. An AI with the means and motive to kill 10% of humanity probably also has the means and motive to kill 100%.
Natural pandemics DO have common mini-versions, as you point out.
It’s less clear with engineered pandemics. That depends on how easy they are to engineer to kill everyone vs. how easy they are to engineer to kill not-everyone-but-at-least-10%, and it depends on how motivated various potential engineers are.
Accidental physics risks (like igniting the atmosphere, creating a false vacuum collapse or black hole or something with a particle collider) are way more likely to kill 100% of humanity than 10%. They do not have common mini-versions.
So what about unknown risks? Well, we don’t know. But from the track record of known risks, it seems that probably there are many diverse unknown risks, and so probably at least a few of them do not have common mini-versions.
And by the argument you just gave, the “unknown” risks that have common mini-versions won’t actually be unknown, since we’ll see their mini-versions. So “unknown” risks are going to be disproportionately the kind of risk that doesn’t have common mini-versions.
...
As for what I meant about making the exact same argument in the past: I was just saying that we’ve discovered various risks that don’t have common mini-versions, which at one point were unknown and then became known. Your argument basically rules out discovering such things ever again. Had we listened to your argument before learning about AI, for example, we would have concluded that AI was impossible, or that somehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threats.
I think I agree with this general approach to thinking about this.
From what I’ve seen of AI risk discussions, I think I’d stand by my prior statement, which I’d paraphrase now as: There are a variety of different types of AI catastrophe scenario that have been discussed. Some seem like they might be more likely or similarly likely to totally wipe us out that to cause a 5-25% death toll. But some don’t. And I haven’t seen super strong arguments for considering the former much more likely than the latter. And it seems like the AI safety community as a whole has become more diverse in their thinking on this sort of thing over the last few years.
For engineered pandemics, it still seems to me that literally 100% of people dying from the pathogens themselves seems much less likely than a very high number dying, perhaps even enough to cause existential catastrophe slightly “indirectly”. However “well” engineered, pathogens themselves aren’t agents which explicitly seek the complete extinction of humanity. (Again, Defence in Depth seems relevant here.) Though this is slightly different from a conversation about the relative likelihood of 10% vs other percentages. (Also, I feel hesitant to discuss this in great deal, for vague information hazards reasons.)
I agree regarding accidental physics risks. But I think the risks from those is far lower than the risks from AI and bio, and probably nanotech, nuclear, etc. (I don’t really bring any independent evidence to the table; this is just based on the views I’ve seen from x-risk researchers.)
from the track record of known risks, it seems that probably there are many diverse unknown risks, and so probably at least a few of them do not have common mini-versions.
I think that’d logically follow from your prior statements. But I’m not strongly convinced about those statements, except regarding accidental physics risks, which seem very unlikely.
And by the argument you just gave, the “unknown” risks that have common mini-versions won’t actually be unknown, since we’ll see their mini-versions. So “unknown” risks are going to be disproportionately the kind of risk that doesn’t have common mini-versions.
I think this is an interesting point. It does tentatively update me towards thinking that, conditional on there indeed being “unknown risks” that are already “in play”, they’re more likely than I’d otherwise thing to jump straight to 100%, without “mini versions”.
However, I think the most concerning source of “unknown risks” are new technologies or new actions (risks that aren’t yet “in play”). The unknown equivalents of risks from nanotech, space exploration, unprecedented consolidation of governments across the globe, etc. “Drawing a new ball from the urn”, in Bostrom’s metaphor. So even if such risks do have “common mini-versions”, we wouldn’t yet have seen them.
Also, regarding the portion of unknown risks that are in play, it seems to be appropriate to respond to the argument “Most risks have common mini-versions, but we haven’t seen these for unknown risks (pretty much by definition)” partly by updating towards thinking the unknown risks lack such common mini-versions, but also partly by updating towards thinking unknown risks are unlikely. We aren’t forced to fully take the former interpretation.
Tobias’ original point was ” Also, if engineered pandemics, or “unforeseen” and “other” anthropogenic risks have a chance of 3% each of causing extinction, wouldn’t you expect to see smaller versions of these risks (that kill, say, 10% of people, but don’t result in extinction) much more frequently? But we don’t observe that. ”
Thus he is saying there aren’t any “unknown” risks that do have common mini-versions but just haven’t had time to develop yet. That’s way too strong a claim, I think. Perhaps in my argument against this claim I ended up making claims that were also too strong. But I think my central point is still right: Tobias’ argument rules out things arising in the future that clearly shouldn’t be ruled out, because if we had run that argument in the past it would have ruled out various things (e.g. AI, nukes, physics risks, and come to think of it even asteroid strikes and pandemics if we go far enough back in the past) that in fact happened.
1. I interpreted the original claim—“wouldn’t you expect”—as being basically one in which observation X was evidence against hypothesis Y. Not conclusive evidence, just an update. I didn’t interpret it as “ruling things out” (in a strong way) or saying that there aren’t any unknown risks without common mini-versions (just that it’s less likely that there are than one would otherwise think). Note that his point seemed to be in defence of “Ord’s estimates seem too high to me”, rather than “the risks are 0″.
2. I do think that Tobias’s point, even interpreted that way, was probably too strong, or missing a key detail, in that the key sources of risks are probably emerging or new things, so we wouldn’t expect to have observed their mini-versions yet. Though I do tentatively think I’d expect to see mini-versions before the “full thing”, once the new things do start arising. (I’m aware this is all pretty hand-wavey phrasing.)
3i. As I went into more in my other comment, I think the general expectation that we’ll expect to see very small versions before and more often than small ones, which we expect to see before and more often than medium, which we expect to see before and more often than large, etc., probably would’ve served well in the past. There was progressively more advanced tech before AI, and AI is progressively advancing more. There were progressively more advanced weapons, progressively more destructive wars, progressively larger numbers of nukes, etc. I’d guess the biggest pandemics and asteroid strikes weren’t the first, because the biggest are rare.
3ii. AI is the least clear of those examples, because:
(a) it seems like destruction from AI so far has been very minimal (handful of fatalities from driverless cars, the “flash crash”, etc.), yet it seems plausible major destruction could occur in future
(b) we do have specific arguments, though of somewhat unclear strength, that the same AI might actively avoid causing any destruction for a while, and then suddenly seize decisive strategic advantage etc.
But on (a), I do think most relevant researchers would say the risk this month from AI is extremely low; the risks will rise in future as systems become more capable. So there’s still time in which we may see mini-versions.
And on (b), I’d consider that a case where a specific argument updates us away from a generally pretty handy prior that we’ll see small things earlier and more often than extremely large things. And we also don’t yet have super strong reason to believe that those arguments are really painting the right picture, as far as I’m aware.
3iii. I think if we interpreted Tobias’s point as something like “We’ll never see anything that’s unlike the past”, then yes, of course that’s ridiculous. So as I mentioned elsewhere, I think it partly depends on how we carve up reality, how we define things, etc. E.g., do we put nukes in a totally new bucket, or consider it a part of trends in weaponry/warfare/explosives?
But in any case, my interpretation of Tobias’s point, where it’s just about it being unlikely to see extreme things before smaller versions, would seem to work with e.g. nukes, even if we put them in their own special category—we’d be surprise by the first nuke, but we’d indeed see there’s one nuke before there are thousands, and there are two detonations on cities before there’s a full-scale nuclear war (if there ever is one, which hopefully and plausibly there won’t be).
In general I think you’ve thought this through more carefully than me so without having read all your points I’m just gonna agree with you.
So yeah, I think the main problem with Tobias’ original point was that unknown risks are probably mostly new things that haven’t arisen yet and thus the lack of observed mini-versions of them is no evidence against them. But I still think it’s also true that some risks just don’t have mini-versions, or rather are as likely or more likely to have big versions than mini-versions. I agree that most risks are not like this, including some of the examples I reached for initially.
As for what I meant about making the exact same argument in the past: I was just saying that we’ve discovered various risks that don’t have common mini-versions, which at one point were unknown and then became known. Your argument basically rules out discovering such things ever again. Had we listened to your argument before learning about AI, for example, we would have concluded that AI was impossible, or that somehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threats.
Hmm. I’m not sure I’m understanding you correctly. But I’ll respond to what I think you’re saying.
Firstly, the risk of natural pandemics, which the Spanish Flu was a strong example of, did have “common mini-versions”. In fact, Wikipedia says the Black Death was the “most fatal pandemic recorded in human history”. So I really don’t think we’d have ruled out the Spanish Flu happening by using the sort of argument I’m discussing (which I’m not sure I’d call “my argument”) - we’d have seen it as unlikely in any given year, and that would’ve been correct. But I imagine we’d have given it approximately the correct ex ante probability.
Secondly, even nuclear weapons, which I assume is what your reference to 1944 is about, seem like they could fit neatly in this sort of argument. It’s a new weapon, but weapons and wars existed for a long time. And the first nukes really couldn’t have killed everyone. And then we gradually had more nukes, more test explosions, more Cold War events, as we got closer and closer to it being possible for 100% of people to die from it. And we haven’t had 100% die. So it again seems like we wouldn’t have ruled out what ended up happening.
Likewise, AI can arguably be seen as a continuation of past technological, intellectual, scientific, etc. progress in various ways. Of course, various trends might change in shape, speed up, etc. But so far they do seem to have mostly done so somewhat gradually, such that none of the developments would’ve been “ruled out” by expecting the future to looking roughly similar to the past or the past+extrapolation. (I’m not an expert on this, but I think this is roughly the conclusion AI Impacts is arriving at based on their research.)
Perhaps a key point is that we indeed shouldn’t say “The future will be exactly like the past.” But instead “The future seems likely to typically be fairly well modelled as a rough extrapolation of some macro trends. But there’ll be black swans sometimes. And we can’t totally rule out totally surprising things, especially if we do very new things.”
This is essentially me trying to lay out a certain way of looking at things. It’s not necessarily the one I strongly adopt. (I actually hadn’t thought about this viewpoint much before, so I’ve found trying to lay it out/defend it here interesting.)
In fact, as I said, I (at least sort-of) disagreed with Tobias’ original comment, and I’m very concerned about existential risks. And I think a key point is that new technologies and actions can change the distributions we’re drawing from, in ways that we don’t understand. I’m just saying it still seems quite plausible to me (and probably likely, though not guaranteed) that we’d see a 5-25%-style catastrophe from a particular type of risk before a 100% catastrophe from it. And I think history seems consistent with that, and that that idea probably would’ve done fairly well in the past.
(Also, as I noted in my other comment, I haven’t yet seen very strong evidence or arguments against the idea that “somehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threats”—or more specifically, that AI might result in that, whether or not it has the “motive” to do that. It seems to me the jury is still out. So I don’t think I’d use the fact an argument reaches that conclusion as a point against that argument.)
Likewise, AI can arguably be seen as a continuation of past technological, intellectual, scientific, etc. progress in various ways. Of course, various trends might change in shape, speed up, etc. But so far they do seem to have mostly done so somewhat gradually, such that none of the developments would’ve been “ruled out” by expecting the future to looking roughly similar to the past or the past+extrapolation. (I’m not an expert on this, but I think this is roughly the conclusion AI Impacts is arriving at based on their research.)
I agree with all this and don’t think it significantly undermines anything I said.
I think the community has indeed developed more diverse views over the years, but I still think the original take (as seen in Bostrom’s Superintelligence) is the closest to the truth. The fact that the community has gotten more diverse can be easily explained as the result of it growing a lot bigger and having a lot more time to think. (Having a lot more time to think means more scenarios can be considered, more distinctions made, etc. More time for disagreements to arise and more time for those disagreements to seem like big deals when really they are fairly minor; the important things are mostly agreed on but not discussed anymore.) Or maybe you are right and this is evidence that Bostrom is wrong. Idk. But currently I think it is weak evidence, given the above.
Yeah in retrospect I really shouldn’t have picked nukes and natural pandemics as my two examples. Natural pandemics do have common mini-versions, and nukes, well, the jury is still out on that one. (I think it could go either way. I think that nukes maybe can kill everyone, because the people who survive the initial blasts might die from various other causes, e.g. civilizational collapse or nuclear winter. But insofar as we think that isn’t plausible, then yeah killing 10% is way more likely than killing 100%. (I’m assuming we count killing 99% as killing 10% here?) )
I think AI, climate change tail risks, physics risks, grey goo, etc. would be better examples for me to talk about.
With nukes, I do share the view that they could plausibly kill everyone. If there’s a nuclear war, followed by nuclear winter, and everyone dies during that winter, rather than most people dying and then the rest succumbing 10 years later from something else or never recovering, I’d consider that nuclear war causing 100% deaths.
My point was instead that that really couldn’t have happened in 1945. So there was one nuke, and a couple explosions, and gradually more nukes and test explosions, etc., before there was a present risk of 100% of people dying from this source. So we did see something like “mini-versions”—Hiroshima and Nagasaki, test explosions, Cuban Missile Crisis—before we saw 100% (which indeed, we still haven’t and hopefully won’t).
With climate change, we’re already seeing mini-versions. I do think it’s plausible that there could be a relatively sudden jump due to amplifying feedback loops. But “relatively sudden” might mean over months or years or something like that. And it wouldn’t be a total bolt from the blue in any case—the damage is already accruing and increasing, and likely would do in the lead up to such tail risks.
AI, physics risks, and nanotech are all plausible cases where there’d be a sudden jump. And I’m very concerned about AI and somewhat about nanotech. But note that we don’t actually have clear evidence that those things could cause such sudden jumps. I obviously don’t think we should wait for such evidence, because if it came we’d be dead. But it just seems worth remembering that before using “Hypothesis X predicts no sudden jump in destruction from Y” as an argument against hypothesis X.
Also, as I mentioned in my other comment, I’m now thinking maybe the best way to look at that is specific arguments in the case of AI, physics risks, and nanotech updating us away from the generally useful prior that we’ll see small things before extreme versions of the same things.
Minor: some recent papers argue the death toll from the Plague of Justinian has been exaggerated.
Existing mortality estimates assert that the Justinianic Plague (circa 541 to 750 CE) caused tens of millions of deaths throughout the Mediterranean world and Europe, helping to end antiquity and start the Middle Ages. In this article, we argue that this paradigm does not fit the evidence.
It concludes that the Justinianic Plague had an overall limited effect on late antique society. Although on some occasions the plague might have caused high mortality in specific places, leaving strong impressions on contemporaries, it neither caused widespread demographic decline nor kept Mediterranean populations low.
I think, for basically Richard Ngo’s reasons, I weakly disagree with a strong version of Tobias’s original claim that:
if engineered pandemics, or “unforeseen” and “other” anthropogenic risks have a chance of 3% each of causing extinction, wouldn’t you expect to see smaller versions of these risks (that kill, say, 10% of people, but don’t result in extinction) much more frequently? But we don’t observe that.
(Whether/how much I disagree depends in part on what “much more frequently” is meant to imply.)
I also might agree with:
I think a reasonable prior on this sort of thing would have killing 10% of people not much more likely than killing 100% of people
(Again, depends in part on what “much more likely” would mean.)
But I was very surprised to read:
[...] and actually IMO it should be substantially less likely.
And I mostly agree with Tobias’s response to that.
The point that there’s a narrower band of asteroid sizes that would cause ~10% of the population’s death than of asteroid sizes that would cause 100% makes sense. But I believe there are also many more asteroids in that narrow band. E.g., Ord writes:
While an impact with an asteroid in this smaller size range [0-10km across] would be much less likely to cause an existential catastrophe [than an impact from one that’s greater than 10km across], this may be more than offset by their much higher probability of impact.
[...] the probability of an EArth’s impact in an average century is about one in 6,000 for asteroids between one and ten kilometres in size, and about one in 1.5 million for those above ten kilometres.
And for pandemics it seems clearly far more likely to have one that kills a massive number of people than one that kills everyone. Indeed, my impression is that most x-risk researchers think it’d be notably more likely for a pandemic to cause existential catastrophe through something like causing collapse or reduction in population to below minimum viable population levels, rather than by “directly” killing everyone. (I’m not certain of that, though.)
I’d guess, without much basis, that:
the distribution of impacts from a source of risk would vary between the different sources of risks
For pandemics, killing (let’s say) 5-25% really is much more likely than killing 100%. (This isn’t why I make this guess, but it seems worth noting: Metaculus currently predicts a 6% chance COVID-19 kills >100million people by the end of the year, which would put it close to that 5-25% range. I really don’t know how much stock to put in that.)
For asteroids, it appears based on Ord that the same is true (“smaller” catastrophes much likelier than complete extinction events).
For AI, the Bostrom/Yudkowsky-style scenario might actually be more likely to kill 100% than 5-25%, or at least not much less likely. But issues involving less “discontinuous” development, some misuse by humans rather than a “treacherous turn”, multiple AI systems at similar levels, etc., might be much more likely to kill 5-25% rather than 100%. (I haven’t thought about this much.)
You could have made the exact same argument in 1917, in 1944, etc. and you would have been wildly wrong.
I guess I’d have to see the details of what the modelling would’ve looked like, but it seems plausible to me that a model in which 10% events are much less likely than 1% events, which are in turn much less likely than 0.1% events, would’ve led to good predictions. And cherry-picking two years where that would’ve failed doesn’t mean the predictions would’ve been foolish ex ante.
E.g., if the model said something as bad as the Spanish Flu would happen every two centuries, then predicting a 0.5% (or whatever) chance of it in 1917 would’ve made sense. And if you looked at that prediction alongside the surrounding 199 predictions, the predictor might indeed seem well-calibrated. (These numbers are made up for the example only.)
(Also, the Defence in Depth paper seems somewhat relevant to this matter, and is great in any case.)
I feel the need to clarify, by the way, that I’m being a bit overly aggressive in my tone here and I apologize for that. I think I was writing quickly and didn’t realize how I came across. I think you are making good points and have been upvoting them even as I disagree with them.
Just FYI: I don’t think I found your tone overly aggressive in either of your comments. Perhaps a slight vibe of strawmanning or accidental misinterpretation (which are hard to tease apart), but nothing major, and no sense of aggression :)
Something I’ve just realised I forgot to mention in reply: These are estimates of existential risk, not extinction risk. I do get the impression that Ord thinks extinction is the most likely form of existential catastrophe, but the fact that these estimates aren’t just aboutextinction might help explain at least part of why they’re higher than you see as plausible.
In particular, your response based on it seeming very unlikely an event could kill even people in remote villages etc. is less important if we’re talking about existential risk than extinction risk. That’s because if everyone except the people in the places you mentioned were killed, it seems very plausible we’d have something like an unrecoverable collapse, or the population lingering at low levels for a short while before some “minor” catastrophe finishes them off.
This relates to Richard’s question “Are you suggesting that six people could repopulate the human species?”
I haven’t looked at this in much detail, but Ord’s estimates seem too high to me. It seems really hard for humanity to go extinct, considering that there are people in remote villages, people in submarines, people in mid-flight at the time a disaster strikes, and even people on the International Space Station. (And yes, there are women on the ISS, I looked that up.) I just don’t see how e.g. a pandemic would plausibly kill all those people.
Also, if engineered pandemics, or “unforeseen” and “other” anthropogenic risks have a chance of 3% each of causing extinction, wouldn’t you expect to see smaller versions of these risks (that kill, say, 10% of people, but don’t result in extinction) much more frequently? But we don’t observe that.
(I haven’t read Ord’s book so I don’t know if he addresses these points.)
What is the significance of the people on the ISS? Are you suggesting that six people could repopulate the human species? And what sort of disaster takes less time than a flight, and only kills people on the ground?
Also, I expect to see small engineered pandemics, but only after effective genetic engineering is widespread. So the fact that we haven’t seen any so far is not much evidence.
Yes, that was broadly the response I had in mind as well. Same goes for most of the “unforeseen”/”other” anthropogenic risks; those categories are in the chapter on “Future risks”, and are mostly things Ord appears to think either will or may get riskier as certain technologies are developed/advanced.
Sleepy reply to Tobias’ “Ord’s estimates seem too high to me”: An important idea in the book is that “the per-century extinction risks from “natural” causes must be very low, based in part on our long history of surviving such risks” (as I phrase it in this post). The flipside of that is roughly the argument that we haven’t got strong evidence of our ability to survive (uncollapsed and sans dystopia) a long period with various technologies that will be developed later, but haven’t been yet.
Of course, that doesn’t seem sufficient by itself as a reason for a high level of concern, as some version of that could’ve been said at every point in history when “things were changing”. But if you couple that general argument with specific reasons to believe upcoming technologies could be notably risky, you could perhaps reasonably arrive at Ord’s estimates. (And there are obviously a lot of specific details and arguments and caveats that I’m omitting here.)
I don’t think so. I think a reasonable prior on this sort of thing would have killing 10% of people not much more likely than killing 100% of people, and actually IMO it should be substantially less likely. (Consider a distribution over asteroid impacts. The range of asteroids that kill around 10% of humans is a narrow band of asteroid sizes, whereas any asteroid above a certain size will kill 100%.)
Moreover, if you think 10% is much more likely than 100%, you should think 1% is much more likely than 10%, and so on. But we don’t in fact see lots of unknown risks killing even 0.1% of the population. So that means the probability of x-risk from unknown causes according to you must be really really really small. But that’s just implausible. You could have made the exact same argument in 1917, in 1944, etc. and you would have been wildly wrong.
Well, historically, there have been quite a few pandemics that killed more than 10% of people, e.g. the Black Death or Plague of Justinian. There’s been no pandemic that killed everyone.
Is your point that it’s different for anthropogenic risks? Then I guess we could look at wars for historic examples. Indeed, there have been wars that killed something on the order of 10% of people, at least in the warring nations, and IMO that is a good argument to take the risk of a major war quite seriously.
But there have been far more wars that killed fewer people, and none that caused extinction. The literature usually models the number of casualties as a Pareto distribution, which means that the probability density is monotonically decreasing in the number of deaths. (For a broader reference class of atrocities, genocides, civil wars etc., I think the picture is similar.)
Smoking, lack of exercise, and unhealthy diets each kill more than 0.1% of the population each year. Coronavirus may kill 0.1% in some countries. The advent of cars in the 20th century resulted in 60 million road deaths, which is maybe 0.5% of everyone alive over that time (I haven’t checked this in detail). That can be seen as an unknown from the perspective of someone in 1900. Granted, some of those are more gradual than the sort of catastrophe people have in mind—but actually I’m not sure why that matters.
Looking at individual nations, I’m sure you can find many examples of civil wars, famines, etc. killing 0.1% of the population of a certain country, but far fewer examples killing 10% (though there are some). I’m not claiming the latter is 100x less likely but it is clearly much less likely.
I don’t understand this. What do you think the exact same argument would have been, and why was that wildly wrong?
I think there are some risks which have “common mini-versions,” to coin a phrase, and others which don’t. Asteroids have mini-versions (10%-killer-versions), and depending on how common they are the 10%-killers might be more likely than the 100%-killers, or vice versa. I actually don’t know which is more likely in that case.
AI risk is the sort of thing that doesn’t have common mini-versions, I think. An AI with the means and motive to kill 10% of humanity probably also has the means and motive to kill 100%.
Natural pandemics DO have common mini-versions, as you point out.
It’s less clear with engineered pandemics. That depends on how easy they are to engineer to kill everyone vs. how easy they are to engineer to kill not-everyone-but-at-least-10%, and it depends on how motivated various potential engineers are.
Accidental physics risks (like igniting the atmosphere, creating a false vacuum collapse or black hole or something with a particle collider) are way more likely to kill 100% of humanity than 10%. They do not have common mini-versions.
So what about unknown risks? Well, we don’t know. But from the track record of known risks, it seems that probably there are many diverse unknown risks, and so probably at least a few of them do not have common mini-versions.
And by the argument you just gave, the “unknown” risks that have common mini-versions won’t actually be unknown, since we’ll see their mini-versions. So “unknown” risks are going to be disproportionately the kind of risk that doesn’t have common mini-versions.
...
As for what I meant about making the exact same argument in the past: I was just saying that we’ve discovered various risks that don’t have common mini-versions, which at one point were unknown and then became known. Your argument basically rules out discovering such things ever again. Had we listened to your argument before learning about AI, for example, we would have concluded that AI was impossible, or that somehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threats.
I think I agree with this general approach to thinking about this.
From what I’ve seen of AI risk discussions, I think I’d stand by my prior statement, which I’d paraphrase now as: There are a variety of different types of AI catastrophe scenario that have been discussed. Some seem like they might be more likely or similarly likely to totally wipe us out that to cause a 5-25% death toll. But some don’t. And I haven’t seen super strong arguments for considering the former much more likely than the latter. And it seems like the AI safety community as a whole has become more diverse in their thinking on this sort of thing over the last few years.
For engineered pandemics, it still seems to me that literally 100% of people dying from the pathogens themselves seems much less likely than a very high number dying, perhaps even enough to cause existential catastrophe slightly “indirectly”. However “well” engineered, pathogens themselves aren’t agents which explicitly seek the complete extinction of humanity. (Again, Defence in Depth seems relevant here.) Though this is slightly different from a conversation about the relative likelihood of 10% vs other percentages. (Also, I feel hesitant to discuss this in great deal, for vague information hazards reasons.)
I agree regarding accidental physics risks. But I think the risks from those is far lower than the risks from AI and bio, and probably nanotech, nuclear, etc. (I don’t really bring any independent evidence to the table; this is just based on the views I’ve seen from x-risk researchers.)
I think that’d logically follow from your prior statements. But I’m not strongly convinced about those statements, except regarding accidental physics risks, which seem very unlikely.
I think this is an interesting point. It does tentatively update me towards thinking that, conditional on there indeed being “unknown risks” that are already “in play”, they’re more likely than I’d otherwise thing to jump straight to 100%, without “mini versions”.
However, I think the most concerning source of “unknown risks” are new technologies or new actions (risks that aren’t yet “in play”). The unknown equivalents of risks from nanotech, space exploration, unprecedented consolidation of governments across the globe, etc. “Drawing a new ball from the urn”, in Bostrom’s metaphor. So even if such risks do have “common mini-versions”, we wouldn’t yet have seen them.
Also, regarding the portion of unknown risks that are in play, it seems to be appropriate to respond to the argument “Most risks have common mini-versions, but we haven’t seen these for unknown risks (pretty much by definition)” partly by updating towards thinking the unknown risks lack such common mini-versions, but also partly by updating towards thinking unknown risks are unlikely. We aren’t forced to fully take the former interpretation.
Tobias’ original point was ” Also, if engineered pandemics, or “unforeseen” and “other” anthropogenic risks have a chance of 3% each of causing extinction, wouldn’t you expect to see smaller versions of these risks (that kill, say, 10% of people, but don’t result in extinction) much more frequently? But we don’t observe that. ”
Thus he is saying there aren’t any “unknown” risks that do have common mini-versions but just haven’t had time to develop yet. That’s way too strong a claim, I think. Perhaps in my argument against this claim I ended up making claims that were also too strong. But I think my central point is still right: Tobias’ argument rules out things arising in the future that clearly shouldn’t be ruled out, because if we had run that argument in the past it would have ruled out various things (e.g. AI, nukes, physics risks, and come to think of it even asteroid strikes and pandemics if we go far enough back in the past) that in fact happened.
1. I interpreted the original claim—“wouldn’t you expect”—as being basically one in which observation X was evidence against hypothesis Y. Not conclusive evidence, just an update. I didn’t interpret it as “ruling things out” (in a strong way) or saying that there aren’t any unknown risks without common mini-versions (just that it’s less likely that there are than one would otherwise think). Note that his point seemed to be in defence of “Ord’s estimates seem too high to me”, rather than “the risks are 0″.
2. I do think that Tobias’s point, even interpreted that way, was probably too strong, or missing a key detail, in that the key sources of risks are probably emerging or new things, so we wouldn’t expect to have observed their mini-versions yet. Though I do tentatively think I’d expect to see mini-versions before the “full thing”, once the new things do start arising. (I’m aware this is all pretty hand-wavey phrasing.)
3i. As I went into more in my other comment, I think the general expectation that we’ll expect to see very small versions before and more often than small ones, which we expect to see before and more often than medium, which we expect to see before and more often than large, etc., probably would’ve served well in the past. There was progressively more advanced tech before AI, and AI is progressively advancing more. There were progressively more advanced weapons, progressively more destructive wars, progressively larger numbers of nukes, etc. I’d guess the biggest pandemics and asteroid strikes weren’t the first, because the biggest are rare.
3ii. AI is the least clear of those examples, because:
(a) it seems like destruction from AI so far has been very minimal (handful of fatalities from driverless cars, the “flash crash”, etc.), yet it seems plausible major destruction could occur in future
(b) we do have specific arguments, though of somewhat unclear strength, that the same AI might actively avoid causing any destruction for a while, and then suddenly seize decisive strategic advantage etc.
But on (a), I do think most relevant researchers would say the risk this month from AI is extremely low; the risks will rise in future as systems become more capable. So there’s still time in which we may see mini-versions.
And on (b), I’d consider that a case where a specific argument updates us away from a generally pretty handy prior that we’ll see small things earlier and more often than extremely large things. And we also don’t yet have super strong reason to believe that those arguments are really painting the right picture, as far as I’m aware.
3iii. I think if we interpreted Tobias’s point as something like “We’ll never see anything that’s unlike the past”, then yes, of course that’s ridiculous. So as I mentioned elsewhere, I think it partly depends on how we carve up reality, how we define things, etc. E.g., do we put nukes in a totally new bucket, or consider it a part of trends in weaponry/warfare/explosives?
But in any case, my interpretation of Tobias’s point, where it’s just about it being unlikely to see extreme things before smaller versions, would seem to work with e.g. nukes, even if we put them in their own special category—we’d be surprise by the first nuke, but we’d indeed see there’s one nuke before there are thousands, and there are two detonations on cities before there’s a full-scale nuclear war (if there ever is one, which hopefully and plausibly there won’t be).
In general I think you’ve thought this through more carefully than me so without having read all your points I’m just gonna agree with you.
So yeah, I think the main problem with Tobias’ original point was that unknown risks are probably mostly new things that haven’t arisen yet and thus the lack of observed mini-versions of them is no evidence against them. But I still think it’s also true that some risks just don’t have mini-versions, or rather are as likely or more likely to have big versions than mini-versions. I agree that most risks are not like this, including some of the examples I reached for initially.
Hmm. I’m not sure I’m understanding you correctly. But I’ll respond to what I think you’re saying.
Firstly, the risk of natural pandemics, which the Spanish Flu was a strong example of, did have “common mini-versions”. In fact, Wikipedia says the Black Death was the “most fatal pandemic recorded in human history”. So I really don’t think we’d have ruled out the Spanish Flu happening by using the sort of argument I’m discussing (which I’m not sure I’d call “my argument”) - we’d have seen it as unlikely in any given year, and that would’ve been correct. But I imagine we’d have given it approximately the correct ex ante probability.
Secondly, even nuclear weapons, which I assume is what your reference to 1944 is about, seem like they could fit neatly in this sort of argument. It’s a new weapon, but weapons and wars existed for a long time. And the first nukes really couldn’t have killed everyone. And then we gradually had more nukes, more test explosions, more Cold War events, as we got closer and closer to it being possible for 100% of people to die from it. And we haven’t had 100% die. So it again seems like we wouldn’t have ruled out what ended up happening.
Likewise, AI can arguably be seen as a continuation of past technological, intellectual, scientific, etc. progress in various ways. Of course, various trends might change in shape, speed up, etc. But so far they do seem to have mostly done so somewhat gradually, such that none of the developments would’ve been “ruled out” by expecting the future to looking roughly similar to the past or the past+extrapolation. (I’m not an expert on this, but I think this is roughly the conclusion AI Impacts is arriving at based on their research.)
Perhaps a key point is that we indeed shouldn’t say “The future will be exactly like the past.” But instead “The future seems likely to typically be fairly well modelled as a rough extrapolation of some macro trends. But there’ll be black swans sometimes. And we can’t totally rule out totally surprising things, especially if we do very new things.”
This is essentially me trying to lay out a certain way of looking at things. It’s not necessarily the one I strongly adopt. (I actually hadn’t thought about this viewpoint much before, so I’ve found trying to lay it out/defend it here interesting.)
In fact, as I said, I (at least sort-of) disagreed with Tobias’ original comment, and I’m very concerned about existential risks. And I think a key point is that new technologies and actions can change the distributions we’re drawing from, in ways that we don’t understand. I’m just saying it still seems quite plausible to me (and probably likely, though not guaranteed) that we’d see a 5-25%-style catastrophe from a particular type of risk before a 100% catastrophe from it. And I think history seems consistent with that, and that that idea probably would’ve done fairly well in the past.
(Also, as I noted in my other comment, I haven’t yet seen very strong evidence or arguments against the idea that “somehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threats”—or more specifically, that AI might result in that, whether or not it has the “motive” to do that. It seems to me the jury is still out. So I don’t think I’d use the fact an argument reaches that conclusion as a point against that argument.)
I agree with all this and don’t think it significantly undermines anything I said.
I think the community has indeed developed more diverse views over the years, but I still think the original take (as seen in Bostrom’s Superintelligence) is the closest to the truth. The fact that the community has gotten more diverse can be easily explained as the result of it growing a lot bigger and having a lot more time to think. (Having a lot more time to think means more scenarios can be considered, more distinctions made, etc. More time for disagreements to arise and more time for those disagreements to seem like big deals when really they are fairly minor; the important things are mostly agreed on but not discussed anymore.) Or maybe you are right and this is evidence that Bostrom is wrong. Idk. But currently I think it is weak evidence, given the above.
Yeah in retrospect I really shouldn’t have picked nukes and natural pandemics as my two examples. Natural pandemics do have common mini-versions, and nukes, well, the jury is still out on that one. (I think it could go either way. I think that nukes maybe can kill everyone, because the people who survive the initial blasts might die from various other causes, e.g. civilizational collapse or nuclear winter. But insofar as we think that isn’t plausible, then yeah killing 10% is way more likely than killing 100%. (I’m assuming we count killing 99% as killing 10% here?) )
I think AI, climate change tail risks, physics risks, grey goo, etc. would be better examples for me to talk about.
With nukes, I do share the view that they could plausibly kill everyone. If there’s a nuclear war, followed by nuclear winter, and everyone dies during that winter, rather than most people dying and then the rest succumbing 10 years later from something else or never recovering, I’d consider that nuclear war causing 100% deaths.
My point was instead that that really couldn’t have happened in 1945. So there was one nuke, and a couple explosions, and gradually more nukes and test explosions, etc., before there was a present risk of 100% of people dying from this source. So we did see something like “mini-versions”—Hiroshima and Nagasaki, test explosions, Cuban Missile Crisis—before we saw 100% (which indeed, we still haven’t and hopefully won’t).
With climate change, we’re already seeing mini-versions. I do think it’s plausible that there could be a relatively sudden jump due to amplifying feedback loops. But “relatively sudden” might mean over months or years or something like that. And it wouldn’t be a total bolt from the blue in any case—the damage is already accruing and increasing, and likely would do in the lead up to such tail risks.
AI, physics risks, and nanotech are all plausible cases where there’d be a sudden jump. And I’m very concerned about AI and somewhat about nanotech. But note that we don’t actually have clear evidence that those things could cause such sudden jumps. I obviously don’t think we should wait for such evidence, because if it came we’d be dead. But it just seems worth remembering that before using “Hypothesis X predicts no sudden jump in destruction from Y” as an argument against hypothesis X.
Also, as I mentioned in my other comment, I’m now thinking maybe the best way to look at that is specific arguments in the case of AI, physics risks, and nanotech updating us away from the generally useful prior that we’ll see small things before extreme versions of the same things.
Minor: some recent papers argue the death toll from the Plague of Justinian has been exaggerated.
https://www.pnas.org/content/116/51/25546?fbclid=IwAR1bN1LgbMI-CVUNxGsm3QxCEhGMVMB50IkEVoKpEIfSySEmxY6Ug5IhRTE
https://academic.oup.com/past/article/244/1/3/5532056
(Two authors appear on both papers.)
I think, for basically Richard Ngo’s reasons, I weakly disagree with a strong version of Tobias’s original claim that:
(Whether/how much I disagree depends in part on what “much more frequently” is meant to imply.)
I also might agree with:
(Again, depends in part on what “much more likely” would mean.)
But I was very surprised to read:
And I mostly agree with Tobias’s response to that.
The point that there’s a narrower band of asteroid sizes that would cause ~10% of the population’s death than of asteroid sizes that would cause 100% makes sense. But I believe there are also many more asteroids in that narrow band. E.g., Ord writes:
And for pandemics it seems clearly far more likely to have one that kills a massive number of people than one that kills everyone. Indeed, my impression is that most x-risk researchers think it’d be notably more likely for a pandemic to cause existential catastrophe through something like causing collapse or reduction in population to below minimum viable population levels, rather than by “directly” killing everyone. (I’m not certain of that, though.)
I’d guess, without much basis, that:
the distribution of impacts from a source of risk would vary between the different sources of risks
For pandemics, killing (let’s say) 5-25% really is much more likely than killing 100%. (This isn’t why I make this guess, but it seems worth noting: Metaculus currently predicts a 6% chance COVID-19 kills >100million people by the end of the year, which would put it close to that 5-25% range. I really don’t know how much stock to put in that.)
For asteroids, it appears based on Ord that the same is true (“smaller” catastrophes much likelier than complete extinction events).
For AI, the Bostrom/Yudkowsky-style scenario might actually be more likely to kill 100% than 5-25%, or at least not much less likely. But issues involving less “discontinuous” development, some misuse by humans rather than a “treacherous turn”, multiple AI systems at similar levels, etc., might be much more likely to kill 5-25% rather than 100%. (I haven’t thought about this much.)
I guess I’d have to see the details of what the modelling would’ve looked like, but it seems plausible to me that a model in which 10% events are much less likely than 1% events, which are in turn much less likely than 0.1% events, would’ve led to good predictions. And cherry-picking two years where that would’ve failed doesn’t mean the predictions would’ve been foolish ex ante.
E.g., if the model said something as bad as the Spanish Flu would happen every two centuries, then predicting a 0.5% (or whatever) chance of it in 1917 would’ve made sense. And if you looked at that prediction alongside the surrounding 199 predictions, the predictor might indeed seem well-calibrated. (These numbers are made up for the example only.)
(Also, the Defence in Depth paper seems somewhat relevant to this matter, and is great in any case.)
Yeah I take back what I said about it being substantially less likely, that seems wrong.
I feel the need to clarify, by the way, that I’m being a bit overly aggressive in my tone here and I apologize for that. I think I was writing quickly and didn’t realize how I came across. I think you are making good points and have been upvoting them even as I disagree with them.
Just FYI: I don’t think I found your tone overly aggressive in either of your comments. Perhaps a slight vibe of strawmanning or accidental misinterpretation (which are hard to tease apart), but nothing major, and no sense of aggression :)
Something I’ve just realised I forgot to mention in reply: These are estimates of existential risk, not extinction risk. I do get the impression that Ord thinks extinction is the most likely form of existential catastrophe, but the fact that these estimates aren’t just about extinction might help explain at least part of why they’re higher than you see as plausible.
In particular, your response based on it seeming very unlikely an event could kill even people in remote villages etc. is less important if we’re talking about existential risk than extinction risk. That’s because if everyone except the people in the places you mentioned were killed, it seems very plausible we’d have something like an unrecoverable collapse, or the population lingering at low levels for a short while before some “minor” catastrophe finishes them off.
This relates to Richard’s question “Are you suggesting that six people could repopulate the human species?”