As for what I meant about making the exact same argument in the past: I was just saying that weāve discovered various risks that donāt have common mini-versions, which at one point were unknown and then became known. Your argument basically rules out discovering such things ever again. Had we listened to your argument before learning about AI, for example, we would have concluded that AI was impossible, or that somehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threats.
Hmm. Iām not sure Iām understanding you correctly. But Iāll respond to what I think youāre saying.
Firstly, the risk of natural pandemics, which the Spanish Flu was a strong example of, did have ācommon mini-versionsā. In fact, Wikipedia says the Black Death was the āmost fatal pandemic recorded in human historyā. So I really donāt think weād have ruled out the Spanish Flu happening by using the sort of argument Iām discussing (which Iām not sure Iād call āmy argumentā) - weād have seen it as unlikely in any given year, and that wouldāve been correct. But I imagine weād have given it approximately the correct ex ante probability.
Secondly, even nuclear weapons, which I assume is what your reference to 1944 is about, seem like they could fit neatly in this sort of argument. Itās a new weapon, but weapons and wars existed for a long time. And the first nukes really couldnāt have killed everyone. And then we gradually had more nukes, more test explosions, more Cold War events, as we got closer and closer to it being possible for 100% of people to die from it. And we havenāt had 100% die. So it again seems like we wouldnāt have ruled out what ended up happening.
Likewise, AI can arguably be seen as a continuation of past technological, intellectual, scientific, etc. progress in various ways. Of course, various trends might change in shape, speed up, etc. But so far they do seem to have mostly done so somewhat gradually, such that none of the developments wouldāve been āruled outā by expecting the future to looking roughly similar to the past or the past+extrapolation. (Iām not an expert on this, but I think this is roughly the conclusion AI Impacts is arriving at based on their research.)
Perhaps a key point is that we indeed shouldnāt say āThe future will be exactly like the past.ā But instead āThe future seems likely to typically be fairly well modelled as a rough extrapolation of some macro trends. But thereāll be black swans sometimes. And we canāt totally rule out totally surprising things, especially if we do very new things.ā
This is essentially me trying to lay out a certain way of looking at things. Itās not necessarily the one I strongly adopt. (I actually hadnāt thought about this viewpoint much before, so Iāve found trying to lay it out/ādefend it here interesting.)
In fact, as I said, I (at least sort-of) disagreed with Tobiasā original comment, and Iām very concerned about existential risks. And I think a key point is that new technologies and actions can change the distributions weāre drawing from, in ways that we donāt understand. Iām just saying it still seems quite plausible to me (and probably likely, though not guaranteed) that weād see a 5-25%-style catastrophe from a particular type of risk before a 100% catastrophe from it. And I think history seems consistent with that, and that that idea probably wouldāve done fairly well in the past.
(Also, as I noted in my other comment, I havenāt yet seen very strong evidence or arguments against the idea that āsomehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threatsāāor more specifically, that AI might result in that, whether or not it has the āmotiveā to do that. It seems to me the jury is still out. So I donāt think Iād use the fact an argument reaches that conclusion as a point against that argument.)
Likewise, AI can arguably be seen as a continuation of past technological, intellectual, scientific, etc. progress in various ways. Of course, various trends might change in shape, speed up, etc. But so far they do seem to have mostly done so somewhat gradually, such that none of the developments wouldāve been āruled outā by expecting the future to looking roughly similar to the past or the past+extrapolation. (Iām not an expert on this, but I think this is roughly the conclusion AI Impacts is arriving at based on their research.)
I agree with all this and donāt think it significantly undermines anything I said.
I think the community has indeed developed more diverse views over the years, but I still think the original take (as seen in Bostromās Superintelligence) is the closest to the truth. The fact that the community has gotten more diverse can be easily explained as the result of it growing a lot bigger and having a lot more time to think. (Having a lot more time to think means more scenarios can be considered, more distinctions made, etc. More time for disagreements to arise and more time for those disagreements to seem like big deals when really they are fairly minor; the important things are mostly agreed on but not discussed anymore.) Or maybe you are right and this is evidence that Bostrom is wrong. Idk. But currently I think it is weak evidence, given the above.
Yeah in retrospect I really shouldnāt have picked nukes and natural pandemics as my two examples. Natural pandemics do have common mini-versions, and nukes, well, the jury is still out on that one. (I think it could go either way. I think that nukes maybe can kill everyone, because the people who survive the initial blasts might die from various other causes, e.g. civilizational collapse or nuclear winter. But insofar as we think that isnāt plausible, then yeah killing 10% is way more likely than killing 100%. (Iām assuming we count killing 99% as killing 10% here?) )
I think AI, climate change tail risks, physics risks, grey goo, etc. would be better examples for me to talk about.
With nukes, I do share the view that they could plausibly kill everyone. If thereās a nuclear war, followed by nuclear winter, and everyone dies during that winter, rather than most people dying and then the rest succumbing 10 years later from something else or never recovering, Iād consider that nuclear war causing 100% deaths.
My point was instead that that really couldnāt have happened in 1945. So there was one nuke, and a couple explosions, and gradually more nukes and test explosions, etc., before there was a present risk of 100% of people dying from this source. So we did see something like āmini-versionsāāHiroshima and Nagasaki, test explosions, Cuban Missile Crisisābefore we saw 100% (which indeed, we still havenāt and hopefully wonāt).
With climate change, weāre already seeing mini-versions. I do think itās plausible that there could be a relatively sudden jump due to amplifying feedback loops. But ārelatively suddenā might mean over months or years or something like that. And it wouldnāt be a total bolt from the blue in any caseāthe damage is already accruing and increasing, and likely would do in the lead up to such tail risks.
AI, physics risks, and nanotech are all plausible cases where thereād be a sudden jump. And Iām very concerned about AI and somewhat about nanotech. But note that we donāt actually have clear evidence that those things could cause such sudden jumps. I obviously donāt think we should wait for such evidence, because if it came weād be dead. But it just seems worth remembering that before using āHypothesis X predicts no sudden jump in destruction from Yā as an argument against hypothesis X.
Also, as I mentioned in my other comment, Iām now thinking maybe the best way to look at that is specific arguments in the case of AI, physics risks, and nanotech updating us away from the generally useful prior that weāll see small things before extreme versions of the same things.
Hmm. Iām not sure Iām understanding you correctly. But Iāll respond to what I think youāre saying.
Firstly, the risk of natural pandemics, which the Spanish Flu was a strong example of, did have ācommon mini-versionsā. In fact, Wikipedia says the Black Death was the āmost fatal pandemic recorded in human historyā. So I really donāt think weād have ruled out the Spanish Flu happening by using the sort of argument Iām discussing (which Iām not sure Iād call āmy argumentā) - weād have seen it as unlikely in any given year, and that wouldāve been correct. But I imagine weād have given it approximately the correct ex ante probability.
Secondly, even nuclear weapons, which I assume is what your reference to 1944 is about, seem like they could fit neatly in this sort of argument. Itās a new weapon, but weapons and wars existed for a long time. And the first nukes really couldnāt have killed everyone. And then we gradually had more nukes, more test explosions, more Cold War events, as we got closer and closer to it being possible for 100% of people to die from it. And we havenāt had 100% die. So it again seems like we wouldnāt have ruled out what ended up happening.
Likewise, AI can arguably be seen as a continuation of past technological, intellectual, scientific, etc. progress in various ways. Of course, various trends might change in shape, speed up, etc. But so far they do seem to have mostly done so somewhat gradually, such that none of the developments wouldāve been āruled outā by expecting the future to looking roughly similar to the past or the past+extrapolation. (Iām not an expert on this, but I think this is roughly the conclusion AI Impacts is arriving at based on their research.)
Perhaps a key point is that we indeed shouldnāt say āThe future will be exactly like the past.ā But instead āThe future seems likely to typically be fairly well modelled as a rough extrapolation of some macro trends. But thereāll be black swans sometimes. And we canāt totally rule out totally surprising things, especially if we do very new things.ā
This is essentially me trying to lay out a certain way of looking at things. Itās not necessarily the one I strongly adopt. (I actually hadnāt thought about this viewpoint much before, so Iāve found trying to lay it out/ādefend it here interesting.)
In fact, as I said, I (at least sort-of) disagreed with Tobiasā original comment, and Iām very concerned about existential risks. And I think a key point is that new technologies and actions can change the distributions weāre drawing from, in ways that we donāt understand. Iām just saying it still seems quite plausible to me (and probably likely, though not guaranteed) that weād see a 5-25%-style catastrophe from a particular type of risk before a 100% catastrophe from it. And I think history seems consistent with that, and that that idea probably wouldāve done fairly well in the past.
(Also, as I noted in my other comment, I havenāt yet seen very strong evidence or arguments against the idea that āsomehow AIs which have the means and motive to kill 10% of people are more likely than AIs which pose existential threatsāāor more specifically, that AI might result in that, whether or not it has the āmotiveā to do that. It seems to me the jury is still out. So I donāt think Iād use the fact an argument reaches that conclusion as a point against that argument.)
I agree with all this and donāt think it significantly undermines anything I said.
I think the community has indeed developed more diverse views over the years, but I still think the original take (as seen in Bostromās Superintelligence) is the closest to the truth. The fact that the community has gotten more diverse can be easily explained as the result of it growing a lot bigger and having a lot more time to think. (Having a lot more time to think means more scenarios can be considered, more distinctions made, etc. More time for disagreements to arise and more time for those disagreements to seem like big deals when really they are fairly minor; the important things are mostly agreed on but not discussed anymore.) Or maybe you are right and this is evidence that Bostrom is wrong. Idk. But currently I think it is weak evidence, given the above.
Yeah in retrospect I really shouldnāt have picked nukes and natural pandemics as my two examples. Natural pandemics do have common mini-versions, and nukes, well, the jury is still out on that one. (I think it could go either way. I think that nukes maybe can kill everyone, because the people who survive the initial blasts might die from various other causes, e.g. civilizational collapse or nuclear winter. But insofar as we think that isnāt plausible, then yeah killing 10% is way more likely than killing 100%. (Iām assuming we count killing 99% as killing 10% here?) )
I think AI, climate change tail risks, physics risks, grey goo, etc. would be better examples for me to talk about.
With nukes, I do share the view that they could plausibly kill everyone. If thereās a nuclear war, followed by nuclear winter, and everyone dies during that winter, rather than most people dying and then the rest succumbing 10 years later from something else or never recovering, Iād consider that nuclear war causing 100% deaths.
My point was instead that that really couldnāt have happened in 1945. So there was one nuke, and a couple explosions, and gradually more nukes and test explosions, etc., before there was a present risk of 100% of people dying from this source. So we did see something like āmini-versionsāāHiroshima and Nagasaki, test explosions, Cuban Missile Crisisābefore we saw 100% (which indeed, we still havenāt and hopefully wonāt).
With climate change, weāre already seeing mini-versions. I do think itās plausible that there could be a relatively sudden jump due to amplifying feedback loops. But ārelatively suddenā might mean over months or years or something like that. And it wouldnāt be a total bolt from the blue in any caseāthe damage is already accruing and increasing, and likely would do in the lead up to such tail risks.
AI, physics risks, and nanotech are all plausible cases where thereād be a sudden jump. And Iām very concerned about AI and somewhat about nanotech. But note that we donāt actually have clear evidence that those things could cause such sudden jumps. I obviously donāt think we should wait for such evidence, because if it came weād be dead. But it just seems worth remembering that before using āHypothesis X predicts no sudden jump in destruction from Yā as an argument against hypothesis X.
Also, as I mentioned in my other comment, Iām now thinking maybe the best way to look at that is specific arguments in the case of AI, physics risks, and nanotech updating us away from the generally useful prior that weāll see small things before extreme versions of the same things.