Thanks for this postâI think this is a very important topic! I largely agree that this argument deserves substantial weight, and that we should probably think more about unknown existential risks and about how we should respond to the possibility of them.
In general, the more recently we discovered a particular existential risk, the more probable it appears. I proposed that this pattern occurs because technological growth introduces increasingly-significant risks. But we have an alternative explanation: Perhaps all existential risks are unlikely, but the recently-discovered risks appear more likely due to bad priors plus wide error bars in our probability estimates.
(Iâm not sure if the following is disagreeing with you or just saying something separate yet consistent)
I agree that recently discovered risks seem the largest source of existential risk, and that two plausible explanations of that are âtechnological growth introduces increasingly significant risksâ and âwe have bad priors in general and wider error bars for probability estimates about recently discovered risks than about risks we discovered earlierâ. But it also seems to me that we can believe all that and yet still think thereâs a fairly high chance weâll later realise that the risks we recently discovered are less risky than we thought, and that future unknown risks in general arenât very risky, even if we didnât have bad priors.
Essentially, this is because, if we have a few decades or centuries without an existential catastrophe (and especially if this happens despite the development of highly advanced AI, nanotechnology, and biotech), thatâll represent new evidence not just about those particular risks, but also about how risky advanced technologies are in general and how resilient to major changes civilization is in general.
Thousands of years where asteroids couldâve wiped humanity out but didnât updates us towards thinking extinction risk from asteroids is low. I think that, in the same way, developing more things in the reference class of âvery advanced tech with hard-to-predict consequencesâ without an existential catastrophe occurring should update us towards thinking existential risk from developing more things like that is relatively low.
I think we can make that update in 2070 (or whatever) without saying our beliefs in 2020 were foolish, given what we knew. And I donât think we should make that update yet, as we havenât yet seen that evidence of âgeneral civilizational stabilityâ (or something like that).
Thanks for this postâI think this is a very important topic! I largely agree that this argument deserves substantial weight, and that we should probably think more about unknown existential risks and about how we should respond to the possibility of them.
(Iâm not sure if the following is disagreeing with you or just saying something separate yet consistent)
I agree that recently discovered risks seem the largest source of existential risk, and that two plausible explanations of that are âtechnological growth introduces increasingly significant risksâ and âwe have bad priors in general and wider error bars for probability estimates about recently discovered risks than about risks we discovered earlierâ. But it also seems to me that we can believe all that and yet still think thereâs a fairly high chance weâll later realise that the risks we recently discovered are less risky than we thought, and that future unknown risks in general arenât very risky, even if we didnât have bad priors.
Essentially, this is because, if we have a few decades or centuries without an existential catastrophe (and especially if this happens despite the development of highly advanced AI, nanotechnology, and biotech), thatâll represent new evidence not just about those particular risks, but also about how risky advanced technologies are in general and how resilient to major changes civilization is in general.
Thousands of years where asteroids couldâve wiped humanity out but didnât updates us towards thinking extinction risk from asteroids is low. I think that, in the same way, developing more things in the reference class of âvery advanced tech with hard-to-predict consequencesâ without an existential catastrophe occurring should update us towards thinking existential risk from developing more things like that is relatively low.
I think we can make that update in 2070 (or whatever) without saying our beliefs in 2020 were foolish, given what we knew. And I donât think we should make that update yet, as we havenât yet seen that evidence of âgeneral civilizational stabilityâ (or something like that).
Does that make sense?