Hi! What do you think about x-risk cause prioritization showing that regardless of whether the chance of a pandemic is 0.1% or 10%, the enormous size of the future makes this worthwhile anyway, especially given tractable solutions? Asking as I think the crux for working on biosec is less about how likely something is to happen, than what might happen. I am really happy with the recent clarification on E2H, but still feel like the low protection needed is under explained, meaning I think the lowest confidence from my point of view is whether current plans builds sufficient protection. That said, I would agree with “we don’t know what protection level is required, but let’s start building what we can, some protection is better than none”.
I don’t think we should take the enormous size of the future into account because I don’t think we have sufficient knowledge about the future to have any good guesses on how large humanity will become or whether it will create a world that we would approve of. As such, I would measure the value of existential risk reduction much more conservatively, such as the number of humans over the next two hundred years who wouldn’t be born.
I find the chance of an existential risk from biorisks to be extremely low (I.e. 10^-7). This is because I think that, for a biorisk to be an existential risk, I think that it would have to make the surface of the Earth unlivable for an extended period of time. Otherwise, a significant number of humans will survive and be able to continue civilization. I find this unlikely to occur because, even if this is technically possible, it would be extraordinarily difficult. (That said, I haven’t looked into mirror bacteria.)
Even among people who are worried about engineered pandemics, virtually none of the are worried about existential risk. As such, if we’re basing our view on other people’s view, and (1) very few experts are concerned about engineered pandemics (2) very few of those experts are worried about existential risk (3) we have the prior that we should significantly discount fringe views because they’re almost always wrong, then we can reasonably get to quite a low risk of engineered pandemics.
I definitely agree with the idea of doing a bit of a numbers game though. If the numbers suggest we should be really focused on biorisks over other concerns, I think it makes sense to allocate a lot of resources towards it.
you might want to read these, the short is that it’s not obvious that if we collapse the modern economy that we can actually get back to interstellar, because we will have way less non renewables (phosphorus, oil/coal) at our disposal next time (don’t totally agree, but worth considering).
Point 1, strong agree esp w/ we don’t know if per human (or human descendant) ev is positive or negative (relative to counterfactual, which could be nothing or aliens), wish this was more mainstream dogma here, not sure why longtermists think they can just not engage with this.
Point 3, again I’ll re route to my response to point 2. Covid (the virus) wasn’t even that bad in a sense and it was still catastrophic (for how much it affected society). Imagine something slightly worse than covid + a record heat wave that causes a massive refugee crisis or + a war. I’m not so confident this wouldn’t massively collapse the global economy, and then route back to the fruit picking, we might only get 1-3 tries to go interstellar, so collapsing the global economy prob not == death but would == reduced chance of becoming grabby, which is ~= death from POV of total utilitarian.
1 - I think it is good to be sceptical about this. I guess you’ve read about Pascal’s mugging and other related topics. I am happy you keep this debate alive as it’s not really settled.
2 - Yeah look at mirror bacteria, which is just one example of engineered, environment-to-human threats (air turning toxic)
3 - I think this is a useful intuition, but it also has limits. That said, there are step changes in history, such as powered flight invented only after thousands of years of longing and attempts by humans. This is about step changes in technology. If you believe DNA is the code of life, it is not a large leap to believe we can create really bad stuff. And it has happened before, like the great oxygenation event that ~killed all previous biological life on earth.
But basically, please be skeptical—it is super healthy!
Hi! What do you think about x-risk cause prioritization showing that regardless of whether the chance of a pandemic is 0.1% or 10%, the enormous size of the future makes this worthwhile anyway, especially given tractable solutions? Asking as I think the crux for working on biosec is less about how likely something is to happen, than what might happen. I am really happy with the recent clarification on E2H, but still feel like the low protection needed is under explained, meaning I think the lowest confidence from my point of view is whether current plans builds sufficient protection. That said, I would agree with “we don’t know what protection level is required, but let’s start building what we can, some protection is better than none”.
Three things:
I don’t think we should take the enormous size of the future into account because I don’t think we have sufficient knowledge about the future to have any good guesses on how large humanity will become or whether it will create a world that we would approve of. As such, I would measure the value of existential risk reduction much more conservatively, such as the number of humans over the next two hundred years who wouldn’t be born.
I find the chance of an existential risk from biorisks to be extremely low (I.e. 10^-7). This is because I think that, for a biorisk to be an existential risk, I think that it would have to make the surface of the Earth unlivable for an extended period of time. Otherwise, a significant number of humans will survive and be able to continue civilization. I find this unlikely to occur because, even if this is technically possible, it would be extraordinarily difficult. (That said, I haven’t looked into mirror bacteria.)
Even among people who are worried about engineered pandemics, virtually none of the are worried about existential risk. As such, if we’re basing our view on other people’s view, and (1) very few experts are concerned about engineered pandemics (2) very few of those experts are worried about existential risk (3) we have the prior that we should significantly discount fringe views because they’re almost always wrong, then we can reasonably get to quite a low risk of engineered pandemics.
I definitely agree with the idea of doing a bit of a numbers game though. If the numbers suggest we should be really focused on biorisks over other concerns, I think it makes sense to allocate a lot of resources towards it.
Re point 2,
https://forum.effectivealtruism.org/posts/CxMusuX8E5hiTXEWX/fruit-picking-as-an-existential-risk
https://forum.effectivealtruism.org/posts/Nc9fCzjBKYDaDJGiX/what-is-the-likelihood-that-civilizational-collapse-would-1
you might want to read these, the short is that it’s not obvious that if we collapse the modern economy that we can actually get back to interstellar, because we will have way less non renewables (phosphorus, oil/coal) at our disposal next time (don’t totally agree, but worth considering).
Point 1, strong agree esp w/ we don’t know if per human (or human descendant) ev is positive or negative (relative to counterfactual, which could be nothing or aliens), wish this was more mainstream dogma here, not sure why longtermists think they can just not engage with this.
Point 3, again I’ll re route to my response to point 2. Covid (the virus) wasn’t even that bad in a sense and it was still catastrophic (for how much it affected society). Imagine something slightly worse than covid + a record heat wave that causes a massive refugee crisis or + a war. I’m not so confident this wouldn’t massively collapse the global economy, and then route back to the fruit picking, we might only get 1-3 tries to go interstellar, so collapsing the global economy prob not == death but would == reduced chance of becoming grabby, which is ~= death from POV of total utilitarian.
1 - I think it is good to be sceptical about this. I guess you’ve read about Pascal’s mugging and other related topics. I am happy you keep this debate alive as it’s not really settled.
2 - Yeah look at mirror bacteria, which is just one example of engineered, environment-to-human threats (air turning toxic)
3 - I think this is a useful intuition, but it also has limits. That said, there are step changes in history, such as powered flight invented only after thousands of years of longing and attempts by humans. This is about step changes in technology. If you believe DNA is the code of life, it is not a large leap to believe we can create really bad stuff. And it has happened before, like the great oxygenation event that ~killed all previous biological life on earth.
But basically, please be skeptical—it is super healthy!
Ah, thank you!