I think it would be true if there were other X-risks. I just think that there is no other literal X-risk. I think that there are huge catastrophic risks. But there’s still a huge difference between killing 99% of people and killing a 100%.
I’d recommend reading (or skimming through) this to have a better sense of how different the 2 are.
I think that in general the sense that it’s cool to work on every risks come precisely from the fact that very few people have thought about every risks and thus people in AI for instance IMO tend to overestimate risks in other areas.
“no other literal X-risk” seems too strong. There are certainly some potential ways that nuclear war or a bioweapon could cause human extinction. They’re not just catastrophic risks.
In addition, catastrophic risks don’t just involve massive immediate suffering. They drastically change global circumstances in a way which will have knock-on effects on whether, when, and how we build AGI.
All that said, I directionally agree with you, and I think that probably all longtermists should have a model of the effects their work has on the potentiality of aligned AGI, and that they should seriously consider switching to working more directly on AI, even if their competencies appear to lie elsewhere. I just think that your post takes this point too far.
Just tell me a story with probabilities of how nuclear war or bioweapons could cause human extinction and you’ll see that when you’ll multiply the probabilities, it will go down to a very low number. I repeat but I think that you don’t still have a good sense of how difficult it is to kill every humans if the minimal viable population (MVP) is around 1000 as argued in the post linked above.
”knock-on effects” I think that it’s true but I think that on the first-order, not dying from AGI is the most important thing compared with developing it in like 100 years.
I have a slight problem with the “tell me a story” framing. Scenarios are useful, but also lend themselves general to crude rather than complex risks. In asking this question, you implicitly downplay complex risks. For a more thorough discussion, the “Democratising Risk” paper by Cramer and Kemp has some useful ideas in it (I disagree with parts of the paper but still)
It also continues to priorities epistemically neat and “sexy” risks which whilst possibly the most worrying are not exclusive.
Also probabilities on scenarios in many contexts can be somewhat problematic, and the methodologies used to come up with very high xrisk values for AGI vs other xrisks have very high uncertainties. To this degree, I think the certainty you have is somewhat problematic
Yes, scenarios are a good way to put a lower bound but if you’re not able to create one single scenario that’s a bad sign in my opinion.
For AGI there are many plausible scenarios where I can reach ~1-10% likelihood of dying. With biorisks it’s impossible with my current belief on the MVP (minimum viable population)
I think that if you take these infohazards seriously enough, you probably even shouldn’t do that. Because if everyone has a 95% likelihood to keep it secret, with 10 persons in the know is 60%.
I see what you mean, but if you value cause prioritization seriously enough, it is really stifling to have literally no place to discuss x-risks in detail. Carefully managed private spaces are the best compromise I’ve seen so far, but if there’s something better then I’d be really glad to learn about it.
I think that I’d be glad to stay as long as we can in the domain of aggregate probabilities and proxies of real scenarios, particularly for biorisks. Mostly because I think that most people can’t do a lot about infohazardy things so the first-order effect is just net negative.
Yes I mostly agree but even conditional on info hazardy things I still think that the aggregate probability of likelihood of collapse is a very important parameter.
I’m not sure what you mean—I agree the aggregate probability of collapse is an important parameter, but I was talking about the kinds of bio-risk scenarios that simeon_c was asking for above? Do I understand you right that overall risk levels should be estimated/communicated even though their components might involve info-hazards? If so, I agree, and it’s tricky. They’ll likely be some progress on this over the next 6-12 months with Open Phil’s project to quantify bio-risk, and (to some extent) the results of UPenn’s hybrid forecasting/persuasion tournament on existential risks.
Thanks for this information! What’s the probability we go extinct due to biorisks by 2045 according to you?
Also, I think that things that are extremely infohazardy shouldn’t be thought of too strongly bc without the info revelation they will likely remain very unlikely
I’m currently involved in the UPenn tournament so can’t communicate my forecasts or rationales to maintain experimental conditions, but it’s at least substantially higher than 1⁄10,000.
And yeah, I agree complicated plans where an info-hazard makes the difference are unlikely, but info-hazards also preclude much activity and open communication about scenarios even in general.
I don’t have a deep model of AI—I mostly defer to some bodged-together aggregate of reasonable seeming approaches/people (e.g. Carlsmith/Cotra/Davidson/Karnofsky/Ord/surveys).
I think that it’s one of the problems that explains why many people find my claim far too strong: in the EA community, very few people have a strong inside view on both advanced AIs and biorisks. (I think that’s it’s more generally true for most combinations of cause areas).
And I think that indeed, with the kind of uncertainty one must have when one’s deferring , it becomes harder to do claims as strong as the one I’m making here.
Also, I think that things that are extremely infohazardy shouldn’t be thought of too strongly bc without the info revelation they will likely remain very unlikely
I don’t think this reasoning works in general. A highly dangerous technology could become obvious in 2035, but we could still want actors to not know about it until as late as possible. Or the probability of a leak over the next 10 years could be high, yet it could still be worth trying to maintain secrecy.
Yes, I think you’re right actually. Here’s a weaker claim which is I think it true: - When someone knows and has thought on a infohazard, the baseline is that he’s way more likely to cause harm via it than to cause good. - Thus, I’d recommend anyone who’s not actively thinking about ways to solve to prevent classes of scenario where this infohazard would end up being very bad, to try to forget this infohazard and not talk about it even to trusted individuals. Otherwise it will most likely be net negative.
Luisa’s post addresses our chance of getting killed ‘within decades’ of a civilisational collapse, but that’s not the same as the chance that it prevents us ever becoming a happy intergalactic civilisation, which is the end state we’re seeking. If you think that probability is 90%, given a global collapse, then the effective x-risk of that collapse is 0.1 * <its probability of happening>. One order of magnitude doesn’t seem like that big a deal here, given all the other uncertainties around our future.
That’s right! I just think that the base rate for “civilisation collapse prevents us from ever becoming a happy intergalactic civilisation” is very low. And multiplying any probability by 0.1 also does matter because when we’re talking about AGI, we’re talking about things are >=10% likely to happen for a lot of people (I put a higher likelihood than that but Toby Ord putting 10% is sufficient).
So it means that even if you condition on biorisks being the same as AGI (which is the point I argue against) for everything else, you still need biorisks to be >5% likely to lead to a civilizational collapse by the end of the century for my point to not hold, i.e that 95% of longtermists should work AI (19/20 of the people + assumption of linear returns for the few first thousands ppl).
Thanks for the comment.
I think it would be true if there were other X-risks. I just think that there is no other literal X-risk. I think that there are huge catastrophic risks. But there’s still a huge difference between killing 99% of people and killing a 100%.
I’d recommend reading (or skimming through) this to have a better sense of how different the 2 are.
I think that in general the sense that it’s cool to work on every risks come precisely from the fact that very few people have thought about every risks and thus people in AI for instance IMO tend to overestimate risks in other areas.
“no other literal X-risk” seems too strong. There are certainly some potential ways that nuclear war or a bioweapon could cause human extinction. They’re not just catastrophic risks.
In addition, catastrophic risks don’t just involve massive immediate suffering. They drastically change global circumstances in a way which will have knock-on effects on whether, when, and how we build AGI.
All that said, I directionally agree with you, and I think that probably all longtermists should have a model of the effects their work has on the potentiality of aligned AGI, and that they should seriously consider switching to working more directly on AI, even if their competencies appear to lie elsewhere. I just think that your post takes this point too far.
Just tell me a story with probabilities of how nuclear war or bioweapons could cause human extinction and you’ll see that when you’ll multiply the probabilities, it will go down to a very low number.
I repeat but I think that you don’t still have a good sense of how difficult it is to kill every humans if the minimal viable population (MVP) is around 1000 as argued in the post linked above.
”knock-on effects”
I think that it’s true but I think that on the first-order, not dying from AGI is the most important thing compared with developing it in like 100 years.
I have a slight problem with the “tell me a story” framing. Scenarios are useful, but also lend themselves general to crude rather than complex risks. In asking this question, you implicitly downplay complex risks. For a more thorough discussion, the “Democratising Risk” paper by Cramer and Kemp has some useful ideas in it (I disagree with parts of the paper but still) It also continues to priorities epistemically neat and “sexy” risks which whilst possibly the most worrying are not exclusive. Also probabilities on scenarios in many contexts can be somewhat problematic, and the methodologies used to come up with very high xrisk values for AGI vs other xrisks have very high uncertainties. To this degree, I think the certainty you have is somewhat problematic
Yes, scenarios are a good way to put a lower bound but if you’re not able to create one single scenario that’s a bad sign in my opinion.
For AGI there are many plausible scenarios where I can reach ~1-10% likelihood of dying. With biorisks it’s impossible with my current belief on the MVP (minimum viable population)
Sketching specific bio-risk extinction scenarios would likely involve substantial info-hazards.
You could avoid such infohazards by drawing up the scenarios in a private message or private doc that’s only shared with select people.
I think that if you take these infohazards seriously enough, you probably even shouldn’t do that. Because if everyone has a 95% likelihood to keep it secret, with 10 persons in the know is 60%.
I see what you mean, but if you value cause prioritization seriously enough, it is really stifling to have literally no place to discuss x-risks in detail. Carefully managed private spaces are the best compromise I’ve seen so far, but if there’s something better then I’d be really glad to learn about it.
I think that I’d be glad to stay as long as we can in the domain of aggregate probabilities and proxies of real scenarios, particularly for biorisks.
Mostly because I think that most people can’t do a lot about infohazardy things so the first-order effect is just net negative.
Yes I mostly agree but even conditional on info hazardy things I still think that the aggregate probability of likelihood of collapse is a very important parameter.
I’m not sure what you mean—I agree the aggregate probability of collapse is an important parameter, but I was talking about the kinds of bio-risk scenarios that simeon_c was asking for above?
Do I understand you right that overall risk levels should be estimated/communicated even though their components might involve info-hazards? If so, I agree, and it’s tricky. They’ll likely be some progress on this over the next 6-12 months with Open Phil’s project to quantify bio-risk, and (to some extent) the results of UPenn’s hybrid forecasting/persuasion tournament on existential risks.
Thanks for this information!
What’s the probability we go extinct due to biorisks by 2045 according to you?
Also, I think that things that are extremely infohazardy shouldn’t be thought of too strongly bc without the info revelation they will likely remain very unlikely
I’m currently involved in the UPenn tournament so can’t communicate my forecasts or rationales to maintain experimental conditions, but it’s at least substantially higher than 1⁄10,000.
And yeah, I agree complicated plans where an info-hazard makes the difference are unlikely, but info-hazards also preclude much activity and open communication about scenarios even in general.
And on AI, do you have timelines + P(doom|AGI)?
I don’t have a deep model of AI—I mostly defer to some bodged-together aggregate of reasonable seeming approaches/people (e.g. Carlsmith/Cotra/Davidson/Karnofsky/Ord/surveys).
I think that it’s one of the problems that explains why many people find my claim far too strong: in the EA community, very few people have a strong inside view on both advanced AIs and biorisks. (I think that’s it’s more generally true for most combinations of cause areas).
And I think that indeed, with the kind of uncertainty one must have when one’s deferring , it becomes harder to do claims as strong as the one I’m making here.
I don’t think this reasoning works in general. A highly dangerous technology could become obvious in 2035, but we could still want actors to not know about it until as late as possible. Or the probability of a leak over the next 10 years could be high, yet it could still be worth trying to maintain secrecy.
Yes, I think you’re right actually.
Here’s a weaker claim which is I think it true:
- When someone knows and has thought on a infohazard, the baseline is that he’s way more likely to cause harm via it than to cause good.
- Thus, I’d recommend anyone who’s not actively thinking about ways to solve to prevent classes of scenario where this infohazard would end up being very bad, to try to forget this infohazard and not talk about it even to trusted individuals. Otherwise it will most likely be net negative.
Luisa’s post addresses our chance of getting killed ‘within decades’ of a civilisational collapse, but that’s not the same as the chance that it prevents us ever becoming a happy intergalactic civilisation, which is the end state we’re seeking. If you think that probability is 90%, given a global collapse, then the effective x-risk of that collapse is 0.1 * <its probability of happening>. One order of magnitude doesn’t seem like that big a deal here, given all the other uncertainties around our future.
That’s right! I just think that the base rate for “civilisation collapse prevents us from ever becoming a happy intergalactic civilisation” is very low.
And multiplying any probability by 0.1 also does matter because when we’re talking about AGI, we’re talking about things are >=10% likely to happen for a lot of people (I put a higher likelihood than that but Toby Ord putting 10% is sufficient).
So it means that even if you condition on biorisks being the same as AGI (which is the point I argue against) for everything else, you still need biorisks to be >5% likely to lead to a civilizational collapse by the end of the century for my point to not hold, i.e that 95% of longtermists should work AI (19/20 of the people + assumption of linear returns for the few first thousands ppl).