Amazing thread, Vasco! Your post hits like a fresh blast of reason in the middle of a doomsday, conspiracy-like fringe. I think you are on the right track by addressing ‘the risk hype.’
I have a Ph.D. in gene technology and have served on the bioadvisory board of an EU member state as a representative of environmental protection agencies for 13 years. To be honest, the “Biosecurity & Pandemics” topic enticed me to join the EA Forum, and I have been having a hard time understanding how this fits with EA.
There are only a few things more wasteful and frankly counterproductive to spend money on than mitigating obscure pandemic/bioweapon threats. It could turn out to be useful, but it lands smack in the middle of other very high-risk, low reward investments. For example, the US has spent something like $40-50 billion dollars since 2001 on anthrax research alone – a disease that only has a few cases in the US and a few thousand globally per year. Incredibly miserable investment-reward balance. I know that working with some random adenovirus or Mycobacterium tuberculosis doesn’t sound so sexy as running bioreactors with Y. pestis in BSL4, but would be orders of magnitude more effective for humanity.
This brings me to my second point: the incident with the Ames strain from USAMRIID in the 2001 anthrax letters perfectly illustrates the self-fulfilling prophecy generated by circulating these agents in labs/industry in order to develop countermeasures. In fact, such activities and initiatives are the main force increasing the risk of existential catastrophe imposed by these agents. Thereby, I cannot see reaching anywhere near the 1% chance of existential catastrophe from biological causes by 2100 without spreading corresponding infrastructure and agents unnecessarily. Even covert bioweapon development by nation-states is much smaller a problem to deal with.
And thirdly, I would address this notion – probably doing some heavy lifting to prop up the chances of existential catastrophe in some eyes – that any day now, some nut will self-educate on YouTube or some skilled professional with lab access will flip and construct a DIY bioweapon capable of posing a critical threat to society. I will give it some rope in terms of somebody starting that “secret project” not being too far-fetched. Can happen, people can be very weird! However, I can see difficulties even if the person gains access to free and unlimited NA printing resources. There is a reason why the Soviet Union had tons of anthrax and smallpox – you are going to need a large-scale, sophisticated delivery system for the initial release. Otherwise, the list of victims will include only the bioterrorist or close people, and it will never be more than a regional incident.
Not to mention all the DIY genetic engineering projects that people are much more likely to work on. From doing home gene therapy on a pet (or on oneself) to larger-scale synthetic biology projects to enable a yeast to synthesize the original special ingredient of the original version of Coca-Cola. Moreover, during times when environmental activism swings to many strange places, the next iteration of Ted Kaczynski can easily be a person that seeks to modify the biosphere in order to protect it from humanity – e.g., an agent that impacts fish, making them unedible for people. Or plastic-degrading bacteria released into the ocean.
Don’t get me wrong—I do think we have to prepare and stay vigilant. There will be a new pandemic—only question is when and which agent. Hopefully we don’t manifest it artificially out of fear. And way to go, Vasco, for being the tip of the spear to unmask delusion of grandeur behind of this ‘risk reasoning’.
I wonder if you’d be willing to be a bit more vocal about this. For example, the second most upvoted comment (27 karma right now) takes me to task for saying that “most experts are deeply skeptical of Ord’s claim” (1/30 existential biorisk in the next 100 years).
I take that to be uncontroversial. Would you be willing to say so?
David, as someone who’s generally a big fan of your work, it’s kind of on you to provide evidence that most experts are ‘deeply skeptical’ of Ord’s claim. And here’s the thing, you might not even be wrong about it! But your level of confidence in this claim is ‘uncontroversial’, and yet the evidence you provide does not match it. I find it strange/disappointing that you don’t address this, given that it’s a common theme on your blog that EAs often make overconfident claims.
“A group of health researchers from King’s College”
“an expert panel on risks posed by bioweapons” convened by one of the above researchers
“David Sarpong and colleagues”
Which you then use to conclude “Experts widely believe that existential biorisk in this century is quite low. The rest of us could do worse than to follow their example.” But you haven’t argued for this. What’s the numerator and denominator here? How are you so certain without calculating the proportion? What does ‘widely believe’ mean? Doesn’t Ord also think existential biorisk is ‘quite low’? 3.33% makes sense as ‘quite low’ to me, maybe you mean ‘exceedinly low’/‘vanishingly small chance’ or something like that instead?
Then, in part 11, you appeal to how in the recent XPT study Superforecasters reduced their median risk in existential risk from bio to 0.1% to 0.01%, but you don’t mention in the same study[1] that domain experts increasedtheir x-risk on the same question from 0.7% to 1%. So in this study when the “experts” don’t match your viewpoint, you suddenly only mention the non-experts and decline to mention that expert consensus moved in the opposite direction to what you expect, or your case expects. And even then, a1% vs 3.33% difference in subjective risk estimation doesn’t sound like a gap that merits describing a ‘deep scepticism’ of the latter from the former to me.
I like your work, and I think that you successfully ‘kicked the tires’ on the Aum Shinrikyo case present in The Precipice, for example. But you conclude this mini-series in part 11 by saying this:
“But experts are largely unconvinced that there is a serious risk of large-scale biological attacks, particularly on a scale that could lead to existential catastrophe.”
But it also turns out, from what I can tell, that most EAs don’t think so either! So maybe you’re just going after Ord here, but then again I think that ~1% v 3.33% estimation of risk doesn’t seem as big a difference as you claim. But I don’t think that’s what you’re restricting your claims to, since you also mention ‘many leading effective altruists’ and also use this to push-back on your perceived issue with how EAs treat ‘expert’ evidence, for example. But much like your critiques of EA xrisk work, I think you continually fail to produce either arguments, or good arguments, for this particular claim that can justify the strength of your position.
the second most upvoted comment (27 karma right now) takes me to task for saying that “most experts are deeply skeptical of Ord’s claim” (1/30 existential biorisk in the next 100 years).
I take that to be uncontroversial. Would you be willing to say so?
I asked because I’m interested—what makes you think most experts don’t think biorisk is such a big threat, beyond a couple of papers?
I’m glad you’re bringing your expertise into this area, thanks for jumping in! Reading your comment, however, it sounds to me like you’re responding as if EAs concerned about biosecurity are advocating work that’s pretty different than what we actually are? Some examples:
the US has spent something like $40-50 billion dollars since 2001 on anthrax research alone
I don’t know any EAs who think this has been a good use of funds. Biosecurity isn’t a knob that we turn between ‘less’ and ‘more’, it’s a broad field where we can try to discover and fund the best interventions. To make an analogy to global health and development, if we learn that funding for textbooks in low income countries has generally been very low impact (say, because of issues with absenteeism, nutrition, etc) that isn’t very relevant when deciding whether to distribute anti-malarial nets.
running bioreactors with Y. pestis in BSL4, but would be orders of magnitude more effective for humanity.
I don’t think any EAs are doing this kind of work, and the ones I’ve talked to generally think this is harmful and should stop.
the incident with the Ames strain from USAMRIID in the 2001 anthrax letters perfectly illustrates the self-fulfilling prophecy generated by circulating these agents in labs/industry in order to develop countermeasures. In fact, such activities and initiatives are the main force increasing the risk of existential catastrophe imposed by these agents.
This is another thing that EAs don’t do, and generally don’t think others should do.
Later, you do get into areas where EAs do work. For example:
that any day now, some nut will self-educate on YouTube or some skilled professional with lab access will flip and construct a DIY bioweapon capable of posing a critical threat to society.
There is a reason why the Soviet Union had tons of anthrax and smallpox – you are going to need a large-scale, sophisticated delivery system for the initial release. Otherwise, the list of victims will include only the bioterrorist or close people, and it will never be more than a regional incident.
This seems to miss one of the main reasons that biological attacks are risky: contagion. With a contagious pathogen you can infect almost the whole world with only a small amount of seeding. This gives two main patterns, ‘wildfire’ pandemics (ex: a worse Ebola) which are obvious but so contagious that they’re extremely challenging to stop, and ‘stealth’ pandemics (ex: a worse HIV) that first infect many people and only much later cause massive harm. See Securing Civilisation Against Catastrophic Pandemics.
Biosecurity isn’t a knob that we turn between ‘less’ and ‘more’, it’s a broad field where we can try to discover and fund the best interventions. To make an analogy to global health and development, if we learn that funding for textbooks in low income countries has generally been very low impact (say, because of issues with absenteeism, nutrition, etc) that isn’t very relevant when deciding whether to distribute anti-malarial nets.
I wonder how much interventions in biosecurity differ in their cost-effectiveness. From Ben Todd’s related in-depth analysis, which I should note does not look into biosecurity interventions:
Overall, I roughly estimate that the most effective measurable interventions in an area are usually around 3–10 times more cost effective than the mean of measurable interventions (where the mean is the expected effectiveness you’d get from picking randomly). If you also include interventions whose effectiveness can’t be measured in advance, then I’d expect the spread to be larger by another factor of 2–10, though it’s hard to say how the results would generalise to areas without data.
The above suggests that, in a given area, the most effective interventions are 24.5 (= (3*10)^0.5*(2*10)^0.5) times as cost-effectivene as randomly selected ones. For education in low income countries, the ratio is around 20. These ratios are not super large, so there is a sense in which knowing about the cost-effectiveness of a bunch of random interventions in a given area could inform us about the cost-effectiveness of the best ones.
Yet, it might be the case that the anthrax research is much worse than a random biosecurity intervention, despite the large investment. If so, the best biosecurity interventions could still easily be orders of magnitude more cost-effective.
it might be the case that the anthrax research is much worse than a random biosecurity intervention
I think many biosecurity interventions have historically made us less safe, likely including the anthrax research, and probably also including the median intervention. So an analysis that works by scaling the cost effectiveness of a random intervention doesn’t look so good!
Your post hits like a fresh blast of reason in the middle of a doomsday, conspiracy-like fringe. I think you are on the right track by addressing ‘the risk hype.’
Stay tuned for more! Just one note. While I do think people in the effective altruism community have often been exaggerating the risks, I would say using terms like “conspiracy”, “fringe” and “hype” tends to increase the chance of adversarial rather than constructive discussions, and therefore can be counterproductive. Yet, I appreciate you sharing your honest feelings.
To be honest, the “Biosecurity & Pandemics” topic enticed me to join the EA Forum, and I have been having a hard time understanding how this fits with EA.
I am glad you joined. I think people with expertise in bio who have not been exposed to effective altruism since their early age may have different takes which are worth listening to. You can check 80,000 Hours’ profile on preventing catastrophic pandemics for an overview of why it is a top cause area in EA. If you see yourself disagreeing with many points, and would like a side project, you can then consider sharing your thoughts in a post.
There are only a few things more wasteful and frankly counterproductive to spend money on than mitigating obscure pandemic/bioweapon threats.
This may not apply in all cases. Charity Entrepeneurship has estimated that advocating for academic guidelines to limit dual-use research of concern (DURC) can save a life for just 30 $ (related post), which is around 200 (= (5*10^3)/30) times as cost-effective as GiveWell’s top charities (often consided the best interventions in global health and development).
For example, the US has spent something like $40-50 billion dollars since 2001 on anthrax research alone – a disease that only has a few cases in the US and a few thousand globally per year.
That does look like a bad investment. Considering the value of a statistical life (VSL) used by the Federal Emergency Management Agency (FEMA) of 7.5 M$, an investment of 45 G$ (= (40 + 50)/2*10^9) would have to save 6.00 k (= 45*10^9/(7.5*10^6)) lives in the US in expectation, i.e. 261 per year (= 6.00*10^3/23) over the 23 years from 2001 to 2023. This looks like a high bar considering that anthrax is not contagious, which limits the probability of having lots of deaths.
That being said, it is worth noting the deaths from pandemics are very heavy-tailed in general, so the actual cost-effectiveness is not a good proxy for the expected cost-effectiveness, which is what one should care about. I can imagine investments in mRNA vaccines also saved few lives before Covid-19, but their expected cost-effectiveness was driven by relatively rare events, which in this case did happen.
the incident with the Ames strain from USAMRIID in the 2001 anthrax letters
For reference, Peter is referring to the 2001 anthrax attacks. People may also want to check Wikipedia’s list of bioterrorist attacks. As a side note, I think your comment would benefit from having a few links, but I appreciate this takes time!
And thirdly, I would address this notion – probably doing some heavy lifting to prop up the chances of existential catastrophe in some eyes – that any day now, some nut will self-educate on YouTube or some skilled professional with lab access will flip and construct a DIY bioweapon capable of posing a critical threat to society. I will give it some rope in terms of somebody starting that “secret project” not being too far-fetched. Can happen, people can be very weird! However, I can see difficulties even if the person gains access to free and unlimited NA printing resources. There is a reason why the Soviet Union had tons of anthrax and smallpox – you are going to need a large-scale, sophisticated delivery system for the initial release. Otherwise, the list of victims will include only the bioterrorist or close people, and it will never be more than a regional incident.
Amazing thread, Vasco! Your post hits like a fresh blast of reason in the middle of a doomsday, conspiracy-like fringe. I think you are on the right track by addressing ‘the risk hype.’
I have a Ph.D. in gene technology and have served on the bioadvisory board of an EU member state as a representative of environmental protection agencies for 13 years. To be honest, the “Biosecurity & Pandemics” topic enticed me to join the EA Forum, and I have been having a hard time understanding how this fits with EA.
There are only a few things more wasteful and frankly counterproductive to spend money on than mitigating obscure pandemic/bioweapon threats. It could turn out to be useful, but it lands smack in the middle of other very high-risk, low reward investments. For example, the US has spent something like $40-50 billion dollars since 2001 on anthrax research alone – a disease that only has a few cases in the US and a few thousand globally per year. Incredibly miserable investment-reward balance. I know that working with some random adenovirus or Mycobacterium tuberculosis doesn’t sound so sexy as running bioreactors with Y. pestis in BSL4, but would be orders of magnitude more effective for humanity.
This brings me to my second point: the incident with the Ames strain from USAMRIID in the 2001 anthrax letters perfectly illustrates the self-fulfilling prophecy generated by circulating these agents in labs/industry in order to develop countermeasures. In fact, such activities and initiatives are the main force increasing the risk of existential catastrophe imposed by these agents. Thereby, I cannot see reaching anywhere near the 1% chance of existential catastrophe from biological causes by 2100 without spreading corresponding infrastructure and agents unnecessarily. Even covert bioweapon development by nation-states is much smaller a problem to deal with.
And thirdly, I would address this notion – probably doing some heavy lifting to prop up the chances of existential catastrophe in some eyes – that any day now, some nut will self-educate on YouTube or some skilled professional with lab access will flip and construct a DIY bioweapon capable of posing a critical threat to society. I will give it some rope in terms of somebody starting that “secret project” not being too far-fetched. Can happen, people can be very weird! However, I can see difficulties even if the person gains access to free and unlimited NA printing resources. There is a reason why the Soviet Union had tons of anthrax and smallpox – you are going to need a large-scale, sophisticated delivery system for the initial release. Otherwise, the list of victims will include only the bioterrorist or close people, and it will never be more than a regional incident.
Not to mention all the DIY genetic engineering projects that people are much more likely to work on. From doing home gene therapy on a pet (or on oneself) to larger-scale synthetic biology projects to enable a yeast to synthesize the original special ingredient of the original version of Coca-Cola. Moreover, during times when environmental activism swings to many strange places, the next iteration of Ted Kaczynski can easily be a person that seeks to modify the biosphere in order to protect it from humanity – e.g., an agent that impacts fish, making them unedible for people. Or plastic-degrading bacteria released into the ocean.
Don’t get me wrong—I do think we have to prepare and stay vigilant. There will be a new pandemic—only question is when and which agent. Hopefully we don’t manifest it artificially out of fear. And way to go, Vasco, for being the tip of the spear to unmask delusion of grandeur behind of this ‘risk reasoning’.
Thanks Peter!
I wonder if you’d be willing to be a bit more vocal about this. For example, the second most upvoted comment (27 karma right now) takes me to task for saying that “most experts are deeply skeptical of Ord’s claim” (1/30 existential biorisk in the next 100 years).
I take that to be uncontroversial. Would you be willing to say so?
David, as someone who’s generally a big fan of your work, it’s kind of on you to provide evidence that most experts are ‘deeply skeptical’ of Ord’s claim. And here’s the thing, you might not even be wrong about it! But your level of confidence in this claim is ‘uncontroversial’, and yet the evidence you provide does not match it. I find it strange/disappointing that you don’t address this, given that it’s a common theme on your blog that EAs often make overconfident claims.
For example, in Part 10 of ‘Exaggerating the risks’ you evidence for the claim of ‘most experts’ is only:
“A group of health researchers from King’s College”
“an expert panel on risks posed by bioweapons” convened by one of the above researchers
“David Sarpong and colleagues”
Which you then use to conclude “Experts widely believe that existential biorisk in this century is quite low. The rest of us could do worse than to follow their example.” But you haven’t argued for this. What’s the numerator and denominator here? How are you so certain without calculating the proportion? What does ‘widely believe’ mean? Doesn’t Ord also think existential biorisk is ‘quite low’? 3.33% makes sense as ‘quite low’ to me, maybe you mean ‘exceedinly low’/‘vanishingly small chance’ or something like that instead?
Then, in part 11, you appeal to how in the recent XPT study Superforecasters reduced their median risk in existential risk from bio to 0.1% to 0.01%, but you don’t mention in the same study[1] that domain experts increased their x-risk on the same question from 0.7% to 1%. So in this study when the “experts” don’t match your viewpoint, you suddenly only mention the non-experts and decline to mention that expert consensus moved in the opposite direction to what you expect, or your case expects. And even then, a 1% vs 3.33% difference in subjective risk estimation doesn’t sound like a gap that merits describing a ‘deep scepticism’ of the latter from the former to me.
I like your work, and I think that you successfully ‘kicked the tires’ on the Aum Shinrikyo case present in The Precipice, for example. But you conclude this mini-series in part 11 by saying this:
“But experts are largely unconvinced that there is a serious risk of large-scale biological attacks, particularly on a scale that could lead to existential catastrophe.”
But it also turns out, from what I can tell, that most EAs don’t think so either! So maybe you’re just going after Ord here, but then again I think that ~1% v 3.33% estimation of risk doesn’t seem as big a difference as you claim. But I don’t think that’s what you’re restricting your claims to, since you also mention ‘many leading effective altruists’ and also use this to push-back on your perceived issue with how EAs treat ‘expert’ evidence, for example. But much like your critiques of EA xrisk work, I think you continually fail to produce either arguments, or good arguments, for this particular claim that can justify the strength of your position.
Page 66 on the pdf
I asked because I’m interested—what makes you think most experts don’t think biorisk is such a big threat, beyond a couple of papers?
I’m glad you’re bringing your expertise into this area, thanks for jumping in! Reading your comment, however, it sounds to me like you’re responding as if EAs concerned about biosecurity are advocating work that’s pretty different than what we actually are? Some examples:
I don’t know any EAs who think this has been a good use of funds. Biosecurity isn’t a knob that we turn between ‘less’ and ‘more’, it’s a broad field where we can try to discover and fund the best interventions. To make an analogy to global health and development, if we learn that funding for textbooks in low income countries has generally been very low impact (say, because of issues with absenteeism, nutrition, etc) that isn’t very relevant when deciding whether to distribute anti-malarial nets.
I don’t think any EAs are doing this kind of work, and the ones I’ve talked to generally think this is harmful and should stop.
This is another thing that EAs don’t do, and generally don’t think others should do.
Later, you do get into areas where EAs do work. For example:
Yes, this is a real concern for many of us. I wrote a case for it recently in the post Out-of-distribution Bioattacks.
This seems to miss one of the main reasons that biological attacks are risky: contagion. With a contagious pathogen you can infect almost the whole world with only a small amount of seeding. This gives two main patterns, ‘wildfire’ pandemics (ex: a worse Ebola) which are obvious but so contagious that they’re extremely challenging to stop, and ‘stealth’ pandemics (ex: a worse HIV) that first infect many people and only much later cause massive harm. See Securing Civilisation Against Catastrophic Pandemics.
Happy to get into any of this more!
Thanks for the clarifications, Jeff!
I wonder how much interventions in biosecurity differ in their cost-effectiveness. From Ben Todd’s related in-depth analysis, which I should note does not look into biosecurity interventions:
The above suggests that, in a given area, the most effective interventions are 24.5 (= (3*10)^0.5*(2*10)^0.5) times as cost-effectivene as randomly selected ones. For education in low income countries, the ratio is around 20. These ratios are not super large, so there is a sense in which knowing about the cost-effectiveness of a bunch of random interventions in a given area could inform us about the cost-effectiveness of the best ones.
Yet, it might be the case that the anthrax research is much worse than a random biosecurity intervention, despite the large investment. If so, the best biosecurity interventions could still easily be orders of magnitude more cost-effective.
I think many biosecurity interventions have historically made us less safe, likely including the anthrax research, and probably also including the median intervention. So an analysis that works by scaling the cost effectiveness of a random intervention doesn’t look so good!
Thanks, Peeter!
Stay tuned for more! Just one note. While I do think people in the effective altruism community have often been exaggerating the risks, I would say using terms like “conspiracy”, “fringe” and “hype” tends to increase the chance of adversarial rather than constructive discussions, and therefore can be counterproductive. Yet, I appreciate you sharing your honest feelings.
I am glad you joined. I think people with expertise in bio who have not been exposed to effective altruism since their early age may have different takes which are worth listening to. You can check 80,000 Hours’ profile on preventing catastrophic pandemics for an overview of why it is a top cause area in EA. If you see yourself disagreeing with many points, and would like a side project, you can then consider sharing your thoughts in a post.
This may not apply in all cases. Charity Entrepeneurship has estimated that advocating for academic guidelines to limit dual-use research of concern (DURC) can save a life for just 30 $ (related post), which is around 200 (= (5*10^3)/30) times as cost-effective as GiveWell’s top charities (often consided the best interventions in global health and development).
That does look like a bad investment. Considering the value of a statistical life (VSL) used by the Federal Emergency Management Agency (FEMA) of 7.5 M$, an investment of 45 G$ (= (40 + 50)/2*10^9) would have to save 6.00 k (= 45*10^9/(7.5*10^6)) lives in the US in expectation, i.e. 261 per year (= 6.00*10^3/23) over the 23 years from 2001 to 2023. This looks like a high bar considering that anthrax is not contagious, which limits the probability of having lots of deaths.
That being said, it is worth noting the deaths from pandemics are very heavy-tailed in general, so the actual cost-effectiveness is not a good proxy for the expected cost-effectiveness, which is what one should care about. I can imagine investments in mRNA vaccines also saved few lives before Covid-19, but their expected cost-effectiveness was driven by relatively rare events, which in this case did happen.
For reference, Peter is referring to the 2001 anthrax attacks. People may also want to check Wikipedia’s list of bioterrorist attacks. As a side note, I think your comment would benefit from having a few links, but I appreciate this takes time!
Relatedly, I enjoyed listening to Sonia Ben Ouagrham-Gormley on Barriers to Bioweapons, which I plan to linkpost on the EA Forum in the coming weeks.