I’m a Research Fellow at Open Philanthropy, though all views I express here are my own.
Before OP I was an independent researcher in global health and biosecurity, and a Charity Entrepreneurship incubatee. I have an MD and undergrad degrees in philosophy, international relations, and neuroscience, all from the University of Sydney.
Yeah, though to be fair the CEA for Malawi was b/c it was LEEP’s literal first campaign. I’d imagine LEEP has CEAs for all their country work which include adjustments for likelihood of success, though I don’t know whether they intend to publish them any time soon.
Yeah makes sense, and that the early research could have been heavily discounted by pessimism about a charity achieving big wins.
One example I know of off the top of my head is LEEP—their CEA for their Malawi campaign found a median of $14/DALY. CE’s original report on lead paint regulation suggested $156/DALY (as a central estimate, I think). That direction and magnitude is pretty surprising to me. I expect it would be explicable based on the details of the different approaches/considerations, but I’d need to look into the details. Maybe a motivating story is that LEEP’s Malawi campaign was surprisingly fast and effective compared to the original report’s hopes?
Another is Family Empowerment Media. An ex post Rethink Priorities report mentions FEM used a Givewell model to estimate a cost-effectiveness 26.9x cash transfers, and Founders Pledge estimated 22x. The original CE report links to a CEA that estimates $984/DALY averted, which is lower than Givewell top charities—though I don’t know the exact comparison to cash transfers, and there are other benefits to family planning than just DALYs.
I suspect a strong selection effect is in play—i.e. I know of these examples and their CEAs are prominent because they were successful—and the ideas survived the gauntlet of further research, selection, founding, piloting, and scaling.
I sympathise with this view, but I think I see it in more continuous terms than ex ante vs. ex post, and maybe akin to quality. This is because even ex post, I think there would still be substantial guess-work and assumptions, and the bottom line still relies on interpretation. But the difference for ex post is how empirically informed that analysis can be, and how specific. I.e an ex post analysis can ground estimates on data for that specific org, with that program, in that community. Ex ante analyses can also differ in quality for how empirically informed they are, and how specific they are. But a great ex ante CEA could be more empirically informed and specific than an sub-par ex post CEA. But all this is ~semantics—I think we basically agree.
Not sure about the geography question—from this it looks like the 2012 estimate was based on distribution in Malawi. In 2022 they distribute in DRC, Ghana, Guinea, Malawi, Papua New Guinea, Togo, Uganda, and Zambia, and my guess is that the Givewell figure is an average of those programs? Read into that what you will.
Ooh another angle would be to compare Charity Entrepreneurship’s ex ante CEAs with the eventual charities’ own ex post CEAs. But there’d be a strong selection effect given it depends on eventual charity success/stability, plus the interventions change a lot from the research to implementaiton.
I think a similar view is found in ‘Why we can’t take expected value estimates literally even when they’re unbiased’ I.e. we should have a pretty low prior that any particular intervention is above (e.g.) 10x cash transfers, but the strength and robustness of top charities’ CEAs are sufficient to clear them over the bar. And most CEAs of specific interventions written up on the forum aren’t compelling enough to bring the estimate all that much higher from the low prior.
I agree it’d be informative to see what ‘naive’ versions of top charity CEAs would be like. As a quick and dirty version, I looked at Givewell’s stuff on AMF—their 2023 central figure is $5,500 per life saved, with 226 rows in the spreadsheet. If we look at their 2012 CEA (downloadable here), they have 45 rows with their optimistic value being $1819 per life saved. Leaving aside inter-temporal confounders, naively a 3x cut is reasonable if the random forum CEA is equivalent to 2012 optimistic Givewell. Though it depends on the quality of the random CEA, I’d guess a 2x-10x cut is a reasonable prior? Plus a stronger cut for really high estimates—eg a 500x cost-effectiveness is more likely to be due to an over-generous methodology
I weakly agree with the claim that the offense/defense balance is not a useful way to project the implications of AI. However, I disagree strongly with how the post got there. Considering only cyber-security and per-capita death rate is not a sufficient basis for the claim that there is “little historical evidence for large changes in the O/D balance, even in response to technological revolutions.”
There are good examples where technology greatly shifts the nature of war: castles favouring defense, before becoming negated by cannons. The machine gun and barbed wire are typically held as technologies that gave a significant defensive advantage in WWI, and were crucial in the development of trench warfare. Tanks similarly for offense. And so on. One would need to consider and reject these examples (which is certainly plausible, but needs to be actually engaged with).
I don’t think the per capita mortality from war is a very useful measure of the O/D balance—it will be driven by many other confounding factors, and as you point out technologies alter the decision calculus of both sides of a conflict. Part of the problem with the O/D theory is a lack of well-defined measures (e.g.). From my impression of the literature, it also appears to fail in empirical predictions (e.g.). Cyber-security is a good example of an arena where O/D theory applies poorly (e.g.). I think this section does well to support the claim that “shifting O/D balance doesn’t dominantly affect the frequency and intensity of war”, but it doesn’t show that “technologies don’t shift the O/D balance”.
So I disagree with the post by thinking that technologies can systematically favour aggressive vs. defensive actions and actors, but agree that this pattern doesn’t necessarily result in strong implications about the rate or intensity of large-scale conflict. However, I think there can still be implications for non-state violence, especially when the technology enables an ‘asymmetric weapon’ - a cheap way to inflict very costly damages or demand costly defence. The prototypical example here being terrorist use of improvised explosives. This is an important channel for worries about future bioweapons.
Nitpick. I think you meant bioterrorism, not terrorism which includes more data.
Thanks! Fixed.
I don’t know the nuclear field well, so don’t have much to add. If I’m following your comment though, it seems like you have your own estimate of the chance of nuclear war raising 47+ Tg of soot, and on the basis of that infer the implied probability supers give to extinction conditional on such a war. Why not instead infer that supers have a higher forecast of nuclear war than your 0.39% by 2100? E.g. a ~1.6% chance of nuclear war with 47+ Tg and a 5% chance of extinction conditional on it. I may be misunderstanding your comment. Though to be clear, I think it’s very possible the supers were not thinking things through in similar detail to you—there were a fair number of questions in the XPT.
I am left wondering about how much higher extinction risk will become accounting not only for increased capabilities, but also increased safety
I don’t think I follow this sentence? Is it that one might expect future advances in defensive biotech/other tech to counterbalance offensive tech development, and that without a detailed quant model you expect the defensive side to be under-counted?
Hi Vasco, nice post thanks for writing it! I haven’t had the time to look into all your details so these are some thoughts written quickly.
I worked on a project for Open Phil quantifying the likely number of terrorist groups pursuing bioweapons over the next 30 years, but didn’t look specifically at attack magnitudes (I appreciate the push to get a public-facing version of the report published—I’m on it!). That work was as an independent contractor for OP, but I now work for them on the GCR Cause Prio team. All that to say these are my own views, not OP’s.
I think this is a great post grappling with the empirics of terrorism. And I agree with the claim that the history of terrorism implies an extinction-level terrorist attack is unlikely. However, for similar reasons to Jeff Kaufman, I don’t think this strongly undermines the existential threat from non-state actors. This is for three reasons, one methodological and two qualitative:
The track record of bioterrorism in particular is too sparse to make empirical projections with much confidence. I think the rarity of bioterror and general small magnitudes of attacks justifies a prior against bioterror as a significant threat, but only a weak one. It also justifies a prior that we should expect at least a handful of groups to attempt bioterror over the next 10-30 years. To take the broader set of terror attacks as having strong implications for future bioterror, one would need to think that ‘terrorism’ is a compelling reference class for bio x-risk—which my next two points dispute.
The vast majority of terror groups (‘violent non-state actors’ is a more generally applicable handle) would not want to cause extinction. Omnicidality is a fairly rare motivation—most groups have specific political aims, or ideological motivations that are predicated on a particular people/country/sect/whatever thriving and overcoming its enemies. Aiming for civilisational collapse is slightly more prevalent, though still uncommon. And for all of history, there hasn’t been a viable path to omnicide or x-risk anyway. So the kind of actor that presents a bio x-risk is probably going to be very different to the kinds of actor that make up the track record of terrorism.
The vast majority of terror attacks are kinetic—involving explosives, firearms, vehicles, melee weapons. The exceptions are chemical and biological weapons. The biological weapons chosen are generally non-transmissible—anthrax, botulinum toxin, ricin, etc. This means that chem and bio attacks also rely on delivery mechanisms that have to get each individual victim to come into contact with the agent. An attack with a pandemic-class agent would not rely on such delivery. It would be strikingly different in complexity of development, attack modality, targeting specificity, and many other dimensions. I.e. it would be very unlike almost all previous terrorist attacks. The ability to carry out such an attack is also fairly unprecedented—it may only emerge with subsequent developments in biotechnology, especially from the convergence of AI and biotechnology.
So overall, compared to the threat model of future bio x-risk, I think the empirical track record of bioterrorism is too weak (point 1), and the broader terrorism track record is based on actors with very different motivations (point 2) using very different attack modalities (point 3). The latter two points are grounded in a particular worldview—that within coming years/decades biotechnology will enable biological weapons with catastrophic potential. I think that worldview is certainly contestable, but I think the track record of terrorism is not the most fruitful line of attack against it.
On a meta-level, the fact that XPT superforecasters are so much higher than what your model outputs suggests that they also think the right reference class approach is OOMs higher. And this is despite my suspicion that the XPT supers are too low and too indexed on past base-rates.
You emailed asking for reading recommendations—in lieu of my actual report (which will take some time to get to a publishable state), here’s my structured bibliography! In particular I’d recommend Binder & Ackermann 2023 (CBRN Terrorism) and McCann 2021 (Outbreak: A Comprehensive Analysis of Biological Terrorism).
Although focused on civil conflicts, Lauren Gilbert’s shallow explores some possible interventions in this space, including:
Disarmament, Demobilization, and Reintegration (DDR) Programs
Community-Driven Development
Cognitive Behavioral Therapy
Cash Transfers and/or Job Training
Alternative Dispute Resolution (ADR)
Contact Interventions and Mass Media
Investigative Journalism
Mediation and Diplomacy
Open Phil had this issue—they now use ‘Global Health & Wellbeing’ and ‘Global Catastrophic Risks’, which I think captures the substantive focus of each.
As one data point: I was interested in global health from a young age, and found 80K during med school in 2019, which led to opportunities in biosecurity research, and now I’m a researcher on global catastrophic risks. I’m really glad I’ve made this transition! However, it’s possible that I would have not applied to 80K (and not gone down this path) if I had gotten the impression they weren’t interested in near-termist causes.
Looking back at my 80K 1on1 application materials, I can see I was aware that 80K thought global health was less neglected than biosecurity, and I was considering bio as a career (though perhaps only with 20-30% credence compared to global health). If I’d been aware at the time just how longtermist 80K is, I think there’s a 20-40% chance I would have not applied.
I think Elika’s is a great example of having a lot of impact, but I agree that an example shifting from global health is maybe unnecessarily dismissive. I don’t think the tobacco thing is good—surely any remotely moral career advisor would advise moving away from that. Ideally a reader who shifted from a neutral or only very-mildly-good career to a great career would be better (as they do for their other examples). I’d guess 80K know some great examples? Maybe someone working exclusively on rich-country health or pharma who moved into bio-risk?
Happy to end this thread here. On a meta-point, I think paying attention to nuance/tone/implicatures is a better communication strategy than retreating to legalese, but it does need practice. I think reflecting on one’s own communicative ability is more productive than calling others irrational or being passive-aggressive. But it sucks that this has been a bad experience for you. Hope your day goes better!
Things can be ‘not the best’, but still good. For example, let’s say a systematic, well-run, whistleblower organisation was the ‘best’ way. And compare it to ‘telling your friends about a bad org’. ‘Telling your friends’ is not the best strategy, but it still might be good to do, or worth doing. Saying “telling your friends is not the best way” is consistent with this. Saying “telling your friends is a bad idea” is not consistent with this.
I.e. ‘bad idea’ connotes much more than just ‘sub-optimal, all things considered’.
Your top-level post did not claim ‘public exposés are not the best strategy’, you claimed “public exposés are often a bad idea in EA”. That is a different claim, and far from a default view. It is also the view I have been arguing against. I think you’ve greatly misunderstood others’ positions, and have rudely dismissed them rather than trying to understand them. You’ve ignored the arguments given by others, while not defending your own assertions. So it’s frustrating to see you playing the ‘I’m being cool-headed and rational here’ card. This has been a pretty disappointing negative update for me. Thanks
You didn’t provide an alternative, other than the example of you conducting your own private investigation. That option is not open to most, and the beneficial results do not accrue to most. I agree hundreds of hours of work is a cost; that is a pretty banal point. I think we agree that a more systematic solution would be better than relying on a single individual’s decision to put in a lot of work and take on a lot of risk. But you are, blithely in my view, dismissing one of the few responses that have the potential to protect people. Nonlinear have their own funding, and lots of pre-existing ties to the community and EA public materials. A public expose has a much better chance of protecting newcomers from serious harm than some high-up EAs having a private critical doc. The impression I have of your view is that it would have been better if Ben hadn’t written or published his post and instead saved his time, and prefer that Nonlinear was quietly rejected by those in the know. Is that an accurate picture of your view? If you think there are better solutions, it would be good to name them up front, rather than just denigrate public criticism.
Not everyone is well connected enough to hear rumours. Newcomers and/or less-well-connected people need protection from bad actors too. If someone new to the community was considering an opportunity with Nonlinear, they wouldn’t have the same epistemic access as a central and long-standing grant-maker. They could, however, see a public exposé.
What a fantastic resource, thanks all! Also may be worth adding, the new National Security Commission on Emerging Biotechnology, which will be delivering a 2024 report based on “a thorough review of how advances in emerging biotechnology and related technologies will shape current and future activities of the Department of Defense“ - delivering it to the DoD, White House, and Congress.
Ooh what about Bob Fischer? He’s a philosophy professor who ran Rethink’s moral weights project and is now on their new Worldview Investigations team! [edit: just saw him suggested in a different comment]
Big +1
Awesome work, thanks! And this model resonates with my experience getting more involved with bio over the last few years.