Thank so much you for writing this I think it is an excellent piece and makes a really strong case for how longtermists should consider approaching policy. I agree with most of your conclusions here.
I have been working in the space for a number of years advocating (with some limited successes) for a cost effectiveness approach to government policy making on risks in the UK (and am a contributing author to the Future Proof report your cite). Interestingly despite having made progress in the area I am over time leaning more towards work on specific advocacy focused on known risks (e.g. on pandemic preparedness) than more general work on improve government spending on risks as a whole. I have a number of unpublished notes on how to assess the value of such work that might be useful so thought I would share below.
I think there is three points my notes might helpfully add to your work
Some more depth about how to think about cost benefit analysis and in particular what the threshold is for government to take action. I think the cost benefit you describe is below the threshold for government action.
An independent literature review type analysis on what the benefit cost ratio is for on the margin additional funds going into disaster prevention. (Literature list in Annex section).
Some vague reflections as a practitioner in this space on the paths to impact
Note: Some of this research was carried out for Charity Entrepreneurship and should be considered Charity Entrepreneurship work. This post is in an independent capacity and does not represent views of any employer
1. The cost benefit analysis here is not enough to suggests government action
I think it is worth putting some though into how to interpret cost benefit analyses and how a government policy maker might interpret and use them. Your conservative estimate suggests a benefit $646 billion to a cost of $400 billion – this is a benefit cost ratio (BCR) of 1.6 to 1.
Naively a benefit cost ratio of >1 to 1 suggests that a project is worth funding. However given the overhead costs of government policy, to governments propensity to make even cost effective projects go wrong and public preferences for money in hand it may be more appropriate to apply a higher bar for cost-effective government spending. I remember I used to have a 3 to 1 ratio, perhaps picked up when I worked in Government although I cannot find a source for this now.
According to https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5537512/ the average cost benefit ratio of government investment into health programs is 8.3 to 1. I highly expect there are many actions the US government could take to improve citizens healthcare with a CBA in the 5-10 to 1 range. In comparison on a 1.6 to 1 does not look like a priority.
Some Copenhagen Consensus analysis I read considers robust evidence for benefits between 5 to 15 times higher than costs as “good” interventions.
So overall if making the case to government or the public I think making the case that there is a 1.6 to 1 BCR is not sufficient to suggest action. I would consider 3 to 1 to be a reasonable bar and 5 to 1 to be a good case for action.
2. On the other hand the benefit cost ratio is probably higher than your analysis suggests
As mentioned you directly calculate a benefit cost ratio of 1.6 to 1 (i.e. $646 billion to $400 billion).
Firstly I note that reading your workings this is clearly a conservative estimate. I would be interested to see a midline estimate of the BCR too.
I made a separate estimate that I thought I would share. It was a bit more optimistic than this. It suggested that the benefit costs ratios (BCR) for disaster prevention are that, on the margin, additional spending on disaster preparedness to be in the region of 10 to 1, maybe a bit below that. I copy my sources into an annex section below.
(That said, spending $400 billion is arguably more than “on the margin” and is a big jump in spending so we might expect that spending at that level to have a somewhat lower value. Of course in practice I don’t think advocates are going to get government to spend $400bn tomorrow and that a gradual ramp up in spending is likely justified.)
3. A few reflections on political tractability and value
My experience (based on the UK) is that I expect governments to be relatively open to change and improvement in this area. I expect the technocratic elements of government respond well to highlighting inconsistencies in process and decision making and the UK government has committed to improvements to how they asses risks. I expect governments to be a bit more reticent to make changes that necessitate significant spending or to put in place mechanisms and oversight that can hold them to account for not spending sufficiently on high-impact risks that might ensure future spending.
I am also becoming a bit more sceptical of the value of this kind of general longtermist work when put in comparison to work focusing on known risks. Based on my analysis to date I believe some of the more specific policy change ideas about preventing dangerous research or developing new technology to tackle pandemics (or AI regulation) to be a bit more tractable and a bit higher benefit to cost than then this more general work to increase spending on risks. That said in policy you may want to have many avenues you are working on at once so as to capitalise on key opportunities so these approaches should not be seen as mutually exclusive. Additionally there is a case for more general system improvements from a more patient longtermist view or from a higher focus on unknown unknown risks being critical.
ANNEX: My estimate
On the value of general disaster preparedness
We can set a prior for the value of pandemic preparedness by looking at other disaster preparedness spending.
Real-world evidence. Most of the evidence for this comes from assessments of the value of physical infrastructure preparation for natural disasters, such as building buildings that can withstand floods. See table below.
“The value of hazard mitigation is well known: the Multihazard Mitigation Council (MMC) upped their initial estimate of $4 (MMC 2005) saved for every $1 spent on hazard mitigation to $6, and $7 with regard to flood mitigation (MMC 2017).”
4:1
6½:1
Other estimates. There are also a number of estimates of benefit cost ratios:
Source
Summary
BCR
Does mitigation save? Reviewing cost-benefit analyses of disaster risk reduction
Suggest that disaster relief saves money but at what ratio is unclear and is highly dependent on situation location and kind of disaster.
BCR estimates tend to be less than 10 with a few going into the 10s and a very few much higher (largest was 1800)
BCRs may underestimate by putting a high discount rate
~10:1
Natural disasters challenge paper
(Copenhagen Consensus, 2015)
There are growing economic costs from natural disasters in recent years. This is especially true in developing countries where there may be limited insurance, higher risks and looser building codes.
Looks at retrofitting schools to be earthquake resistant in seismically active countries, suggests this has a BCR close to 1:1
Looks at constructing a one-metre high wall to protect homes or elevating houses by one metre to reduce flooding. Suggests this has a BCR of 60:1.
““We estimate that for each dollar spent on disaster preparedness, an average of four dollars is saved on disaster response and recovery” says Alberto Monguzzi, Disaster Management Coordinator in the IFRC Europe Zone Office.”
4:1
Pandemic preparedness estimates
Other estimates. We found one example of estimates of the value of preparing better for future pandemics.
“expanding the stockpile of AV drugs to encompass the whole UK population (≈60 million) might even be acceptable (≈£6,500 per QALY gained over a no intervention strategy for the 1918 scenario under base-case assumptions).”
Very roughly if we place a value of $50-100k per year of life this suggests a BCR of roughly 10:1.
“Procuring an adequate PPE stockpile in advance at non-pandemic prices would cost only 17% of the projected amount needed to procure it at current pandemic-inflated prices”
6:1
Historical data and estimates suggest the value of increasing preparedness is decent but not very high, with estimated benefit cost ratios (BCR) often around or just below 1:10.
How this changes with scale of the disaster
There is some reason to think that disaster preparedness is more cost effective when targeted at worse disasters. Theoretically this makes sense as disasters are heavy-tailed and most of the impact of preventing and mitigating disasters will be in preventing and mitigating the very worst disasters. This is also supported by models estimating the effect of pandemic preparedness, such as those discussed in this talk. (Doohan and Hauck, 202?)
Pandemics affect more people than natural disasters so we could expect a higher than average BCR. This is more relevant if we pick preparedness interventions that scale with the size of the disaster (an example of an intervention that does not have this effect might be stockpiling, for which the impact is capped by the size of the stockpile, not by the size of the disaster).
However overall I did not find much solid evidence to suggest that the BCR ratio is higher for higher scale disasters.
Naively a benefit cost ratio of >1 to 1 suggests that a project is worth funding. However given the overhead costs of government policy, to governments propensity to make even cost effective projects go wrong and public preferences for money in hand it may be more appropriate to apply a higher bar for cost-effective government spending. I remember I used to have a 3 to 1 ratio, perhaps picked up when I worked in Government although I cannot find a source for this now.
This is good to know. Our BCR of 1.6 is based on very conservative assumptions. We were basically seeing how conservative we could go while still getting a BCR of over 1. I think Carl and I agree that, on more reasonable estimates, the BCR of the suite is over 5 and maybe even over 10 (certainly I think that’s the case for some of the interventions within the suite). If, as you say, many people in government are looking for interventions with BCRs significantly higher than 1, then I think we should place more emphasis on our less conservative estimates going forward.
I made a separate estimate that I thought I would share. It was a bit more optimistic than this. It suggested that the benefit costs ratios (BCR) for disaster prevention are that, on the margin, additional spending on disaster preparedness to be in the region of 10 to 1, maybe a bit below that. I copy my sources into an annex section below.
Thanks very much for this! I might try to get some of these references into the final paper.
I am also becoming a bit more sceptical of the value of this kind of general longtermist work when put in comparison to work focusing on known risks. Based on my analysis to date I believe some of the more specific policy change ideas about preventing dangerous research or developing new technology to tackle pandemics (or AI regulation) to be a bit more tractable and a bit higher benefit to cost than then this more general work to increase spending on risks.
Thank so much you for writing this I think it is an excellent piece and makes a really strong case for how longtermists should consider approaching policy. I agree with most of your conclusions here.
I have been working in the space for a number of years advocating (with some limited successes) for a cost effectiveness approach to government policy making on risks in the UK (and am a contributing author to the Future Proof report your cite). Interestingly despite having made progress in the area I am over time leaning more towards work on specific advocacy focused on known risks (e.g. on pandemic preparedness) than more general work on improve government spending on risks as a whole. I have a number of unpublished notes on how to assess the value of such work that might be useful so thought I would share below.
I think there is three points my notes might helpfully add to your work
Some more depth about how to think about cost benefit analysis and in particular what the threshold is for government to take action. I think the cost benefit you describe is below the threshold for government action.
An independent literature review type analysis on what the benefit cost ratio is for on the margin additional funds going into disaster prevention. (Literature list in Annex section).
Some vague reflections as a practitioner in this space on the paths to impact
Note: Some of this research was carried out for Charity Entrepreneurship and should be considered Charity Entrepreneurship work. This post is in an independent capacity and does not represent views of any employer
1. The cost benefit analysis here is not enough to suggests government action
I think it is worth putting some though into how to interpret cost benefit analyses and how a government policy maker might interpret and use them. Your conservative estimate suggests a benefit $646 billion to a cost of $400 billion – this is a benefit cost ratio (BCR) of 1.6 to 1.
Naively a benefit cost ratio of >1 to 1 suggests that a project is worth funding. However given the overhead costs of government policy, to governments propensity to make even cost effective projects go wrong and public preferences for money in hand it may be more appropriate to apply a higher bar for cost-effective government spending. I remember I used to have a 3 to 1 ratio, perhaps picked up when I worked in Government although I cannot find a source for this now.
According to https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5537512/ the average cost benefit ratio of government investment into health programs is 8.3 to 1. I highly expect there are many actions the US government could take to improve citizens healthcare with a CBA in the 5-10 to 1 range. In comparison on a 1.6 to 1 does not look like a priority.
Some Copenhagen Consensus analysis I read considers robust evidence for benefits between 5 to 15 times higher than costs as “good” interventions.
So overall if making the case to government or the public I think making the case that there is a 1.6 to 1 BCR is not sufficient to suggest action. I would consider 3 to 1 to be a reasonable bar and 5 to 1 to be a good case for action.
2. On the other hand the benefit cost ratio is probably higher than your analysis suggests
As mentioned you directly calculate a benefit cost ratio of 1.6 to 1 (i.e. $646 billion to $400 billion).
Firstly I note that reading your workings this is clearly a conservative estimate. I would be interested to see a midline estimate of the BCR too.
I made a separate estimate that I thought I would share. It was a bit more optimistic than this. It suggested that the benefit costs ratios (BCR) for disaster prevention are that, on the margin, additional spending on disaster preparedness to be in the region of 10 to 1, maybe a bit below that. I copy my sources into an annex section below.
(That said, spending $400 billion is arguably more than “on the margin” and is a big jump in spending so we might expect that spending at that level to have a somewhat lower value. Of course in practice I don’t think advocates are going to get government to spend $400bn tomorrow and that a gradual ramp up in spending is likely justified.)
3. A few reflections on political tractability and value
My experience (based on the UK) is that I expect governments to be relatively open to change and improvement in this area. I expect the technocratic elements of government respond well to highlighting inconsistencies in process and decision making and the UK government has committed to improvements to how they asses risks. I expect governments to be a bit more reticent to make changes that necessitate significant spending or to put in place mechanisms and oversight that can hold them to account for not spending sufficiently on high-impact risks that might ensure future spending.
I am also becoming a bit more sceptical of the value of this kind of general longtermist work when put in comparison to work focusing on known risks. Based on my analysis to date I believe some of the more specific policy change ideas about preventing dangerous research or developing new technology to tackle pandemics (or AI regulation) to be a bit more tractable and a bit higher benefit to cost than then this more general work to increase spending on risks. That said in policy you may want to have many avenues you are working on at once so as to capitalise on key opportunities so these approaches should not be seen as mutually exclusive. Additionally there is a case for more general system improvements from a more patient longtermist view or from a higher focus on unknown unknown risks being critical.
ANNEX: My estimate
On the value of general disaster preparedness
We can set a prior for the value of pandemic preparedness by looking at other disaster preparedness spending.
Real-world evidence. Most of the evidence for this comes from assessments of the value of physical infrastructure preparation for natural disasters, such as building buildings that can withstand floods. See table below.
Natural Hazard Mitigation Saves: 2019 Report
Link
Looks at the BCR for different disaster mitigation prevention. For example:
Adopting building codes 11:1
Changing buildings 4:1
(We think this source has some risk of bias, although it does appear to be high quality.)
11:1
4:1
If Mitigation Saves $6 Per Every $1 …
(Gall and Friedland, 2020)
Link
“The value of hazard mitigation is well known: the Multihazard Mitigation Council (MMC) upped their initial estimate of $4 (MMC 2005) saved for every $1 spent on hazard mitigation to $6, and $7 with regard to flood mitigation (MMC 2017).”
4:1
6½:1
Other estimates. There are also a number of estimates of benefit cost ratios:
Does mitigation save? Reviewing cost-benefit analyses of disaster risk reduction
(Shreve and Kelman, 2014)
Link
Suggest that disaster relief saves money but at what ratio is unclear and is highly dependent on situation location and kind of disaster.
BCR estimates tend to be less than 10 with a few going into the 10s and a very few much higher (largest was 1800)
BCRs may underestimate by putting a high discount rate
Natural disasters challenge paper
(Copenhagen Consensus, 2015)
There are growing economic costs from natural disasters in recent years. This is especially true in developing countries where there may be limited insurance, higher risks and looser building codes.
Looks at retrofitting schools to be earthquake resistant in seismically active countries, suggests this has a BCR close to 1:1
Looks at constructing a one-metre high wall to protect homes or elevating houses by one metre to reduce flooding. Suggests this has a BCR of 60:1.
1:1
60:1
Link
““We estimate that for each dollar spent on disaster preparedness, an average of four dollars is saved on disaster response and recovery” says Alberto Monguzzi, Disaster Management Coordinator in the IFRC Europe Zone Office.”
Pandemic preparedness estimates
Other estimates. We found one example of estimates of the value of preparing better for future pandemics.
Not the last pandemic: … (Craven et al., 2021)
Link
Suggests a cost over 10 years of $285bn-$430bn would partially mitigate a damage $16,000bn every 50 years.
This roughly implies a BCR of <9:1
(16000/50) / (((285+430)/2)/10) = 8.95
We also found three examples of estimates of the value stockpiling for future pandemics.
The Cost Efectiveness of Stockpiling Drugs, Vaccines and Other Health Resources for Pandemic …
(Plans‑Rubió, 2020)
Link
Looked at estimates for the cost effectiveness of stockpiling drugs for a pandemic. Example estimates:
“US$8550 per LYS in very high severity pandemics and US$13,447 per LYS in moderate severity pandemics”
“₤3800 per QALY and ₤28,000 per QALY for the 1918 and 1957⁄69 pandemic scenarios”
Very roughly if we place a value of $50-100k per year of life this suggests a BCR of roughly 5:1 − 10:1.
Cost-Benefit of Stockpiling Drugs …
(Balicer et al, 2005)
Link
Suggest various options for stockpiling in Israel for an influenza pandemic have a cost benefit ratios of 0.37, 0.38, 2.49, 2.44, 3.68
“investments in antiviral agents can be expected to yield a substantial economic return of >$3.68 per $1 invested, while saving many lives”
“expanding the stockpile of AV drugs to encompass the whole UK population (≈60 million) might even be acceptable (≈£6,500 per QALY gained over a no intervention strategy for the 1918 scenario under base-case assumptions).”
Very roughly if we place a value of $50-100k per year of life this suggests a BCR of roughly 10:1.
()
Link
“Procuring an adequate PPE stockpile in advance at non-pandemic prices would cost only 17% of the projected amount needed to procure it at current pandemic-inflated prices”
Historical data and estimates suggest the value of increasing preparedness is decent but not very high, with estimated benefit cost ratios (BCR) often around or just below 1:10.
How this changes with scale of the disaster
There is some reason to think that disaster preparedness is more cost effective when targeted at worse disasters. Theoretically this makes sense as disasters are heavy-tailed and most of the impact of preventing and mitigating disasters will be in preventing and mitigating the very worst disasters. This is also supported by models estimating the effect of pandemic preparedness, such as those discussed in this talk. (Doohan and Hauck, 202?)
Pandemics affect more people than natural disasters so we could expect a higher than average BCR. This is more relevant if we pick preparedness interventions that scale with the size of the disaster (an example of an intervention that does not have this effect might be stockpiling, for which the impact is capped by the size of the stockpile, not by the size of the disaster).
However overall I did not find much solid evidence to suggest that the BCR ratio is higher for higher scale disasters.
Thanks for this! All extremely helpful info.
This is good to know. Our BCR of 1.6 is based on very conservative assumptions. We were basically seeing how conservative we could go while still getting a BCR of over 1. I think Carl and I agree that, on more reasonable estimates, the BCR of the suite is over 5 and maybe even over 10 (certainly I think that’s the case for some of the interventions within the suite). If, as you say, many people in government are looking for interventions with BCRs significantly higher than 1, then I think we should place more emphasis on our less conservative estimates going forward.
Thanks very much for this! I might try to get some of these references into the final paper.
This is really good to know as well.