Hey Oscar, I am indeed reading this! (albeit a bit late)
First, I really appreciate you looking into this and writing this. I’m excited to see people explore more cause areas and give rough estimates of how promising they are.
There are quite a few details in the cost-effectiveness analysis that I think are inaccurate, and I’ve mentioned a few at the bottom of the comment in case it is of interest. However, I still think this is a good and valuable shallow investigation. If there weren’t discrepancies between the conclusions from 50 hours of investigation and 6 years of work by different people that would be quite surprising (and Google would have wasted quite a lot of money on external analyses, RCTs, etc.).
Details aside, I broadly agree with what you wrote in this comment. There’s a big difference between statements about the promisingness of a field as a whole, vs. claiming that there aren’t uniquely impactful opportunities in a field—and I think we have a uniquely impactful opportunity. I think flooding is an incredibly important problem and there are many organizations I would love to see investing more in it, but I would not make the case that this should be a top priority focus area for GiveWell / OpenPhil / CE.
It’s also worth noting explicitly that even if I had more meaningful disagreements with the conclusions of this investigation I’d still be very appreciative of it. These questions are complex, multiple perspectives are incredibly valuable, and even well-informed intelligent people can disagree.
Finally, just in case it is of interest and useful, I’ll note the largest drivers of where my estimates diverge from those in the report:
Total impact—the EMDAT is definitely a candidate for the most comprehensive source of recorded flood (and other natural disaster) deaths, but it’s not a great source for an estimate of total deaths. This is partially due to undercounting of deaths (and other estimates that use EMDAT alongside other sources find higher numbers[1]<sup><strong>,</strong></sup>[2]), but more significantly due to these types of counts only counting immediate deaths (due to reasons such as trauma) and not due to longer term health effects[3]. Tracking the relationship between floods and all-cause mortality (or other health indicators)[4] leads to vastly larger numbers. Both these methodologies are probably far from the true numbers, but in opposite directions. The World Bank report you linked to thinks warning systems can save 23,000 lives per year, and though this may be an overestimate of the number of lives that would be saved by that plan, I think it might indeed be a reasonable lower bound on the number of lives lost annually in total. My current best estimate of fatalities is about an order of magnitude higher than yours.
Comparison between top intervention to all harms vs. all investment—we both already commented on how unique opportunities might be more promising than the field as a whole. I want to make a related but broader point here, because I’m worried the methodology here will systematically bias us in favor of existing cause areas over new ones. At least in parts, your analysis asks how much funding it would take to address all flood harms globally. You then compare your estimate of the cost-effectiveness of that to GiveDirectly, one of the top interventions globally, which invests immensely in targeting[5] to maximize its cost-effectiveness. Due to diminishing returns, almost any domain you’d look into would be very cost-ineffective to try and completely solve the problem globally (for example, trying to solve all poverty via cash transfers would also be far less cost-effective than GiveDirectly’s current margin). Specifically in floods this can lead to a difference in orders of magnitude. While on average the cost-benefit ratio of early warning systems has been estimated as 1:9[6], but in highly-affected low income countries the cost-benefit ratio can rise to the hundreds[7]. Also, I won’t name names, but most $100M+ programs in this space that I’ve seen were never actually completed or used—so the average cost-effectiveness numbers in this space are very far from the effectiveness of a well-functioning organization doing good work. Concretely, I think that your statements might be more than an order of magnitude off if we’re considering investing in promising projects to mitigate flood harms (focusing on severely affected low and middle income countries, early warning, and organizations/solutions with a strong track record).
Note: my views are disproportionately influenced by work on early warning systems, which are only a part of the work you’ve aimed to review, and you noted towards monitoring, forecasting, and alerting, which are the areas I’ve been most involved in, while your report touched on other areas in flood management as well.
Finally, you might also be interested in a report from 2016 by CEA, which also includes a review of the cost-effectiveness of flood management. I think it misses different nuances but again provides another interesting perspective.
I have A LOT more things to say about the empirical statements, framework for evaluation, and assumptions that went into this—happy to chat if you’re interested.
Hi Sela, thanks for the long and thoughtful comment, and for your kind words. That is reassuring that you also do not feel this is a key area for GiveWell/OP to expand into.
Really interesting re EMDAT possibly being off by ~10x, I was aware that longer-term harms are a lot harder to measure but wasn’t expecting the effect to be that large.
Re my references to ending all flooding harms, that makes sense; I wasn’t trying to suggest that the average cost-effectiveness would be the same as marginal cost-effectiveness. Perhaps a better thing to say would be that in order to be competitive with top charities, marginal targeted interventions would need to be far better than the average of existing interventions.
Hmm yes I was a bit surprised at how expensive EWS were made out to be, particularly when I would have thought a lot of costs could be saved by rolling out the same model and infrastructure across different countries.
Thanks for the offer, I am not currently working on this and don’t expect to go back to it, so I don’t think there would be much value in talking further—I’ll let you know if I am coming back to this though. I hope you make great progress on your flood forecasting work!
Hey Oscar, I am indeed reading this! (albeit a bit late)
First, I really appreciate you looking into this and writing this. I’m excited to see people explore more cause areas and give rough estimates of how promising they are.
There are quite a few details in the cost-effectiveness analysis that I think are inaccurate, and I’ve mentioned a few at the bottom of the comment in case it is of interest. However, I still think this is a good and valuable shallow investigation. If there weren’t discrepancies between the conclusions from 50 hours of investigation and 6 years of work by different people that would be quite surprising (and Google would have wasted quite a lot of money on external analyses, RCTs, etc.).
Details aside, I broadly agree with what you wrote in this comment. There’s a big difference between statements about the promisingness of a field as a whole, vs. claiming that there aren’t uniquely impactful opportunities in a field—and I think we have a uniquely impactful opportunity. I think flooding is an incredibly important problem and there are many organizations I would love to see investing more in it, but I would not make the case that this should be a top priority focus area for GiveWell / OpenPhil / CE.
It’s also worth noting explicitly that even if I had more meaningful disagreements with the conclusions of this investigation I’d still be very appreciative of it. These questions are complex, multiple perspectives are incredibly valuable, and even well-informed intelligent people can disagree.
Finally, just in case it is of interest and useful, I’ll note the largest drivers of where my estimates diverge from those in the report:
Total impact—the EMDAT is definitely a candidate for the most comprehensive source of recorded flood (and other natural disaster) deaths, but it’s not a great source for an estimate of total deaths. This is partially due to undercounting of deaths (and other estimates that use EMDAT alongside other sources find higher numbers[1]<sup><strong>,</strong></sup>[2]), but more significantly due to these types of counts only counting immediate deaths (due to reasons such as trauma) and not due to longer term health effects[3]. Tracking the relationship between floods and all-cause mortality (or other health indicators)[4] leads to vastly larger numbers. Both these methodologies are probably far from the true numbers, but in opposite directions. The World Bank report you linked to thinks warning systems can save 23,000 lives per year, and though this may be an overestimate of the number of lives that would be saved by that plan, I think it might indeed be a reasonable lower bound on the number of lives lost annually in total. My current best estimate of fatalities is about an order of magnitude higher than yours.
Comparison between top intervention to all harms vs. all investment—we both already commented on how unique opportunities might be more promising than the field as a whole. I want to make a related but broader point here, because I’m worried the methodology here will systematically bias us in favor of existing cause areas over new ones. At least in parts, your analysis asks how much funding it would take to address all flood harms globally. You then compare your estimate of the cost-effectiveness of that to GiveDirectly, one of the top interventions globally, which invests immensely in targeting[5] to maximize its cost-effectiveness. Due to diminishing returns, almost any domain you’d look into would be very cost-ineffective to try and completely solve the problem globally (for example, trying to solve all poverty via cash transfers would also be far less cost-effective than GiveDirectly’s current margin). Specifically in floods this can lead to a difference in orders of magnitude. While on average the cost-benefit ratio of early warning systems has been estimated as 1:9[6], but in highly-affected low income countries the cost-benefit ratio can rise to the hundreds[7]. Also, I won’t name names, but most $100M+ programs in this space that I’ve seen were never actually completed or used—so the average cost-effectiveness numbers in this space are very far from the effectiveness of a well-functioning organization doing good work. Concretely, I think that your statements might be more than an order of magnitude off if we’re considering investing in promising projects to mitigate flood harms (focusing on severely affected low and middle income countries, early warning, and organizations/solutions with a strong track record).
Note: my views are disproportionately influenced by work on early warning systems, which are only a part of the work you’ve aimed to review, and you noted towards monitoring, forecasting, and alerting, which are the areas I’ve been most involved in, while your report touched on other areas in flood management as well.
Finally, you might also be interested in a report from 2016 by CEA, which also includes a review of the cost-effectiveness of flood management. I think it misses different nuances but again provides another interesting perspective.
I have A LOT more things to say about the empirical statements, framework for evaluation, and assumptions that went into this—happy to chat if you’re interested.
A digitized global flood inventory (1998–2008): compilation and preliminary results
The Human Impact of Floods: a Historical Review of Events 1980-2009 and Systematic Literature Review
Health Risks of Flood Disasters
Health Effects of Flooding in Rural Bangladesh
Study: AI targeting helped reach more of the poorest people in Togo
Global Commission on Adaptation’s Adapt Now report
Background Paper on Assessment of the Economics of Early Warning Systems for Disaster Risk Reduction
Hi Sela, thanks for the long and thoughtful comment, and for your kind words. That is reassuring that you also do not feel this is a key area for GiveWell/OP to expand into.
Really interesting re EMDAT possibly being off by ~10x, I was aware that longer-term harms are a lot harder to measure but wasn’t expecting the effect to be that large.
Re my references to ending all flooding harms, that makes sense; I wasn’t trying to suggest that the average cost-effectiveness would be the same as marginal cost-effectiveness. Perhaps a better thing to say would be that in order to be competitive with top charities, marginal targeted interventions would need to be far better than the average of existing interventions.
Hmm yes I was a bit surprised at how expensive EWS were made out to be, particularly when I would have thought a lot of costs could be saved by rolling out the same model and infrastructure across different countries.
Thanks for the offer, I am not currently working on this and don’t expect to go back to it, so I don’t think there would be much value in talking further—I’ll let you know if I am coming back to this though. I hope you make great progress on your flood forecasting work!