Thanks for this thoughtful post, Tom. You’ve definitely raised some thoughts that have been on our mind recently, such as how GiveWell could systematically incorporate more external input into our grant-making process. We’ve taken some steps towards this, such as with a beneficiary survey in 2019, and seeking out more external experts, but we’d like to do more – it was really great to read through your recommendations.
We did want to clarify, however, that GiveWell’s approach to modeling cost-effectiveness and making grant recommendations is heavily context-specific. You’re right that we start with intervention-level analysis in order to get a rough sense of the cost-effectiveness of any given program. But, our next step is to modify our models with many charity- and context-specific data and only fund grant opportunities that are above our 10x bar. For example, when we’re considering making grants to Malaria Consortium’s SMC program, we assess funding opportunities at the country level, taking into consideration the differences in malaria prevalence, demographics (age distribution), mortality rates, program costs, and the spending we might expect from other actors in each setting. The result is that the same program may clear our cost-effectiveness bar in some locations and not in others. For a recent example, see this page about a grant we recommended in January for Malaria Consortium; we decided to extend funding for its SMC program in Nigeria, Burkina Faso, and Togo (where estimated cost-effectiveness was near or above our bar), but provide only exit funding for Chad (which was below our bar).
We realize that this nuance isn’t captured in our external-facing marketing, and we’re working on updates to our website to address this. We really appreciate your engagement with our work; we aspire to account for context-specific variables in our grantmaking and appreciate the push to consider this further and to make this aspect of our work clearer.
Let me first re-emphasise that this blog is very much in a “yes and…” spirit. As I say in the blog, I believe EA is a positive influence on philanthropy and global development and has more potential to continue to shake things up for the better.
Thank you for this detailed response and the clarification that some of your analysis and grant making is indeed context-specific; great to see, particularly as someone who cut their teeth on CEA of malaria interventions. My impression is still that context-differentiated analysis perhaps the exception rather than the rule within the EA space—i.e. including organisations beyond GiveWell and topics beyond global health. It doesn’t yet come out strongly in the back-and-forth in forums like this one nor in presentations at EA Global conferences and it’s usually not emphasised in recommendations of EA donors—perhaps including GiveWell if we look at the evidence in the Top Charities page (as you note). Solving this challenge is not straightforward of course and remains an issue for other donors in the development space.
I don’t mean to suggest there are off-the-shelf lessons from HTA—on this or the other differences I highlight—that could be adopted directly by EA organisations. Equally, the field of HTA has been developing approaches for cost-effectiveness-based decision-making for several decades and may be fertile ground to explore for the development of EA prioritisation.
Thanks for this thoughtful post, Tom. You’ve definitely raised some thoughts that have been on our mind recently, such as how GiveWell could systematically incorporate more external input into our grant-making process. We’ve taken some steps towards this, such as with a beneficiary survey in 2019, and seeking out more external experts, but we’d like to do more – it was really great to read through your recommendations.
We did want to clarify, however, that GiveWell’s approach to modeling cost-effectiveness and making grant recommendations is heavily context-specific. You’re right that we start with intervention-level analysis in order to get a rough sense of the cost-effectiveness of any given program. But, our next step is to modify our models with many charity- and context-specific data and only fund grant opportunities that are above our 10x bar. For example, when we’re considering making grants to Malaria Consortium’s SMC program, we assess funding opportunities at the country level, taking into consideration the differences in malaria prevalence, demographics (age distribution), mortality rates, program costs, and the spending we might expect from other actors in each setting. The result is that the same program may clear our cost-effectiveness bar in some locations and not in others. For a recent example, see this page about a grant we recommended in January for Malaria Consortium; we decided to extend funding for its SMC program in Nigeria, Burkina Faso, and Togo (where estimated cost-effectiveness was near or above our bar), but provide only exit funding for Chad (which was below our bar).
If there’s reason to expect variations within countries, we also build out our model at the subnational level. For example, in 2022 we updated our model of Malaria Consortium’s SMC program in Nigeria with state-level malaria prevalence and mortality data. While we don’t capture every variance one could expect, we’re trying to adjust for the major variances that could exist in different contexts (and which therefore could affect our bottom-line grantmaking decisions).
We realize that this nuance isn’t captured in our external-facing marketing, and we’re working on updates to our website to address this. We really appreciate your engagement with our work; we aspire to account for context-specific variables in our grantmaking and appreciate the push to consider this further and to make this aspect of our work clearer.
Greetings GiveWell Colleagues,
Let me first re-emphasise that this blog is very much in a “yes and…” spirit. As I say in the blog, I believe EA is a positive influence on philanthropy and global development and has more potential to continue to shake things up for the better.
Thank you for this detailed response and the clarification that some of your analysis and grant making is indeed context-specific; great to see, particularly as someone who cut their teeth on CEA of malaria interventions. My impression is still that context-differentiated analysis perhaps the exception rather than the rule within the EA space—i.e. including organisations beyond GiveWell and topics beyond global health. It doesn’t yet come out strongly in the back-and-forth in forums like this one nor in presentations at EA Global conferences and it’s usually not emphasised in recommendations of EA donors—perhaps including GiveWell if we look at the evidence in the Top Charities page (as you note). Solving this challenge is not straightforward of course and remains an issue for other donors in the development space.
I don’t mean to suggest there are off-the-shelf lessons from HTA—on this or the other differences I highlight—that could be adopted directly by EA organisations. Equally, the field of HTA has been developing approaches for cost-effectiveness-based decision-making for several decades and may be fertile ground to explore for the development of EA prioritisation.
Happy to keep in touch as useful.