Hi Everyone—partly inspired by attending the recent EA Global London conference a couple of weeks ago, I’ve written a CGD Blog with some thoughts on EA’s approach to prioritisation and methods in health economics (specifically Health Technology Assessment). This is a link post and as CGD staff I have to post on our platform, but since the key target audience is EAs, I’d be delighted to hear thoughts from this community. I’ll be sure to monitor the comments section and perhaps the discussion will feed into future work.
The differences between EA and health econ I highlight include:
1. Approaches to generalising cost-effectiveness evidence
2. Going beyond cost-effectiveness in determining value
3. Deliberative appraisal
4. Institutionalisation of a participatory process
Please click through for the full blog.
Thanks for this thoughtful post, Tom. You’ve definitely raised some thoughts that have been on our mind recently, such as how GiveWell could systematically incorporate more external input into our grant-making process. We’ve taken some steps towards this, such as with a beneficiary survey in 2019, and seeking out more external experts, but we’d like to do more – it was really great to read through your recommendations.
We did want to clarify, however, that GiveWell’s approach to modeling cost-effectiveness and making grant recommendations is heavily context-specific. You’re right that we start with intervention-level analysis in order to get a rough sense of the cost-effectiveness of any given program. But, our next step is to modify our models with many charity- and context-specific data and only fund grant opportunities that are above our 10x bar. For example, when we’re considering making grants to Malaria Consortium’s SMC program, we assess funding opportunities at the country level, taking into consideration the differences in malaria prevalence, demographics (age distribution), mortality rates, program costs, and the spending we might expect from other actors in each setting. The result is that the same program may clear our cost-effectiveness bar in some locations and not in others. For a recent example, see this page about a grant we recommended in January for Malaria Consortium; we decided to extend funding for its SMC program in Nigeria, Burkina Faso, and Togo (where estimated cost-effectiveness was near or above our bar), but provide only exit funding for Chad (which was below our bar).
If there’s reason to expect variations within countries, we also build out our model at the subnational level. For example, in 2022 we updated our model of Malaria Consortium’s SMC program in Nigeria with state-level malaria prevalence and mortality data. While we don’t capture every variance one could expect, we’re trying to adjust for the major variances that could exist in different contexts (and which therefore could affect our bottom-line grantmaking decisions).
We realize that this nuance isn’t captured in our external-facing marketing, and we’re working on updates to our website to address this. We really appreciate your engagement with our work; we aspire to account for context-specific variables in our grantmaking and appreciate the push to consider this further and to make this aspect of our work clearer.
Greetings GiveWell Colleagues,
Let me first re-emphasise that this blog is very much in a “yes and…” spirit. As I say in the blog, I believe EA is a positive influence on philanthropy and global development and has more potential to continue to shake things up for the better.
Thank you for this detailed response and the clarification that some of your analysis and grant making is indeed context-specific; great to see, particularly as someone who cut their teeth on CEA of malaria interventions. My impression is still that context-differentiated analysis perhaps the exception rather than the rule within the EA space—i.e. including organisations beyond GiveWell and topics beyond global health. It doesn’t yet come out strongly in the back-and-forth in forums like this one nor in presentations at EA Global conferences and it’s usually not emphasised in recommendations of EA donors—perhaps including GiveWell if we look at the evidence in the Top Charities page (as you note). Solving this challenge is not straightforward of course and remains an issue for other donors in the development space.
I don’t mean to suggest there are off-the-shelf lessons from HTA—on this or the other differences I highlight—that could be adopted directly by EA organisations. Equally, the field of HTA has been developing approaches for cost-effectiveness-based decision-making for several decades and may be fertile ground to explore for the development of EA prioritisation.
Happy to keep in touch as useful.
Thanks for this, I found it interesting!
I think HTA offers a concrete path to better emulate moral uncertainty and pluralism, though I worry it could soften and blunt the insight EA contributes through its focus on cost-effectiveness. Do you think this is a real trade-off? Is pluralism better emulated at the field-level, such that it is served by EA continuing its narrow focus?
I liked the other recent CGD blog on PEPFAR, a case where cost-effectiveness analysis was misleading. Seemingly the failures were strategic—not considering how budgetary constraints could be loosened nor how prices could be affected by major efforts. Do you think HTA would have fared better?
Hi Ben
First off—I see HTA as a kind of process wrapper for cost-effectiveness analysis. So in a sense It’s CEA+, rather than an alternative.
I’m not sure I followed you points about moral uncertainty and emulation of pluralism. Would you try again to say what you mean?
On Justin’s blog, CGD welcomes—even encourages—a diversity of views within its staff. I don’t agree with everything in this blog. For example HTA (which as I say, encompasses CEA) is precisely the tool used by many countries in price negotiation. I think there will be a follow up CGD blog with the opposing view, but I would say that CEA/HTA is now adopted by a large and increasing number of countries to inform health policy. If we’re saying it’s good enough domestic policy but not aid policy, we need to be really clear why.
Thanks,
Thanks! To clarify about moral uncertainty/pluralism—a caricatured EA approach might use CEA to evaluate how well different interventions maximise units of a particular benefit (e.g. DALYs, lives saved, ROI), and then follow those results to fulfil a consequentialist ethic. HTA seems like a method that could additionally weight equity, fairness, or benefit to the worst off, such that it would be more inclusive of prioritarian or justice-based views. HTA’s greater emphasis on participation by different stakeholders also seems more amenable to considering a plurality of worldviews, rather than reliance on those of a small set of donors and analysts.
So all that sounds quite good to me, but I’m aware that there may be a trade-off where EA loses some its weirdness, such that high-impact but niche interventions and cause areas would be neglected if EA became more like the rest of the aid landscape. I’m not sure what to think here.
Ok gotcha this time. Similar to some of the response to Jason below I would say that the assessment of value and the assessment of gaps etc are separate steps. We shouldn’t think about adopting a value framework because it allows us to find a practical niche. I suppose it does beg the question; whose value framework should we use? I don’t have that answer, but I do think that it links to the 4th difference in my blog; about institutionalisation and participation.
Btw, you imply that the rest of the aid landscape uses this HTA-like approach to prioritisation.. Sadly this is certainly not the case! I worked in the UK Department for International Development for several years, in a team that was tasked with providing evidence to the rest of the organisation. We did our best but it was a long way from perfect. Indeed I do see a significant opportunity for EA movement and organisations to influence other donors to take a more systematic, evidence-informed approach. For me this absolutely does mean using cost-effectiveness—but it means using it not in a blunt way but as part of an appropriate process and so that—wherever possible—we work with, not around, local systems and institutions.
Thanks for sharing this, Tom!
Some thoughts for future development—I recognize, of course, that you were working within the confines of a blog post:
On difference 1: Would it be more accurate to describe this as a continuum? E.g., EA organizations tend to invest significantly fewer resources into translating and adapting cost-effectiveness work in Country A before applying it in Country B than organizations applying HTA. Although giving no thought to emergent properties like differences in disease burden would be really unwise, I am not aware of any evidence that major EA funders are missing the super-low-hanging fruit there. (Of course, I could be underinformed, in which case some concrete examples of serious misses would help me update on their methodology!)
The idea of the suggested reframing is to paint investing in cost-effectiveness translation as a tradeoff—one can do more extensive work in this area (HTA) or relatively less (EA), and both approaches have their benefits and downsides. It intuitively seems that best approach would depend on the resources that would be going into the intervention in a new location, the costs of additional translation efforts, and the likelihood that the results of those efforts would either shape the intervention-as-deployed or change funding decisions. Indeed, my hunch is that—in general—your suggestions grow stronger as the amount of money EA is putting into global health increases, and the odds of EA funders being able to influence other funders through their actions increases.
On difference 2: I think the crux here is about the effect of other donors and the likelihood of successfully coordinating with them. If one views the decisions of other donors as essentially fixed, then it likely makes sense to allocate one’s own funds in a way that seeks to gives cost-effectiveness (or other atttributes one thought undervalued by the broader funding ecosystem) the appropriate amount of emphasis in the global health ecosystem as a whole—rather than weighing a basket of attributes in the way the donor would if they controlled how all donors spent their monies on global health.
For longer pieces (not blog posts), it would be worth considering that “someone should do X,” “EA as a whole should do X,” and “Specific Organization C should do X” are different claims. I think the kind of thinking in your blog post has a lot to offer EA organizations, but at later stages of development would need to be actionable by specific people and organizations in light of their particular constraints to achieve impact. In particular, I see GiveWell as acting more as a pure advisor to donors, so it faces the constraint of keeping those donors (which it has attracted with a goal of finding “charities that save or improve lives the most per dollar”) happy. Much more so than Open Phil and Founders Pledge, it is also built around the Unix-like philosophy of (mostly) doing one thing and doing it very well and may experience adverse outcomes from even well-intentioned mission creep by funding more “meta” work like your recommendation 4.
So I am not sure how actionable recommendations 2, 3, and 4 (maybe even 1) are for an organization in the style of GiveWell. On the other hand, I think they are quite actionable for OpenPhil. Given that GiveWell alone is a significant part of the ecosystem, a suggestion for longer pieces would be to offer more action items that clearly fit within its operating philosophy and constraints.
Thanks Jason, appreciate your thoughts.
Taking your points in order
Yes I essentially agree—though the other option available is to decide to work more through local institutions to prioritise and fund health services. Again yes this may be better done at scale. Though with the rate of growth in EA funding, I wouldn’t rule out a role for this kind of EA engagement. Or perhaps there is a role for EA as a kind of catalytic funder, investing in frameworks/platforms for more effective aid policy from states and major multilaterals.
I think you mean that if there are gaps for highly cost-effective services that aren’t being delivered by other donors (or indeed the national health system, we should remember that even in the very poorest countries, national institutions are still likely to be the main healthcare funder and provider, not the donors) then it makes sense for EA to fill those gaps. I think gap-filling and prioritisation are separate steps. With the additional criteria we’re still trying to assess value and therefore in a sense a certain kind of more broadly conceived assessment of cost-effectiveness. For example, if intervention A has a better cost per DALY than intervention B but equity considerations mean the HTA committee strongly prefers intervention B, then B is rightly considered a higher priority. We could then look at the practical issues of gaps, organisations that can deliver etc.
On specificity of recommendations, yes that’s fair. As you say this is just a blog so perhaps for future work. I also considered working out a clearer taxonomy of what is being prioritised (causes/programmes/interventions/charities etc). But this one was on the long side for a blog as it is