I mostly agree. Though having worked in research, policy and now policy analysis, it’s also the case that sectors can get stuck in established practices. Big-picture /blue sky thinking is of course also important. But I agree that the value of work to get the basics/details right is under recognised. So thumbs up for your post, except the framing of “boring”; one person’s boring is another’s wildly fascinating—who are we to judge.
TomDrake
Karma: 205
The $1 Trillion Paradox: Why Reforming Research Publishing Should Be a Global Priority
Thanks for sharing—it’s an interesting interview. My first reaction is that interdepartmental bureaucracy is quite a different beast to an evidence-to-policy process. I agree that splitting development policy/programmes across multiple government depts causes lots of problems and is generally to be avoided if possible (I’m thinking about the UK system but imagine the challenges are similar in the US and elsewhere).
Of course you do need some bureaucracy to facilitate evidence-to-policy too, but on the whole I think it’s absolutely worth the time. For public policy we should aim to make a small number of decisions really well. The idea a small efficient group who just know what to do and crack on is appealing; it’s a more heroic narrative than a careful weighing of the evidence. Though I can’t imagine the users of this forum need persuading of the importance of using evidence to do better than our intuitions and overcome our biases.
Incidentally, I feel this kind of we-know-what-to-do-let’s-crack-on instinct is more acceptable in development policy than domestic, and in my view development policy would benefit from being much more considered. We cause a lot of chaos and harm to systems in LMICs in the way we offer development assistance, even through programmes that are supporting valuable services. I think all of the major GHI’s do great work, but all could benefit from substantial reforms. Though again, this is somewhat separate from the point about interdepartmental bureaucracy.
Thanks for your comment and I agree. Modelling (even rigorous modelling) is just that, a model. It’s a simplification of a more complex reality. We should not mistake the map for the territory, but equally, not using a map would be foolish.
Did Economists Really Get Africa’s AIDS Epidemic “Analytically Wrong”? (A Reply)
Greetings GiveWell Colleagues,
Let me first re-emphasise that this blog is very much in a “yes and…” spirit. As I say in the blog, I believe EA is a positive influence on philanthropy and global development and has more potential to continue to shake things up for the better.
Thank you for this detailed response and the clarification that some of your analysis and grant making is indeed context-specific; great to see, particularly as someone who cut their teeth on CEA of malaria interventions. My impression is still that context-differentiated analysis perhaps the exception rather than the rule within the EA space—i.e. including organisations beyond GiveWell and topics beyond global health. It doesn’t yet come out strongly in the back-and-forth in forums like this one nor in presentations at EA Global conferences and it’s usually not emphasised in recommendations of EA donors—perhaps including GiveWell if we look at the evidence in the Top Charities page (as you note). Solving this challenge is not straightforward of course and remains an issue for other donors in the development space.
I don’t mean to suggest there are off-the-shelf lessons from HTA—on this or the other differences I highlight—that could be adopted directly by EA organisations. Equally, the field of HTA has been developing approaches for cost-effectiveness-based decision-making for several decades and may be fertile ground to explore for the development of EA prioritisation.
Happy to keep in touch as useful.
Ok gotcha this time. Similar to some of the response to Jason below I would say that the assessment of value and the assessment of gaps etc are separate steps. We shouldn’t think about adopting a value framework because it allows us to find a practical niche. I suppose it does beg the question; whose value framework should we use? I don’t have that answer, but I do think that it links to the 4th difference in my blog; about institutionalisation and participation.
Btw, you imply that the rest of the aid landscape uses this HTA-like approach to prioritisation.. Sadly this is certainly not the case! I worked in the UK Department for International Development for several years, in a team that was tasked with providing evidence to the rest of the organisation. We did our best but it was a long way from perfect. Indeed I do see a significant opportunity for EA movement and organisations to influence other donors to take a more systematic, evidence-informed approach. For me this absolutely does mean using cost-effectiveness—but it means using it not in a blunt way but as part of an appropriate process and so that—wherever possible—we work with, not around, local systems and institutions.
Thanks Jason, appreciate your thoughts.
Taking your points in order
Yes I essentially agree—though the other option available is to decide to work more through local institutions to prioritise and fund health services. Again yes this may be better done at scale. Though with the rate of growth in EA funding, I wouldn’t rule out a role for this kind of EA engagement. Or perhaps there is a role for EA as a kind of catalytic funder, investing in frameworks/platforms for more effective aid policy from states and major multilaterals.
I think you mean that if there are gaps for highly cost-effective services that aren’t being delivered by other donors (or indeed the national health system, we should remember that even in the very poorest countries, national institutions are still likely to be the main healthcare funder and provider, not the donors) then it makes sense for EA to fill those gaps. I think gap-filling and prioritisation are separate steps. With the additional criteria we’re still trying to assess value and therefore in a sense a certain kind of more broadly conceived assessment of cost-effectiveness. For example, if intervention A has a better cost per DALY than intervention B but equity considerations mean the HTA committee strongly prefers intervention B, then B is rightly considered a higher priority. We could then look at the practical issues of gaps, organisations that can deliver etc.
On specificity of recommendations, yes that’s fair. As you say this is just a blog so perhaps for future work. I also considered working out a clearer taxonomy of what is being prioritised (causes/programmes/interventions/charities etc). But this one was on the long side for a blog as it is
Hi Ben
First off—I see HTA as a kind of process wrapper for cost-effectiveness analysis. So in a sense It’s CEA+, rather than an alternative.
I’m not sure I followed you points about moral uncertainty and emulation of pluralism. Would you try again to say what you mean?
On Justin’s blog, CGD welcomes—even encourages—a diversity of views within its staff. I don’t agree with everything in this blog. For example HTA (which as I say, encompasses CEA) is precisely the tool used by many countries in price negotiation. I think there will be a follow up CGD blog with the opposing view, but I would say that CEA/HTA is now adopted by a large and increasing number of countries to inform health policy. If we’re saying it’s good enough domestic policy but not aid policy, we need to be really clear why.
Thanks,
Beyond Cost-Effectiveness: Insights for Effective Altruism from Health Economics
[CROSS POST] The Malaria Vaccination Era—How Endemic Countries and Donors Can Make the Most of R21
Thanks Ryan!
I see you like the capability approach—I’m a fan too
Thanks for this considered response Jason.
I would summarise your overall message as “we’re not ready for this yet” and I would partly agree. But EA community and orgs could be part of developing the solution and influencing other donors. We shouldn’t consider EA orgs as separate from bilateral, multilateral or other philanthropic donors. At this point, EA is not an insignificant voice in the global health ecosystem. Also the shift doesn’t need to happen for the sector as a whole, it could be country by country and some countries probably are ready for such an approach. We’re discussing it with a couple.
To some of your more specific points:
I disagree that fungibility is the biggest challenge. Excepting cases of corruption, this simply means the country is effectively choosing the spend on other priorities. Indeed some folks in CGD have argued against the marginal aid approach precisely because they favour the fungible quality of current donations. This isn’t my view but it speaks to an opinion I do share, which is that, in many cases, countries should be given more control over healthcare priority setting.
Of course a common concern is whether country institutions are good faith actors and have the necessary capabilities to adopt a marginal aid approach. This will differ between countries and can be considered on a case by case basis—and will change over time (inc with donor support to strengthen priority setting capabilities, ensure comprehensive delivery on the core package etc). No country administrations are perfect, including in many donor countries, but we need a better framework for how countries transition from receiving support to deliver essential health services to doing so themselves. This is an active process in many countries today (inc where EA supported orgs operate) and has been exacerbated by the recent squeeze on global development financing.
Lastly, you touch on some practical challenges with funding marginal services. I agree that donors with country presence will be best placed participate in the coordination mechanisms necessary for a marginal aid approach, but EA individuals or orgs could i) choose to support intermediaries. There are none at present but it’s not hard to imagine existing organisations adopting the approach. For example, the Global Financing Facility (GFF) has done much to work with countries and donors on better coordinated and prioritised investments (with some success and some challenges). ii) they could also choose to support catalytic investments to strengthen country capabilities rather than earmarked aid for specific services.
I would challenge EAs to rethink the rationale for doing global level analysis to set health priorities in low income countries. In absence of local prioritisation it can be better than nothing, and initiatives like the Disease Control Priorities Project have been helpful. But such approaches are both technically and philosophically limited, and inferior to building local capabilities. Before too long, and with the growing decolonisation/localisation movements, I expect this top-down approach will seem increasingly out-dated.
Thanks again Matt. Yes, negative externalities could be a helpful way to think about at least some of those six challenges.
To your question, in the short term I wouldn’t advise individual small donors to change their behaviour. In absence of a coordinated effort to improve donor harmonisation, I support giving based on cost-effectiveness principles and my intuition is not that this kind of giving is a net harm. Our pitch is perhaps to global health institutions—including EA orgs like GiveWell and Open Phil—that we could do better. We don’t yet have the institutions that would allow individuals to support the kind approach we outline (essentially TA + harmonised support to marginal services), but perhaps that’s something we need… Of course that’s a trickier sell but I’m sure some smart strategic comms folk could help.
Hi Jason, you’re right that our proposal is that donors would shift away from funding these kinds of programmes directly but that instead they would not only finance services at the margin but also provide technical support to prioritise and then deliver on those local priorities. I guess it’s the health policy version of the “teach a man to fish” principle. Sure, giving bed nets or antiretrovirals does some good, but helping to build an effective health system is better. And I’m not at all convinced that governments wouldn’t fund these high value services instead. I would say it’s more that administrators take whatever help is being offered and then try to run a health system around it, but managing these donations takes work and makes it harder to strengthen the national system.
Thanks Mattias, glad to know you’d already seen the paper! I would say that absolutely I think governments in many countries can and are working towards improved evidence-informed priority setting. Sure some still have some way to go (including many high-income countries btw) but many countries with less resources are developing strong prioritisation capabilities. Thailand is the classic example though several other countries have engaged in ambitious whole-health-benefits-package assessments including Ethiopia, Malawi and Pakistan. We wouldn’t expect a shift to a marginal aid approach overnight, or for all countries at the same time, but we need to imagine what the transition between the current situation and a world where countries do have effective flourishing health systems. In my view that means strengthening local priority setting and reducing fragmentation (etc) in how we provide financial support for service delivery.
Thanks for the comments on the potential value of gathering additional info, we are indeed hoping to do some work along these lines, including interviewing country officials as well.
Jason, agreed that there are some scale advantages for certain multilaterals. However it could be possible to retain these with a shift to a marginal aid approach. For example, even if countries were to move to support vaccination through domestic financing, Gavi might continue to provide support on pooled procurement.
Thanks Matt, I’ll check out these links.
You say:
“I suspect your proposed model is perhaps more suited to megadonors and government aid, rather than small donors (except perhaps where those donors are influenced by the same recommender, e.g. GiveWell), because I’m not sure how individual donors would be able to know which services would otherwise not be covered by domestic funds? ”
To me this underlines the point that individual donors aren’t best placed to set priorities for what a countries or population needs. If the donor doesn’t have confidence in giving options with more structure, I could see a rationale for cash (such as Give Directly) but the problems that earmarked aid create for countries are increasingly well understood and this is no less of true for small donations than large. We summarise six key issues in the paper: volatility, fragmentation, fungibility, weak prioritisation, exit strategy and local autonomy.
To be clear I think cash is a low bar for effective giving and there are many better options, but I would suggest that it is crucial that we strengthen, rather than undermine, national institutions in the process.
In other words, I would love to see different narratives around giving with a focus on strengthening effective cause prioritisation in countries—rather than this prioritisation being done in London, SF, Geneva etc—and organisations that seek donations from individuals to provide technical assistance and top-up financing.
I always enjoy some good malaria cost-effectiveness chat, however I think we need to move beyond prioritisation of the roll-out of health interventions at the global/donor level. While there is a place for generalised cost-effectiveness analysis, on the whole it’s better to think of cost-effectiveness not as a generalisable property of a technology but as a product of the technology and the context. There is no doubt malaria vaccination (R21 or RTS,S) will be highly cost-effective in some populations and less so in others. The same goes for bed nets and SMC due to mosquito type, seasonality and many other factors. Plus, as others have said, the original post too often treats malaria interventions as alternatives when, in many cases, more important malaria planing will be about where to focus interventions rather than which technology to chose.
There are two broad ways to approach linking cost-effectiveness with context: i) large-scale multi-country, multi-intervention, cost-effectiveness analyses at national or sub-national level. There is a place for this and many organisations do great work in this space, including Givewell. ii) Countries undertake their own analyses and set priorities compared to other healthcare investment options, not so much other malaria interventions both other healthcare. For highly endemic regions, it could well be that nets and vaccination are both very cost-effective and that an MoH should prioritise this over, say, certain oncology treatments. If the MoH still accepts donated vaccine or earmarked vaccine financing (and why wouldn’t it), then the donors aren’t really buying more people vaccinated, they’re buying more people treated for cancer—which might still be a positive contribution, but not what they think they are buying and not the best option.
From where I’m sitting, it seems that global health is rightly moving away from campaigns to back single technologies or disease areas and looking at how to provide less fragmented and more foundational support. That’s not as easy a story to tell to philanthropic donors but there’s increasing recognition of the harm that hundreds of separate earmarked offers of support for this technology or that disease area does. Even in most low-income countries, the government is the main funder of health services and those looking to help should identify appropriate auxiliary programmes to support, not displace or distract.
(Note all the above refers to *roll-out* of vaccines. For R&D a best-guess overall cost-effectiveness estimate makes sense and so does intervention-focused advocacy/engagement.)
Tl;dr it is usually better to consider cost-effectiveness of technologies as context-dependent, rather than an property of the technology, and we should consider systems support over financing technologies that would/should have been bought with domestic financing anyway. We need to empower local planners to consider the full range of health investment options, not try to decide for them.
--
A few bonus comments on some questions in the post:
“Is there a reason why including a mixture of cost-effective and somewhat less cost-effective interventions together could be a high-impact outcome?”
- Yes, for example if the combination of interventions brings local R below 1. But it might also be less than expected too. Effectiveness for infectious diseases is non-linear so it’s better to simulate transmission effects. This has been done by several groups.
“Will the cost-effectiveness of bednets/SMC decline over time (e.g. because of diminishing marginal returns of some sort)? If so, might vaccine cost-effectiveness overtake it? “
- Yes, insecticide or drug resistance can result in declining effectiveness of bed nets or prophylaxis.
And on the age-factor of bed net vs vx cost-effectiveness—I would certainly expect this to be taken into account in all decent modelling so would be wary of over correcting in botecs.