Nick, on behalf of our team at Rethink Priorities, I’d like to thank you for engaging with our work and sharing your insights. We really appreciate it! You raise some excellent points, and I’d like to respond to them in turn.
HSS is not neglected:
I’d agree that looking at total dollar spending on HSS doesn’t by itself give a clear sense of neglectedness. While HSS has certainly been central to the discourse, I’d argue it has often been more rhetoric than reality. Much of the ~$100B labeled as “HSS” may in practice have gone toward narrow or superficial fixes, at least that’s something I’ve seen noted repeatedly in the literature. In some yet-unpublished follow-up work, we’ve also tried to relate those spending figures to the share of the health burden attributable to weak health systems, though doing so is quite tricky and comes with considerable uncertainty.
RP’s best Interventions are barely HSS and focus on Health Workers:
That’s a good point, and I’d clarify that we don’t necessarily think interventions like IMCI, CHW expansion, or community scorecards are the best or most impactful HSS interventions overall. Rather, they are the ones where decent evidence on cost-effectiveness was most readily available, which made them rise to the top in our initial analysis. There may well be higher-level or more systemic interventions that could deliver larger or more sustained health gains, but they often lack the kind of robust evaluation data we need to assess them confidently. In that sense, the ranking reflects the current evidence base more than an absolute judgment about where the biggest opportunities lie.
I share your impression that some nice-sounding or fuzzy interventions can be hard to see as meaningfully improving things. That said, governance is one area I strongly believe in (see also what I wrote on governance further below). While its impact can be difficult to capture from the bottom up when looking only at individual impact evaluations, it becomes more visible from the top down when examining which factors plausibly contributed most to meaningful health systems strengthening in a country over time. This is also something I found quite convincing in the work of Exemplars in Global Health and the Good Health at Low Cost study.
Cost-Effectiveness of HSS interventions can and should be measured:
Regarding the “narrow outcomes,” I’d be thrilled if more HSS evaluations reported actual health outcomes (or direct effects on household finances), but as you’ve noted, that’s not always feasible or practical. In the meantime, we can focus on intermediate outputs, but I see two main concerns. First, these outputs are often so far upstream that linking them to health impact requires a lot of guesswork and assumptions. Second, they often capture only a small slice of the system, even though the reform affected much more, making it hard to understand the broader effects. For example, an evaluation might report that a procurement reform reduced drug prices but provide no data on whether availability or stockouts improved, even though that was a key reform goal. And when prices change, the downstream and general equilibrium effects can be complex and hard to anticipate. I’m often surprised by how little is measured around big, system-wide reforms, given how much they can shift across the system.
I’m very sympathetic to the idea that RCTs could be used more often for HSS interventions, and I think they are technically feasible in many cases. However, I do believe RCTs in the context of HSS tend to be more complicated and expensive, especially when multiple interventions are rolled out simultaneously, timelines are long, and there are potential spillover or general equilibrium effects, typically more so than in vertical programs. I also suspect it’s often a public goods problem: the benefits of generating generalizable evidence mostly accrue to others, while the costs and burdens of implementing the reform in a way that allows an RCT can largely fall on the implementing government. At least, that was my impression when I was involved in evaluating different reforms in Ethiopia, where the government seemed more keen on rolling things out quickly and evenly rather than on randomizing across districts and making some people wait longer for the reform.
Context matters—Country >>> Intervention:
I strongly agree with you that the country context often matters more than the specific intervention. I think many of the classical success stories, like Bangladesh, Rwanda, or Ethiopia, are ultimately stories of strong governance and sustained political commitment. That commitment manifested in a wide range of ways, from institutional reforms to close collaboration with NGOs. In those contexts, my impression is often that any specific intervention mattered less than the broader enabling environment.
I also agree with your point about not overinterpreting success in high-functioning countries. As you noted, interventions like microfinance didn’t translate well from Bangladesh to other settings, and I think the same caution is warranted for health systems. That said, I still see a lot of value in studying what worked and where, especially to understand the combination of factors that enabled success in specific contexts.
IMCI and other guidelines have enormous potential:
What you wrote makes a lot of sense to me and aligns with my general intuition about the importance of guideline-based healthcare. It’s not an area I’ve looked into closely myself, so I really appreciate the insight.
Community Health workers are often not cost-effective:
I agree with you that community health workers (CHWs) are unlikely to be among the most cost-effective interventions. We noted in the report that “CHW interventions are likely to require large amounts of ongoing financial support, and do not offer a clear path toward highly cost-effective or leveraged interventions”, which was meant to reflect this view.
That said, if we look beyond the immediate, short-run effects of CHWs (such as patients treated), their longer-term contributions may be quite substantial. In Bangladesh, for example, CHWs played a transformative role not only in delivering services but also in shifting norms around family planning, increasing female education, and building trust in the formal healthcare system, all of which helped increase demand for institutional care over time. Those broader effects may not show up in short-term cost-effectiveness metrics, but I believe they are likely substantial (though I can’t point to concrete numbers). Exemplars in Global Health illustrate this nicely in their case studies, e.g. here and here.
LMH is far more well known and better funded than Living Goods:
We only had time to look at LMH at a very superficial level, so you might be right about that.
Transitioning to government ownership is a risky bet—that might sometimes be worth it:
I really appreciate this insight from someone experienced on the ground. To be clear, we have not spent any time coming up with a probability of success for this pathway, and I agree that assuming a smooth transition to government ownership would be naive. At best, I think of this as a high-risk, potentially high-reward opportunity, and one that might only make sense if there’s a particularly strong implementing partner (like LMH), unusually strong government buy-in, and a clear reason to believe philanthropic support would be catalytic rather than duplicative. But it’s certainly not a “default” safe bet.
Supply Chains interventions have largely failed:
We actually did some (unpublished) follow-up research that focused more heavily on supply chain interventions, and it left us more cautious as well. The evidence for large-scale structural reforms seems quite mixed, and we struggled to find clear, well-documented success stories. The USAID-Chemonics project you mentioned illustrates this challenge well, though it’s hard to disentangle whether the issue stemmed primarily from the inherent complexity of supply chain reform or from shortcomings in the program’s design and execution.
Regarding the need for last-mile delivery interventions, you probably have a better sense of the situation on the ground than we do. That said, some recent publications make me think that the problem isn’t quite solved yet. For example, a recent last-mile delivery intervention for COVID-19 vaccines in Sierra Leone increased vaccination rates by 26 percentage points within 48-72 hours (Meriggi et al., 2024). Moreover, the Gates Foundation still seems pretty active in supporting last-mile delivery interventions (e.g., here and here), which suggests that they still see meaningful gaps to address. We haven’t reviewed RidersForHealth in detail, so I can’t speak to their specific model.
Thanks so so much for this fantastic response, really appreciate it and this helps clarify a few things :). I don’t have anything significant to add, but a few comments.
- On the measurement issue, yeah I 100% agree that downstream effects are the ones that need be measured. If its a procurement reform, then I would agree the primary outcme would be to reduce stockouts. This might be one of the easier RCTs to do (depending on the system). I would ave thought in many cases randomising by district and following up over 1-2 years should answer the question pretty clearly? I agree some reforms it might be hard to find a clear outcome measure, but on procurement/supply chains it mighe be doable in many cases.
- Yes I agree with community health workers in terms of producing great flow-on effects, although like you said wit those exemplar countries like Bangladesh its difficult to know how much to attribute those changes to CHWs. Also the E”xemplars” case study pointed to a bunch of health interventions, not just CHWs which probably contributed to the amazing improvements there, including Access to health centers, maternal health vouchers, procurement changes etc. I really like the “Exemplars in Global Health” studies.
- Small critique I think saying “In Bangladesh, for example, CHWs played a transformative role not only in delivering services but also in shifting norms around family planning, increasing female education, and building trust in the formal healthcare system, all of which helped increase demand for institutional care over time.” could be true but we can’t say it with much confidence at all. Especially saying they played a “transformative role” seems like overreach. How can we seperate the effect of the CHWs from all the other amazing health system inputs Bangladesh was imputting?
- You’re right that sometimes countries want to move fast and don’t care much about research. That’s great if they are funding things themselves. I think if its being externally funded, its is on the funder to make sure research happens in cases where it is important/possible. If we’re funding it, we’re doing an RCT to test it—take it or leave it. Its an under-appreciated point that governments everywhere and especially in low income countries are often not interested in evidence at all. I’ve been blown away by the extent of it here in Uganda.
- On the vaccine point, I’m not sure $33 per extra person vaccinated (from the Meriggi study) would ever worth it for covid (maybr for some other disease). As a side note I think it was a huge waste of resources to try (and fail) vaccinate most Africans for covid. I think it should have just been older people and those with comorbidities vaccinated,( wrote a bit about Covid in Uganda here)
-I think with Health Systems Strengthening, especially when it comes to areas like Governance and supply chains, the question of whether there are “gaps” isn’t always the most important. Yes there are gaps—huge gaps that need addressing. The bigger question is whether we have a consistently effective method to address those gaps.
The gaps are enormous throughout Health Systems, and I think even more than with other interventions cost-effective tractability is the big question.
On a completely unrelated note, this graph from the Bangladesh Exemplars study is one of the coolest I’ve ever seen :D :D :D.
Nick, on behalf of our team at Rethink Priorities, I’d like to thank you for engaging with our work and sharing your insights. We really appreciate it! You raise some excellent points, and I’d like to respond to them in turn.
HSS is not neglected:
I’d agree that looking at total dollar spending on HSS doesn’t by itself give a clear sense of neglectedness. While HSS has certainly been central to the discourse, I’d argue it has often been more rhetoric than reality. Much of the ~$100B labeled as “HSS” may in practice have gone toward narrow or superficial fixes, at least that’s something I’ve seen noted repeatedly in the literature. In some yet-unpublished follow-up work, we’ve also tried to relate those spending figures to the share of the health burden attributable to weak health systems, though doing so is quite tricky and comes with considerable uncertainty.
RP’s best Interventions are barely HSS and focus on Health Workers:
That’s a good point, and I’d clarify that we don’t necessarily think interventions like IMCI, CHW expansion, or community scorecards are the best or most impactful HSS interventions overall. Rather, they are the ones where decent evidence on cost-effectiveness was most readily available, which made them rise to the top in our initial analysis. There may well be higher-level or more systemic interventions that could deliver larger or more sustained health gains, but they often lack the kind of robust evaluation data we need to assess them confidently. In that sense, the ranking reflects the current evidence base more than an absolute judgment about where the biggest opportunities lie.
I share your impression that some nice-sounding or fuzzy interventions can be hard to see as meaningfully improving things. That said, governance is one area I strongly believe in (see also what I wrote on governance further below). While its impact can be difficult to capture from the bottom up when looking only at individual impact evaluations, it becomes more visible from the top down when examining which factors plausibly contributed most to meaningful health systems strengthening in a country over time. This is also something I found quite convincing in the work of Exemplars in Global Health and the Good Health at Low Cost study.
Cost-Effectiveness of HSS interventions can and should be measured:
Regarding the “narrow outcomes,” I’d be thrilled if more HSS evaluations reported actual health outcomes (or direct effects on household finances), but as you’ve noted, that’s not always feasible or practical. In the meantime, we can focus on intermediate outputs, but I see two main concerns. First, these outputs are often so far upstream that linking them to health impact requires a lot of guesswork and assumptions. Second, they often capture only a small slice of the system, even though the reform affected much more, making it hard to understand the broader effects. For example, an evaluation might report that a procurement reform reduced drug prices but provide no data on whether availability or stockouts improved, even though that was a key reform goal. And when prices change, the downstream and general equilibrium effects can be complex and hard to anticipate. I’m often surprised by how little is measured around big, system-wide reforms, given how much they can shift across the system.
I’m very sympathetic to the idea that RCTs could be used more often for HSS interventions, and I think they are technically feasible in many cases. However, I do believe RCTs in the context of HSS tend to be more complicated and expensive, especially when multiple interventions are rolled out simultaneously, timelines are long, and there are potential spillover or general equilibrium effects, typically more so than in vertical programs. I also suspect it’s often a public goods problem: the benefits of generating generalizable evidence mostly accrue to others, while the costs and burdens of implementing the reform in a way that allows an RCT can largely fall on the implementing government. At least, that was my impression when I was involved in evaluating different reforms in Ethiopia, where the government seemed more keen on rolling things out quickly and evenly rather than on randomizing across districts and making some people wait longer for the reform.
Context matters—Country >>> Intervention:
I strongly agree with you that the country context often matters more than the specific intervention. I think many of the classical success stories, like Bangladesh, Rwanda, or Ethiopia, are ultimately stories of strong governance and sustained political commitment. That commitment manifested in a wide range of ways, from institutional reforms to close collaboration with NGOs. In those contexts, my impression is often that any specific intervention mattered less than the broader enabling environment.
I also agree with your point about not overinterpreting success in high-functioning countries. As you noted, interventions like microfinance didn’t translate well from Bangladesh to other settings, and I think the same caution is warranted for health systems. That said, I still see a lot of value in studying what worked and where, especially to understand the combination of factors that enabled success in specific contexts.
IMCI and other guidelines have enormous potential:
What you wrote makes a lot of sense to me and aligns with my general intuition about the importance of guideline-based healthcare. It’s not an area I’ve looked into closely myself, so I really appreciate the insight.
Community Health workers are often not cost-effective:
I agree with you that community health workers (CHWs) are unlikely to be among the most cost-effective interventions. We noted in the report that “CHW interventions are likely to require large amounts of ongoing financial support, and do not offer a clear path toward highly cost-effective or leveraged interventions”, which was meant to reflect this view.
That said, if we look beyond the immediate, short-run effects of CHWs (such as patients treated), their longer-term contributions may be quite substantial. In Bangladesh, for example, CHWs played a transformative role not only in delivering services but also in shifting norms around family planning, increasing female education, and building trust in the formal healthcare system, all of which helped increase demand for institutional care over time. Those broader effects may not show up in short-term cost-effectiveness metrics, but I believe they are likely substantial (though I can’t point to concrete numbers). Exemplars in Global Health illustrate this nicely in their case studies, e.g. here and here.
LMH is far more well known and better funded than Living Goods:
We only had time to look at LMH at a very superficial level, so you might be right about that.
Transitioning to government ownership is a risky bet—that might sometimes be worth it:
I really appreciate this insight from someone experienced on the ground. To be clear, we have not spent any time coming up with a probability of success for this pathway, and I agree that assuming a smooth transition to government ownership would be naive. At best, I think of this as a high-risk, potentially high-reward opportunity, and one that might only make sense if there’s a particularly strong implementing partner (like LMH), unusually strong government buy-in, and a clear reason to believe philanthropic support would be catalytic rather than duplicative. But it’s certainly not a “default” safe bet.
Supply Chains interventions have largely failed:
We actually did some (unpublished) follow-up research that focused more heavily on supply chain interventions, and it left us more cautious as well. The evidence for large-scale structural reforms seems quite mixed, and we struggled to find clear, well-documented success stories. The USAID-Chemonics project you mentioned illustrates this challenge well, though it’s hard to disentangle whether the issue stemmed primarily from the inherent complexity of supply chain reform or from shortcomings in the program’s design and execution.
Regarding the need for last-mile delivery interventions, you probably have a better sense of the situation on the ground than we do. That said, some recent publications make me think that the problem isn’t quite solved yet. For example, a recent last-mile delivery intervention for COVID-19 vaccines in Sierra Leone increased vaccination rates by 26 percentage points within 48-72 hours (Meriggi et al., 2024). Moreover, the Gates Foundation still seems pretty active in supporting last-mile delivery interventions (e.g., here and here), which suggests that they still see meaningful gaps to address. We haven’t reviewed RidersForHealth in detail, so I can’t speak to their specific model.
Again, thanks a lot for engaging with our report!
Thanks so so much for this fantastic response, really appreciate it and this helps clarify a few things :). I don’t have anything significant to add, but a few comments.
- On the measurement issue, yeah I 100% agree that downstream effects are the ones that need be measured. If its a procurement reform, then I would agree the primary outcme would be to reduce stockouts. This might be one of the easier RCTs to do (depending on the system). I would ave thought in many cases randomising by district and following up over 1-2 years should answer the question pretty clearly? I agree some reforms it might be hard to find a clear outcome measure, but on procurement/supply chains it mighe be doable in many cases.
- Yes I agree with community health workers in terms of producing great flow-on effects, although like you said wit those exemplar countries like Bangladesh its difficult to know how much to attribute those changes to CHWs. Also the E”xemplars” case study pointed to a bunch of health interventions, not just CHWs which probably contributed to the amazing improvements there, including Access to health centers, maternal health vouchers, procurement changes etc. I really like the “Exemplars in Global Health” studies.
- Small critique I think saying “In Bangladesh, for example, CHWs played a transformative role not only in delivering services but also in shifting norms around family planning, increasing female education, and building trust in the formal healthcare system, all of which helped increase demand for institutional care over time.” could be true but we can’t say it with much confidence at all. Especially saying they played a “transformative role” seems like overreach. How can we seperate the effect of the CHWs from all the other amazing health system inputs Bangladesh was imputting?
- You’re right that sometimes countries want to move fast and don’t care much about research. That’s great if they are funding things themselves. I think if its being externally funded, its is on the funder to make sure research happens in cases where it is important/possible. If we’re funding it, we’re doing an RCT to test it—take it or leave it. Its an under-appreciated point that governments everywhere and especially in low income countries are often not interested in evidence at all. I’ve been blown away by the extent of it here in Uganda.
- On the vaccine point, I’m not sure $33 per extra person vaccinated (from the Meriggi study) would ever worth it for covid (maybr for some other disease). As a side note I think it was a huge waste of resources to try (and fail) vaccinate most Africans for covid. I think it should have just been older people and those with comorbidities vaccinated,( wrote a bit about Covid in Uganda here)
-I think with Health Systems Strengthening, especially when it comes to areas like Governance and supply chains, the question of whether there are “gaps” isn’t always the most important. Yes there are gaps—huge gaps that need addressing. The bigger question is whether we have a consistently effective method to address those gaps.
The gaps are enormous throughout Health Systems, and I think even more than with other interventions cost-effective tractability is the big question.
On a completely unrelated note, this graph from the Bangladesh Exemplars study is one of the coolest I’ve ever seen :D :D :D.