Thanks for sharing your critique of our recent grants with Open Philanthropy for technical support units (TSUs). We really appreciate this thoughtful pushback! We’ve recommended (and are considering) a number of grants to help respond to the current situation with cuts in US foreign health assistance. So, getting critiques like yours is helpful since it encourages us to pause and consider whether we’re making the right tradeoffs in these grants. While we share some of your perspectives on the uncertainties of this work, we’re still excited about our decision in this case.
While this grant’s impact is particularly uncertain, we see this as a difference in degree, not kind, compared to other grants we recommend. Most of our funding still goes to Top Charities—proven programs backed by strong evidence and our cost-effectiveness analysis. But we also recommend opportunities through the All Grants Fund. The goal of this fund is to find and fund what we believe are the highest-impact uses of marginal dollars, even when those opportunities are riskier or harder to model. This grant fits squarely in that approach. We’ve funded technical assistance from the All Grants Fund before, alongside grants that are uncertain for other reasons. For example, sometimes we’re trying to generate new evidence, while at other times we’re recommending high expected value bets even when we know we’re unlikely to get a definitive answer on their impact.
We agree that the evidence base for TSUs is thin. In general, we think it’s challenging to evaluate technical assistance programs because
particular programs might not have very convincing controls or comparators and
technical assistance packages can vary widely so aggregating results from many programs might not be that informative, even if they’re targeting the same outcome
So even if the review that Nick cites had found good evidence for past TA programs, we still might not feel sure that it would generalize to the TSUs we recommended funding.
But as discussed above, we don’t consider high uncertainty to be a dealbreaker in grants funded from the All Grants Fund. We still think it can be worth funding TA (see, e.g., our maternal syphilisgrants) and we’re very interested in building up our ability to learn about programs like this over time. (We’re working on a project looking back on a subset of technical assistance grants we’ve funded, but don’t have a publication date for that yet.)
While we don’t have detailed theories of change, we still think it’s plausible that TSUs could be impactful. We are excited about this grant because we think it could help governments to make difficult prioritization and program adaptation decisions in countries affected by US government funding freezes and cuts. We expect that the details of how this could look will vary by country and so we don’t feel confident that any particular mechanism will cash out in impact. But for example, we think TSUs could help governments to:
Adapt programs away from more expensive models implemented with USG support to more cost-efficient models (e.g., by integrating disease-specific verticals or moving from international to local implementing organizations)
Target programs (e.g., to certain regions or populations) based on cost-effectiveness analyses
Crowd in philanthropic or other funding by identifying funding gaps
While we think the above examples are plausible, we agree that the theory of change for these programs is not tightly specified. However, we spoke with senior Ministry of Health officials in each country about this grant, and overall governments voiced support and demand for the proposed TSUs and were eager to have CHAI and PATH’s support on this work. We also think both organizations are well-placed to support this work, as both have supported on malaria-specific TSUs in the past, have teams with specific focus on health systems and health financing, and have established relationships with the governments they’re supporting.
Budget—Thank you for sharing your estimates—this is helpful for us as we continue to update how we review budgets. We’ll share a high level budget breakdown for this grant with our public grant write-ups (which are coming—see below!). One quick clarification (which wasn’t clear in the podcast) is that costs for Nigeria reflect support in seven states as well as national support.
Outside of that, we think that the higher budget reflects both higher salaries and a higher non-salary budget share (to account for travel, coordinating stakeholder engagement, and support from global technical teams). Our understanding is that salaries are set based on globally-benchmarked salary ranges and localized equity adjustments to account for organizational equitable pay standards and differential cost of living across different geographies. A portion of the compensation costs is also due to benefits (such as health insurance) that may be standard to each location.
Learning—We agree that we should try to learn about the impact of these grants and also agree with commenters and Nick’s revision that an RCT isn’t an appropriate strategy. We’ve asked CHAI and PATH to track and report out on the following.
Process indicators (staff hiring, government engagement, TSU recommendations made and implemented);
Case studies of how government prioritization decisions are made;
Promising funding opportunities identified through the TSU work;
Contexts where cost-effectiveness considerations are or aren’t helpful for government decision-making.
We’ll attempt to triangulate these reports through speaking with other stakeholders, though we expect we’ll still have substantial uncertainty about impact given the lack of counterfactuals.
No CEA and grant write up. These are coming! We typically have a lag between making grants and publishing our write-ups, but wanted to share about this grant sooner because we’ve received a lot of interest in our response to the funding cuts. We expect to publish pages for CHAI and PATH (including a rough BOTEC) by the end of June.
Urgency. We see the urgency here as being specifically related to governments’ needs to adapt to frozen or cut US health assistance: we heard when investigating this grant that governments were already beginning this planning process and that lighter touch versions of the support offered by TSUs were already being provided on nights and weekends by CHAI staff in certain countries. We also think this kind of grant is inherently uncertain and it didn’t seem likely that we’d reduce that uncertainty by spending additional time investigating. So, with apparent demand for support at the time and since we didn’t think waiting would lead to a better decision, we chose to recommend funding relatively quickly.
Thanks for the input and helpful reply, definitely clarifies some things and I really hope this work goes well and better than I expect :)
One comment on the above that I don’t think the specified learning approach will yield the most important learnings
Especially in these kind of cases where counterfactuals are difficult to assess, I don’t think self-reports provide very useful data. Incentives are all wrong, and there’s a lot of confirmation bias, implementers see the results they want to see—often unintentionally. BINGOs, with the exception of GiveDirectly basically report nearly all (at least publicly) of their programs as successful. As far as I’ve seen, CHAI and PATH are no exception. Just by nature of doing tricky work, many projects will be complete failures through no fault of anyone, but it won’t be reported usually not even internally.
As a datapoint to suppor/refute my claim, it might be useful to look at the reports for the last 20 times CHAI and PATH have supported TSUs. Due to the nature of TSU work, at very least a handful of these (I would guess 25%-75%) would not achieve anything useful, which is obviously fine and to be expected in this high variance situation. Despite this I would be fairly confident if you looked at the reports of last 20 times CHAI has implemented TSUs, all of them will report overall success (with caveats of course, that’s standard thing to do), and none of them will say that they were overall a failure, or didn’t achieve anything. If you did find that 5 out of 20 reports did say their work was likely useless then I would be happy to be corrected/proved wrong here!
Because of this I think the higher risk/more uncertain the work is, the more it is important to rely on external assessment. Where metrics are clearer and crisper then I think its more reasonable to rely somewhat on implementing orgs own assessments - there’s less room for subjectivity.
Although its very difficult to measure success well, I would suggest this as an alternative to your current suggestion of “Ask PATH and CHAI to report and track...” which I don’t think will yield much useful information.
1. Have the learnings/assessment done not by implementers, perhaps even by GiveWell staff yourselves who are really motivated to find the truth 2. After discussion with CHAI and PATH (in advance) as to what you all think success metrics might look like, then specify some clear numerical/qualitative success metrics in advance as well. Then don’t show your final metrics to CHAI/PATH so this doesn’t become a gameable target can’t be Goodharted. 3. Validate with the PATH/CHAI failure as being likely in many TSU cases—state that half of these TSUs (or whatever) are likely to achieve not much and that’s OK. You might then get more self reflective/honest feedback from implementers—at least you’d be in with a chance.
To clarify I’m not trying to say that these orgs are bad/dishonest in any way, I’m just riding of my experience in programs of this nature. Incentives incentives incentives. With a combination of BINGO implementers and unclear/nebulous success measures, both personal emotional/motivational incentives and organisational incentives for things to “work” mean that with remarkable consistency successes are exaggerated and failures suppressed.
GW’s email newsletter just alerted me to their grant writeup on this, in case you still want to look into it / aren’t subbed to them :) especially the main reservations section
Thanks yes. I don’t think there’s much new in the write-up that hasn’t been discussed—
The BOTEC is clear and well written as usual. I think (predictably) that they are very optimistic about what TSUs might be able do. They think that TSUs will...
1) Have a 70% chance of increasing cost effectiveness of 20 −40 million dollars of health spending in each country (over 100million in total) by around 20%. This through shifting allocation. I struggle to fathom how that could be possible here in Uganda, with my albeit limited knowledge of how the ministry of health works here. If they could move 20 million of funding to be 30% more cost-effective in even one country I’d be impressed.
2) Bring in 40 million dollars of counterfactually new health funding in the countries that they work in (between 2 and 20 million in each country). This would be a truly brilliant lobbying effort that again I’m very skeptical about. Bringing in new counterfactual funding is very difficult.
if a technical support unit can achieve anything like that in only 18 months I would be blown away. The tricky thing thing is that it will be very difficult to tell whether this happened or not—again it would be great to have an external org (or GiveWell themselves) assessing the likely counterfactual here.
On the other hand (to their credit as it hurts their BOTEC) they are very conservative about the chance of the grant being funded by some other funder. They put this chance at 50%, but I think its far far lower. Very few orgs in the world fund this kind of work to the tune of 4.5 million dollars, I would have put it at more like 20%-30%
Actually now that you mention the shifting health budget allocation I’m also skeptical, although I’m mostly thinking of CEAP’s experience finding the development budget ~fixed and the remaining sliver fought over by hundreds of NGOs, I take you to be saying it’s the same story for health budget.
I agree re: impact attribution determination and they don’t seem to be planning to do that in their plans for follow-up section.
Thanks for sharing your critique of our recent grants with Open Philanthropy for technical support units (TSUs). We really appreciate this thoughtful pushback! We’ve recommended (and are considering) a number of grants to help respond to the current situation with cuts in US foreign health assistance. So, getting critiques like yours is helpful since it encourages us to pause and consider whether we’re making the right tradeoffs in these grants. While we share some of your perspectives on the uncertainties of this work, we’re still excited about our decision in this case.
While this grant’s impact is particularly uncertain, we see this as a difference in degree, not kind, compared to other grants we recommend. Most of our funding still goes to Top Charities—proven programs backed by strong evidence and our cost-effectiveness analysis. But we also recommend opportunities through the All Grants Fund. The goal of this fund is to find and fund what we believe are the highest-impact uses of marginal dollars, even when those opportunities are riskier or harder to model. This grant fits squarely in that approach. We’ve funded technical assistance from the All Grants Fund before, alongside grants that are uncertain for other reasons. For example, sometimes we’re trying to generate new evidence, while at other times we’re recommending high expected value bets even when we know we’re unlikely to get a definitive answer on their impact.
We agree that the evidence base for TSUs is thin. In general, we think it’s challenging to evaluate technical assistance programs because
particular programs might not have very convincing controls or comparators and
technical assistance packages can vary widely so aggregating results from many programs might not be that informative, even if they’re targeting the same outcome
So even if the review that Nick cites had found good evidence for past TA programs, we still might not feel sure that it would generalize to the TSUs we recommended funding.
But as discussed above, we don’t consider high uncertainty to be a dealbreaker in grants funded from the All Grants Fund. We still think it can be worth funding TA (see, e.g., our maternal syphilis grants) and we’re very interested in building up our ability to learn about programs like this over time. (We’re working on a project looking back on a subset of technical assistance grants we’ve funded, but don’t have a publication date for that yet.)
While we don’t have detailed theories of change, we still think it’s plausible that TSUs could be impactful. We are excited about this grant because we think it could help governments to make difficult prioritization and program adaptation decisions in countries affected by US government funding freezes and cuts. We expect that the details of how this could look will vary by country and so we don’t feel confident that any particular mechanism will cash out in impact. But for example, we think TSUs could help governments to:
Adapt programs away from more expensive models implemented with USG support to more cost-efficient models (e.g., by integrating disease-specific verticals or moving from international to local implementing organizations)
Target programs (e.g., to certain regions or populations) based on cost-effectiveness analyses
Crowd in philanthropic or other funding by identifying funding gaps
While we think the above examples are plausible, we agree that the theory of change for these programs is not tightly specified. However, we spoke with senior Ministry of Health officials in each country about this grant, and overall governments voiced support and demand for the proposed TSUs and were eager to have CHAI and PATH’s support on this work. We also think both organizations are well-placed to support this work, as both have supported on malaria-specific TSUs in the past, have teams with specific focus on health systems and health financing, and have established relationships with the governments they’re supporting.
Budget—Thank you for sharing your estimates—this is helpful for us as we continue to update how we review budgets. We’ll share a high level budget breakdown for this grant with our public grant write-ups (which are coming—see below!). One quick clarification (which wasn’t clear in the podcast) is that costs for Nigeria reflect support in seven states as well as national support.
Outside of that, we think that the higher budget reflects both higher salaries and a higher non-salary budget share (to account for travel, coordinating stakeholder engagement, and support from global technical teams). Our understanding is that salaries are set based on globally-benchmarked salary ranges and localized equity adjustments to account for organizational equitable pay standards and differential cost of living across different geographies. A portion of the compensation costs is also due to benefits (such as health insurance) that may be standard to each location.
Learning—We agree that we should try to learn about the impact of these grants and also agree with commenters and Nick’s revision that an RCT isn’t an appropriate strategy. We’ve asked CHAI and PATH to track and report out on the following.
Process indicators (staff hiring, government engagement, TSU recommendations made and implemented);
Case studies of how government prioritization decisions are made;
Promising funding opportunities identified through the TSU work;
Contexts where cost-effectiveness considerations are or aren’t helpful for government decision-making.
We’ll attempt to triangulate these reports through speaking with other stakeholders, though we expect we’ll still have substantial uncertainty about impact given the lack of counterfactuals.
No CEA and grant write up. These are coming! We typically have a lag between making grants and publishing our write-ups, but wanted to share about this grant sooner because we’ve received a lot of interest in our response to the funding cuts. We expect to publish pages for CHAI and PATH (including a rough BOTEC) by the end of June.
Urgency. We see the urgency here as being specifically related to governments’ needs to adapt to frozen or cut US health assistance: we heard when investigating this grant that governments were already beginning this planning process and that lighter touch versions of the support offered by TSUs were already being provided on nights and weekends by CHAI staff in certain countries. We also think this kind of grant is inherently uncertain and it didn’t seem likely that we’d reduce that uncertainty by spending additional time investigating. So, with apparent demand for support at the time and since we didn’t think waiting would lead to a better decision, we chose to recommend funding relatively quickly.
Thanks for the input and helpful reply, definitely clarifies some things and I really hope this work goes well and better than I expect :)
One comment on the above that I don’t think the specified learning approach will yield the most important learnings
Especially in these kind of cases where counterfactuals are difficult to assess, I don’t think self-reports provide very useful data. Incentives are all wrong, and there’s a lot of confirmation bias, implementers see the results they want to see—often unintentionally. BINGOs, with the exception of GiveDirectly basically report nearly all (at least publicly) of their programs as successful. As far as I’ve seen, CHAI and PATH are no exception. Just by nature of doing tricky work, many projects will be complete failures through no fault of anyone, but it won’t be reported usually not even internally.
As a datapoint to suppor/refute my claim, it might be useful to look at the reports for the last 20 times CHAI and PATH have supported TSUs. Due to the nature of TSU work, at very least a handful of these (I would guess 25%-75%) would not achieve anything useful, which is obviously fine and to be expected in this high variance situation. Despite this I would be fairly confident if you looked at the reports of last 20 times CHAI has implemented TSUs, all of them will report overall success (with caveats of course, that’s standard thing to do), and none of them will say that they were overall a failure, or didn’t achieve anything. If you did find that 5 out of 20 reports did say their work was likely useless then I would be happy to be corrected/proved wrong here!
Because of this I think the higher risk/more uncertain the work is, the more it is important to rely on external assessment. Where metrics are clearer and crisper then I think its more reasonable to rely somewhat on implementing orgs own assessments - there’s less room for subjectivity.
Although its very difficult to measure success well, I would suggest this as an alternative to your current suggestion of “Ask PATH and CHAI to report and track...” which I don’t think will yield much useful information.
1. Have the learnings/assessment done not by implementers, perhaps even by GiveWell staff yourselves who are really motivated to find the truth
2. After discussion with CHAI and PATH (in advance) as to what you all think success metrics might look like, then specify some clear numerical/qualitative success metrics in advance as well. Then don’t show your final metrics to CHAI/PATH so this doesn’t become a gameable target can’t be Goodharted.
3. Validate with the PATH/CHAI failure as being likely in many TSU cases—state that half of these TSUs (or whatever) are likely to achieve not much and that’s OK. You might then get more self reflective/honest feedback from implementers—at least you’d be in with a chance.
To clarify I’m not trying to say that these orgs are bad/dishonest in any way, I’m just riding of my experience in programs of this nature. Incentives incentives incentives. With a combination of BINGO implementers and unclear/nebulous success measures, both personal emotional/motivational incentives and organisational incentives for things to “work” mean that with remarkable consistency successes are exaggerated and failures suppressed.
GW’s email newsletter just alerted me to their grant writeup on this, in case you still want to look into it / aren’t subbed to them :) especially the main reservations section
Thanks yes. I don’t think there’s much new in the write-up that hasn’t been discussed—
The BOTEC is clear and well written as usual. I think (predictably) that they are very optimistic about what TSUs might be able do. They think that TSUs will...
1) Have a 70% chance of increasing cost effectiveness of 20 −40 million dollars of health spending in each country (over 100million in total) by around 20%. This through shifting allocation. I struggle to fathom how that could be possible here in Uganda, with my albeit limited knowledge of how the ministry of health works here. If they could move 20 million of funding to be 30% more cost-effective in even one country I’d be impressed.
2) Bring in 40 million dollars of counterfactually new health funding in the countries that they work in (between 2 and 20 million in each country). This would be a truly brilliant lobbying effort that again I’m very skeptical about. Bringing in new counterfactual funding is very difficult.
if a technical support unit can achieve anything like that in only 18 months I would be blown away. The tricky thing thing is that it will be very difficult to tell whether this happened or not—again it would be great to have an external org (or GiveWell themselves) assessing the likely counterfactual here.
On the other hand (to their credit as it hurts their BOTEC) they are very conservative about the chance of the grant being funded by some other funder. They put this chance at 50%, but I think its far far lower. Very few orgs in the world fund this kind of work to the tune of 4.5 million dollars, I would have put it at more like 20%-30%
Actually now that you mention the shifting health budget allocation I’m also skeptical, although I’m mostly thinking of CEAP’s experience finding the development budget ~fixed and the remaining sliver fought over by hundreds of NGOs, I take you to be saying it’s the same story for health budget.
I agree re: impact attribution determination and they don’t seem to be planning to do that in their plans for follow-up section.