As a donor to global health and development causes, what I would be looking for are:
Independent evaluation
Randomized controlled trials, probably randomizing at the hospital level, with study hospitals far enough apart that patients aren’t able to choose a treatment hospital over a control one.
Success in other countries (though having something that only works in one country is still valuable, if there is an explanation)
Evidence that the intervention can scale (basically, that the positive results aren’t due to a small number of highly motivated people)
I haven’t spent any time picking apart this particular program but one thing I see missing here is any skepticism about the results. Usually when it comes to charitable interventions, the headline number overstates things by a lot, because when you dig into it, it turns out there are various confounding or offsetting considerations that lower one’s confidence. What are some reasons that the intervention might actually not accomplish anything at all (or even be counterproductive), that would be consistent with these results so far?
You might also ask GiveWell what it would take to get their attention.
Hi Ian, thank you for your thoughtful feedback. We fully agree that independent evaluations and randomized controlled trials (RCTs) are an important tools for understanding impact. However, we must also acknowledge that RCTs come with significant costs, long timelines, and often produce results that don’t fully translate into real-world settings. Our experience has shown that the complexity of low-resource healthcare environments requires adaptable, on-the-ground solutions that are continuously tested and refined in real-time.
That’s why, at GOAL 3, continuous evaluation and learning are embedded into our approach, aimed at Real World Evidence. We feel that waiting for a one-off trial to tell us whether IMPALA works is not appropriate. Instead, we are committed to assessing our effectiveness at every site in the real-world, following up over time, and ensuring year-on-year improvement. This approach allows us to make iterative and localized changes based on the needs of health workers and patients, ensuring we create a sustainable impact that lasts. Our goal is to see a measurable improvement not just immediately after implementation but as a consistent part of the health system’s development.
Of course we do want this to be evaluated independently, to also assess unanticipated negative and/or positive impact from the intervention. We know we are biased and we understand that we are not the best evaluators. That is why we leverage the Founders Pledge grant to ensure independent research institutes can do this evaluation across different settings. This will strenthen the evidence on effectiveness across different settings with a larger poule (7) hospitals, over time (>1year follow-up) while also assessing differences.
On a personal note, while I understand and endorse the rigorous processes that donors and evaluators require, I sometimes feel frustrated by the lack of urgency. The slowness of decision-making in global health is an enormous barrier to innovation. Many life-saving solutions are delayed because innovators and startups simply don’t have the time or resources to wait for trial results. During COVID-19, we saw how innovations were fast-tracked to address an urgent crisis. It feels unfair that the 5 million pediatric deaths every year are not considered urgent enough to prioritize similarly fast solutions. Basically the risk of unanticipated consequences should be balanced against the potential positive impact and use that to navigate decisions about
Ultimately I agree with the process and I should not complain after receiving a grant from Founders Pledge to support us in building the evidence base that is needed. Its just that I am frustrated because it feels like a continuous fight to move things forward or even survive, while deep in my heart I know we are right and that every delay will hinder us from preventing deaths that are unneeded.
On a personal note, while I understand and endorse the rigorous processes that donors and evaluators require, I sometimes feel frustrated by the lack of urgency.
I don’t mean to speak for anyone but myself, but I sense that there has been an increase in skepticism in recent decades, as interventions that seemed promising and which “we know work” turn out not to have any measurable effect, or even to have negative effects, when carefully studied. My prior, at least, regarding any new intervention that I hear about is that any effect is probably illusory or minimal.
It’s not that there is no urgency about the problems in the world — those are indeed urgent and demanding — but rather, our resources are so limited, and good interventions so hard to find, that we should probably dedicate only a small amount of our funding to more speculative projects, so that we can build evidence over time and not waste funding on projects that turn out not to work. As an example of what we want to avoid doing, look at what happened with microfinance.
Of course the situation would be different if we had already fully funded well-studied projects like bednets, deworming, Vitamin A, etc.; but sadly that’s not the world we actually inhabit.
I guess then it’s just my title: In God we trust, but all others must bring data. It just feels frustrating sometimes, also because people actually don’t say they don’t believe you. So if you don’t call me a liar, then why don’t you help or support me?
I would really like it if a more systematic way of identifying, testing and scaling successful innovations was available.
My post are definitely not intended to complain by the way. I am blessed with a great and supportive team and have great partners as well. It’s a genuine interest to understand, learn and improve.
PS. over the next days I will try to add a bit more background about the solution and our evidence, where we think it needs to be strenthened. I was a bit reluctant to put too much information in, because I thougt people might not read it.
As a donor to global health and development causes, what I would be looking for are:
Independent evaluation
Randomized controlled trials, probably randomizing at the hospital level, with study hospitals far enough apart that patients aren’t able to choose a treatment hospital over a control one.
Success in other countries (though having something that only works in one country is still valuable, if there is an explanation)
Evidence that the intervention can scale (basically, that the positive results aren’t due to a small number of highly motivated people)
I haven’t spent any time picking apart this particular program but one thing I see missing here is any skepticism about the results. Usually when it comes to charitable interventions, the headline number overstates things by a lot, because when you dig into it, it turns out there are various confounding or offsetting considerations that lower one’s confidence. What are some reasons that the intervention might actually not accomplish anything at all (or even be counterproductive), that would be consistent with these results so far?
You might also ask GiveWell what it would take to get their attention.
Hi Ian, thank you for your thoughtful feedback. We fully agree that independent evaluations and randomized controlled trials (RCTs) are an important tools for understanding impact. However, we must also acknowledge that RCTs come with significant costs, long timelines, and often produce results that don’t fully translate into real-world settings. Our experience has shown that the complexity of low-resource healthcare environments requires adaptable, on-the-ground solutions that are continuously tested and refined in real-time.
That’s why, at GOAL 3, continuous evaluation and learning are embedded into our approach, aimed at Real World Evidence. We feel that waiting for a one-off trial to tell us whether IMPALA works is not appropriate. Instead, we are committed to assessing our effectiveness at every site in the real-world, following up over time, and ensuring year-on-year improvement. This approach allows us to make iterative and localized changes based on the needs of health workers and patients, ensuring we create a sustainable impact that lasts. Our goal is to see a measurable improvement not just immediately after implementation but as a consistent part of the health system’s development.
Of course we do want this to be evaluated independently, to also assess unanticipated negative and/or positive impact from the intervention. We know we are biased and we understand that we are not the best evaluators. That is why we leverage the Founders Pledge grant to ensure independent research institutes can do this evaluation across different settings. This will strenthen the evidence on effectiveness across different settings with a larger poule (7) hospitals, over time (>1year follow-up) while also assessing differences.
On a personal note, while I understand and endorse the rigorous processes that donors and evaluators require, I sometimes feel frustrated by the lack of urgency. The slowness of decision-making in global health is an enormous barrier to innovation. Many life-saving solutions are delayed because innovators and startups simply don’t have the time or resources to wait for trial results. During COVID-19, we saw how innovations were fast-tracked to address an urgent crisis. It feels unfair that the 5 million pediatric deaths every year are not considered urgent enough to prioritize similarly fast solutions. Basically the risk of unanticipated consequences should be balanced against the potential positive impact and use that to navigate decisions about
Ultimately I agree with the process and I should not complain after receiving a grant from Founders Pledge to support us in building the evidence base that is needed. Its just that I am frustrated because it feels like a continuous fight to move things forward or even survive, while deep in my heart I know we are right and that every delay will hinder us from preventing deaths that are unneeded.
Thanks Niek for your thoughts.
Regarding this:
I don’t mean to speak for anyone but myself, but I sense that there has been an increase in skepticism in recent decades, as interventions that seemed promising and which “we know work” turn out not to have any measurable effect, or even to have negative effects, when carefully studied. My prior, at least, regarding any new intervention that I hear about is that any effect is probably illusory or minimal.
It’s not that there is no urgency about the problems in the world — those are indeed urgent and demanding — but rather, our resources are so limited, and good interventions so hard to find, that we should probably dedicate only a small amount of our funding to more speculative projects, so that we can build evidence over time and not waste funding on projects that turn out not to work. As an example of what we want to avoid doing, look at what happened with microfinance.
Of course the situation would be different if we had already fully funded well-studied projects like bednets, deworming, Vitamin A, etc.; but sadly that’s not the world we actually inhabit.
I guess then it’s just my title: In God we trust, but all others must bring data. It just feels frustrating sometimes, also because people actually don’t say they don’t believe you. So if you don’t call me a liar, then why don’t you help or support me?
I would really like it if a more systematic way of identifying, testing and scaling successful innovations was available.
My post are definitely not intended to complain by the way. I am blessed with a great and supportive team and have great partners as well. It’s a genuine interest to understand, learn and improve.
PS. over the next days I will try to add a bit more background about the solution and our evidence, where we think it needs to be strenthened. I was a bit reluctant to put too much information in, because I thougt people might not read it.