Thank you for looking into it! Definitely interesting. To recap:
GiveWellâs cost-benefit calculations hinge on the relationship between SMC and mortality.
The key mediator there is cases of malaria.
In the provided studies, the estimated relationship between cases of malaria and deaths is likely to bedownwardly biased because of co-delivered interventions (ITN, HMM, and, as youâve identified, just more attentiveness to malaria in general in treated areas).
As SMC is rolled out, is it rolled out along with more general medical care, or without? With co-interventions, or without? This seems like the key question we donât have a handle on and that GiveWellâs materials donât shine much light on.
Letâs say itâs rolled out along with general medical care. In that case, whatâs actually doing the work in reducing mortality, SMC or medical care? And which set of costs (SMC, medical care, or the two combined) should factor into the $-per-life-saved calculation?
Letâs say itâs rolled out without that general medical care. In that case, do we really have a good estimate of the expected effects on mortality of just SMC? because that seems like the number GiveWell is basing its top charity title on, and at first glance, itâs really not clear what percentage of the research actually estimates that directly.
So in sum, either SMC is typically going to be rolled out in places/âcontexts where its effect on mortality is likely to be much lower than broader data about the relationship between malaria and mortality would suggest, which means that our $-per-life-saved metrics might be seriously off-base; or it will be rolled out in places that are very much unlike the settings in which the studies were run, which is a serious external validity problem.
So all in all, a confusing situation. And given the high stakes, I suggest that GiveWell taps a team with expertise in both the subject matter and RCTs to design and run an intervention that maps directly onto the target population.
Two postscripts:
Just curious, is this the kind of thing yâall discuss day-to-day at AMF? Iâm very curious to hear from practitioners on this kind of thing. I am a total outsider who happened to notice that the evidence in Cochrane review didnât map very neatly onto GiveWellâs analyses. Would love to know the âinsiderâ perspective a bit more.
I think we are getting closer to the core of your question here: the relationship between cases of malaria (or severe malaria more specifically) and deaths. I think that it would indeed be good to know more about the circumstances under which children die from malaria, and how this is affected by various kinds of medical care.
The question might partially touch upon SMC. Besides preventing malaria cases, it could also have an effect on severity (Iâm thinking of Covid vaccines as an analogy). That said, the case for SMC (as I understand it) is that itâs an excellent way to prevent malaria infections. This is what the RCTs measure, and this is where its value comes from.
To answer the question, I believe it would be more helpful to do research into malaria as an illness, rather than doing an SMC trial replication. I continue to think that the evidence base for SMC is good enough. You have doubts since âmost published research findings are falseâ, but âmost published research findingsâ might be the wrong reference class here:
It includes observational studies, surveys, and other less reliable methods; here, we have RCTs.
It includes all published studies, also those with small samples and effect sizes. Here, we have >7 trials, >12k participants, and the effect (SMCâs reduction of malaria episodes) is >6 standard deviations away from zero.
It includes studies with effects that are multiple causal steps away from the intervention (e.g., deworming improves income) and have many confounding factors. Here, we are measuring the effect of a malaria medication on malaria, with clearly-understood underlying mechanisms.
You also ask about the settings in which SMC is rolled out. There is no specific answer here, since SMC is often rolled out for entire countries or regions, aiming to fully cover all eligible children. More than 30 million children received SMC last year. In their cost-effectiveness analysis, GiveWell looks at interventions by country and takes a number of relevant factors into account, such as the âmortality rate from malaria for 3-59 month oldsâ.
In general, malaria fatality (deaths per case) is trending downwards a bit, due to factors such as better access to medical care, better diagnosis, better education of parents, and certainly many others. It could make sense to make this explicit when doing a cost-effectiveness analysis.
Iâd expect GiveWell to be mindful about these things and to have thought of the most-relevant factors. I donât think additional RCTs would lead to large changes here.
Regarding the post-script about AMF: We are fortunate to have a board of trustees and leaders that think a lot about high-level questions and trends, both those closer to AMFâs work (e.g., resistance to insecticides used in nets) and those more peripheral (e.g., the impact of new vaccines). There is also good and regular communication between GiveWell and AMF. As for myself, the day-to-day preoccupations are often much more mundane ;-)
Thanks as always for your careful and helpful read! I was just telling someone yesterday that this exchange is a positive reflection on the EA community and ethos â as a comparison point, itâs been way more constructive and collaborative than any of my experiences with academic peer review.
It sounds like I havenât changed your mind on the core subject and thatâs totally understandable. I speculate that this is something of a (professional) culture difference â the academics I discussed this essay with all started nodding along with the general idea the moment I mentioned âuncertainty about external validityâ đ
And thanks for the insight into AMF, yâall do great work.
Thank you for looking into it! Definitely interesting. To recap:
GiveWellâs cost-benefit calculations hinge on the relationship between SMC and mortality.
The key mediator there is cases of malaria.
In the provided studies, the estimated relationship between cases of malaria and deaths is likely to be downwardly biased because of co-delivered interventions (ITN, HMM, and, as youâve identified, just more attentiveness to malaria in general in treated areas).
As SMC is rolled out, is it rolled out along with more general medical care, or without? With co-interventions, or without? This seems like the key question we donât have a handle on and that GiveWellâs materials donât shine much light on.
Letâs say itâs rolled out along with general medical care. In that case, whatâs actually doing the work in reducing mortality, SMC or medical care? And which set of costs (SMC, medical care, or the two combined) should factor into the $-per-life-saved calculation?
Letâs say itâs rolled out without that general medical care. In that case, do we really have a good estimate of the expected effects on mortality of just SMC? because that seems like the number GiveWell is basing its top charity title on, and at first glance, itâs really not clear what percentage of the research actually estimates that directly.
So in sum, either SMC is typically going to be rolled out in places/âcontexts where its effect on mortality is likely to be much lower than broader data about the relationship between malaria and mortality would suggest, which means that our $-per-life-saved metrics might be seriously off-base; or it will be rolled out in places that are very much unlike the settings in which the studies were run, which is a serious external validity problem.
So all in all, a confusing situation. And given the high stakes, I suggest that GiveWell taps a team with expertise in both the subject matter and RCTs to design and run an intervention that maps directly onto the target population.
Two postscripts:
Just curious, is this the kind of thing yâall discuss day-to-day at AMF? Iâm very curious to hear from practitioners on this kind of thing. I am a total outsider who happened to notice that the evidence in Cochrane review didnât map very neatly onto GiveWellâs analyses. Would love to know the âinsiderâ perspective a bit more.
DataColada just published something about some structural issues with conventional meta-analysis that might be of interest.
Thanks for the thoughts!
I think we are getting closer to the core of your question here: the relationship between cases of malaria (or severe malaria more specifically) and deaths. I think that it would indeed be good to know more about the circumstances under which children die from malaria, and how this is affected by various kinds of medical care.
The question might partially touch upon SMC. Besides preventing malaria cases, it could also have an effect on severity (Iâm thinking of Covid vaccines as an analogy). That said, the case for SMC (as I understand it) is that itâs an excellent way to prevent malaria infections. This is what the RCTs measure, and this is where its value comes from.
To answer the question, I believe it would be more helpful to do research into malaria as an illness, rather than doing an SMC trial replication. I continue to think that the evidence base for SMC is good enough. You have doubts since âmost published research findings are falseâ, but âmost published research findingsâ might be the wrong reference class here:
It includes observational studies, surveys, and other less reliable methods; here, we have RCTs.
It includes all published studies, also those with small samples and effect sizes. Here, we have >7 trials, >12k participants, and the effect (SMCâs reduction of malaria episodes) is >6 standard deviations away from zero.
It includes studies with effects that are multiple causal steps away from the intervention (e.g., deworming improves income) and have many confounding factors. Here, we are measuring the effect of a malaria medication on malaria, with clearly-understood underlying mechanisms.
You also ask about the settings in which SMC is rolled out. There is no specific answer here, since SMC is often rolled out for entire countries or regions, aiming to fully cover all eligible children. More than 30 million children received SMC last year. In their cost-effectiveness analysis, GiveWell looks at interventions by country and takes a number of relevant factors into account, such as the âmortality rate from malaria for 3-59 month oldsâ.
In general, malaria fatality (deaths per case) is trending downwards a bit, due to factors such as better access to medical care, better diagnosis, better education of parents, and certainly many others. It could make sense to make this explicit when doing a cost-effectiveness analysis.
Iâd expect GiveWell to be mindful about these things and to have thought of the most-relevant factors. I donât think additional RCTs would lead to large changes here.
Regarding the post-script about AMF: We are fortunate to have a board of trustees and leaders that think a lot about high-level questions and trends, both those closer to AMFâs work (e.g., resistance to insecticides used in nets) and those more peripheral (e.g., the impact of new vaccines). There is also good and regular communication between GiveWell and AMF. As for myself, the day-to-day preoccupations are often much more mundane ;-)
Thanks as always for your careful and helpful read! I was just telling someone yesterday that this exchange is a positive reflection on the EA community and ethos â as a comparison point, itâs been way more constructive and collaborative than any of my experiences with academic peer review.
It sounds like I havenât changed your mind on the core subject and thatâs totally understandable. I speculate that this is something of a (professional) culture difference â the academics I discussed this essay with all started nodding along with the general idea the moment I mentioned âuncertainty about external validityâ đ
And thanks for the insight into AMF, yâall do great work.