I considered providing an estimate like the one you suggest, but shied away for two reasons:
I am not a subject matter expert and I donât have a good sense of what the effect size would be â as GiveWell notes, across all seven studies, mortality in both groups is lower than you expect, so thereâs some disconnect between theory and empirics here that I/âwe lack context on;
the expected value of a new finding hinges on equilibrium effects that I canât really get a handle on. Letâs say that GiveWell finds smaller effects than they expect and then shifts a different charity to be #1. Is that interventionâs evidence really solid, or should that intervention also be closely re-examined and then replicated? I do not know; if I had had more time I would have like to do this type of analysis for the other three interventions as well.
My hope is that if I help point GiveWell in the right direction, people who are more experienced at cost-benefit analysis can take it from there. My comparative advantage is reading RCTs and meta-analyses.
As to the ethical concerns â that depends on whether the control group is likely to have received an anti-malaria treatment in the absence of an intervention, i.e. the point I made in section 2. If everybody is receiving bed nets anyway, letâs study that population.
That seems fair. I agree that my request for an estimate is a big, maybe even unreasonable, request.
I asked because I am wondering if there really is enough reason to doubt the results of existing SMC trials. If I understand your post correctly, your main worry is not about actual errors in the trials; we donât have concrete reasons to believe they are wrong. Indeed, the trials provide high-quality evidence that SMC reduces malaria cases, including severe cases.
Your worries seem to be that (1) studies are underpowered to quantify reduction in malaria deaths. Iâm not sure if that is a big problem, given that there are clear causal links between malaria cases and malaria deaths. (2) The trials did not follow the new best practices that weâve identified since they were published. This indeed makes the trials less reliable than we would wish for, but Iâm not sure whether the problem extends to a meta-analysis of seven trials.
For all these reasons, I keep wondering: how strongly do you really believe these results are wrong? And by how much? Even some rough answer would be OK here⊠and Iâm sure it would also help GiveWell when they evaluate this post.
Iâve been thinking about this and I think youâre right, I do believe that running this replication trial passes a cost-benefit test, and I should try to explain why.
how strongly do you really believe these results are wrong? And by how much?
I think thereâs a 50% chance that a perfectly done SMC replication would find mortality effects that are statistically indistinguishable from a null, for two reasons: 1) the documented empirical effects are strange and donât gel with our underlying theory of malaria; 2) our theory also conflicts with the repeated observation that people living in extreme poverty donât seem to take malaria as seriously as outsiders do, which is prima facie evidence that weâre misunderstanding something big.
My essayâs thesis is that SMCâs underlying RCT evidence, which is the foundation of GiveWellâs cost-benefit analysis, is weaker than it appears at first glance.
Does the use of meta-analysis somewhat or largely obviate this problem? In my opinion, no, aggregation does not paper over structural issues in the data generation process.
One of the most striking things my co-authors found when meta-analyzing the contact hypothesis literature was the gap in effect size between studies that had a pre-analysis plan (d = 0.016) and those that didnât (d = 0.451). This obviously isnât dispositive that thereâs âno there thereâ with intergroup contact; but when subsequent high-quality studies on the subject found much more mixed results (e.g. here and here), at the very least, we can say we had a warning sign.
Can we supplement evidence that SMC reduces malaria cases with other putatively causal[1] evidence that intervening to reduce malaria leads to a sizeable reduction in deaths?
That depends on how seriously we take the argument that most published research findings are false. I myself take this very seriously, and I basically treat all research as provisional until itâs been validated through a seriously well-identified study.
Iâm not saying that we donât know that malaria causes deathsâwe definitely know that people die of malaria. But why did the SMC studies find much smaller overall mortality effects, in both treatment and control, than expected?
That study was on the very low end for âentomological inoculation rates (infective bites per person per year)â (Meremikwu et al. 2012 p.8), which ranged from 10 (Cisse 2006) to 173 bites (Konate 2011).
To my eyes, this looks like a serious disconnect between theory and empirics, and itâs repeated across many settings. Frankly I have no idea whatâs going on. Am I misunderstanding something fundamental here or have I made a mistake? What does GiveWell make of this? I take it you work for AMFâwhat do you make of it?
Back to my experience with the contact hypothesis: I treat âsomething weird that we donât understand in the published findingsâ as a warning bell. So, personally, I think there is at least a 50% chance that a perfectly run SMC study today would find effects on mortality that are indistinguishable from a null.
Letâs say that trial cost $10M to do, and it affected the allocation of hundreds of millions of dollars. I think that passes a cost-benefit test across a wide range of supplementary parameter values.
They also say that they only want to recommend charities that meet a benchmark of being 10X as cost-effective as cash.
In effect, GiveWell is saying that if you give someone living in extreme poverty $10, the things they spend it on will only give them 10% of the utility that they might have gotten if someone else had chosen their bundle of goods for them.
This is actually an extraordinary claim and thus requires extraordinary evidence. It would definitely raise your hackles if someone said it about you â that youâre leaving 90% of potential value on the table because you donât know whatâs best for yourself.
Are we saying that they donât have access to the correct bundle of goods for distribution/âinfrastructure reasons?
A few of the SMC studies note that people have bed nets but that theyâre in bad shape; others have found that âwidely distributed ITNs have been repurposed as fishing nets throughout the worldâ (Larsen et al. 2021).
At face value, this seems like evidence that people in extreme poverty donât value malaria treatments nearly as much as GiveWell does. Whatâs going on there? Is it because theyâre ignorant or short-sighted? Or is it because Westerners are convinced of a theory that conflicts with peopleâs lived experiencesâand also with the actual empirical evidence found by SMC studies? I donât know, and I find these theories about equally plausible.
I know this was all very approximate for a cost-benefit analysis, but IMO,we need a stronger basis for our assumptions about effect sizes than we currently have to be more specific.
putative because Iâm pretty sure it doesnât come from Human Challenge Trials, i.e. malaria was not actually the thing randomly assigned. FWIW I donât think that that trial would pass a cost-benefit test.
I appreciate the thoughts! Iâm going to think about this more thoroughly⊠but hereâs a quick guess about the low death numbers:
These trials involved measuring malaria prevalence in children. Presumably, children with a positive result would then get medication or be referred to a health center. Malaria is a curable disease, so this approach would save lives. Unfortunately, itâs also quite likely that the child would not receive appropriate treatment in the absence of a diagnosis, due to lack of knowledge of the parents, distance to health facilities, etc.
Anyway, itâs just a quick guess. Might be worth checking if the studies describe what happened to children with positive test results.
The study was designed to measure Malaria, not deaths:
The primary outcome measure was a comparison of the occurence of clinical malaria between children in the two study groups.
Children with positive malaria tests received treatment:
Malaria morbidity was monitored through home visits every week and by detection of study children who presented at one of three health centres in the study area. At each assessment, axillary temperature was measured, and if it was 37.5C or greater, or if there was a history of fever or vomiting during the previous 24 h, a blood film was prepared. Results of the blood film examination were usually available within 2 h. Antimalarial treatment was given when appropriate according to the national guidelines: chloroquine as firstline treatment, quinine or sulfadoxine-pyrimethamine as second-line treatment in cases of failure of treatment with chloroquine, and injectable quinine for cases with persistent vomiting or severe malaria. Study children received iron supplementation if they presented at a health centre with an illness suggestive of anaemia, pale mucosae, or both.
Iâll still think more about this⊠but here we have at least a lead towards better understanding of low death numbers in SMC trials.
Thank you for looking into it! Definitely interesting. To recap:
GiveWellâs cost-benefit calculations hinge on the relationship between SMC and mortality.
The key mediator there is cases of malaria.
In the provided studies, the estimated relationship between cases of malaria and deaths is likely to bedownwardly biased because of co-delivered interventions (ITN, HMM, and, as youâve identified, just more attentiveness to malaria in general in treated areas).
As SMC is rolled out, is it rolled out along with more general medical care, or without? With co-interventions, or without? This seems like the key question we donât have a handle on and that GiveWellâs materials donât shine much light on.
Letâs say itâs rolled out along with general medical care. In that case, whatâs actually doing the work in reducing mortality, SMC or medical care? And which set of costs (SMC, medical care, or the two combined) should factor into the $-per-life-saved calculation?
Letâs say itâs rolled out without that general medical care. In that case, do we really have a good estimate of the expected effects on mortality of just SMC? because that seems like the number GiveWell is basing its top charity title on, and at first glance, itâs really not clear what percentage of the research actually estimates that directly.
So in sum, either SMC is typically going to be rolled out in places/âcontexts where its effect on mortality is likely to be much lower than broader data about the relationship between malaria and mortality would suggest, which means that our $-per-life-saved metrics might be seriously off-base; or it will be rolled out in places that are very much unlike the settings in which the studies were run, which is a serious external validity problem.
So all in all, a confusing situation. And given the high stakes, I suggest that GiveWell taps a team with expertise in both the subject matter and RCTs to design and run an intervention that maps directly onto the target population.
Two postscripts:
Just curious, is this the kind of thing yâall discuss day-to-day at AMF? Iâm very curious to hear from practitioners on this kind of thing. I am a total outsider who happened to notice that the evidence in Cochrane review didnât map very neatly onto GiveWellâs analyses. Would love to know the âinsiderâ perspective a bit more.
I think we are getting closer to the core of your question here: the relationship between cases of malaria (or severe malaria more specifically) and deaths. I think that it would indeed be good to know more about the circumstances under which children die from malaria, and how this is affected by various kinds of medical care.
The question might partially touch upon SMC. Besides preventing malaria cases, it could also have an effect on severity (Iâm thinking of Covid vaccines as an analogy). That said, the case for SMC (as I understand it) is that itâs an excellent way to prevent malaria infections. This is what the RCTs measure, and this is where its value comes from.
To answer the question, I believe it would be more helpful to do research into malaria as an illness, rather than doing an SMC trial replication. I continue to think that the evidence base for SMC is good enough. You have doubts since âmost published research findings are falseâ, but âmost published research findingsâ might be the wrong reference class here:
It includes observational studies, surveys, and other less reliable methods; here, we have RCTs.
It includes all published studies, also those with small samples and effect sizes. Here, we have >7 trials, >12k participants, and the effect (SMCâs reduction of malaria episodes) is >6 standard deviations away from zero.
It includes studies with effects that are multiple causal steps away from the intervention (e.g., deworming improves income) and have many confounding factors. Here, we are measuring the effect of a malaria medication on malaria, with clearly-understood underlying mechanisms.
You also ask about the settings in which SMC is rolled out. There is no specific answer here, since SMC is often rolled out for entire countries or regions, aiming to fully cover all eligible children. More than 30 million children received SMC last year. In their cost-effectiveness analysis, GiveWell looks at interventions by country and takes a number of relevant factors into account, such as the âmortality rate from malaria for 3-59 month oldsâ.
In general, malaria fatality (deaths per case) is trending downwards a bit, due to factors such as better access to medical care, better diagnosis, better education of parents, and certainly many others. It could make sense to make this explicit when doing a cost-effectiveness analysis.
Iâd expect GiveWell to be mindful about these things and to have thought of the most-relevant factors. I donât think additional RCTs would lead to large changes here.
Regarding the post-script about AMF: We are fortunate to have a board of trustees and leaders that think a lot about high-level questions and trends, both those closer to AMFâs work (e.g., resistance to insecticides used in nets) and those more peripheral (e.g., the impact of new vaccines). There is also good and regular communication between GiveWell and AMF. As for myself, the day-to-day preoccupations are often much more mundane ;-)
Thanks as always for your careful and helpful read! I was just telling someone yesterday that this exchange is a positive reflection on the EA community and ethos â as a comparison point, itâs been way more constructive and collaborative than any of my experiences with academic peer review.
It sounds like I havenât changed your mind on the core subject and thatâs totally understandable. I speculate that this is something of a (professional) culture difference â the academics I discussed this essay with all started nodding along with the general idea the moment I mentioned âuncertainty about external validityâ đ
And thanks for the insight into AMF, yâall do great work.
Hi, and thanks for giving this a close read!
I considered providing an estimate like the one you suggest, but shied away for two reasons:
I am not a subject matter expert and I donât have a good sense of what the effect size would be â as GiveWell notes, across all seven studies, mortality in both groups is lower than you expect, so thereâs some disconnect between theory and empirics here that I/âwe lack context on;
the expected value of a new finding hinges on equilibrium effects that I canât really get a handle on. Letâs say that GiveWell finds smaller effects than they expect and then shifts a different charity to be #1. Is that interventionâs evidence really solid, or should that intervention also be closely re-examined and then replicated? I do not know; if I had had more time I would have like to do this type of analysis for the other three interventions as well.
My hope is that if I help point GiveWell in the right direction, people who are more experienced at cost-benefit analysis can take it from there. My comparative advantage is reading RCTs and meta-analyses.
As to the ethical concerns â that depends on whether the control group is likely to have received an anti-malaria treatment in the absence of an intervention, i.e. the point I made in section 2. If everybody is receiving bed nets anyway, letâs study that population.
That seems fair. I agree that my request for an estimate is a big, maybe even unreasonable, request.
I asked because I am wondering if there really is enough reason to doubt the results of existing SMC trials. If I understand your post correctly, your main worry is not about actual errors in the trials; we donât have concrete reasons to believe they are wrong. Indeed, the trials provide high-quality evidence that SMC reduces malaria cases, including severe cases.
Your worries seem to be that (1) studies are underpowered to quantify reduction in malaria deaths. Iâm not sure if that is a big problem, given that there are clear causal links between malaria cases and malaria deaths. (2) The trials did not follow the new best practices that weâve identified since they were published. This indeed makes the trials less reliable than we would wish for, but Iâm not sure whether the problem extends to a meta-analysis of seven trials.
For all these reasons, I keep wondering: how strongly do you really believe these results are wrong? And by how much? Even some rough answer would be OK here⊠and Iâm sure it would also help GiveWell when they evaluate this post.
Hi Sjlver,
Iâve been thinking about this and I think youâre right, I do believe that running this replication trial passes a cost-benefit test, and I should try to explain why.
I think thereâs a 50% chance that a perfectly done SMC replication would find mortality effects that are statistically indistinguishable from a null, for two reasons: 1) the documented empirical effects are strange and donât gel with our underlying theory of malaria; 2) our theory also conflicts with the repeated observation that people living in extreme poverty donât seem to take malaria as seriously as outsiders do, which is prima facie evidence that weâre misunderstanding something big.
My essayâs thesis is that SMCâs underlying RCT evidence, which is the foundation of GiveWellâs cost-benefit analysis, is weaker than it appears at first glance.
Does the use of meta-analysis somewhat or largely obviate this problem? In my opinion, no, aggregation does not paper over structural issues in the data generation process.
One of the most striking things my co-authors found when meta-analyzing the contact hypothesis literature was the gap in effect size between studies that had a pre-analysis plan (d = 0.016) and those that didnât (d = 0.451). This obviously isnât dispositive that thereâs âno there thereâ with intergroup contact; but when subsequent high-quality studies on the subject found much more mixed results (e.g. here and here), at the very least, we can say we had a warning sign.
Can we supplement evidence that SMC reduces malaria cases with other putatively causal[1] evidence that intervening to reduce malaria leads to a sizeable reduction in deaths?
That depends on how seriously we take the argument that most published research findings are false. I myself take this very seriously, and I basically treat all research as provisional until itâs been validated through a seriously well-identified study.
Iâm not saying that we donât know that malaria causes deathsâwe definitely know that people die of malaria. But why did the SMC studies find much smaller overall mortality effects, in both treatment and control, than expected?
CissĂ© et al. (2006) studied a region âwhere the mortality rate for children under 5 years of age is 40 deaths per 1000 children per year. Malaria accounts for about a quarter of deaths in those aged 1â5 yearsâ (p. 660).
That study was on the very low end for âentomological inoculation rates (infective bites per person per year)â (Meremikwu et al. 2012 p.8), which ranged from 10 (Cisse 2006) to 173 bites (Konate 2011).
So letâs just take CissĂ© et al.âs estimate as a conservative baseline, and say that if you study 12,589 children in endemic/âhyperendemic regions for a year,[2] you should expect about 500 deaths overall, with 125 of them attributable to malaria.
Instead, we get 26 deaths overall.
To my eyes, this looks like a serious disconnect between theory and empirics, and itâs repeated across many settings. Frankly I have no idea whatâs going on. Am I misunderstanding something fundamental here or have I made a mistake? What does GiveWell make of this? I take it you work for AMFâwhat do you make of it?
Back to my experience with the contact hypothesis: I treat âsomething weird that we donât understand in the published findingsâ as a warning bell. So, personally, I think there is at least a 50% chance that a perfectly run SMC study today would find effects on mortality that are indistinguishable from a null.
Letâs say that trial cost $10M to do, and it affected the allocation of hundreds of millions of dollars. I think that passes a cost-benefit test across a wide range of supplementary parameter values.
GiveWell cautions us not to take its expected value estimates literally, which is why I donât take its 5K per life saved estimate as a baseline.
They also say that they only want to recommend charities that meet a benchmark of being 10X as cost-effective as cash.
In effect, GiveWell is saying that if you give someone living in extreme poverty $10, the things they spend it on will only give them 10% of the utility that they might have gotten if someone else had chosen their bundle of goods for them.
This is actually an extraordinary claim and thus requires extraordinary evidence. It would definitely raise your hackles if someone said it about you â that youâre leaving 90% of potential value on the table because you donât know whatâs best for yourself.
Are we saying that they donât have access to the correct bundle of goods for distribution/âinfrastructure reasons?
A few of the SMC studies note that people have bed nets but that theyâre in bad shape; others have found that âwidely distributed ITNs have been repurposed as fishing nets throughout the worldâ (Larsen et al. 2021).
At face value, this seems like evidence that people in extreme poverty donât value malaria treatments nearly as much as GiveWell does. Whatâs going on there? Is it because theyâre ignorant or short-sighted? Or is it because Westerners are convinced of a theory that conflicts with peopleâs lived experiencesâand also with the actual empirical evidence found by SMC studies? I donât know, and I find these theories about equally plausible.
I know this was all very approximate for a cost-benefit analysis, but IMO,we need a stronger basis for our assumptions about effect sizes than we currently have to be more specific.
putative because Iâm pretty sure it doesnât come from Human Challenge Trials, i.e. malaria was not actually the thing randomly assigned. FWIW I donât think that that trial would pass a cost-benefit test.
âThe length of follow-up for the included trials varied from six months to two years; with one year being most commonâ (Meremikwu et al p. 8)
I appreciate the thoughts! Iâm going to think about this more thoroughly⊠but hereâs a quick guess about the low death numbers:
These trials involved measuring malaria prevalence in children. Presumably, children with a positive result would then get medication or be referred to a health center. Malaria is a curable disease, so this approach would save lives. Unfortunately, itâs also quite likely that the child would not receive appropriate treatment in the absence of a diagnosis, due to lack of knowledge of the parents, distance to health facilities, etc.
Anyway, itâs just a quick guess. Might be worth checking if the studies describe what happened to children with positive test results.
Looks like I can confirm this. Relevant passages from Cissé et al (2006):
The study was designed to measure Malaria, not deaths:
Children with positive malaria tests received treatment:
Iâll still think more about this⊠but here we have at least a lead towards better understanding of low death numbers in SMC trials.
Thank you for looking into it! Definitely interesting. To recap:
GiveWellâs cost-benefit calculations hinge on the relationship between SMC and mortality.
The key mediator there is cases of malaria.
In the provided studies, the estimated relationship between cases of malaria and deaths is likely to be downwardly biased because of co-delivered interventions (ITN, HMM, and, as youâve identified, just more attentiveness to malaria in general in treated areas).
As SMC is rolled out, is it rolled out along with more general medical care, or without? With co-interventions, or without? This seems like the key question we donât have a handle on and that GiveWellâs materials donât shine much light on.
Letâs say itâs rolled out along with general medical care. In that case, whatâs actually doing the work in reducing mortality, SMC or medical care? And which set of costs (SMC, medical care, or the two combined) should factor into the $-per-life-saved calculation?
Letâs say itâs rolled out without that general medical care. In that case, do we really have a good estimate of the expected effects on mortality of just SMC? because that seems like the number GiveWell is basing its top charity title on, and at first glance, itâs really not clear what percentage of the research actually estimates that directly.
So in sum, either SMC is typically going to be rolled out in places/âcontexts where its effect on mortality is likely to be much lower than broader data about the relationship between malaria and mortality would suggest, which means that our $-per-life-saved metrics might be seriously off-base; or it will be rolled out in places that are very much unlike the settings in which the studies were run, which is a serious external validity problem.
So all in all, a confusing situation. And given the high stakes, I suggest that GiveWell taps a team with expertise in both the subject matter and RCTs to design and run an intervention that maps directly onto the target population.
Two postscripts:
Just curious, is this the kind of thing yâall discuss day-to-day at AMF? Iâm very curious to hear from practitioners on this kind of thing. I am a total outsider who happened to notice that the evidence in Cochrane review didnât map very neatly onto GiveWellâs analyses. Would love to know the âinsiderâ perspective a bit more.
DataColada just published something about some structural issues with conventional meta-analysis that might be of interest.
Thanks for the thoughts!
I think we are getting closer to the core of your question here: the relationship between cases of malaria (or severe malaria more specifically) and deaths. I think that it would indeed be good to know more about the circumstances under which children die from malaria, and how this is affected by various kinds of medical care.
The question might partially touch upon SMC. Besides preventing malaria cases, it could also have an effect on severity (Iâm thinking of Covid vaccines as an analogy). That said, the case for SMC (as I understand it) is that itâs an excellent way to prevent malaria infections. This is what the RCTs measure, and this is where its value comes from.
To answer the question, I believe it would be more helpful to do research into malaria as an illness, rather than doing an SMC trial replication. I continue to think that the evidence base for SMC is good enough. You have doubts since âmost published research findings are falseâ, but âmost published research findingsâ might be the wrong reference class here:
It includes observational studies, surveys, and other less reliable methods; here, we have RCTs.
It includes all published studies, also those with small samples and effect sizes. Here, we have >7 trials, >12k participants, and the effect (SMCâs reduction of malaria episodes) is >6 standard deviations away from zero.
It includes studies with effects that are multiple causal steps away from the intervention (e.g., deworming improves income) and have many confounding factors. Here, we are measuring the effect of a malaria medication on malaria, with clearly-understood underlying mechanisms.
You also ask about the settings in which SMC is rolled out. There is no specific answer here, since SMC is often rolled out for entire countries or regions, aiming to fully cover all eligible children. More than 30 million children received SMC last year. In their cost-effectiveness analysis, GiveWell looks at interventions by country and takes a number of relevant factors into account, such as the âmortality rate from malaria for 3-59 month oldsâ.
In general, malaria fatality (deaths per case) is trending downwards a bit, due to factors such as better access to medical care, better diagnosis, better education of parents, and certainly many others. It could make sense to make this explicit when doing a cost-effectiveness analysis.
Iâd expect GiveWell to be mindful about these things and to have thought of the most-relevant factors. I donât think additional RCTs would lead to large changes here.
Regarding the post-script about AMF: We are fortunate to have a board of trustees and leaders that think a lot about high-level questions and trends, both those closer to AMFâs work (e.g., resistance to insecticides used in nets) and those more peripheral (e.g., the impact of new vaccines). There is also good and regular communication between GiveWell and AMF. As for myself, the day-to-day preoccupations are often much more mundane ;-)
Thanks as always for your careful and helpful read! I was just telling someone yesterday that this exchange is a positive reflection on the EA community and ethos â as a comparison point, itâs been way more constructive and collaborative than any of my experiences with academic peer review.
It sounds like I havenât changed your mind on the core subject and thatâs totally understandable. I speculate that this is something of a (professional) culture difference â the academics I discussed this essay with all started nodding along with the general idea the moment I mentioned âuncertainty about external validityâ đ
And thanks for the insight into AMF, yâall do great work.