This is really great work and quite interesting to read! Regarding the deworming section, I’m not sure how much it would make sense to update based on Liu & Liu. Liu & Liu isn’t the only paper that’s looked at the impacts of deworming based on historical natural experiment data—Bleakley (2007) found evidence of long-term effects on income as a result of hookworm eradication in the American south. Since GiveWell is aware of the Bleakley paper, I don’t know how much it would make sense to shift their priors based on Liu & Liu. Perhaps if Liu & Liu was thought to be more robust, it would be a big shift, but if not, I think it would be a smaller update.
We’re also uncertain how much we should update on Liu & Liu. But one reason it’d be strange for GiveWell to assign zero weight to Bleakley (2007) or Liu and Liu (2019) is that this is the only type of evidence (historical natural experiments) that they use for estimating the income effects of malaria prevention.
Note that if observational (i.e. non-experimental) studies are being included, one would probably also want to consider Croke 2019, which shows null effects on literacy and numeracy.
There is also Makamu et al. 2018, but I don’t think the natural experiment is very plausible (they use variation in which regions had deworming campaigns, but this is likely to be correlated with other policies/economic factors).
To be clear, I primarily cite natural experiment which I’d argue are not evidentially equivalent to observational studies.
Aside: I only skimmed Croke (2019), but are the null effects surprising given that control children are already given a big boost? -- Plausibly including other GiveWell top charity interventions such as vitamin-A supplementation and vaccines?
The intervention was delivered via a Child Health Day (CHD). Child Health Days are public events, held biannually, in which all parents in a given catchment area are requested to bring pre-school age children to a treatment site to receive a set of basic health services such as Vitamin A supplementation, growth monitoring, and vaccines. At the five Child Health Days held over the course of the project, children in the control parishes who attended the event were offered the standard CHD interventions, while those in the treatment group were also given deworming treatment (400 mg albendazole).
Ah yes, my apologies, I meant natural experiments (or in the case of Croke 2019, a natural experiment caused by an actual experiment).
I suppose it is possible deworming would have a much smaller effect when children also receive these other interventions. However, I would’ve thought many children currently being treated for worms are also receiving such interventions, therefore making it decision-relevant for GiveWell-funded deworming programs?
Very nice find on Croke (2019), I wasn’t aware of it! I think it’s worth mentioning that though their results aren’t statistically significant, their point estimates are positive for the literacy and numeracy effects. Since GiveWell is basically meta-analyzing the existing literature in trying to come up with its effect size estimates, there’s a meaningful difference between a positive point estimate and a zero or negative point estimate in the case where a study doesn’t find statistically significant results.
This is really great work and quite interesting to read! Regarding the deworming section, I’m not sure how much it would make sense to update based on Liu & Liu. Liu & Liu isn’t the only paper that’s looked at the impacts of deworming based on historical natural experiment data—Bleakley (2007) found evidence of long-term effects on income as a result of hookworm eradication in the American south. Since GiveWell is aware of the Bleakley paper, I don’t know how much it would make sense to shift their priors based on Liu & Liu. Perhaps if Liu & Liu was thought to be more robust, it would be a big shift, but if not, I think it would be a smaller update.
We’re also uncertain how much we should update on Liu & Liu. But one reason it’d be strange for GiveWell to assign zero weight to Bleakley (2007) or Liu and Liu (2019) is that this is the only type of evidence (historical natural experiments) that they use for estimating the income effects of malaria prevention.
Note that if observational (i.e. non-experimental) studies are being included, one would probably also want to consider Croke 2019, which shows null effects on literacy and numeracy.
There is also Makamu et al. 2018, but I don’t think the natural experiment is very plausible (they use variation in which regions had deworming campaigns, but this is likely to be correlated with other policies/economic factors).
To be clear, I primarily cite natural experiment which I’d argue are not evidentially equivalent to observational studies.
Aside: I only skimmed Croke (2019), but are the null effects surprising given that control children are already given a big boost? -- Plausibly including other GiveWell top charity interventions such as vitamin-A supplementation and vaccines?
Ah yes, my apologies, I meant natural experiments (or in the case of Croke 2019, a natural experiment caused by an actual experiment).
I suppose it is possible deworming would have a much smaller effect when children also receive these other interventions. However, I would’ve thought many children currently being treated for worms are also receiving such interventions, therefore making it decision-relevant for GiveWell-funded deworming programs?
Very nice find on Croke (2019), I wasn’t aware of it! I think it’s worth mentioning that though their results aren’t statistically significant, their point estimates are positive for the literacy and numeracy effects. Since GiveWell is basically meta-analyzing the existing literature in trying to come up with its effect size estimates, there’s a meaningful difference between a positive point estimate and a zero or negative point estimate in the case where a study doesn’t find statistically significant results.