More work building an empirical prior seems likely to change the estimated decay of income effects and thus deworming’s cost-effectiveness, although it’s unclear what direction.
Further progress appears easy to make.
This work doesn’t update HLI’s view of deworming much because:
The long-term income effects of deworming remain uncertain.
In either case, analysing deworming’s long-term effects still relies on a judgement-driven analysis of a single (well-done) but noisy study.
[Note: I threw this comment together rather quickly, but I wanted to get something out there quickly that gave my approximate views.]
1. There are several things I like about this update:
In several ways, it clarifies GiveWell’s analysis of deworming.
It succinctly explains where many of the numbers come from.
It clarifies the importance of explicit subjective assumptions (they seem pretty important).
It lays out the evidence it uses to build its prior in a manner that’s pretty easy to follow.
Helpfully, it lists the sources and reasons for the studies not included.
2. There are a few things that I think could be a bit clearer:
The decay rate from the raw (unadjusted) data is 13% yearly.
Assuming the same starting value as GiveWell, using this decay rate would lead to a total present value of 0.06 log-income units, compared to 0.09 for their “3% decay” model and 0.11 for their “no-decay” model.
Different decay rates imply very different discounts relative to the “no-decay” baseline / prior, 13% decay→ 49% discount. 3% → 19% discount.
They arrive at a decay rate of 3% instead of 13% because they subjectively discount the effect size from earlier time points more, which reduces the decay rate to 3%. While some of their justifications seem quite plausible[1] -- after some light spreadsheet crawling, I’m still confused about what’s happening underneath the hood here.
The 10% decrease in effectiveness comes because they assign 50% of the weight to their prior that there’s no decay and 50% to their estimate of a 3% decay rate. So whether the overall adjustment is 0% or 50% depends primarily on two factors:
How much to subjectively (unclear if this has an empirical element) discount each follow-up.
How much weight should be assigned to the prior for deworming’s time-trajectory, which they inform with a literature review.
All this being said, I think this update is a big improvement to the clarity of GiveWell’s deworming analysis.
My next two comments are related to some limitations of this update that Alex acknowledges:
It’s possible we’ve missed some relevant studies altogether.
We have not tried to formally combine these to get point estimates over time or attempted to weight studies based on relevance, study quality, etc.
We are combining studies that may have little ability to inform what we’d expect from deworming (twin studies, childcare programs, etc.).
It could be possible to re-assess other studies measuring long-term benefits of early childhood health interventions. When we set our prior, we excluded studies that did not report separate effects on income at different time periods. We guess that for several of these studies, it would be possible to re-analyze the primary data and create estimates of the effect on income at different time periods.
GiveWell takes a “vote counting” approach where the studies are weighted equally[2]. But I would be very surprised if further research assigned equal weight to these studies because they appear to vary considerably in relevance, sample size, and quality.
Deworming analogies include preschool, schooling, low birth weight, early childhood stimulation, pollution, twin height differences, and nutritional school lunches. It’s unclear how relevant these are to deworming because the mechanisms for deworming to benefit income seem poorly understood.
Sample sizes aren’t noted. This could matter as one of the “pro-growth trajectory” studies, Gertler et al. (2021) have a follow-up sample size of around ~50. That seems unusually small, so it’s unclear how much weight that should receive relative to others. However, it is one of the only studies in an LMIC!
There are also two observational studies, which typically receive less weight than quasi-experimental trials or RCTs (Case et al. 2005, Currie and Hyson 1999).
4. Progress towards building a firmer prior seems straightforward. Is GiveWell planning on refining its prior for deworming’s trajectory? Or incentivizing more research on this topic, e.g., via a prize or a bounty? Here are some reasons why I think further progress may not be difficult:
The literature review seems like it could be somewhat easily expanded:
It seems plausible that you could use Liu and Liu (2019), another causal study of deworming’s long-term effects on income, to see if the long-term effects change depending on age. They were helpful when we asked them for assistance.
Somewhat at random, I looked at Duflo et al. (2021), which was passed over for inclusion in the review and found that it contained multiple follow-ups and found weak evidence for incomes increasing over time due to additional education.
The existing literature review on priors could be upgraded to a meta-analysis with time (data extraction is more tedious than technically challenging). A resulting meta-analysis where each study is weighted by precision and potentially a subjective assessment of relevance would be more clarifying than the present “vote counting” method.
It’s unclear if all the conclusions were warranted. GiveWell reads Lang and Nystedt (2018) as finding “Increases for males; mixed for females” and notes some quotes from the original study:
“From ages 30–34 and onwards, the height premium increases over the life cycle for men, starting at approximately 5%, reaching 10% at ages 45-64 and approximately 11-12% at ages 65-79 (i.e., in retirement).” [...] “Almost the opposite trend is found for women. Being one decimeter taller is associated with over 11% higher earnings for women aged 25–29. As the women age, the height premium decreases and levels off at approximately 6–7%.” [...] “The path of the height premium profile over the female adult life cycle is quite unstable, and no obvious trend can be seen (see Fig. 2).” (17-18)
But when I look up that same table (shown below), I see decay for women and growth for men.
Higher ln earnings effects from KLPS-2 to KLPS-3 are driven by lower control group earnings in KLPS-2 ($330 vs. $1165).[8] In KLPS-3, researchers started measuring farming profits in addition to other forms of earnings,[9]so part of the apparent increase in control group earnings from KLPS-2 to KLPS-3 is likely driven by a change in measurement, not real standards of living or catch-up growth.”
“We found 10 longitudinal studies with at least two adult follow-ups from a number of countries examining the impact of a range of childhood interventions or conditions (see this table), in addition to the deworming study (Hamory et al. 2021). Of those 10 studies, 3 found decreasing effects on income, 3 found increasing effects, and 4 found mixed effects (either similar effects across time periods, different patterns across males and females, or increases and then decreases over the life cycle). Based on this, we think it makes sense to continue to assume as a prior that income effects would be constant over time. I have low confidence in these estimates, though, and it’s possible further work could lead to a different conclusion.”
Alex here, responding to your comment. Thank you for taking the time to give us this feedback!
In response to some of your specific points:
You’re right that we should have characterized the results from Lång and Nystedt (2018) as mixed rather than positive. Thanks for pointing out that mistake. We will update the spreadsheet so that study is correctly color-coded, and update the relevant part of the post. With this adjustment, among the studies we looked at, 3 suggest decreasing effects over time, 2 suggest increasing effects over time, and 5 show mixed effects. This still doesn’t seem like it adds up to strong evidence for either increasing or decreasing effects, so my prior of a flat effect over time remains the same.
We excluded Duflo et al. 2021 because it didn’t appear to include much about life cycle impacts on income from the intervention. It does report some increases in income for women in the treatment group between 2019 and 2020. However, I’d be reluctant to interpret that as evidence for increases over adulthood, because it represents only one year and because it compares pre-COVID results with results during COVID, which means other factors are probably at play.
That said, I agree that a more in-depth analysis might lead to a different prior for how we should expect early-life health interventions to affect income over the life cycle. We didn’t prioritize an in-depth analysis for this adjustment, but we would be open to more work to create a better-informed prior of deworming’s income effects over time. This would require deeper engagement with the studies we looked at to better understand their methodologies, relevance to deworming, and other factors. At the moment, it’s not a high-priority project for GiveWell staff, but we’re considering an external partnership to explore this further. We imagine that having a better grasp on how income effects change over time could inform our analysis not just of deworming but also of other programs we support, including vitamin A supplementation and seasonal malaria chemoprevention.
We’ll continue to share here if more work on this leads us to further updates.
Hi Alex, I’m heartened to see GiveWell engage with and update based on our previous work!
[Edited to expand on takeaway]
My overall impression is:
This update clearly improves GiveWell’s deworming analysis.
Each % point change in deworming cost-effectiveness could affect where hundreds of thousands of dollars are allocated. So getting it right seems important.
More work building an empirical prior seems likely to change the estimated decay of income effects and thus deworming’s cost-effectiveness, although it’s unclear what direction.
Further progress appears easy to make.
This work doesn’t update HLI’s view of deworming much because:
We primarily focus on subjective wellbeing as an outcome, which deworming doesn’t appear to affect in the long run.
The long-term income effects of deworming remain uncertain.
In either case, analysing deworming’s long-term effects still relies on a judgement-driven analysis of a single (well-done) but noisy study.
[Note: I threw this comment together rather quickly, but I wanted to get something out there quickly that gave my approximate views.]
1. There are several things I like about this update:
In several ways, it clarifies GiveWell’s analysis of deworming.
It succinctly explains where many of the numbers come from.
It clarifies the importance of explicit subjective assumptions (they seem pretty important).
It lays out the evidence it uses to build its prior in a manner that’s pretty easy to follow.
Helpfully, it lists the sources and reasons for the studies not included.
2. There are a few things that I think could be a bit clearer:
The decay rate from the raw (unadjusted) data is 13% yearly.
Assuming the same starting value as GiveWell, using this decay rate would lead to a total present value of 0.06 log-income units, compared to 0.09 for their “3% decay” model and 0.11 for their “no-decay” model.
Different decay rates imply very different discounts relative to the “no-decay” baseline / prior, 13% decay→ 49% discount. 3% → 19% discount.
They arrive at a decay rate of 3% instead of 13% because they subjectively discount the effect size from earlier time points more, which reduces the decay rate to 3%. While some of their justifications seem quite plausible[1] -- after some light spreadsheet crawling, I’m still confused about what’s happening underneath the hood here.
The 10% decrease in effectiveness comes because they assign 50% of the weight to their prior that there’s no decay and 50% to their estimate of a 3% decay rate. So whether the overall adjustment is 0% or 50% depends primarily on two factors:
How much to subjectively (unclear if this has an empirical element) discount each follow-up.
How much weight should be assigned to the prior for deworming’s time-trajectory, which they inform with a literature review.
All this being said, I think this update is a big improvement to the clarity of GiveWell’s deworming analysis.
My next two comments are related to some limitations of this update that Alex acknowledges:
3. After briefly looking over the literature review GiveWell uses to build a prior on the long-term effects of deworming, it seems like further research would lead to different results.
GiveWell takes a “vote counting” approach where the studies are weighted equally[2]. But I would be very surprised if further research assigned equal weight to these studies because they appear to vary considerably in relevance, sample size, and quality.
Deworming analogies include preschool, schooling, low birth weight, early childhood stimulation, pollution, twin height differences, and nutritional school lunches. It’s unclear how relevant these are to deworming because the mechanisms for deworming to benefit income seem poorly understood.
Sample sizes aren’t noted. This could matter as one of the “pro-growth trajectory” studies, Gertler et al. (2021) have a follow-up sample size of around ~50. That seems unusually small, so it’s unclear how much weight that should receive relative to others. However, it is one of the only studies in an LMIC!
There are also two observational studies, which typically receive less weight than quasi-experimental trials or RCTs (Case et al. 2005, Currie and Hyson 1999).
4. Progress towards building a firmer prior seems straightforward. Is GiveWell planning on refining its prior for deworming’s trajectory? Or incentivizing more research on this topic, e.g., via a prize or a bounty? Here are some reasons why I think further progress may not be difficult:
The literature review seems like it could be somewhat easily expanded:
It seems plausible that you could use Liu and Liu (2019), another causal study of deworming’s long-term effects on income, to see if the long-term effects change depending on age. They were helpful when we asked them for assistance.
Somewhat at random, I looked at Duflo et al. (2021), which was passed over for inclusion in the review and found that it contained multiple follow-ups and found weak evidence for incomes increasing over time due to additional education.
The existing literature review on priors could be upgraded to a meta-analysis with time (data extraction is more tedious than technically challenging). A resulting meta-analysis where each study is weighted by precision and potentially a subjective assessment of relevance would be more clarifying than the present “vote counting” method.
It’s unclear if all the conclusions were warranted. GiveWell reads Lang and Nystedt (2018) as finding “Increases for males; mixed for females” and notes some quotes from the original study:
“From ages 30–34 and onwards, the height premium increases over the life cycle for men, starting at approximately 5%, reaching 10% at ages 45-64 and approximately 11-12% at ages 65-79 (i.e., in retirement).” [...] “Almost the opposite trend is found for women. Being one decimeter taller is associated with over 11% higher earnings for women aged 25–29. As the women age, the height premium decreases and levels off at approximately 6–7%.” [...] “The path of the height premium profile over the female adult life cycle is quite unstable, and no obvious trend can be seen (see Fig. 2).” (17-18)
But when I look up that same table (shown below), I see decay for women and growth for men.
Higher ln earnings effects from KLPS-2 to KLPS-3 are driven by lower control group earnings in KLPS-2 ($330 vs. $1165).[8] In KLPS-3, researchers started measuring farming profits in addition to other forms of earnings,[9]so part of the apparent increase in control group earnings from KLPS-2 to KLPS-3 is likely driven by a change in measurement, not real standards of living or catch-up growth.”
“We found 10 longitudinal studies with at least two adult follow-ups from a number of countries examining the impact of a range of childhood interventions or conditions (see this table), in addition to the deworming study (Hamory et al. 2021). Of those 10 studies, 3 found decreasing effects on income, 3 found increasing effects, and 4 found mixed effects (either similar effects across time periods, different patterns across males and females, or increases and then decreases over the life cycle). Based on this, we think it makes sense to continue to assume as a prior that income effects would be constant over time. I have low confidence in these estimates, though, and it’s possible further work could lead to a different conclusion.”
Hi, Joel,
Alex here, responding to your comment. Thank you for taking the time to give us this feedback!
In response to some of your specific points:
You’re right that we should have characterized the results from Lång and Nystedt (2018) as mixed rather than positive. Thanks for pointing out that mistake. We will update the spreadsheet so that study is correctly color-coded, and update the relevant part of the post. With this adjustment, among the studies we looked at, 3 suggest decreasing effects over time, 2 suggest increasing effects over time, and 5 show mixed effects. This still doesn’t seem like it adds up to strong evidence for either increasing or decreasing effects, so my prior of a flat effect over time remains the same.
We excluded Duflo et al. 2021 because it didn’t appear to include much about life cycle impacts on income from the intervention. It does report some increases in income for women in the treatment group between 2019 and 2020. However, I’d be reluctant to interpret that as evidence for increases over adulthood, because it represents only one year and because it compares pre-COVID results with results during COVID, which means other factors are probably at play.
That said, I agree that a more in-depth analysis might lead to a different prior for how we should expect early-life health interventions to affect income over the life cycle. We didn’t prioritize an in-depth analysis for this adjustment, but we would be open to more work to create a better-informed prior of deworming’s income effects over time. This would require deeper engagement with the studies we looked at to better understand their methodologies, relevance to deworming, and other factors. At the moment, it’s not a high-priority project for GiveWell staff, but we’re considering an external partnership to explore this further. We imagine that having a better grasp on how income effects change over time could inform our analysis not just of deworming but also of other programs we support, including vitamin A supplementation and seasonal malaria chemoprevention.
We’ll continue to share here if more work on this leads us to further updates.
Best,
Alex