Full disclosure: I’m the primary author of a yet to be published SoGive report on deworming, however I’m commenting here in a personal capacity.
Thanks for this thought provoking and well-written analysis!
I have a query about whether the exponential decay model appropriately reflects the evidence:
If I understand the model correctly, this cell seems to imply that the annual consumption effect of deworming in the first year of adulthood is 0.006 logs.
As HLI is aware, this is based on GiveWell’s estimated annual earnings effect—GiveWell gets 0.006 by applying some adjustments to the original effect of 0.109.
However, 0.109 is not the effect for the first year of adulthood. Rather, it is the effect across the first ~11 years of adulthood (ie. pooled earnings across KLPS rounds conducted ~10-20 years after treatment). *
I think this implies that the total effect over the first 11 years of adulthood (without discounting) is 0.006*11 = 0.061.
Currently, the HLI exponential decay / no discounting model suggests the total effect over these 11 years is only 0.035. Should this instead be 0.061 to reflect the 11 years of evidence we have?
To make the total effect 0.061 over these first 11 years (without discounting), the first year annual effect would need to be 0.010 rather than 0.006 (I used the Goal Seek function to get this number).
As a result, HLI’s exponential decay model with 4% discounting produces lifetime earnings of 0.061 (coincidently the same number as above). This is still a lot lower than GiveWell’s figure (0.115), but is higher than HLI’s (0.035, also coincidently the same number as above).
Under this new approach, decaying earnings would reduce cost-effectiveness by 46%, compared to 69% in the HLI model.
As a sense check, we can set the number of years of impact in GiveWell’s model to 11 years (instead of 40 years), which gives us total earnings of 0.051. Therefore, I don’t think it would make sense if the decay model produced lifetime earnings of only 0.035.
Looking forward to hearing HLI’s thoughts on whether this approach better reflects the evidence or if I have misunderstood.
* Note that I have included both the 10th and 20th year, hence the 11 years.
Thank you for your comment Lucas! Looking forward to seeing your forthcoming report.
Firstly, to clarify, we are doing a comparison between GiveWell’s model without decay and with decay. So to make the closest comparison possible we use the starting value and the time values that GiveWell uses. Rows 17, 18, and 19 of their CEA show the values they use for these. They consider the effects of starting 8 years after the deworming ends (~when participants start joining the labour force, see here) and continuing for 40 years with 0.006 each year. We get the same (similar because of our discretisation) total effects as GiveWell of 0.115 (0.113) for their model and show that if we use the exponential decay, we get a ~60% smaller total effect of 0.047.
While it’s plausible there’s a better value to start with; we’re trying to illustrate what would happen if GiveWell added decay to their model. It’s unclear if they would also change the starting value too, but seems like a plausible choice.
The advantage of exponential decay is that it is based on % and so we can extract it from the study and use it on any start value and period, as long as we use the same as GW on these, we can get a proportional decrease in the effect.
We also considered linear decay. When we used linear decay, we found that the reduction in benefits is more dramatic: an 88% reduction. With linear decay, we had to change the start value, but we did this both for the constant effect model and the decay models so we could compare the proportional change.
Of course, a more complex analysis, which neither ourselves nor GiveWell present, would be to model this with the whole individual data.
The main point here is that the effect is very sensitive to the choice of modelling over time and thereby should be explicitly mentioned in GiveWell’s analysis and reporting. I think this point holds.
I think my concern is that we can only “illustrate what would happen if GiveWell added decay to their model” if we have the right starting value. In the decay model’s current form, I believe the model is not only adding decay, but also inadvertently changes the total earnings effect over the first 11 years of adulthood (yet we already have evidence on the total earnings effect for these years).
However, as you noted, the main point certainly still holds either way.
As a separate note, I’m not sure if it was intentional, but it appears HLI has calculated log effects slightly differently to GiveWell.
GiveWell takes the average of earnings and consumption, and then calculates the log change.
HLI does the reverse, i.e. calculates the log of earnings and the log of consumption, and then takes the average.
If we were to follow the GiveWell method, the effect at the second follow-up would be 0.239 instead of 0.185, i.e. there would be no decay between the first and second follow-up (but the size of the decay between the first and third follow-up would be unaffected).
If the decay theory relies only on a single data point, does this place the theory on slightly shakier ground?
I don’t have a good intuition on which of these approaches is better. Was there any rationale for applying the second approach for this calculation?
Average of the log is more principled and I’m pretty surprised that givewell did it the reverse. These two quantities are always different (Jensen’s inequality) and only one of them is what we care about. Log increase in consumption/income represents the % increase in that quantity. We want to find the average % increase across all people, so we should take the average of the log increase.
Indeed, we did take the average of the logs instead of the log of the averages. This doesn’t change the end and start point, so it wouldn’t change the overall decay rate we estimate. We could do more complex modelling where effects between KLPS2 and KLPS3 see small growth and KLPS3 and KLPS4 see large decay. I think this shows that the overall results are sensitive to how we model effect across time.
See Figure 4 of the appendix, which shows, whether in earnings or in consumption, that the relative gains, as shown by the log difference, decrease over time.
We used the pooled data because it is what GiveWell does. In the appendix we note that the consumption and earnings data look different. So, perhaps a more principle way would be to look at the decay within earnings and within consumption. The decay within earnings (84%) and the decay within consumption (81%) are both stronger (i.e., would lead to smaller effects) than the 88% pooled decay.
Full disclosure: I’m the primary author of a yet to be published SoGive report on deworming, however I’m commenting here in a personal capacity.
Thanks for this thought provoking and well-written analysis!
I have a query about whether the exponential decay model appropriately reflects the evidence:
If I understand the model correctly, this cell seems to imply that the annual consumption effect of deworming in the first year of adulthood is 0.006 logs.
As HLI is aware, this is based on GiveWell’s estimated annual earnings effect—GiveWell gets 0.006 by applying some adjustments to the original effect of 0.109.
However, 0.109 is not the effect for the first year of adulthood. Rather, it is the effect across the first ~11 years of adulthood (ie. pooled earnings across KLPS rounds conducted ~10-20 years after treatment). *
I think this implies that the total effect over the first 11 years of adulthood (without discounting) is 0.006*11 = 0.061.
Currently, the HLI exponential decay / no discounting model suggests the total effect over these 11 years is only 0.035. Should this instead be 0.061 to reflect the 11 years of evidence we have?
To make the total effect 0.061 over these first 11 years (without discounting), the first year annual effect would need to be 0.010 rather than 0.006 (I used the Goal Seek function to get this number).
As a result, HLI’s exponential decay model with 4% discounting produces lifetime earnings of 0.061 (coincidently the same number as above). This is still a lot lower than GiveWell’s figure (0.115), but is higher than HLI’s (0.035, also coincidently the same number as above).
Under this new approach, decaying earnings would reduce cost-effectiveness by 46%, compared to 69% in the HLI model.
As a sense check, we can set the number of years of impact in GiveWell’s model to 11 years (instead of 40 years), which gives us total earnings of 0.051. Therefore, I don’t think it would make sense if the decay model produced lifetime earnings of only 0.035.
Looking forward to hearing HLI’s thoughts on whether this approach better reflects the evidence or if I have misunderstood.
* Note that I have included both the 10th and 20th year, hence the 11 years.
Thank you for your comment Lucas! Looking forward to seeing your forthcoming report.
Firstly, to clarify, we are doing a comparison between GiveWell’s model without decay and with decay. So to make the closest comparison possible we use the starting value and the time values that GiveWell uses. Rows 17, 18, and 19 of their CEA show the values they use for these. They consider the effects of starting 8 years after the deworming ends (~when participants start joining the labour force, see here) and continuing for 40 years with 0.006 each year. We get the same (similar because of our discretisation) total effects as GiveWell of 0.115 (0.113) for their model and show that if we use the exponential decay, we get a ~60% smaller total effect of 0.047.
While it’s plausible there’s a better value to start with; we’re trying to illustrate what would happen if GiveWell added decay to their model. It’s unclear if they would also change the starting value too, but seems like a plausible choice.
The advantage of exponential decay is that it is based on % and so we can extract it from the study and use it on any start value and period, as long as we use the same as GW on these, we can get a proportional decrease in the effect.
We also considered linear decay. When we used linear decay, we found that the reduction in benefits is more dramatic: an 88% reduction. With linear decay, we had to change the start value, but we did this both for the constant effect model and the decay models so we could compare the proportional change.
Of course, a more complex analysis, which neither ourselves nor GiveWell present, would be to model this with the whole individual data.
The main point here is that the effect is very sensitive to the choice of modelling over time and thereby should be explicitly mentioned in GiveWell’s analysis and reporting. I think this point holds.
Hi Joel, thanks for your response on this!
I think my concern is that we can only “illustrate what would happen if GiveWell added decay to their model” if we have the right starting value. In the decay model’s current form, I believe the model is not only adding decay, but also inadvertently changes the total earnings effect over the first 11 years of adulthood (yet we already have evidence on the total earnings effect for these years).
However, as you noted, the main point certainly still holds either way.
As a separate note, I’m not sure if it was intentional, but it appears HLI has calculated log effects slightly differently to GiveWell.
GiveWell takes the average of earnings and consumption, and then calculates the log change.
HLI does the reverse, i.e. calculates the log of earnings and the log of consumption, and then takes the average.
If we were to follow the GiveWell method, the effect at the second follow-up would be 0.239 instead of 0.185, i.e. there would be no decay between the first and second follow-up (but the size of the decay between the first and third follow-up would be unaffected).
If the decay theory relies only on a single data point, does this place the theory on slightly shakier ground?
I don’t have a good intuition on which of these approaches is better. Was there any rationale for applying the second approach for this calculation?
Average of the log is more principled and I’m pretty surprised that givewell did it the reverse. These two quantities are always different (Jensen’s inequality) and only one of them is what we care about. Log increase in consumption/income represents the % increase in that quantity. We want to find the average % increase across all people, so we should take the average of the log increase.
Thank you for your comment!
Indeed, we did take the average of the logs instead of the log of the averages. This doesn’t change the end and start point, so it wouldn’t change the overall decay rate we estimate. We could do more complex modelling where effects between KLPS2 and KLPS3 see small growth and KLPS3 and KLPS4 see large decay. I think this shows that the overall results are sensitive to how we model effect across time.
See Figure 4 of the appendix, which shows, whether in earnings or in consumption, that the relative gains, as shown by the log difference, decrease over time.
We used the pooled data because it is what GiveWell does. In the appendix we note that the consumption and earnings data look different. So, perhaps a more principle way would be to look at the decay within earnings and within consumption. The decay within earnings (84%) and the decay within consumption (81%) are both stronger (i.e., would lead to smaller effects) than the 88% pooled decay.
Thanks for the response Samuel, would be interesting to hear GiveWell’s rationale on using the log of average(earnings+consumption).