As a separate note, I’m not sure if it was intentional, but it appears HLI has calculated log effects slightly differently to GiveWell.
GiveWell takes the average of earnings and consumption, and then calculates the log change.
HLI does the reverse, i.e. calculates the log of earnings and the log of consumption, and then takes the average.
If we were to follow the GiveWell method, the effect at the second follow-up would be 0.239 instead of 0.185, i.e. there would be no decay between the first and second follow-up (but the size of the decay between the first and third follow-up would be unaffected).
If the decay theory relies only on a single data point, does this place the theory on slightly shakier ground?
I don’t have a good intuition on which of these approaches is better. Was there any rationale for applying the second approach for this calculation?
Average of the log is more principled and I’m pretty surprised that givewell did it the reverse. These two quantities are always different (Jensen’s inequality) and only one of them is what we care about. Log increase in consumption/income represents the % increase in that quantity. We want to find the average % increase across all people, so we should take the average of the log increase.
Indeed, we did take the average of the logs instead of the log of the averages. This doesn’t change the end and start point, so it wouldn’t change the overall decay rate we estimate. We could do more complex modelling where effects between KLPS2 and KLPS3 see small growth and KLPS3 and KLPS4 see large decay. I think this shows that the overall results are sensitive to how we model effect across time.
See Figure 4 of the appendix, which shows, whether in earnings or in consumption, that the relative gains, as shown by the log difference, decrease over time.
We used the pooled data because it is what GiveWell does. In the appendix we note that the consumption and earnings data look different. So, perhaps a more principle way would be to look at the decay within earnings and within consumption. The decay within earnings (84%) and the decay within consumption (81%) are both stronger (i.e., would lead to smaller effects) than the 88% pooled decay.
As a separate note, I’m not sure if it was intentional, but it appears HLI has calculated log effects slightly differently to GiveWell.
GiveWell takes the average of earnings and consumption, and then calculates the log change.
HLI does the reverse, i.e. calculates the log of earnings and the log of consumption, and then takes the average.
If we were to follow the GiveWell method, the effect at the second follow-up would be 0.239 instead of 0.185, i.e. there would be no decay between the first and second follow-up (but the size of the decay between the first and third follow-up would be unaffected).
If the decay theory relies only on a single data point, does this place the theory on slightly shakier ground?
I don’t have a good intuition on which of these approaches is better. Was there any rationale for applying the second approach for this calculation?
Average of the log is more principled and I’m pretty surprised that givewell did it the reverse. These two quantities are always different (Jensen’s inequality) and only one of them is what we care about. Log increase in consumption/income represents the % increase in that quantity. We want to find the average % increase across all people, so we should take the average of the log increase.
Thank you for your comment!
Indeed, we did take the average of the logs instead of the log of the averages. This doesn’t change the end and start point, so it wouldn’t change the overall decay rate we estimate. We could do more complex modelling where effects between KLPS2 and KLPS3 see small growth and KLPS3 and KLPS4 see large decay. I think this shows that the overall results are sensitive to how we model effect across time.
See Figure 4 of the appendix, which shows, whether in earnings or in consumption, that the relative gains, as shown by the log difference, decrease over time.
We used the pooled data because it is what GiveWell does. In the appendix we note that the consumption and earnings data look different. So, perhaps a more principle way would be to look at the decay within earnings and within consumption. The decay within earnings (84%) and the decay within consumption (81%) are both stronger (i.e., would lead to smaller effects) than the 88% pooled decay.
Thanks for the response Samuel, would be interesting to hear GiveWell’s rationale on using the log of average(earnings+consumption).