This is Adam Salisbury, a senior research associate at GiveWell, responding from GiveWell’s EA Forum account. Thank you for taking the time to engage with our work. Changes to how much we value the health benefits of lead reduction could change how much we could recommend funding to charities working in this area in the future, and the conceptual approach you lay out is both sensible and easy to follow.
I wanted to clarify how we came to the previous adjustment figure, in case it provides helpful context, and then raise a question about your methodology.
Our approach to estimating the health benefits of lead reduction
Our 120% adjustment is a best guess based on i) baseline mortality from lead exposure and ii) the hypothetical value derived from the Pure Earth intervention averting 5%, 15%, or 30% of these deaths. We thought we could get a more accurate estimate if we spent more time looking at the lead mortality/morbidity literature, but we ultimately decided not to do this, as we did not think this parameter would make a difference to this particular grant decision. This was because the economic benefits alone put Pure Earth above our ‘bar’ of cost-effectiveness.
The decision not to dive deeper was broadly reflective of GiveWell’s research approach: in general, the amount of time we spend researching a parameter is roughly proportional to how critical we think that parameter is in influencing our bottom-line estimate/overall decision. We think this helps us to prioritize our time effectively, by keeping us focused on the areas most relevant to our funding decisions.
Question on the approach you recommended
With this said, if we were to consider more lead grants in the future, it is possible that the economic/qualitative arguments would change, which might make us want to dig deeper on the health impacts of lead exposure. For this reason, feedback such as this is really useful to us. On this note, I have a question about your methodology:
I have tried to replicate your back-of the-envelope calculation here, and have a query related to your approach to discounting. In sum: I don’t think it’s necessary to discount the economic benefits calculated on line 17, because discount rates are already ‘baked-in’ to the 1.2% of GDP estimate this figure derives from.
As I understand it, the 1.2% estimate comes from Attina & Trasande (2013), who estimate the economic burden of lead exposure by: i) estimating baseline blood lead levels (BLLs) in LMICs; ii) estimating the relationship between BLLs and IQ; iii) estimating the relationship between IQ and lifetime economic productivity (LEP); iv) applying this to LEP estimates across LMICs, to ‘back out’ lost LEP due to lead exposure. Their LEP estimates are benchmarked against estimates from the US, which “assume annual growth in productivity of 1% and a 3% discount rate” (page 3).
In other words, I think the 1.2% figure comes from the following equation:
This is admittedly not easy to infer from the text, but I think is clearer in the Supplementary Appendix, where they report lost LEP per cohort for each LMIC (see Table S2). In any case, I think that because discount rates enter the numerator of this equation, we don’t want to further discount the 1.2% figure, since then we’d be ‘double-discounting’ future benefits.
If I don’t discount the economic benefits in line 17, then I get an implied adjustment for health benefits of 115% (line 24). This is broadly similar to the 120% adjustment we had before. Again, this could easily change if we decided to dip deeper into this parameter in the future.
I could easily be missing something, so please let me know if you disagree with my interpretation of Attina and Trasande (2013). And thanks again for your post – we really appreciate the feedback.
Hi Adam! Thanks for the detailed reply. From a brief look at your model, it seems you’ve understood my reasoning in this post correctly. I had indeed overlooked that their numbers were already discounted.
However, since they use a 3% discount rate and you use a 4% discount rate, you would still need to adjust for the difference. If we still assume that the economic impacts hit throughout your entire career, from 15 to 60 years into the future (note: 15 years into the future is not the average, but the initial year of impacts!), then you get to around $0.7 of NPV for each $1 today—much better than the $0.28 in my analysis, but still less than the $1 without discounting. Using this number, the result would be very close to GiveWell’s 20% estimate. How curious!
This is Adam Salisbury, a senior research associate at GiveWell, responding from GiveWell’s EA Forum account. Thank you for taking the time to engage with our work. Changes to how much we value the health benefits of lead reduction could change how much we could recommend funding to charities working in this area in the future, and the conceptual approach you lay out is both sensible and easy to follow.
I wanted to clarify how we came to the previous adjustment figure, in case it provides helpful context, and then raise a question about your methodology.
Our approach to estimating the health benefits of lead reduction
Our 120% adjustment is a best guess based on i) baseline mortality from lead exposure and ii) the hypothetical value derived from the Pure Earth intervention averting 5%, 15%, or 30% of these deaths. We thought we could get a more accurate estimate if we spent more time looking at the lead mortality/morbidity literature, but we ultimately decided not to do this, as we did not think this parameter would make a difference to this particular grant decision. This was because the economic benefits alone put Pure Earth above our ‘bar’ of cost-effectiveness.
The decision not to dive deeper was broadly reflective of GiveWell’s research approach: in general, the amount of time we spend researching a parameter is roughly proportional to how critical we think that parameter is in influencing our bottom-line estimate/overall decision. We think this helps us to prioritize our time effectively, by keeping us focused on the areas most relevant to our funding decisions.
Question on the approach you recommended
With this said, if we were to consider more lead grants in the future, it is possible that the economic/qualitative arguments would change, which might make us want to dig deeper on the health impacts of lead exposure. For this reason, feedback such as this is really useful to us. On this note, I have a question about your methodology:
I have tried to replicate your back-of the-envelope calculation here, and have a query related to your approach to discounting. In sum: I don’t think it’s necessary to discount the economic benefits calculated on line 17, because discount rates are already ‘baked-in’ to the 1.2% of GDP estimate this figure derives from.
As I understand it, the 1.2% estimate comes from Attina & Trasande (2013), who estimate the economic burden of lead exposure by: i) estimating baseline blood lead levels (BLLs) in LMICs; ii) estimating the relationship between BLLs and IQ; iii) estimating the relationship between IQ and lifetime economic productivity (LEP); iv) applying this to LEP estimates across LMICs, to ‘back out’ lost LEP due to lead exposure. Their LEP estimates are benchmarked against estimates from the US, which “assume annual growth in productivity of 1% and a 3% discount rate” (page 3).
In other words, I think the 1.2% figure comes from the following equation:
This is admittedly not easy to infer from the text, but I think is clearer in the Supplementary Appendix, where they report lost LEP per cohort for each LMIC (see Table S2). In any case, I think that because discount rates enter the numerator of this equation, we don’t want to further discount the 1.2% figure, since then we’d be ‘double-discounting’ future benefits.
If I don’t discount the economic benefits in line 17, then I get an implied adjustment for health benefits of 115% (line 24). This is broadly similar to the 120% adjustment we had before. Again, this could easily change if we decided to dip deeper into this parameter in the future.
I could easily be missing something, so please let me know if you disagree with my interpretation of Attina and Trasande (2013). And thanks again for your post – we really appreciate the feedback.
Hi Adam! Thanks for the detailed reply. From a brief look at your model, it seems you’ve understood my reasoning in this post correctly. I had indeed overlooked that their numbers were already discounted.
However, since they use a 3% discount rate and you use a 4% discount rate, you would still need to adjust for the difference. If we still assume that the economic impacts hit throughout your entire career, from 15 to 60 years into the future (note: 15 years into the future is not the average, but the initial year of impacts!), then you get to around $0.7 of NPV for each $1 today—much better than the $0.28 in my analysis, but still less than the $1 without discounting. Using this number, the result would be very close to GiveWell’s 20% estimate. How curious!
Best,
Jakob
Thanks for the flag, Jakob!