Really happy you found the critique valuable. I agree with everywhere you’ve said the issue is too small to be worth addressing, and also with the correction that because AMF and SMC used different data sources it was legitimate not to explicitly model indirect deaths for the AMF (I’m also pleased you’ve standardised your methodology in the updated model, because this was quite a big discrepancy between approaches).
I don’t totally agree with the other areas where you’ve said you’re confident there is no error at all. I think a better interpretation might be that you’ve got a reasonable simplifying assumption that isn’t worth adjusting. For example, you say that it is appropriate to use different methodologies to account for the spillover effects of cash transfer vs income increase from deworming, which results in larger effective household sizes for cash transfer than for deworming. This is because deworming increases income via improving the earning potential of one member of the household, so there needs to be an adjustment for multi-earner households. Obviously you’re the topic area experts so I’d defer to you on whether or not this is reasonable, but the way you’ve implemented this seems contestable to me in the abstract; around 40% of the benefits of GiveDirectly are assumed to be accrued in the form of dividends on a long-term capital investment, but those benefits are distributed across the entire family of 4.7 for the entire ten-year period of the intervention, whereas in the deworming models you explicitly downweight family size to account for dependents leaving the household over time and the possibility of having two earners in a household. It seems more defensible to me to use a constant family size over the two interventions, and instead adjust the number of consumption doublings the deworming intervention provides to account for some families having two wage earners (so a doubling in individual income from the intervention doesn’t translate to a doubling in household income in the model). That is to say; either approach seems completely reasonable in isolation, but using a different approach in the two interventions favours GiveDirectly (because a given transfer spread across more people results in more utility overall, due to utility being a function of log income increase). In hindsight it is probably a bit harsh to describe this as an ‘error’ by GiveWell because it is a good-faith effort to capture the most important part of the model dynamics, but I also don’t think it is unambiguously correct as you’ve indicated in your review document; there’s a genuine inconsistency here. Overall I feel the same about most of the other areas you’ve highlighted—I was probably a bit over-harsh describing them as ‘errors’, but nevertheless they are potentially risky methodological steps which seem to have simpler and less risky alternative implementations.
Also on a small point of clarification—your comments in the linked document suggest that you are not worried about the hard-coded cells because they don’t have any impact on calculations. Sorry if I wasn’t clear in the text, but the main impact of hard coded cells is that the model becomes really hard to maintain—you won’t have an intuitive sense about whether you can make certain changes or not, because certain calculations you might expect to take place actually won’t happen because of the hard coding.
Really happy you found the critique valuable. I agree with everywhere you’ve said the issue is too small to be worth addressing, and also with the correction that because AMF and SMC used different data sources it was legitimate not to explicitly model indirect deaths for the AMF (I’m also pleased you’ve standardised your methodology in the updated model, because this was quite a big discrepancy between approaches).
I don’t totally agree with the other areas where you’ve said you’re confident there is no error at all. I think a better interpretation might be that you’ve got a reasonable simplifying assumption that isn’t worth adjusting. For example, you say that it is appropriate to use different methodologies to account for the spillover effects of cash transfer vs income increase from deworming, which results in larger effective household sizes for cash transfer than for deworming. This is because deworming increases income via improving the earning potential of one member of the household, so there needs to be an adjustment for multi-earner households. Obviously you’re the topic area experts so I’d defer to you on whether or not this is reasonable, but the way you’ve implemented this seems contestable to me in the abstract; around 40% of the benefits of GiveDirectly are assumed to be accrued in the form of dividends on a long-term capital investment, but those benefits are distributed across the entire family of 4.7 for the entire ten-year period of the intervention, whereas in the deworming models you explicitly downweight family size to account for dependents leaving the household over time and the possibility of having two earners in a household. It seems more defensible to me to use a constant family size over the two interventions, and instead adjust the number of consumption doublings the deworming intervention provides to account for some families having two wage earners (so a doubling in individual income from the intervention doesn’t translate to a doubling in household income in the model). That is to say; either approach seems completely reasonable in isolation, but using a different approach in the two interventions favours GiveDirectly (because a given transfer spread across more people results in more utility overall, due to utility being a function of log income increase). In hindsight it is probably a bit harsh to describe this as an ‘error’ by GiveWell because it is a good-faith effort to capture the most important part of the model dynamics, but I also don’t think it is unambiguously correct as you’ve indicated in your review document; there’s a genuine inconsistency here. Overall I feel the same about most of the other areas you’ve highlighted—I was probably a bit over-harsh describing them as ‘errors’, but nevertheless they are potentially risky methodological steps which seem to have simpler and less risky alternative implementations.
Also on a small point of clarification—your comments in the linked document suggest that you are not worried about the hard-coded cells because they don’t have any impact on calculations. Sorry if I wasn’t clear in the text, but the main impact of hard coded cells is that the model becomes really hard to maintain—you won’t have an intuitive sense about whether you can make certain changes or not, because certain calculations you might expect to take place actually won’t happen because of the hard coding.