So I went over the additional documents and I owe you an apology for being dismissive. There is indeed more to the analysis than I thought, and it was flippant to suggest that your or GiveWell’s replicability adjustment was just “this number looks too high” and thus incorporates this. Having gone through the replicability adjustment document, I think it makes a lot more sense than I gave it credit for.
What I couldn’t gather from the document was where exactly you differed from GiveWell. Is it only in the economic losers weighting? Were your components from weight gain, years of schooling and cognition the same as GiveWell’s? In the sheet where you calculate the replicability adjustment, there is no factoring of economic losers as far as I can tell, so in order to arrive at the new replicability adjustment you must have had to differ from GiveWell in the mechanism adjustment, right?
Hi Karthik, we’re glad to hear our additional documents provided useful detail. We apologise if our deviation from GiveWell wasn’t clear, but hopefully our explanation below is clarifying.
While we made a few minor amendments to GiveWell’s replicability adjustment, including to the weight/cognition/schooling components (as outlined in this section), our most material change was to account for economic losers. We account for economic losers by adjusting the size of the effect from the 20-year Busia follow-up (if you would like to see the calculations you can look at this cell, but it may be easier to follow our rationale for the calculation in this section). This “adjusted” effect size is then combined with the weight/cognition/schooling components in a Bayesian analysis.
As noted in the post, we believe others could reasonably disagree with the details of how we calculated this “adjusted” effect size, particularly the details around how the figures relating to wage-employed and self-employed people were used to calibrate the size of the adjustment. We do however think the outcome of the calculation gets us to an adjustment which is materially higher than the 3% adjustment used by GiveWell, which was the main aim of calculating the adjustment.
So I went over the additional documents and I owe you an apology for being dismissive. There is indeed more to the analysis than I thought, and it was flippant to suggest that your or GiveWell’s replicability adjustment was just “this number looks too high” and thus incorporates this. Having gone through the replicability adjustment document, I think it makes a lot more sense than I gave it credit for.
What I couldn’t gather from the document was where exactly you differed from GiveWell. Is it only in the economic losers weighting? Were your components from weight gain, years of schooling and cognition the same as GiveWell’s? In the sheet where you calculate the replicability adjustment, there is no factoring of economic losers as far as I can tell, so in order to arrive at the new replicability adjustment you must have had to differ from GiveWell in the mechanism adjustment, right?
Hi Karthik, we’re glad to hear our additional documents provided useful detail. We apologise if our deviation from GiveWell wasn’t clear, but hopefully our explanation below is clarifying.
While we made a few minor amendments to GiveWell’s replicability adjustment, including to the weight/cognition/schooling components (as outlined in this section), our most material change was to account for economic losers. We account for economic losers by adjusting the size of the effect from the 20-year Busia follow-up (if you would like to see the calculations you can look at this cell, but it may be easier to follow our rationale for the calculation in this section). This “adjusted” effect size is then combined with the weight/cognition/schooling components in a Bayesian analysis.
As noted in the post, we believe others could reasonably disagree with the details of how we calculated this “adjusted” effect size, particularly the details around how the figures relating to wage-employed and self-employed people were used to calibrate the size of the adjustment. We do however think the outcome of the calculation gets us to an adjustment which is materially higher than the 3% adjustment used by GiveWell, which was the main aim of calculating the adjustment.