I believe that someone who had read through all of this would have been less disappointed, however it’s also not reasonable to expect a reader to wade through that much detail.
The amount of detail that we should put in this summarised EA Forum and the extent to which we should keep this post short was something we struggled with. It may be that we got the judgement wrong this time.
We have not claimed that labour market competition is the most likely channel for what we observe.
The details here slightly subtle.
The end result of our analysis is not a dramatic reappraisal of deworming—the cost-effectiveness metrics are non-trivially lower than GiveWell’s, but still close to the bar for GiveWell recommended status.
If we believed that labour market competition / economic losers were the most likely channel for the collection of evidence that we see, we would have applied a more dramatic adjustment, with more dramatic conclusions.
We agree with your claim that labour market effects could include positive spillovers as well as negative ones, and this point is discussed in our fuller analysis document. However, our claim is that given that we see the evidence we see, it is more likely that the negative spillovers (economic losers) outweigh the positive ones, and to a more material degree than GiveWell suggests.
As our replication of the replicability adjustment indicates, and as we understand from our conversations with GiveWell, it is not true to say that GiveWell’s adjustment accounts for this already...
… instead there is a Bayesian method which explicitly accounts for different considerations, and GiveWell’s version of this does not reflect the economic losers consideration, and our replicability adjustment does. (I’ve glossed over some subtleties, full details can be found in the relevant document.) Hence our perspective being different from GiveWell’s.
So I went over the additional documents and I owe you an apology for being dismissive. There is indeed more to the analysis than I thought, and it was flippant to suggest that your or GiveWell’s replicability adjustment was just “this number looks too high” and thus incorporates this. Having gone through the replicability adjustment document, I think it makes a lot more sense than I gave it credit for.
What I couldn’t gather from the document was where exactly you differed from GiveWell. Is it only in the economic losers weighting? Were your components from weight gain, years of schooling and cognition the same as GiveWell’s? In the sheet where you calculate the replicability adjustment, there is no factoring of economic losers as far as I can tell, so in order to arrive at the new replicability adjustment you must have had to differ from GiveWell in the mechanism adjustment, right?
Hi Karthik, we’re glad to hear our additional documents provided useful detail. We apologise if our deviation from GiveWell wasn’t clear, but hopefully our explanation below is clarifying.
While we made a few minor amendments to GiveWell’s replicability adjustment, including to the weight/cognition/schooling components (as outlined in this section), our most material change was to account for economic losers. We account for economic losers by adjusting the size of the effect from the 20-year Busia follow-up (if you would like to see the calculations you can look at this cell, but it may be easier to follow our rationale for the calculation in this section). This “adjusted” effect size is then combined with the weight/cognition/schooling components in a Bayesian analysis.
As noted in the post, we believe others could reasonably disagree with the details of how we calculated this “adjusted” effect size, particularly the details around how the figures relating to wage-employed and self-employed people were used to calibrate the size of the adjustment. We do however think the outcome of the calculation gets us to an adjustment which is materially higher than the 3% adjustment used by GiveWell, which was the main aim of calculating the adjustment.
Thank you for your comment.
I’m sorry you were disappointed by the lack of detail on this post.
The full analysis for this includes a longer document which provides more details on the analysis, a supplementary analysis, a document which sets out the details behind our full replication of the replicability adjustment that GiveWell applies.
I believe that someone who had read through all of this would have been less disappointed, however it’s also not reasonable to expect a reader to wade through that much detail.
The amount of detail that we should put in this summarised EA Forum and the extent to which we should keep this post short was something we struggled with. It may be that we got the judgement wrong this time.
We have not claimed that labour market competition is the most likely channel for what we observe.
The details here slightly subtle.
The end result of our analysis is not a dramatic reappraisal of deworming—the cost-effectiveness metrics are non-trivially lower than GiveWell’s, but still close to the bar for GiveWell recommended status.
If we believed that labour market competition / economic losers were the most likely channel for the collection of evidence that we see, we would have applied a more dramatic adjustment, with more dramatic conclusions.
We agree with your claim that labour market effects could include positive spillovers as well as negative ones, and this point is discussed in our fuller analysis document. However, our claim is that given that we see the evidence we see, it is more likely that the negative spillovers (economic losers) outweigh the positive ones, and to a more material degree than GiveWell suggests.
As our replication of the replicability adjustment indicates, and as we understand from our conversations with GiveWell, it is not true to say that GiveWell’s adjustment accounts for this already...
… instead there is a Bayesian method which explicitly accounts for different considerations, and GiveWell’s version of this does not reflect the economic losers consideration, and our replicability adjustment does. (I’ve glossed over some subtleties, full details can be found in the relevant document.) Hence our perspective being different from GiveWell’s.
So I went over the additional documents and I owe you an apology for being dismissive. There is indeed more to the analysis than I thought, and it was flippant to suggest that your or GiveWell’s replicability adjustment was just “this number looks too high” and thus incorporates this. Having gone through the replicability adjustment document, I think it makes a lot more sense than I gave it credit for.
What I couldn’t gather from the document was where exactly you differed from GiveWell. Is it only in the economic losers weighting? Were your components from weight gain, years of schooling and cognition the same as GiveWell’s? In the sheet where you calculate the replicability adjustment, there is no factoring of economic losers as far as I can tell, so in order to arrive at the new replicability adjustment you must have had to differ from GiveWell in the mechanism adjustment, right?
Hi Karthik, we’re glad to hear our additional documents provided useful detail. We apologise if our deviation from GiveWell wasn’t clear, but hopefully our explanation below is clarifying.
While we made a few minor amendments to GiveWell’s replicability adjustment, including to the weight/cognition/schooling components (as outlined in this section), our most material change was to account for economic losers. We account for economic losers by adjusting the size of the effect from the 20-year Busia follow-up (if you would like to see the calculations you can look at this cell, but it may be easier to follow our rationale for the calculation in this section). This “adjusted” effect size is then combined with the weight/cognition/schooling components in a Bayesian analysis.
As noted in the post, we believe others could reasonably disagree with the details of how we calculated this “adjusted” effect size, particularly the details around how the figures relating to wage-employed and self-employed people were used to calibrate the size of the adjustment. We do however think the outcome of the calculation gets us to an adjustment which is materially higher than the 3% adjustment used by GiveWell, which was the main aim of calculating the adjustment.