This is a good analysis. But I’m disappointed by how little there is to back the economic losers claim. The way I understand it, your evidence for the economic losers claim is
Treated children work fewer hours in agriculture
There is no other good explanation for why long term effects are large when short term effects are small
Both of these are enough to make the economic losers argument an interesting conjecture. Neither are enough to show it is true. They do not explain why labor market competition is the most likely channel for what we observe. There are many other plausible stories: treated children could be more likely to migrate for work in other markets, they could be more likely to start independent businesses, etc. These would all generate the same result without enhancing labor market competition. Indeed, if deworming led to increased consumption --> increased demand for goods --> increased production of goods, then it might even have increased jobs for control children.
Are any of these conjectures grounded in evidence? No. But neither is the economic losers story. We can simply trade pet theories as to what is going on, but without a way to adjudicate between these theories, it seems unprincipled to apply an ad hoc adjustment for one possible factor. At the end of the day, your strongest reason for adjusting is “the number looks too high to be plausible”, which is exactly what GiveWell’s adjustment does.
I believe that someone who had read through all of this would have been less disappointed, however it’s also not reasonable to expect a reader to wade through that much detail.
The amount of detail that we should put in this summarised EA Forum and the extent to which we should keep this post short was something we struggled with. It may be that we got the judgement wrong this time.
We have not claimed that labour market competition is the most likely channel for what we observe.
The details here slightly subtle.
The end result of our analysis is not a dramatic reappraisal of deworming—the cost-effectiveness metrics are non-trivially lower than GiveWell’s, but still close to the bar for GiveWell recommended status.
If we believed that labour market competition / economic losers were the most likely channel for the collection of evidence that we see, we would have applied a more dramatic adjustment, with more dramatic conclusions.
We agree with your claim that labour market effects could include positive spillovers as well as negative ones, and this point is discussed in our fuller analysis document. However, our claim is that given that we see the evidence we see, it is more likely that the negative spillovers (economic losers) outweigh the positive ones, and to a more material degree than GiveWell suggests.
As our replication of the replicability adjustment indicates, and as we understand from our conversations with GiveWell, it is not true to say that GiveWell’s adjustment accounts for this already...
… instead there is a Bayesian method which explicitly accounts for different considerations, and GiveWell’s version of this does not reflect the economic losers consideration, and our replicability adjustment does. (I’ve glossed over some subtleties, full details can be found in the relevant document.) Hence our perspective being different from GiveWell’s.
So I went over the additional documents and I owe you an apology for being dismissive. There is indeed more to the analysis than I thought, and it was flippant to suggest that your or GiveWell’s replicability adjustment was just “this number looks too high” and thus incorporates this. Having gone through the replicability adjustment document, I think it makes a lot more sense than I gave it credit for.
What I couldn’t gather from the document was where exactly you differed from GiveWell. Is it only in the economic losers weighting? Were your components from weight gain, years of schooling and cognition the same as GiveWell’s? In the sheet where you calculate the replicability adjustment, there is no factoring of economic losers as far as I can tell, so in order to arrive at the new replicability adjustment you must have had to differ from GiveWell in the mechanism adjustment, right?
Hi Karthik, we’re glad to hear our additional documents provided useful detail. We apologise if our deviation from GiveWell wasn’t clear, but hopefully our explanation below is clarifying.
While we made a few minor amendments to GiveWell’s replicability adjustment, including to the weight/cognition/schooling components (as outlined in this section), our most material change was to account for economic losers. We account for economic losers by adjusting the size of the effect from the 20-year Busia follow-up (if you would like to see the calculations you can look at this cell, but it may be easier to follow our rationale for the calculation in this section). This “adjusted” effect size is then combined with the weight/cognition/schooling components in a Bayesian analysis.
As noted in the post, we believe others could reasonably disagree with the details of how we calculated this “adjusted” effect size, particularly the details around how the figures relating to wage-employed and self-employed people were used to calibrate the size of the adjustment. We do however think the outcome of the calculation gets us to an adjustment which is materially higher than the 3% adjustment used by GiveWell, which was the main aim of calculating the adjustment.
This is a good analysis. But I’m disappointed by how little there is to back the economic losers claim. The way I understand it, your evidence for the economic losers claim is
Treated children work fewer hours in agriculture
There is no other good explanation for why long term effects are large when short term effects are small
Both of these are enough to make the economic losers argument an interesting conjecture. Neither are enough to show it is true. They do not explain why labor market competition is the most likely channel for what we observe. There are many other plausible stories: treated children could be more likely to migrate for work in other markets, they could be more likely to start independent businesses, etc. These would all generate the same result without enhancing labor market competition. Indeed, if deworming led to increased consumption --> increased demand for goods --> increased production of goods, then it might even have increased jobs for control children.
Are any of these conjectures grounded in evidence? No. But neither is the economic losers story. We can simply trade pet theories as to what is going on, but without a way to adjudicate between these theories, it seems unprincipled to apply an ad hoc adjustment for one possible factor. At the end of the day, your strongest reason for adjusting is “the number looks too high to be plausible”, which is exactly what GiveWell’s adjustment does.
Thank you for your comment.
I’m sorry you were disappointed by the lack of detail on this post.
The full analysis for this includes a longer document which provides more details on the analysis, a supplementary analysis, a document which sets out the details behind our full replication of the replicability adjustment that GiveWell applies.
I believe that someone who had read through all of this would have been less disappointed, however it’s also not reasonable to expect a reader to wade through that much detail.
The amount of detail that we should put in this summarised EA Forum and the extent to which we should keep this post short was something we struggled with. It may be that we got the judgement wrong this time.
We have not claimed that labour market competition is the most likely channel for what we observe.
The details here slightly subtle.
The end result of our analysis is not a dramatic reappraisal of deworming—the cost-effectiveness metrics are non-trivially lower than GiveWell’s, but still close to the bar for GiveWell recommended status.
If we believed that labour market competition / economic losers were the most likely channel for the collection of evidence that we see, we would have applied a more dramatic adjustment, with more dramatic conclusions.
We agree with your claim that labour market effects could include positive spillovers as well as negative ones, and this point is discussed in our fuller analysis document. However, our claim is that given that we see the evidence we see, it is more likely that the negative spillovers (economic losers) outweigh the positive ones, and to a more material degree than GiveWell suggests.
As our replication of the replicability adjustment indicates, and as we understand from our conversations with GiveWell, it is not true to say that GiveWell’s adjustment accounts for this already...
… instead there is a Bayesian method which explicitly accounts for different considerations, and GiveWell’s version of this does not reflect the economic losers consideration, and our replicability adjustment does. (I’ve glossed over some subtleties, full details can be found in the relevant document.) Hence our perspective being different from GiveWell’s.
So I went over the additional documents and I owe you an apology for being dismissive. There is indeed more to the analysis than I thought, and it was flippant to suggest that your or GiveWell’s replicability adjustment was just “this number looks too high” and thus incorporates this. Having gone through the replicability adjustment document, I think it makes a lot more sense than I gave it credit for.
What I couldn’t gather from the document was where exactly you differed from GiveWell. Is it only in the economic losers weighting? Were your components from weight gain, years of schooling and cognition the same as GiveWell’s? In the sheet where you calculate the replicability adjustment, there is no factoring of economic losers as far as I can tell, so in order to arrive at the new replicability adjustment you must have had to differ from GiveWell in the mechanism adjustment, right?
Hi Karthik, we’re glad to hear our additional documents provided useful detail. We apologise if our deviation from GiveWell wasn’t clear, but hopefully our explanation below is clarifying.
While we made a few minor amendments to GiveWell’s replicability adjustment, including to the weight/cognition/schooling components (as outlined in this section), our most material change was to account for economic losers. We account for economic losers by adjusting the size of the effect from the 20-year Busia follow-up (if you would like to see the calculations you can look at this cell, but it may be easier to follow our rationale for the calculation in this section). This “adjusted” effect size is then combined with the weight/cognition/schooling components in a Bayesian analysis.
As noted in the post, we believe others could reasonably disagree with the details of how we calculated this “adjusted” effect size, particularly the details around how the figures relating to wage-employed and self-employed people were used to calibrate the size of the adjustment. We do however think the outcome of the calculation gets us to an adjustment which is materially higher than the 3% adjustment used by GiveWell, which was the main aim of calculating the adjustment.