Another point of agreement: the economics profession currently focuses too much on empirical work. Meanwhile my own personal view is that people like Esther and Chris B are slightly ‘too far’ in the pro-RCT camp, and that people like Lant (and you) are ‘too far’ in the anti-RCT camp. But I don’t see anyone in this discussion as being extreme (except possibly Lant...); healthy disagreement is to be expected and encouraged. Note that Esther and Abhijit’s most recent book tackles macro issues like migration, trade, climate change, and yes growth—using RCTs when possible / relevant but also plenty of other results (including lots of theory! Abhijit started life as a theorist, like I did). Meanwhile Chris has a forthcoming book on war and peace (macro level! no easy RCTs) for which he uses other approaches like machine learning. You can find all sorts of quotes, but the proof is in the pudding. Final point on this is that one can easily combine RCTs with admin data, ML, etc, and researchers (including me) are doing more and more of that, which imo is great—it’s not always one or the other.
As you say, the efficacy of deworming seems to be a point of disagreement between us. Again pulling back somewhat, you link to Eva’s paper as supporting your claim that RCTs have minimal external validity, but her paper is about all forms of impact evaluation (and she notes in the conclusion that the subset of RCTs aren’t special). So this would be extremely damning for economics if true, but her results don’t support your claim. For instance she notes that bednets and conditional cash transfers seem to do very well on this front. More relevantly, her point (as I read it) is to see how much of the nominal variation in effect sizes can be explained by other contextual variables, and she finds that typically a nontrivial amount of it can be. This is good news for external validity, since it means we can often explain / predict the differences even when they do arise.
I think I haven’t been very clear about ‘apples to oranges’ - I agree that these can & should absolutely be compared. I just felt like the way you were doing it glossed over an important difference. I can write a check to AMF and feel very confident that something will change in the world; we can then debate the expected magnitude of the impact of that change. But I can’t write a check to “growth reform in the developing world”, so even before we debate the relative benefits of changing immigration policy vs distributing bednets we have to calculate the probability that the desired policy will get implemented. I realize you’re fully aware of this, but that’s the part I keep coming back to because that’s the part where I’m pessimistic (partly having worked for the US government, although for a counterargument I liked this recent forum post) and suspect that our intuitions disagree, and mostly you keep talking about the benefits of more migration and of GDP growth (which are great!) and not so much about how we sit down and estimate the likelihoods of bringing those about. I’ll admit that the “pessimistic” estimate of 1% on ICRIER in the original post with Hauke really made me distrust everything afterward, since the pessimistic estimate in that case is a negative number and a plausible median estimate seems to be about 1 in a million.
On China I suppose my main point is still that I think it’s simply very very hard to quantitatively estimate most of this. Just because you (or I, or anyone) thinks that something is extremely conservative (when you admit you haven’t put in as much time on all this as you’d like, and indeed it’s not your job to do so) doesn’t make it so. In this specific case, if you forced me to take a stand, my best guess is to agree with you that economists have helped push policy in a better direction and that that made a big difference to global welfare. Even if I felt more confident about that, what is the counterfactual you are comparing to? Did some NGO or the WB cause that to happen on the margin, or would economists have tried to learn about the world and influence policy anyway? Are there similar opportunities going forward? The Taliban says they want economics expertise, so perhaps. But I don’t think we know the answers to these questions (yet), even within orders of magnitude, and whether or not this type of approach will beat RCT-type approaches depends entirely on those particular probabilities.
Another point of agreement: the economics profession currently focuses too much on empirical work. Meanwhile my own personal view is that people like Esther and Chris B are slightly ‘too far’ in the pro-RCT camp, and that people like Lant (and you) are ‘too far’ in the anti-RCT camp. But I don’t see anyone in this discussion as being extreme (except possibly Lant...); healthy disagreement is to be expected and encouraged. Note that Esther and Abhijit’s most recent book tackles macro issues like migration, trade, climate change, and yes growth—using RCTs when possible / relevant but also plenty of other results (including lots of theory! Abhijit started life as a theorist, like I did). Meanwhile Chris has a forthcoming book on war and peace (macro level! no easy RCTs) for which he uses other approaches like machine learning. You can find all sorts of quotes, but the proof is in the pudding. Final point on this is that one can easily combine RCTs with admin data, ML, etc, and researchers (including me) are doing more and more of that, which imo is great—it’s not always one or the other.
As you say, the efficacy of deworming seems to be a point of disagreement between us. Again pulling back somewhat, you link to Eva’s paper as supporting your claim that RCTs have minimal external validity, but her paper is about all forms of impact evaluation (and she notes in the conclusion that the subset of RCTs aren’t special). So this would be extremely damning for economics if true, but her results don’t support your claim. For instance she notes that bednets and conditional cash transfers seem to do very well on this front. More relevantly, her point (as I read it) is to see how much of the nominal variation in effect sizes can be explained by other contextual variables, and she finds that typically a nontrivial amount of it can be. This is good news for external validity, since it means we can often explain / predict the differences even when they do arise.
I think I haven’t been very clear about ‘apples to oranges’ - I agree that these can & should absolutely be compared. I just felt like the way you were doing it glossed over an important difference. I can write a check to AMF and feel very confident that something will change in the world; we can then debate the expected magnitude of the impact of that change. But I can’t write a check to “growth reform in the developing world”, so even before we debate the relative benefits of changing immigration policy vs distributing bednets we have to calculate the probability that the desired policy will get implemented. I realize you’re fully aware of this, but that’s the part I keep coming back to because that’s the part where I’m pessimistic (partly having worked for the US government, although for a counterargument I liked this recent forum post) and suspect that our intuitions disagree, and mostly you keep talking about the benefits of more migration and of GDP growth (which are great!) and not so much about how we sit down and estimate the likelihoods of bringing those about. I’ll admit that the “pessimistic” estimate of 1% on ICRIER in the original post with Hauke really made me distrust everything afterward, since the pessimistic estimate in that case is a negative number and a plausible median estimate seems to be about 1 in a million.
On China I suppose my main point is still that I think it’s simply very very hard to quantitatively estimate most of this. Just because you (or I, or anyone) thinks that something is extremely conservative (when you admit you haven’t put in as much time on all this as you’d like, and indeed it’s not your job to do so) doesn’t make it so. In this specific case, if you forced me to take a stand, my best guess is to agree with you that economists have helped push policy in a better direction and that that made a big difference to global welfare. Even if I felt more confident about that, what is the counterfactual you are comparing to? Did some NGO or the WB cause that to happen on the margin, or would economists have tried to learn about the world and influence policy anyway? Are there similar opportunities going forward? The Taliban says they want economics expertise, so perhaps. But I don’t think we know the answers to these questions (yet), even within orders of magnitude, and whether or not this type of approach will beat RCT-type approaches depends entirely on those particular probabilities.