There is a tension between different EA ideas here in my view. Early on, I recall, the emphasis was on how you need charity evaluators like GiveWell, and RTCs by randomista development economists, because you can’t predict what interventions will work well, or even do more good than harm, on the basis of common sense intuition. (I remember Will giving examples like “actually, having prisoners talk to children about how horrible being in prison is seems to make the children more likely to grow up to commit crimes.) But it turns out that when assessing interventions, there are always points where there just isn’t high quality data on whether some particular factor importantly reduces (or increases) the intervention’s effectiveness. So at that point, we have to rely on commonsense, mildly disciplined by back of the envelope calculations, possibly supplemented by poor quality data if we’re lucky. And then it feels unfair when this is criticized by outsiders (like the recent polemical anti-EA piece in Wired) because well, what else can you possibly do if high-quality studies aren’t available and it’s not feasible to do them yourself? But I guess from the outsider’s perspective, it’s easy to see why this looks like hypocrisy: they criticized other people for relying on their general hunches about how things work, but now the EAs are doing it themselves! I’m not really sure what the general solution (if any) to this is. But it does feel to my like there are a vast number of choice points in GiveWell’s analyses where they are mostly guessing, and if those guesses are all biased in some direction rather than uncorrelated, assessments of interventions will be way off.
There is a tension between different EA ideas here in my view. Early on, I recall, the emphasis was on how you need charity evaluators like GiveWell, and RTCs by randomista development economists, because you can’t predict what interventions will work well, or even do more good than harm, on the basis of common sense intuition. (I remember Will giving examples like “actually, having prisoners talk to children about how horrible being in prison is seems to make the children more likely to grow up to commit crimes.) But it turns out that when assessing interventions, there are always points where there just isn’t high quality data on whether some particular factor importantly reduces (or increases) the intervention’s effectiveness. So at that point, we have to rely on commonsense, mildly disciplined by back of the envelope calculations, possibly supplemented by poor quality data if we’re lucky. And then it feels unfair when this is criticized by outsiders (like the recent polemical anti-EA piece in Wired) because well, what else can you possibly do if high-quality studies aren’t available and it’s not feasible to do them yourself? But I guess from the outsider’s perspective, it’s easy to see why this looks like hypocrisy: they criticized other people for relying on their general hunches about how things work, but now the EAs are doing it themselves! I’m not really sure what the general solution (if any) to this is. But it does feel to my like there are a vast number of choice points in GiveWell’s analyses where they are mostly guessing, and if those guesses are all biased in some direction rather than uncorrelated, assessments of interventions will be way off.