Even if it is self-defeating to request evidence in some respect (even if you have evidence for evidence, do you have evidence for evidence for evidence?), the opposite position “we need absolutely no evidence” is also flatly ridiculous (though, admittedly, not self-defeating) and we end up chasing after Pascal’s Mugging.
So we need some evidence, clearly—the question is, how much?
This is a matter of epistemology and priors, but I think the “evidence crowd” has done quite well. Remember that EA was founded on the “evidence crowd”. The crowd that didn’t request evidence gave us Play Pumps, whereas the crowd that did request evidence has saved thousands of lives through AMF.
But what about moonshots? How much evidence went into eradicating smallpox or producing vaccines? Is GiveWell underinvesting in these sorts of things? I don’t really know personally and haven’t thought about it much, but it’s possible.
I still think, as I’ve argued before, that it really comes down to what the optimal strategy is under uncertainty and I believe that game theory has established the optimal strategy is to do an explore-exploit pattern, where you start out by exploring the space widely and then slowly transition into exploiting the best things found so far.
I think this explore-exploit is done very well by Good Ventures and GiveWell with OpenPhil doing a lot of the exploring but still exploiting the best so far by investing in AMF.
However, it’s pointless to explore unless you actually learn—otherwise you’re not really exploring, you’re just exploiting randomly, which is even worse. And the way you learn is by collecting evidence—evidence that is high-quality enough that you can use it to improve your future actions.
Does this need to be 22 RCTs, as which backs the idea of distributing malaria nets? No—notably, GiveWell still gives millions to orgs that have much less RCTs and there still are several outstanding questions about AMF’s impact.
But on the other hand I see EAs acting without thinking statistically about their activities, running surveys, and showing any signs of skepticism in their work.
I think the “evidence crowd” has done quite well. Remember that EA was founded on the “evidence crowd”. The crowd that didn’t request evidence gave us Play Pumps, whereas the crowd that did request evidence has saved thousands of lives through AMF.
This seems really hard to settle.
Arguably, all of medical research is in the non evidence crowd in the sense that they can’t have empirical evidence that a research program is going to work ahead of time. You can work out whether a medical intervention works later, but you have to invest hugely up front. Medical research has done an absolutely huge amount of good, far more than people focused on scaling up evidence-backed charity and government programs (so far).
Who’s done the most good doesn’t settle it. We want to know something more like who’s done the most good per unit of input. But even then the average biomedical research does pretty well.
You could also put all social movements in the non evidence crowd, including the end of slavery, expansion of the vote, civil rights etc. Likewise, all technological and political innovation.
Perhaps the odd playpump is worth it? And playpump isn’t a good example because sensible non-evidence people would rule it out as well. Looking at facts like how they cost 4x more than regular water pumps and the recipients didn’t want them should make anyone cautious about scaling them up, whether or not you’ve got an RCT.
In the meantime, I agree explore-exploit is a good approach. I’d also say being modest about which causes and methods are best. Expert common sense as a prior seems to be significant weight on things like research, politics, innovation, and social advocacy being major ways to make the world a better place. Finally, there’s looking for other arguments, such as considerations around neglectedness.
[My mini defense of “Charity Science”-style empirical EA]
Earlier I said I was skeptical of uncertain causes. I still think this approach is largely correct (though with some revisions) and I have hoped for quite some time to elaborate again in length.
Even if it is self-defeating to request evidence in some respect (even if you have evidence for evidence, do you have evidence for evidence for evidence?), the opposite position “we need absolutely no evidence” is also flatly ridiculous (though, admittedly, not self-defeating) and we end up chasing after Pascal’s Mugging.
So we need some evidence, clearly—the question is, how much?
This is a matter of epistemology and priors, but I think the “evidence crowd” has done quite well. Remember that EA was founded on the “evidence crowd”. The crowd that didn’t request evidence gave us Play Pumps, whereas the crowd that did request evidence has saved thousands of lives through AMF.
But what about moonshots? How much evidence went into eradicating smallpox or producing vaccines? Is GiveWell underinvesting in these sorts of things? I don’t really know personally and haven’t thought about it much, but it’s possible.
I still think, as I’ve argued before, that it really comes down to what the optimal strategy is under uncertainty and I believe that game theory has established the optimal strategy is to do an explore-exploit pattern, where you start out by exploring the space widely and then slowly transition into exploiting the best things found so far.
I think this explore-exploit is done very well by Good Ventures and GiveWell with OpenPhil doing a lot of the exploring but still exploiting the best so far by investing in AMF.
However, it’s pointless to explore unless you actually learn—otherwise you’re not really exploring, you’re just exploiting randomly, which is even worse. And the way you learn is by collecting evidence—evidence that is high-quality enough that you can use it to improve your future actions.
Does this need to be 22 RCTs, as which backs the idea of distributing malaria nets? No—notably, GiveWell still gives millions to orgs that have much less RCTs and there still are several outstanding questions about AMF’s impact.
But on the other hand I see EAs acting without thinking statistically about their activities, running surveys, and showing any signs of skepticism in their work.
This seems really hard to settle.
Arguably, all of medical research is in the non evidence crowd in the sense that they can’t have empirical evidence that a research program is going to work ahead of time. You can work out whether a medical intervention works later, but you have to invest hugely up front. Medical research has done an absolutely huge amount of good, far more than people focused on scaling up evidence-backed charity and government programs (so far).
Who’s done the most good doesn’t settle it. We want to know something more like who’s done the most good per unit of input. But even then the average biomedical research does pretty well.
https://80000hours.org/career-guide/top-careers/profiles/biomedical-research/
You could also put all social movements in the non evidence crowd, including the end of slavery, expansion of the vote, civil rights etc. Likewise, all technological and political innovation.
Perhaps the odd playpump is worth it? And playpump isn’t a good example because sensible non-evidence people would rule it out as well. Looking at facts like how they cost 4x more than regular water pumps and the recipients didn’t want them should make anyone cautious about scaling them up, whether or not you’ve got an RCT.
In the meantime, I agree explore-exploit is a good approach. I’d also say being modest about which causes and methods are best. Expert common sense as a prior seems to be significant weight on things like research, politics, innovation, and social advocacy being major ways to make the world a better place. Finally, there’s looking for other arguments, such as considerations around neglectedness.