You say that the distribution needs to be “very” fat tailed—implying that we have a decent chance of finding interventions order of mangitude more eefective than bed-nets. I disagree. The very most effective possible interventions, where the cost-benefit ratio is insanely large, are things that we don’t need to run as interventions. For instance, telling people to eat when they have food so they don’t starve would be really impactful if it weren’t unnecessary because of how obviously beneficial it is.
So I don’t think bednets are a massive outlier—they just have a relatively low saturation compared to most comparably effective interventions. The implication of my model is that most really effective interventions are saturated, often very quickly. Even expensive systemic efforts like vaccinations for smallpox got funded fairly rapidly after such universal eradication was possible, and the less used vaccines are either less effective, for less critical diseases, or are more expensive and/or harder to distribute. (And governments and foundations are running those campaigns, successfully, without needing EA pushing or funding.) And that’s why we see few very effective outliers—and since the underlying distribution isn’t fat tailed, even more effective interventions are even rarer, and those that did exist are gone very quickly.
On prediction, I agree that the conclusion is one of epistemic modesty rather than confident claims of non-effectiveness. But the practical implication of that modesty is that for any specific intervention, if we fund it thinking it may be really impactful, we’re incredibly unlikely to be correct.
Also, I’m far more skeptical than you about ‘sophisticated’ estimates. Having taken graduate courses in econometrics, I’ll say that the methods are sometimes really useful, but the assumptions never apply, and unless the system model is really fantastic, the prediction error once accounting for model specification uncertainty is large enough that most such econometric analyses of these sorts of really complex, poorly understood systems like corruption or poverty simply don’t say anything.
You say that the distribution needs to be “very” fat tailed—implying that we have a decent chance of finding interventions order of mangitude more eefective than bed-nets. I disagree. The very most effective possible interventions, where the cost-benefit ratio is insanely large, are things that we don’t need to run as interventions. For instance, telling people to eat when they have food so they don’t starve would be really impactful if it weren’t unnecessary because of how obviously beneficial it is.
So I don’t think bednets are a massive outlier—they just have a relatively low saturation compared to most comparably effective interventions. The implication of my model is that most really effective interventions are saturated, often very quickly. Even expensive systemic efforts like vaccinations for smallpox got funded fairly rapidly after such universal eradication was possible, and the less used vaccines are either less effective, for less critical diseases, or are more expensive and/or harder to distribute. (And governments and foundations are running those campaigns, successfully, without needing EA pushing or funding.) And that’s why we see few very effective outliers—and since the underlying distribution isn’t fat tailed, even more effective interventions are even rarer, and those that did exist are gone very quickly.
On prediction, I agree that the conclusion is one of epistemic modesty rather than confident claims of non-effectiveness. But the practical implication of that modesty is that for any specific intervention, if we fund it thinking it may be really impactful, we’re incredibly unlikely to be correct.
Also, I’m far more skeptical than you about ‘sophisticated’ estimates. Having taken graduate courses in econometrics, I’ll say that the methods are sometimes really useful, but the assumptions never apply, and unless the system model is really fantastic, the prediction error once accounting for model specification uncertainty is large enough that most such econometric analyses of these sorts of really complex, poorly understood systems like corruption or poverty simply don’t say anything.