Good observation, and I would add one other implication: this distribution of returns supports “hits-based evaluation” (in the spirit of hits-based giving) for interventions in global health and development. For example, consider two strategies for spending a $10 million on RCTs to evaluate some intervention:
we can evaluate 10 interventions for $1 million each, developing large samples with high statistical power.
we can evaluate 100 interventions for $100,000 each, developing smaller samples with lower statistical power.
The traditional scientific/economic approach favors 1), since it values precisely estimating the effects of an intervention. But we care about identifying the most effective interventions—and if the returns follow an exponential distribution, the biggest effects are an order of magnitude larger than the average effects, so they will be detected even with low statistical power. Instead, we would rather maximize coverage of interventions to maximize our probably of evaluating those high-impact interventions. If the casualty is that 60 interventions with modest effects go undetected, so be it.
Good observation, and I would add one other implication: this distribution of returns supports “hits-based evaluation” (in the spirit of hits-based giving) for interventions in global health and development. For example, consider two strategies for spending a $10 million on RCTs to evaluate some intervention:
we can evaluate 10 interventions for $1 million each, developing large samples with high statistical power.
we can evaluate 100 interventions for $100,000 each, developing smaller samples with lower statistical power.
The traditional scientific/economic approach favors 1), since it values precisely estimating the effects of an intervention. But we care about identifying the most effective interventions—and if the returns follow an exponential distribution, the biggest effects are an order of magnitude larger than the average effects, so they will be detected even with low statistical power. Instead, we would rather maximize coverage of interventions to maximize our probably of evaluating those high-impact interventions. If the casualty is that 60 interventions with modest effects go undetected, so be it.
Really interesting, thanks for pointing this out!