Executive summary: When you rank interventions by noisy estimates and pick the top one, you systematically overestimate its impact and bias toward more uncertain options, but a simple Bayesian shrinkage correction can reduce this effect in a toy model, though applying it in practice is difficult.
Key points:
The optimiser’s curse shows that selecting the intervention with the highest estimated value will, in many normal situations, both overestimate its true impact and favor more uncertain interventions.
In a toy model where true effects are normally distributed with mean 0 and SD 100 and errors are normally distributed with mean 0 and SD 50, the top-ranked intervention is overestimated by about 50 lives in the median case, roughly a 25% overestimate.
When speculative interventions have error spreads four times larger than grounded ones but identical true-effect distributions, the speculative option is chosen 93% of the time and is usually the wrong choice, while ignoring speculative options yields nearly twice the average lives saved.
A Bayesian correction from Smith and Winkler shrinks estimates toward a prior mean using a factor α = 1/(1 + (σ_V/σ_μ)^2), which in the toy model eliminates systematic overestimation and improves average performance.
Implementing such corrections in practice is hard because the true spread of intervention effects, the spread and correlation of errors, distribution shapes, and post-selection scrutiny are all difficult to estimate.
GiveWell does not explicitly apply an optimiser’s curse adjustment but uses measures such as a “replicability adjustment” (e.g., multiplying deworming estimates by 0.13) and focusing on interventions with strong RCT evidence, which the author argues may partially but not fully address the selection effect.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, andcontact us if you have feedback.
Executive summary: When you rank interventions by noisy estimates and pick the top one, you systematically overestimate its impact and bias toward more uncertain options, but a simple Bayesian shrinkage correction can reduce this effect in a toy model, though applying it in practice is difficult.
Key points:
The optimiser’s curse shows that selecting the intervention with the highest estimated value will, in many normal situations, both overestimate its true impact and favor more uncertain interventions.
In a toy model where true effects are normally distributed with mean 0 and SD 100 and errors are normally distributed with mean 0 and SD 50, the top-ranked intervention is overestimated by about 50 lives in the median case, roughly a 25% overestimate.
When speculative interventions have error spreads four times larger than grounded ones but identical true-effect distributions, the speculative option is chosen 93% of the time and is usually the wrong choice, while ignoring speculative options yields nearly twice the average lives saved.
A Bayesian correction from Smith and Winkler shrinks estimates toward a prior mean using a factor α = 1/(1 + (σ_V/σ_μ)^2), which in the toy model eliminates systematic overestimation and improves average performance.
Implementing such corrections in practice is hard because the true spread of intervention effects, the spread and correlation of errors, distribution shapes, and post-selection scrutiny are all difficult to estimate.
GiveWell does not explicitly apply an optimiser’s curse adjustment but uses measures such as a “replicability adjustment” (e.g., multiplying deworming estimates by 0.13) and focusing on interventions with strong RCT evidence, which the author argues may partially but not fully address the selection effect.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.