Hey, thanks for the question! I’m Alex Cohen, a researcher at GiveWell, and wanted to chime in.
We did say we’d include a 25th/75th percentile range on bottom line cost-effectiveness (in addition to the one-way sensitivity checks). We haven’t added that yet, and we should. We ran into some issues running the full sensitivity analyses (instead of the one-way sensitivity checks we do have), and we prioritized publishing updated intervention reports and cost-effective analyses without them.
We’ll add those percentile ranges to our top charity intervention reports (so the simple cost-effective analyses will also include a bottom line cost-effectiveness 25⁄75 range, in addition to one-way sensitivity checks) and ensure that new intervention reports/grant pages have them included before publishing. We think it’s worth emphasizing how uncertain our cost-effectiveness estimates are, and this is one way to do that (though it has limitations).
We’re still not planning to base our decision-making on this uncertainty in the bottom line cost-effectiveness (like the “Change Our Mind Contest” post recommended) or model uncertainty on every parameter. To defend against the Optimizer’s Curse, we prefer our approach of skeptically adjusting our inputs, rather than an all-in adjustment to bottom-line cost-effectiveness. We explain why in the uncertainty post.
Really appreciate you raising this. Sorry this has taken so long, and grateful for the nudge!
Hey, thanks for the question! I’m Alex Cohen, a researcher at GiveWell, and wanted to chime in.
We did say we’d include a 25th/75th percentile range on bottom line cost-effectiveness (in addition to the one-way sensitivity checks). We haven’t added that yet, and we should. We ran into some issues running the full sensitivity analyses (instead of the one-way sensitivity checks we do have), and we prioritized publishing updated intervention reports and cost-effective analyses without them.
We’ll add those percentile ranges to our top charity intervention reports (so the simple cost-effective analyses will also include a bottom line cost-effectiveness 25⁄75 range, in addition to one-way sensitivity checks) and ensure that new intervention reports/grant pages have them included before publishing. We think it’s worth emphasizing how uncertain our cost-effectiveness estimates are, and this is one way to do that (though it has limitations).
We’re still not planning to base our decision-making on this uncertainty in the bottom line cost-effectiveness (like the “Change Our Mind Contest” post recommended) or model uncertainty on every parameter. To defend against the Optimizer’s Curse, we prefer our approach of skeptically adjusting our inputs, rather than an all-in adjustment to bottom-line cost-effectiveness. We explain why in the uncertainty post.
Really appreciate you raising this. Sorry this has taken so long, and grateful for the nudge!