I’m interested in what you think about using subjective confidence intervals to estimate effectiveness of charities and then comparing them. To account for the optimizer’s curse, we can penalize charities that have wider confidence intervals. Not sure how it would be done in practice, but there probably is a mathematical method to calculate how much they should be penalized. Confidence intervals communicate both, value and uncertainty at the same time and therefore avoid some of the problems that you talk about.
I’m interested in what you think about using subjective confidence intervals to estimate effectiveness of charities and then comparing them. To account for the optimizer’s curse, we can penalize charities that have wider confidence intervals. Not sure how it would be done in practice, but there probably is a mathematical method to calculate how much they should be penalized. Confidence intervals communicate both, value and uncertainty at the same time and therefore avoid some of the problems that you talk about.