RSS

Op­ti­mizer’s curse

TagLast edit: Sep 8, 2022, 9:51 AM by Pablo

The optimizer’s curse is the phenomenon that causes an agent optimizing over a set set of options with uncertain value to systematically overestimate the value of the chosen option.

Illustration

The authors of the paper that originally introduced the optimizer’s curse provide a simple illustration of the phenomenon:[1]

suppose that we evaluate three alternatives that all have true values of exactly zero. The value of each alternative is estimated and the estimates are independent and normally distributed with mean equal to the true value of zero (they are conditionally unbiased) and a standard deviation of one. Selecting the highest value estimate then amounts to selecting the maximum of three draws from a standard normal distribution. The distribution of this maximal value estimate is easily determined by simulation or using results from order statistics and is displayed in Figure 1. The mean of this distribution is 0.85, so in this case, the expected disappointment,, is 0.85.

From Smith, James E. & Robert L. Winkler (2006) The optimizer’s curse: skepticism and postdecision surprise in decision analysis, Management Science, vol. 52, p. 312.

Further reading

Muehlhauser, Luke (2011) The optimizer’s curse and how to beat it, LessWrong, September 15.

Smith, James E. & Robert L. Winkler (2006) The optimizer’s curse: skepticism and postdecision surprise in decision analysis, Management Science, vol. 52, pp. 311–322.

Related entries

expected value | unilateralist’s curse

  1. ^

    Smith, James E. & Robert L. Winkler (2006) The optimizer’s curse: skepticism and postdecision surprise in decision analysis, Management Science, vol. 52, pp. 312.

[Link] The Op­ti­mizer’s Curse & Wrong-Way Reductions

Chris SmithApr 4, 2019, 1:28 PM
94 points
61 comments1 min readEA link

How much do you be­lieve your re­sults?

Eric NeymanMay 5, 2023, 7:51 PM
211 points
14 comments1 min readEA link

Po­ten­tial down­sides of us­ing ex­plicit probabilities

MichaelA🔸Jan 20, 2020, 2:14 AM
57 points
22 comments18 min readEA link

The Multi-Class Op­ti­mizer’s Curse

FeelingSentientJul 4, 2021, 12:04 PM
23 points
0 comments2 min readEA link

When should EAs al­lo­cate fund­ing ran­domly? An in­con­clu­sive liter­a­ture re­view.

Max_DanielNov 17, 2018, 2:53 PM
42 points
10 comments41 min readEA link

Don’t Over-Op­ti­mize Things

Owen Cotton-BarrattJun 16, 2022, 4:28 PM
53 points
7 comments4 min readEA link

List of ways in which cost-effec­tive­ness es­ti­mates can be misleading

sauliusAug 20, 2019, 6:05 PM
236 points
37 comments13 min readEA link

Con­trol­ling for a thinker’s big idea

Vasco Grilo🔸Oct 21, 2023, 7:56 AM
60 points
11 comments8 min readEA link
(magnusvinding.com)

Wide­spread val­ues brainstorming

brb243Mar 15, 2022, 12:45 PM
9 points
2 comments1 min readEA link

Why GiveWell should use com­plete un­cer­tainty quantification

TanaeDec 27, 2022, 8:11 PM
32 points
1 comment1 min readEA link
(suboptimal.substack.com)

Se­quence think­ing vs. cluster thinking

GiveWellJul 25, 2016, 10:43 AM
17 points
0 comments28 min readEA link
(blog.givewell.org)