RSS

Op­ti­mizer’s curse

TagLast edit: 8 Sep 2022 9:51 UTC by Pablo

The optimizer’s curse is the phenomenon that causes an agent optimizing over a set set of options with uncertain value to systematically overestimate the value of the chosen option.

Illustration

The authors of the paper that originally introduced the optimizer’s curse provide a simple illustration of the phenomenon:[1]

suppose that we evaluate three alternatives that all have true values of exactly zero. The value of each alternative is estimated and the estimates are independent and normally distributed with mean equal to the true value of zero (they are conditionally unbiased) and a standard deviation of one. Selecting the highest value estimate then amounts to selecting the maximum of three draws from a standard normal distribution. The distribution of this maximal value estimate is easily determined by simulation or using results from order statistics and is displayed in Figure 1. The mean of this distribution is 0.85, so in this case, the expected disappointment,, is 0.85.

From Smith, James E. & Robert L. Winkler (2006) The optimizer’s curse: skepticism and postdecision surprise in decision analysis, Management Science, vol. 52, p. 312.

Further reading

Muehlhauser, Luke (2011) The optimizer’s curse and how to beat it, LessWrong, September 15.

Smith, James E. & Robert L. Winkler (2006) The optimizer’s curse: skepticism and postdecision surprise in decision analysis, Management Science, vol. 52, pp. 311–322.

Related entries

expected value | unilateralist’s curse

  1. ^

    Smith, James E. & Robert L. Winkler (2006) The optimizer’s curse: skepticism and postdecision surprise in decision analysis, Management Science, vol. 52, pp. 312.

[Link] The Op­ti­mizer’s Curse & Wrong-Way Reductions

Chris Smith4 Apr 2019 13:28 UTC
94 points
61 comments1 min readEA link

How much do you be­lieve your re­sults?

Eric Neyman5 May 2023 19:51 UTC
211 points
14 comments1 min readEA link

Po­ten­tial down­sides of us­ing ex­plicit probabilities

MichaelA20 Jan 2020 2:14 UTC
57 points
22 comments18 min readEA link

The Multi-Class Op­ti­mizer’s Curse

FeelingSentient4 Jul 2021 12:04 UTC
23 points
0 comments2 min readEA link

When should EAs al­lo­cate fund­ing ran­domly? An in­con­clu­sive liter­a­ture re­view.

Max_Daniel17 Nov 2018 14:53 UTC
42 points
10 comments41 min readEA link

Don’t Over-Op­ti­mize Things

Owen Cotton-Barratt16 Jun 2022 16:28 UTC
53 points
7 comments4 min readEA link

List of ways in which cost-effec­tive­ness es­ti­mates can be misleading

saulius20 Aug 2019 18:05 UTC
234 points
37 comments13 min readEA link

Con­trol­ling for a thinker’s big idea

Vasco Grilo🔸21 Oct 2023 7:56 UTC
60 points
11 comments8 min readEA link
(magnusvinding.com)

Wide­spread val­ues brainstorming

brb24315 Mar 2022 12:45 UTC
9 points
2 comments1 min readEA link

Why GiveWell should use com­plete un­cer­tainty quantification

Tanae27 Dec 2022 20:11 UTC
32 points
1 comment1 min readEA link
(suboptimal.substack.com)

Se­quence think­ing vs. cluster thinking

GiveWell25 Jul 2016 10:43 UTC
17 points
0 comments28 min readEA link
(blog.givewell.org)