In the your literature review you summarize the Smith and Winkler (2006) paper as “Prove that nonrandom, non-Bayesian decision strategies systematically overestimate the value of the selected option.”
On first sight, this claim seems like it might be stronger than the claim i have taken away from the paper (which is similar to what you write later in the text): if your decision strategy is to just choose the option you (naively) expect to be best, you will systematically overestimate the value of the selected option.
If you think the first claim is implied by the second (or something in the paper i missed) in some sense, i’d love to learn about your arguments!
“In fact, I believe that choosing the winning option does maximize expected value if all measurements are unbiased and their reliability doesn’t vary too much.”
I think you are basically right, but the amount of available options also plays a role here. If you consider a lot of non-optimal options, for which your measurements are only slightly noisier than for the best option, you can still systematically underselect the best option. (For example, simulations suggest that with 99 N(0,1.1) and 1 N(0.1,1) variables, the last one will only be maximal among the 100 only 0.7% of the time, despite having the highest expected value).
In this case, randomly taking one option would in fact have a higher expected value. (But it still seems very unclear, how one would identify similar situations in reality, even if they existed).
Some combination of moderately varying noise and lots of options seems like the most plausible condition, under which not taking the winning option might be better for some real world decisions.
On your first point: I agree that the paper just shows that, as you wrote, “if your decision strategy is to just choose the option you (naively) expect to be best, you will systematically overestimate the value of the selected option”.
I also think that “just choose the option you (naively) expect to be best” is an example of a “nonrandom, non-Bayesian decision strategy”. Now, the first sentence you quoted might reasonably be read to make the stronger claim that all nonrandom, non-Bayesian decision strategies have a certain property. However, the paper actually just shows that one of them does.
Is this what you were pointing to? If so, I’ll edit the quoted sentence accordingly, but I first wanted to check if I understood you correctly.
Very interesting!
In the your literature review you summarize the Smith and Winkler (2006) paper as “Prove that nonrandom, non-Bayesian decision strategies systematically overestimate the value of the selected option.”
On first sight, this claim seems like it might be stronger than the claim i have taken away from the paper (which is similar to what you write later in the text): if your decision strategy is to just choose the option you (naively) expect to be best, you will systematically overestimate the value of the selected option.
If you think the first claim is implied by the second (or something in the paper i missed) in some sense, i’d love to learn about your arguments!
“In fact, I believe that choosing the winning option does maximize expected value if all measurements are unbiased and their reliability doesn’t vary too much.”
I think you are basically right, but the amount of available options also plays a role here. If you consider a lot of non-optimal options, for which your measurements are only slightly noisier than for the best option, you can still systematically underselect the best option. (For example, simulations suggest that with 99 N(0,1.1) and 1 N(0.1,1) variables, the last one will only be maximal among the 100 only 0.7% of the time, despite having the highest expected value).
In this case, randomly taking one option would in fact have a higher expected value. (But it still seems very unclear, how one would identify similar situations in reality, even if they existed).
Some combination of moderately varying noise and lots of options seems like the most plausible condition, under which not taking the winning option might be better for some real world decisions.
On your first point: I agree that the paper just shows that, as you wrote, “if your decision strategy is to just choose the option you (naively) expect to be best, you will systematically overestimate the value of the selected option”.
I also think that “just choose the option you (naively) expect to be best” is an example of a “nonrandom, non-Bayesian decision strategy”. Now, the first sentence you quoted might reasonably be read to make the stronger claim that all nonrandom, non-Bayesian decision strategies have a certain property. However, the paper actually just shows that one of them does.
Is this what you were pointing to? If so, I’ll edit the quoted sentence accordingly, but I first wanted to check if I understood you correctly.
In any case, thank you for your comment!
Yes, exactly. When first reading your summary i interpreted it as the “for all” claim.
Ok, thanks, I now say “Prove that a certain nonrandom, non-Bayesian …”.
On your second point: I think you’re right, and that’s a great example. I’ve added a link to your comment to the post.