I second the importance of these three terms/concepts.
In case anyone stumbles across this post in future, here are two sources on the optimizer’s curse. (I thought the first was great. I personally disagree with the second on various points, but mention it as other people seemed to find it good and there’s good discussion in the comments.)
Related: Goodhart’s Law
“When a measure becomes a target, it ceases to be a good measure”
https://en.m.wikipedia.org/wiki/Goodhart’s_law
I second the importance of these three terms/concepts.
In case anyone stumbles across this post in future, here are two sources on the optimizer’s curse. (I thought the first was great. I personally disagree with the second on various points, but mention it as other people seemed to find it good and there’s good discussion in the comments.)
And here are some sources on Goodhart’s law.