I second the importance of these three terms/concepts.
In case anyone stumbles across this post in future, here are two sources on the optimizer’s curse. (I thought the first was great. I personally disagree with the second on various points, but mention it as other people seemed to find it good and there’s good discussion in the comments.)
Optimisers curse / Regression to the mean
On how trying to optimise can lead you to make mistakes
Related: Goodhart’s Law
“When a measure becomes a target, it ceases to be a good measure”
https://en.m.wikipedia.org/wiki/Goodhart’s_law
I second the importance of these three terms/concepts.
In case anyone stumbles across this post in future, here are two sources on the optimizer’s curse. (I thought the first was great. I personally disagree with the second on various points, but mention it as other people seemed to find it good and there’s good discussion in the comments.)
And here are some sources on Goodhart’s law.