Benchmarks will become useless due to eval awareness¹
The question isn’t whether benchmarks will become useless. The question is, “is the probability high enough that we can’t count on benchmarks?” To which the answer is yes.
This is a good way of looking at. I think this may not apply for propensity evals if there are ways eval-awareness does not equal eval gaming which often are not the same thing
The question isn’t whether benchmarks will become useless. The question is, “is the probability high enough that we can’t count on benchmarks?” To which the answer is yes.
This is a good way of looking at. I think this may not apply for propensity evals if there are ways eval-awareness does not equal eval gaming which often are not the same thing