Benchmarks will become useless due to eval awareness¹
Some might, but this seems unlikely to be universal. Since for instance it can’t really cheat at an evaluation of its ability to produce easily checkable mathematical proofs. None of that is to say that many very useful benchmarks might not become useless, though, just not benchmarks period.
Some might, but this seems unlikely to be universal. Since for instance it can’t really cheat at an evaluation of its ability to produce easily checkable mathematical proofs. None of that is to say that many very useful benchmarks might not become useless, though, just not benchmarks period.