Benchmarks will become useless due to eval awareness¹
It seems like there are still hard problems that are easy to verify once you have the answer, so knowing it’s an eval doesn’t make it useless.
It seems like there are still hard problems that are easy to verify once you have the answer, so knowing it’s an eval doesn’t make it useless.