What did you have in mind as unconventional benchmarks? There’s a lot of different places you could take benchmarks in the future and people have different ideas on what would be useful
blinded continual evaluation, so that eval awareness is rendered useless as a factor since the model is always being evaluated. A particular favorite implementation of mine would be adversarial proposal markets. I think this mode will be needed for RSI anyway and caps the eval awareness compute tax at something reasonable like 2%
conventional benchmarks will become less useful due to eval awareness
What did you have in mind as unconventional benchmarks? There’s a lot of different places you could take benchmarks in the future and people have different ideas on what would be useful
blinded continual evaluation, so that eval awareness is rendered useless as a factor since the model is always being evaluated. A particular favorite implementation of mine would be adversarial proposal markets. I think this mode will be needed for RSI anyway and caps the eval awareness compute tax at something reasonable like 2%