Evals are being gamed not because the methodology is insufficient but the models on which the compliance audit run are sophisticated enough to game the audit. IC methodology already solved the problem of denied human capabilities through triangulation by using independent behavioural signals not better direct elicitation .The AI safety community needs to make the same epistemological shift. The question isn’t how to make evals harder to game, it’s whether evals are the right instrument at all.
Evals are being gamed not because the methodology is insufficient but the models on which the compliance audit run are sophisticated enough to game the audit.
IC methodology already solved the problem of denied human capabilities through triangulation by using independent behavioural signals not better direct elicitation .The AI safety community needs to make the same epistemological shift.
The question isn’t how to make evals harder to game, it’s whether evals are the right instrument at all.