I agree with some of this comment but I really don’t get the link to the paper you linked:
tendency to armchair analyze deep learning systems instead of having experiments drive decisions was historically off
The paper seems to mostly be evidence that the benchmarks that, you and other who have been focused on certain kinds of ML experiments have created, are not really helping much with AI alignment.
I also disagree some with the methodology of this paper, but I have trouble seeing how its evidence of people doing too much armchair analyzing, when as far as I can tell the flaws with these benchmarks were the result of people doing too much “IDK what alignment is, but maybe if we measure this vaguely related thing it will help” and too little “man, I should really understand what I would learn if this benchmark improved and whether it would cause me to actually update the system that has improved on this benchmark is more aligned and less likely to cause catastrophic consequences”.
I agree with some of this comment but I really don’t get the link to the paper you linked:
The paper seems to mostly be evidence that the benchmarks that, you and other who have been focused on certain kinds of ML experiments have created, are not really helping much with AI alignment.
I also disagree some with the methodology of this paper, but I have trouble seeing how its evidence of people doing too much armchair analyzing, when as far as I can tell the flaws with these benchmarks were the result of people doing too much “IDK what alignment is, but maybe if we measure this vaguely related thing it will help” and too little “man, I should really understand what I would learn if this benchmark improved and whether it would cause me to actually update the system that has improved on this benchmark is more aligned and less likely to cause catastrophic consequences”.