I was going to come back to this and write a comment saying why I either agree or disagree and why, but I keep flipping back and forth.
I now think there are some classes of problems for which I could easily get under 1%, and some for which I can’t, and this partially depends on whether I can learn new material (if I can, I think I’d need to exhaust every promising-looking paper). The question is which is the better reference class for real problems.
You could argue that not learning new material is the better model, because we can’t get external help in real life. But on the other hand, the large action space of real life feels more similar to me to a situation in which we can learn new material—the intuition that the high school student will just “get stuck” seems less strong with an entire academic subfield working on alignment, say.
I was going to come back to this and write a comment saying why I either agree or disagree and why, but I keep flipping back and forth.
I now think there are some classes of problems for which I could easily get under 1%, and some for which I can’t, and this partially depends on whether I can learn new material (if I can, I think I’d need to exhaust every promising-looking paper). The question is which is the better reference class for real problems.
You could argue that not learning new material is the better model, because we can’t get external help in real life. But on the other hand, the large action space of real life feels more similar to me to a situation in which we can learn new material—the intuition that the high school student will just “get stuck” seems less strong with an entire academic subfield working on alignment, say.