Let’s start with the third caveat: maybe the real crux is what we think are the best outputs; what I consider some of the best outputs by young researchers of AI alignment is easier to point at via examples—so it’s e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth. As far as I can tell, none of these seems to be following the proposed ‘formula for successful early-career research’.
My impression is PhD students in AI in Berkeley need to optimise, and actually optimise a lot for success in an established field (ML/AI), and subsequently, the advice should be more applicable to them. I would even say part of what makes a field “established” actually is something like “somewhat clear direction in the space of unknown in which people are trying to push the boundary” and “shared taste in what is good, according to the direction”. (The general direction or at least the taste seems to be ~ self-perpetuating once the field is “established”, sometimes beyond the point of usefulness).
In contrast to your experience with AI students in Berkeley, in my experience about ~20% of ESPR students have generally good ideas even while at high school/first year in college, and I would often prefer these people to think about ways in which their teachers, professors or seniors are possibly confused, as opposed to learning that their ideas are now generally bad and they should seek someone senior to tell them what to work on. (Ok—the actual advice would be more complex and nuanced, something like “update on the idea taste of people who are better/are comparable and have spent more time thinking about something, but be sceptical and picky about your selection of people”). (ESPR is also very selective, although differently.)
With hypothetical surveys, the conclusion (young researchers should mostly defer to seniors in idea taste) does not seem to follow from estimates like “over 80% of them would think their initial ideas were significantly worse than their later ideas”. Relevant comparison is something like “over 80% of them would think they should have spent marginally more time thinking about ideas of more senior AI people at Berkeley, and more time on problems they were given by senior people, and smaller amount of time thinking about their own ideas, and working on projects based on their ideas”. Would you guess the answer would still be 80%?
so it’s e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth. As far as I can tell, none of these seems to be following the proposed ‘formula for successful early-career research’.
I think the mesa optimizers paper fits the formula pretty well? My understanding is that the junior authors on that paper interacted a lot with researchers at MIRI (and elsewhere) while writing that paper.
I don’t know John Wentworth’s history. I think it’s plausible that if I did, I wouldn’t have thought of him as a junior researcher (even before seeing his posts). If that isn’t true, I agree that’s a good counterexample.
My impression is PhD students in AI in Berkeley [...]
I agree the advice is particularly suited to this audience, for the reasons you describe.
the actual advice would be more complex and nuanced, something like “update on the idea taste of people who are better/are comparable and have spent more time thinking about something, but be sceptical and picky about your selection of people”
That sounds like the advice in this post? You’ve added a clause about being picky about the selection of people, which I agree with, but other than that it sounds pretty similar to what Toby is suggesting. If so I’m not sure why a caveat is needed.
Perhaps you think something like “if someone [who is better or who is comparable and has spent more time thinking about something than you] provides feedback, then you should update, but it isn’t that important and you don’t need to seek it out”?
Relevant comparison is something like “over 80% of them would think they should have spent marginally more time thinking about ideas of more senior AI people at Berkeley, and more time on problems they were given by senior people, and smaller amount of time thinking about their own ideas, and working on projects based on their ideas”. Would you guess the answer would still be 80%?
I agree that’s more clearly targeting the right thing, but still not great, for a couple of reasons:
The question is getting pretty complicated, which I think makes answers a bit more random.
Many students are too deferential throughout their PhD, and might correctly say that they should have explored their own ideas more—without this implying that the advice in this post is wrong.
Lots of people do in fact take an approach that is roughly “do stuff your advisor says, and over time become more independent and opinionated”; idk what they would say.
I do predict though that they mostly won’t say things like “my ideas during my first year were good, I would have had more impact had I just followed my instincts and ignored my advisor”. (I guess one exception is that if they hated the project their advisor suggested, but slogged through it anyway, then they might say that—but I feel like that’s more about motivation rather than impact.)
Let’s start with the third caveat: maybe the real crux is what we think are the best outputs; what I consider some of the best outputs by young researchers of AI alignment is easier to point at via examples—so it’s e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth. As far as I can tell, none of these seems to be following the proposed ‘formula for successful early-career research’.
My impression is PhD students in AI in Berkeley need to optimise, and actually optimise a lot for success in an established field (ML/AI), and subsequently, the advice should be more applicable to them. I would even say part of what makes a field “established” actually is something like “somewhat clear direction in the space of unknown in which people are trying to push the boundary” and “shared taste in what is good, according to the direction”. (The general direction or at least the taste seems to be ~ self-perpetuating once the field is “established”, sometimes beyond the point of usefulness).
In contrast to your experience with AI students in Berkeley, in my experience about ~20% of ESPR students have generally good ideas even while at high school/first year in college, and I would often prefer these people to think about ways in which their teachers, professors or seniors are possibly confused, as opposed to learning that their ideas are now generally bad and they should seek someone senior to tell them what to work on. (Ok—the actual advice would be more complex and nuanced, something like “update on the idea taste of people who are better/are comparable and have spent more time thinking about something, but be sceptical and picky about your selection of people”). (ESPR is also very selective, although differently.)
With hypothetical surveys, the conclusion (young researchers should mostly defer to seniors in idea taste) does not seem to follow from estimates like “over 80% of them would think their initial ideas were significantly worse than their later ideas”. Relevant comparison is something like “over 80% of them would think they should have spent marginally more time thinking about ideas of more senior AI people at Berkeley, and more time on problems they were given by senior people, and smaller amount of time thinking about their own ideas, and working on projects based on their ideas”. Would you guess the answer would still be 80%?
I think the mesa optimizers paper fits the formula pretty well? My understanding is that the junior authors on that paper interacted a lot with researchers at MIRI (and elsewhere) while writing that paper.
I don’t know John Wentworth’s history. I think it’s plausible that if I did, I wouldn’t have thought of him as a junior researcher (even before seeing his posts). If that isn’t true, I agree that’s a good counterexample.
I agree the advice is particularly suited to this audience, for the reasons you describe.
That sounds like the advice in this post? You’ve added a clause about being picky about the selection of people, which I agree with, but other than that it sounds pretty similar to what Toby is suggesting. If so I’m not sure why a caveat is needed.
Perhaps you think something like “if someone [who is better or who is comparable and has spent more time thinking about something than you] provides feedback, then you should update, but it isn’t that important and you don’t need to seek it out”?
I agree that’s more clearly targeting the right thing, but still not great, for a couple of reasons:
The question is getting pretty complicated, which I think makes answers a bit more random.
Many students are too deferential throughout their PhD, and might correctly say that they should have explored their own ideas more—without this implying that the advice in this post is wrong.
Lots of people do in fact take an approach that is roughly “do stuff your advisor says, and over time become more independent and opinionated”; idk what they would say.
I do predict though that they mostly won’t say things like “my ideas during my first year were good, I would have had more impact had I just followed my instincts and ignored my advisor”. (I guess one exception is that if they hated the project their advisor suggested, but slogged through it anyway, then they might say that—but I feel like that’s more about motivation rather than impact.)