In my view this text should come with multiple caveats.
- Beware ‘typical young researcher fallacy’. Young researchers are very diverse, and while some of them will benefit from the advice, some of them will not. I do not believe there is a general ‘formula for successful early-career research’. Different people have different styles of doing research, and even different metrics for what ‘successful research’ means. While certainly many people would benefit from the advice ‘your ideas are bad’, some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the”senior researchers”.
- Beware ‘generalisation out of training distribution’ problems. Compared to some other fields, AI governance as studied by Allan Dafoe is relatively well decomposed into a hierarchy of problems and you can meaningfully scale it by adding junior people and telling them what to do (work on sub-problems senior people consider interesting). This seems more typical for research fields with established paradigms than for fields which are pre-paradigmatic, or fields in need of a change of paradigm.
- Large part of the described formula for success seems to be optimised for success in the direction getting attention of senior researchers, writing something well received, or similar. This is highly practical, and likely good for many people in fields like Ai governance; at the same time, it seems the best research outputs by early career researchers in eg AI safety do not follow this generative pattern, and seem to be motivated more by curiosity, reasoning from first principles, and ignoring authority opinions.
I’m not going to go into much detail here, but I disagree with all of these caveats. I think this would be a worse post if it included the first and third caveats (less sure about the second).
First caveat: I think > 95% of incoming PhD students in AI at Berkeley have bad ideas (in the way this post uses the phrase). I predict that if you did a survey of people who have finished their PhD in AI at Berkeley, over 80% of them would think their initial ideas were significantly worse than their later ideas. (Note also that AI @ Berkeley is a very selective program.)
Second caveat: I’d say that the post applies to technical AI safety, at the very least, though it’s plausible it doesn’t generalize further. (That would surprise me though.)
Third caveat: This doesn’t seem true to me in AI safety according to my definition of “best”, though idk exactly which outputs you’re thinking of and why you think they’re “best”.
I think > 95% of incoming PhD students in AI at Berkeley have bad ideas (in the way this post uses the phrase).[...](Note also that AI @ Berkeley is a very selective program.)
What % do you think this is true for, quality-weighted?
I remember an interview with Geoffrey Hinton where (paraphrased) Hinton was basically like “just trust your intuitions man. Either your intuitions are good or they’re bad. If they are good you should mostly trust your intuitions regardless of what other people say, and if they’re bad, well, you aren’t going to be a good researcher anyway.”
And I remember finding that logic really suspicious and his experiences selection-biased like heck (My understanding is that Hinton “got lucky” by calling neural nets early but his views aren’t obviously more principled than his close contemporaries).
But to steelman(steel-alien?) his view a little, I worry that EA is overinvested in outside-view/forecasting types (like myself?), rather than people with strong and true convictions/extremely high-quality initial research taste, which (quality-weighted) may be making up the majority of revolutionary progress.
And if we tell the future Geoffrey Hintons (and Eliezer Yudkowskys) of the world to be more deferential and trust their intuitions less relative to elite consensus or the literature, we’re doing the world/our movement a disservice, even if the advice is likely to be individually useful/good for most researchers in terms of expected correctness of beliefs or career advancement.
What % do you think this is true for, quality-weighted?
Weighted by quality after graduating? Still > 50%, probably > 80%, but it’s really just a lot harder to tell (I don’t have enough data). I’d guess that the best people still had “bad ideas” when they were starting out.
(I think a lot of what makes an junior researcher’s idea “bad” is that the researcher doesn’t know about existing work, or has misinterpreted the goal of the field, or lacks intuitions gained from hands-on experience, etc. It is really hard to compensate for a lack of knowledge with good intuition or strong armchair reasoning, and I think junior researchers should make it a priority to learn this sort of stuff.)
Re: the rest of your comment, I think you’re reading more into my comment than I said or meant. I do not think researchers should generally be deferential; I think they should have strong beliefs, that may in fact go against expert consensus. I just don’t think this is the right attitude while you are junior. Some quotes from my FAQ:
When selecting research projects, when you’re junior you should generally defer to your advisor. As time passes you should have more conviction. I very rarely see a first year’s research intuitions beat a professor’s; I have seen this happen more often for fourth years and above.
[...]
There’s a longstandingdebate about whether one should defer to some aggregation of experts (an “outside view”), or try to understand the arguments and come to your own conclusion (an “inside view”). This debate mostly focuses on which method tends to arrive at correct conclusions. I am not taking a stance on this debate; I think it’s mostly irrelevant to the problem of doing good research. Research is typically meant to advance the frontiers of human knowledge; this is not the same goal as arriving at correct conclusions. If you want to advance human knowledge, you’re going to need a detailed inside view.
[followed by a longer example in which the correct thing to do is to ignore the expert]
Thanks for the link to your FAQ, I’m excited to read it further now!
Re: the rest of your comment, I think you’re reading more into my comment than I said or meant. I do not think researchers should generally be deferential; I think they should have strong beliefs, that may in fact go against expert consensus. I just don’t think this is the right attitude while you are junior
To be clear, I think Geoffrey Hinton’s advice was targeted at very junior people. In context, the interview was conducted for Andrew Ng’s online deep learning course, which for many people would be their first exposure to deep learning. I also got the impression that he would stand by this advice for early PhDs (though I could definitely have misunderstood him), and by “future Geoffrey Hintons and Eliezer Yudkowskys” I was thinking about pretty junior people rather than established researchers.
“Defer to experts for ~3 years, then trust your intuitions”
“Always trust your intuitions”
When you said
But to steelman(steel-alien?) his view a little, I worry that EA is overinvested in outside-view/forecasting types (like myself?), rather than people with strong and true convictions/extremely high-quality initial research taste, which (quality-weighted) may be making up the majority of revolutionary progress.
And if we tell the future Geoffrey Hintons (and Eliezer Yudkowskys) of the world to be more deferential and trust their intuitions less relative to elite consensus or the literature, we’re doing the world/our movement a disservice, even if the advice is likely to be individually useful/good for most researchers in terms of expected correctness of beliefs or career advancement.
I thought you were claiming “maybe 3 > 1”, so my response was “don’t do 1 or 3, do 2″.
If you’re instead claiming “maybe 3 > 2”, I don’t really get the argument. It doesn’t seem like advice #2 is obviously worse than advice #3 even for junior Eliezers and Geoffreys. (It’s hard to say for those two people: in Eliezer’s case, since there were no experts to defer to at the time, and I don’t know enough details about Geoffrey to evaluate which advice would be good for him.)
I think Geoffrey Hinton’s advice was targeted at very junior people.
Oh, I agree that’s probably true. I think he’s wrong to give that advice. I’m generally pretty okay with ignoring expert advice to amateurs if you have reason to believe it’s bad; experts usually don’t remember what it was like to be an amateur and so it’s not that surprising that their advice on what amateurs should do is not great. (EDIT: Here’s a new post that goes into more detail on this.)
I would guess the ‘typical young researcher fallacy’ also applies to Hinton - my impression is he is basically advising his past self, similarly to Toby. As a consequence, the advice is likely sensible for people-much-like-past-Hinton, but not a good general advice for everyone.
In ~3 years most people are able to re-train their intuitions a lot (which is part of the point!). This seems particularly dangerous in cases where expertise in the thing you are actually interested in does not exist, but expertise in something somewhat close does - instead of following your curiosity, you ‘substitute the question’ with a different question, for which a PhD program exists, or senior researchers exist, or established directions exist. If your initial taste/questions was better than the expert’s, you run a risk of overwriting your taste with something less interesting/impactful.
Anecdotal illustrative story:
Arguably, large part of what are now the foundations of quantum information theory / quantum computing could have been discovered much sooner, together with taking more sensible interpretations of quantum mechanics than Copenhagen interpretation seriously. My guess what was happening during multiple decades (!) was many early career researchers were curious what’s going on, dissatisfied with the answers, interested in thinking about the topic more… but they were given the advice along the lines ‘this is not a good topic for PhDs or even undergrads; don’t trust your intuition; problems here are distasteful mix of physics and philosophy; shut up and calculate, that’s how a real progress happens’ … and they followed it; acquired a taste telling them that solving difficult scattering amplitudes integrals using advanced calculus techniques is tasty, and thinking about deep things formulated using tools of high-school algebra is for fools. (Also if you did run a survey in year 4 of their PhDs, large fraction of quantum physicists would probably endorse the learned update from thinking about young foolish questions about QM interpretations to the serious and publishable thinking they have learned.)
I agree substituting the question would be bad, and sometimes there aren’t any relevant experts in which case you shouldn’t defer to people. (Though even then I’d consider doing research in an unrelated area for a couple of years, and then coming back to work on the question of interest.)
I admit I don’t really understand how people manage to have a “driving question” overwritten—I can’t really imagine that happening to me and I am confused about how it happens to other people.
(I think sometimes it is justified, e.g. you realize that your question was confused, and the other work you’ve done has deconfused it, but it does seem like often it’s just that they pick up the surrounding culture and just forget about the question they cared about in the first place.)
So I guess this seems like a possible risk. I’d still bet pretty strongly against any particular junior researcher’s intuition being better, so I still think this advice is good on net.
(I’m mostly not engaging with the quantum example because it sounds like a very just-so story to me and I don’t know enough about the area to evaluate the just-so story.)
I’m confused about your FAQ’s advice here. Some quotes from the longer example:
Let’s say that Alice is an expert in AI alignment, and Bob wants to get into the field, and trusts Alice’s judgment. Bob asks Alice what she thinks is most valuable to work on, and she replies, “probably robustness of neural networks”. [...] I think Bob should instead spend some time thinking about how a solution to robustness would mean that AI risk has been meaningfully reduced. [...] It’s possible that after all this reflection, Bob concludes that impact regularization is more valuable than robustness. [...] It’s probably not the case that progress in robustness is 50x more valuable than progress in impact regularization, and so Bob should go with [impact regularization].
In the example, Bob “wants to get into the field”, so this seems like an example of how junior people shouldn’t defer to experts when picking research projects.
(Specualative differences: Maybe you think there’s a huge difference between Alice giving a recommendation about an area vs a specific research project? Or maybe you think that working on impact regularization is the best Bob can do if he can’t find a senior researcher to supervise him, but if Alice could supervise his work on robustness he should go with robustness? If so, maybe it’s worth clarifying that in the FAQ.)
Edit: TBC, I interpret Toby Shevlane as saying ~you should probably work on whatever senior people find interesting; while Jan Kulveit says that “some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the ‘senior researchers’”. The quoted FAQ example is consistent with going against Jan’s strong claim, but I’m not sure it’s consistent with agreeing with Toby’s initial advice, and I interpret you as agreeing with that advice when writing e.g. “Defer to experts for ~3 years, then trust your intuitions”.
In that example, Alice has ~5 min of time to give feedback to Bob; in Toby’s case the senior researchers are (in aggregate) spending at least multiple hours providing feedback (where “Bob spent 15 min talking to Alice and seeing what she got excited about” counts as 15 min of feedback from Alice). That’s the major difference.
I guess one way you could interpret Toby’s advice is to simply get a project idea from a senior person, and then go work on it yourself without feedback from that senior person—I would disagree with that particular advice. I think it’s important to have iterative / continual feedback from senior people.
Let’s start with the third caveat: maybe the real crux is what we think are the best outputs; what I consider some of the best outputs by young researchers of AI alignment is easier to point at via examples—so it’s e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth. As far as I can tell, none of these seems to be following the proposed ‘formula for successful early-career research’.
My impression is PhD students in AI in Berkeley need to optimise, and actually optimise a lot for success in an established field (ML/AI), and subsequently, the advice should be more applicable to them. I would even say part of what makes a field “established” actually is something like “somewhat clear direction in the space of unknown in which people are trying to push the boundary” and “shared taste in what is good, according to the direction”. (The general direction or at least the taste seems to be ~ self-perpetuating once the field is “established”, sometimes beyond the point of usefulness).
In contrast to your experience with AI students in Berkeley, in my experience about ~20% of ESPR students have generally good ideas even while at high school/first year in college, and I would often prefer these people to think about ways in which their teachers, professors or seniors are possibly confused, as opposed to learning that their ideas are now generally bad and they should seek someone senior to tell them what to work on. (Ok—the actual advice would be more complex and nuanced, something like “update on the idea taste of people who are better/are comparable and have spent more time thinking about something, but be sceptical and picky about your selection of people”). (ESPR is also very selective, although differently.)
With hypothetical surveys, the conclusion (young researchers should mostly defer to seniors in idea taste) does not seem to follow from estimates like “over 80% of them would think their initial ideas were significantly worse than their later ideas”. Relevant comparison is something like “over 80% of them would think they should have spent marginally more time thinking about ideas of more senior AI people at Berkeley, and more time on problems they were given by senior people, and smaller amount of time thinking about their own ideas, and working on projects based on their ideas”. Would you guess the answer would still be 80%?
so it’s e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth. As far as I can tell, none of these seems to be following the proposed ‘formula for successful early-career research’.
I think the mesa optimizers paper fits the formula pretty well? My understanding is that the junior authors on that paper interacted a lot with researchers at MIRI (and elsewhere) while writing that paper.
I don’t know John Wentworth’s history. I think it’s plausible that if I did, I wouldn’t have thought of him as a junior researcher (even before seeing his posts). If that isn’t true, I agree that’s a good counterexample.
My impression is PhD students in AI in Berkeley [...]
I agree the advice is particularly suited to this audience, for the reasons you describe.
the actual advice would be more complex and nuanced, something like “update on the idea taste of people who are better/are comparable and have spent more time thinking about something, but be sceptical and picky about your selection of people”
That sounds like the advice in this post? You’ve added a clause about being picky about the selection of people, which I agree with, but other than that it sounds pretty similar to what Toby is suggesting. If so I’m not sure why a caveat is needed.
Perhaps you think something like “if someone [who is better or who is comparable and has spent more time thinking about something than you] provides feedback, then you should update, but it isn’t that important and you don’t need to seek it out”?
Relevant comparison is something like “over 80% of them would think they should have spent marginally more time thinking about ideas of more senior AI people at Berkeley, and more time on problems they were given by senior people, and smaller amount of time thinking about their own ideas, and working on projects based on their ideas”. Would you guess the answer would still be 80%?
I agree that’s more clearly targeting the right thing, but still not great, for a couple of reasons:
The question is getting pretty complicated, which I think makes answers a bit more random.
Many students are too deferential throughout their PhD, and might correctly say that they should have explored their own ideas more—without this implying that the advice in this post is wrong.
Lots of people do in fact take an approach that is roughly “do stuff your advisor says, and over time become more independent and opinionated”; idk what they would say.
I do predict though that they mostly won’t say things like “my ideas during my first year were good, I would have had more impact had I just followed my instincts and ignored my advisor”. (I guess one exception is that if they hated the project their advisor suggested, but slogged through it anyway, then they might say that—but I feel like that’s more about motivation rather than impact.)
Thanks for the caveats Jan, I think that’s helpful.
It’s true that my views have been formed from within the field of AI governance, and I am open to the idea that they won’t fully generalise to other fields. I have inserted a line in the introduction that clarifies this.
In my view this text should come with multiple caveats.
- Beware ‘typical young researcher fallacy’. Young researchers are very diverse, and while some of them will benefit from the advice, some of them will not. I do not believe there is a general ‘formula for successful early-career research’. Different people have different styles of doing research, and even different metrics for what ‘successful research’ means. While certainly many people would benefit from the advice ‘your ideas are bad’, some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the”senior researchers”.
- Beware ‘generalisation out of training distribution’ problems. Compared to some other fields, AI governance as studied by Allan Dafoe is relatively well decomposed into a hierarchy of problems and you can meaningfully scale it by adding junior people and telling them what to do (work on sub-problems senior people consider interesting). This seems more typical for research fields with established paradigms than for fields which are pre-paradigmatic, or fields in need of a change of paradigm.
- Large part of the described formula for success seems to be optimised for success in the direction getting attention of senior researchers, writing something well received, or similar. This is highly practical, and likely good for many people in fields like Ai governance; at the same time, it seems the best research outputs by early career researchers in eg AI safety do not follow this generative pattern, and seem to be motivated more by curiosity, reasoning from first principles, and ignoring authority opinions.
I’m not going to go into much detail here, but I disagree with all of these caveats. I think this would be a worse post if it included the first and third caveats (less sure about the second).
First caveat: I think > 95% of incoming PhD students in AI at Berkeley have bad ideas (in the way this post uses the phrase). I predict that if you did a survey of people who have finished their PhD in AI at Berkeley, over 80% of them would think their initial ideas were significantly worse than their later ideas. (Note also that AI @ Berkeley is a very selective program.)
Second caveat: I’d say that the post applies to technical AI safety, at the very least, though it’s plausible it doesn’t generalize further. (That would surprise me though.)
Third caveat: This doesn’t seem true to me in AI safety according to my definition of “best”, though idk exactly which outputs you’re thinking of and why you think they’re “best”.
What % do you think this is true for, quality-weighted?
I remember an interview with Geoffrey Hinton where (paraphrased) Hinton was basically like “just trust your intuitions man. Either your intuitions are good or they’re bad. If they are good you should mostly trust your intuitions regardless of what other people say, and if they’re bad, well, you aren’t going to be a good researcher anyway.”
And I remember finding that logic really suspicious and his experiences selection-biased like heck (My understanding is that Hinton “got lucky” by calling neural nets early but his views aren’t obviously more principled than his close contemporaries).
But to steelman(steel-alien?) his view a little, I worry that EA is overinvested in outside-view/forecasting types (like myself?), rather than people with strong and true convictions/extremely high-quality initial research taste, which (quality-weighted) may be making up the majority of revolutionary progress.
And if we tell the future Geoffrey Hintons (and Eliezer Yudkowskys) of the world to be more deferential and trust their intuitions less relative to elite consensus or the literature, we’re doing the world/our movement a disservice, even if the advice is likely to be individually useful/good for most researchers in terms of expected correctness of beliefs or career advancement.
Weighted by quality after graduating? Still > 50%, probably > 80%, but it’s really just a lot harder to tell (I don’t have enough data). I’d guess that the best people still had “bad ideas” when they were starting out.
(I think a lot of what makes an junior researcher’s idea “bad” is that the researcher doesn’t know about existing work, or has misinterpreted the goal of the field, or lacks intuitions gained from hands-on experience, etc. It is really hard to compensate for a lack of knowledge with good intuition or strong armchair reasoning, and I think junior researchers should make it a priority to learn this sort of stuff.)
Re: the rest of your comment, I think you’re reading more into my comment than I said or meant. I do not think researchers should generally be deferential; I think they should have strong beliefs, that may in fact go against expert consensus. I just don’t think this is the right attitude while you are junior. Some quotes from my FAQ:
Thanks for the link to your FAQ, I’m excited to read it further now!
To be clear, I think Geoffrey Hinton’s advice was targeted at very junior people. In context, the interview was conducted for Andrew Ng’s online deep learning course, which for many people would be their first exposure to deep learning. I also got the impression that he would stand by this advice for early PhDs (though I could definitely have misunderstood him), and by “future Geoffrey Hintons and Eliezer Yudkowskys” I was thinking about pretty junior people rather than established researchers.
I’m considering three types of advice:
“Always defer to experts”
“Defer to experts for ~3 years, then trust your intuitions”
“Always trust your intuitions”
When you said
I thought you were claiming “maybe 3 > 1”, so my response was “don’t do 1 or 3, do 2″.
If you’re instead claiming “maybe 3 > 2”, I don’t really get the argument. It doesn’t seem like advice #2 is obviously worse than advice #3 even for junior Eliezers and Geoffreys. (It’s hard to say for those two people: in Eliezer’s case, since there were no experts to defer to at the time, and I don’t know enough details about Geoffrey to evaluate which advice would be good for him.)
Oh, I agree that’s probably true. I think he’s wrong to give that advice. I’m generally pretty okay with ignoring expert advice to amateurs if you have reason to believe it’s bad; experts usually don’t remember what it was like to be an amateur and so it’s not that surprising that their advice on what amateurs should do is not great. (EDIT: Here’s a new post that goes into more detail on this.)
I would guess the ‘typical young researcher fallacy’ also applies to Hinton - my impression is he is basically advising his past self, similarly to Toby. As a consequence, the advice is likely sensible for people-much-like-past-Hinton, but not a good general advice for everyone.
In ~3 years most people are able to re-train their intuitions a lot (which is part of the point!). This seems particularly dangerous in cases where expertise in the thing you are actually interested in does not exist, but expertise in something somewhat close does - instead of following your curiosity, you ‘substitute the question’ with a different question, for which a PhD program exists, or senior researchers exist, or established directions exist. If your initial taste/questions was better than the expert’s, you run a risk of overwriting your taste with something less interesting/impactful.
Anecdotal illustrative story:
Arguably, large part of what are now the foundations of quantum information theory / quantum computing could have been discovered much sooner, together with taking more sensible interpretations of quantum mechanics than Copenhagen interpretation seriously. My guess what was happening during multiple decades (!) was many early career researchers were curious what’s going on, dissatisfied with the answers, interested in thinking about the topic more… but they were given the advice along the lines ‘this is not a good topic for PhDs or even undergrads; don’t trust your intuition; problems here are distasteful mix of physics and philosophy; shut up and calculate, that’s how a real progress happens’ … and they followed it; acquired a taste telling them that solving difficult scattering amplitudes integrals using advanced calculus techniques is tasty, and thinking about deep things formulated using tools of high-school algebra is for fools. (Also if you did run a survey in year 4 of their PhDs, large fraction of quantum physicists would probably endorse the learned update from thinking about young foolish questions about QM interpretations to the serious and publishable thinking they have learned.)
I agree substituting the question would be bad, and sometimes there aren’t any relevant experts in which case you shouldn’t defer to people. (Though even then I’d consider doing research in an unrelated area for a couple of years, and then coming back to work on the question of interest.)
I admit I don’t really understand how people manage to have a “driving question” overwritten—I can’t really imagine that happening to me and I am confused about how it happens to other people.
(I think sometimes it is justified, e.g. you realize that your question was confused, and the other work you’ve done has deconfused it, but it does seem like often it’s just that they pick up the surrounding culture and just forget about the question they cared about in the first place.)
So I guess this seems like a possible risk. I’d still bet pretty strongly against any particular junior researcher’s intuition being better, so I still think this advice is good on net.
(I’m mostly not engaging with the quantum example because it sounds like a very just-so story to me and I don’t know enough about the area to evaluate the just-so story.)
(As an aside, I read your FAQ and enjoyed it, so thanks for the share!)
I’m confused about your FAQ’s advice here. Some quotes from the longer example:
In the example, Bob “wants to get into the field”, so this seems like an example of how junior people shouldn’t defer to experts when picking research projects.
(Specualative differences: Maybe you think there’s a huge difference between Alice giving a recommendation about an area vs a specific research project? Or maybe you think that working on impact regularization is the best Bob can do if he can’t find a senior researcher to supervise him, but if Alice could supervise his work on robustness he should go with robustness? If so, maybe it’s worth clarifying that in the FAQ.)
Edit: TBC, I interpret Toby Shevlane as saying ~you should probably work on whatever senior people find interesting; while Jan Kulveit says that “some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the ‘senior researchers’”. The quoted FAQ example is consistent with going against Jan’s strong claim, but I’m not sure it’s consistent with agreeing with Toby’s initial advice, and I interpret you as agreeing with that advice when writing e.g. “Defer to experts for ~3 years, then trust your intuitions”.
In that example, Alice has ~5 min of time to give feedback to Bob; in Toby’s case the senior researchers are (in aggregate) spending at least multiple hours providing feedback (where “Bob spent 15 min talking to Alice and seeing what she got excited about” counts as 15 min of feedback from Alice). That’s the major difference.
I guess one way you could interpret Toby’s advice is to simply get a project idea from a senior person, and then go work on it yourself without feedback from that senior person—I would disagree with that particular advice. I think it’s important to have iterative / continual feedback from senior people.
Let’s start with the third caveat: maybe the real crux is what we think are the best outputs; what I consider some of the best outputs by young researchers of AI alignment is easier to point at via examples—so it’s e.g. the mesa-optimizers paper or multiple LW posts by John Wentworth. As far as I can tell, none of these seems to be following the proposed ‘formula for successful early-career research’.
My impression is PhD students in AI in Berkeley need to optimise, and actually optimise a lot for success in an established field (ML/AI), and subsequently, the advice should be more applicable to them. I would even say part of what makes a field “established” actually is something like “somewhat clear direction in the space of unknown in which people are trying to push the boundary” and “shared taste in what is good, according to the direction”. (The general direction or at least the taste seems to be ~ self-perpetuating once the field is “established”, sometimes beyond the point of usefulness).
In contrast to your experience with AI students in Berkeley, in my experience about ~20% of ESPR students have generally good ideas even while at high school/first year in college, and I would often prefer these people to think about ways in which their teachers, professors or seniors are possibly confused, as opposed to learning that their ideas are now generally bad and they should seek someone senior to tell them what to work on. (Ok—the actual advice would be more complex and nuanced, something like “update on the idea taste of people who are better/are comparable and have spent more time thinking about something, but be sceptical and picky about your selection of people”). (ESPR is also very selective, although differently.)
With hypothetical surveys, the conclusion (young researchers should mostly defer to seniors in idea taste) does not seem to follow from estimates like “over 80% of them would think their initial ideas were significantly worse than their later ideas”. Relevant comparison is something like “over 80% of them would think they should have spent marginally more time thinking about ideas of more senior AI people at Berkeley, and more time on problems they were given by senior people, and smaller amount of time thinking about their own ideas, and working on projects based on their ideas”. Would you guess the answer would still be 80%?
I think the mesa optimizers paper fits the formula pretty well? My understanding is that the junior authors on that paper interacted a lot with researchers at MIRI (and elsewhere) while writing that paper.
I don’t know John Wentworth’s history. I think it’s plausible that if I did, I wouldn’t have thought of him as a junior researcher (even before seeing his posts). If that isn’t true, I agree that’s a good counterexample.
I agree the advice is particularly suited to this audience, for the reasons you describe.
That sounds like the advice in this post? You’ve added a clause about being picky about the selection of people, which I agree with, but other than that it sounds pretty similar to what Toby is suggesting. If so I’m not sure why a caveat is needed.
Perhaps you think something like “if someone [who is better or who is comparable and has spent more time thinking about something than you] provides feedback, then you should update, but it isn’t that important and you don’t need to seek it out”?
I agree that’s more clearly targeting the right thing, but still not great, for a couple of reasons:
The question is getting pretty complicated, which I think makes answers a bit more random.
Many students are too deferential throughout their PhD, and might correctly say that they should have explored their own ideas more—without this implying that the advice in this post is wrong.
Lots of people do in fact take an approach that is roughly “do stuff your advisor says, and over time become more independent and opinionated”; idk what they would say.
I do predict though that they mostly won’t say things like “my ideas during my first year were good, I would have had more impact had I just followed my instincts and ignored my advisor”. (I guess one exception is that if they hated the project their advisor suggested, but slogged through it anyway, then they might say that—but I feel like that’s more about motivation rather than impact.)
Thanks for the caveats Jan, I think that’s helpful.
It’s true that my views have been formed from within the field of AI governance, and I am open to the idea that they won’t fully generalise to other fields. I have inserted a line in the introduction that clarifies this.