I’m confused about your FAQ’s advice here. Some quotes from the longer example:
Let’s say that Alice is an expert in AI alignment, and Bob wants to get into the field, and trusts Alice’s judgment. Bob asks Alice what she thinks is most valuable to work on, and she replies, “probably robustness of neural networks”. [...] I think Bob should instead spend some time thinking about how a solution to robustness would mean that AI risk has been meaningfully reduced. [...] It’s possible that after all this reflection, Bob concludes that impact regularization is more valuable than robustness. [...] It’s probably not the case that progress in robustness is 50x more valuable than progress in impact regularization, and so Bob should go with [impact regularization].
In the example, Bob “wants to get into the field”, so this seems like an example of how junior people shouldn’t defer to experts when picking research projects.
(Specualative differences: Maybe you think there’s a huge difference between Alice giving a recommendation about an area vs a specific research project? Or maybe you think that working on impact regularization is the best Bob can do if he can’t find a senior researcher to supervise him, but if Alice could supervise his work on robustness he should go with robustness? If so, maybe it’s worth clarifying that in the FAQ.)
Edit: TBC, I interpret Toby Shevlane as saying ~you should probably work on whatever senior people find interesting; while Jan Kulveit says that “some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the ‘senior researchers’”. The quoted FAQ example is consistent with going against Jan’s strong claim, but I’m not sure it’s consistent with agreeing with Toby’s initial advice, and I interpret you as agreeing with that advice when writing e.g. “Defer to experts for ~3 years, then trust your intuitions”.
In that example, Alice has ~5 min of time to give feedback to Bob; in Toby’s case the senior researchers are (in aggregate) spending at least multiple hours providing feedback (where “Bob spent 15 min talking to Alice and seeing what she got excited about” counts as 15 min of feedback from Alice). That’s the major difference.
I guess one way you could interpret Toby’s advice is to simply get a project idea from a senior person, and then go work on it yourself without feedback from that senior person—I would disagree with that particular advice. I think it’s important to have iterative / continual feedback from senior people.
I’m confused about your FAQ’s advice here. Some quotes from the longer example:
In the example, Bob “wants to get into the field”, so this seems like an example of how junior people shouldn’t defer to experts when picking research projects.
(Specualative differences: Maybe you think there’s a huge difference between Alice giving a recommendation about an area vs a specific research project? Or maybe you think that working on impact regularization is the best Bob can do if he can’t find a senior researcher to supervise him, but if Alice could supervise his work on robustness he should go with robustness? If so, maybe it’s worth clarifying that in the FAQ.)
Edit: TBC, I interpret Toby Shevlane as saying ~you should probably work on whatever senior people find interesting; while Jan Kulveit says that “some young researchers actually have great ideas, should work on them, and avoid generally updating on research taste of most of the ‘senior researchers’”. The quoted FAQ example is consistent with going against Jan’s strong claim, but I’m not sure it’s consistent with agreeing with Toby’s initial advice, and I interpret you as agreeing with that advice when writing e.g. “Defer to experts for ~3 years, then trust your intuitions”.
In that example, Alice has ~5 min of time to give feedback to Bob; in Toby’s case the senior researchers are (in aggregate) spending at least multiple hours providing feedback (where “Bob spent 15 min talking to Alice and seeing what she got excited about” counts as 15 min of feedback from Alice). That’s the major difference.
I guess one way you could interpret Toby’s advice is to simply get a project idea from a senior person, and then go work on it yourself without feedback from that senior person—I would disagree with that particular advice. I think it’s important to have iterative / continual feedback from senior people.