A trend I’ve noticed in the AI safety independent research grants for the past two rounds (April and August) is that most of the grantees have little to no online presence as far as I know (they could be using pseudonyms I am unaware of); I believe Alex Turner and David Manheim are the only exceptions. However, when I think about “who am I most excited to give individual research grants to, if I had that kind of money?”, the names I come up with are people who leave interesting comments and posts on LessWrong about AI safety. (This isn’t surprising because I mostly interact with the AI safety community publicly online, so I don’t have much access to private info.) To give an idea of the kind of people I am thinking of, I would name John Wentworth, Steve Byrnes, Ofer G., Morgan Sinclaire, and Evan Hubinger as examples.
This has me wondering what’s going on. Some possibilities I can think of:
the people who contribute on LW aren’t applying for grants
the private people are higher quality than the online people
the private people have more credentials than the online people (e.g. Hertz Fellowship, math contests experience)
fund managers are more receptive offline than online and it’s easier to network offline
fund managers don’t follow online discussions closely
I would appreciate if the fund managers could weigh in on this so I have a better sense of why my own thinking seems to diverge so much from the actual grant recommendations.
As one of the people you mentioned (I’m flattered!), I’ve also been curious about this.
As for my own anecdata, I basically haven’t applied yet. Technically I did apply and get declined last round, but a) it was a fairly low-effort application since I didn’t really need the money then which b) I said so on the application and c) I didn’t have any public posts until 2 months ago so I wasn’t in your demographic and d) I didn’t have any references because I don’t really know many people in the research community.
I’m about to submit a serious application for this round, where of those only (d) is still true. At least, I haven’t extensively interacted with any high-status researchers for it to make sense to ask anyone for references. And I think maybe there’s a correlation there that explains part of your question: I post/comment online when I’m up to it because it’s one of the best ways for me to get good feedback (this being a great example), even though I’m a slow writer and it’s a laborious process for me to get from “this seems like a coherent, nontrivial idea probably worth writing up” to feeling like I’ve covered the all the inferential gaps, noted all the caveats, taken into account relevant prior writings, and thought of possible objections enough to feel ready to hit the submit button. But anyways, I would guess that maybe online people slightly skew towards being isolated (else they’d get feedback or spread their ideas by just talking to e.g. coworkers), hence not having references. But I don’t think this is a large effect (and I defer to Habryka’s comment). Of the people you mentioned, I believe Evan is currently working with Christiano at OpenAI and has been “clued-in” for a while, and I have no idea about the first 3.
Also, I often wonder how much Alignment research is going on that I’m just not clued into from “merely” reading the Alignment Forum, Alignment Newsletter, papers by OpenAI/DeepMind/CHAI etc. I know that MIRI is nondisclosed-by-default now, and I get that. But they laid out their reasons for that in detail, and that’s on top of the trust they’ve earned from me as an institution for their past research. When I hear about people who are doing their own research but not posting anything, I get pretty skeptical unless they’ve produced good Alignment research in the past (producing other technical research counts for something, my own intuition is that the pre-paradigmatic nature of Alignment research is different enough that the tails come apart), and my system 1 says (especially if they’re getting funded):
Oh come on! I would love to sit around and do my own private “research” uninterrupted without the hard work of writing things up, but that’s what you have to do if you want to be a part of the research community collectively working toward solving a problem. If everyone just lounged around in their own thoughts and notes without distilling that information for others to build on, there just wouldn’t be any intellectual progress. That’s the whole point of academic publication, and forum posting is actually a step down from that norm, and even that’s only possible because the community of < 100 is small, young, and non-specialized enough that medium-effort ways of distilling ideas still work (less inferential gaps to cross etc.)
(My system 2 would obviously use a different tone than that, but it largely agrees with the substance.)
Also, to echo points made by Jan, LW is not the best place for a broad impression of current research, the Alignment Forum is strictly better. But even the latter is somewhat skewed towards MIRI-esque things over CHAI, OpenAI, and Deepmind’s stuff, here’s another decent comment thread discussing that.
My sense is mostly (1), with maybe some additional disagreement over what online contributions are actually a sign of competence. But usually I am quite excited when an active online contributor applies.
I share your perspective that I am most excited about people who participate in AI Alignment discussion online, but we’ve received relatively few applications from people in that reference class.
Some of the people we’ve given grants to, were the result of Alex Zhu doing a lot of networking with people who are interested in AI alignment, which tends to select on some slightly different things, but given the lack of applications from people with a history of contributing online, that still seems pretty good to me.
The reason may be somewhat simple: most AI alignment researchers do not participate (post or comment) on LW/AF or participate only a little. For more understanding why, check this post of Wei Dai and the discussion under it.
(Also: if you follow just LW, your understanding of the field of AI safety is likely somewhat distorted)
With hypothesis 4.&5. I expect at least Oli to have strong bias of being more enthusiastic in funding people who like to interact with LW (all other research qualities being equal), so I’m pretty sure it’s not the case
2.&3. is somewhat true at least on average: if we operationalize “private people” as “people who do you meet participating in private research retreats or visiting places like MIRI or FHI”, and “online people” as “people posting and commenting on AI safety on LW” than the first group is on average better.
1. is likely true in the sense that best LW contributors are not applying for grants
The reason may be somewhat simple: most AI alignment researchers do not participate (post or comment) on LW/AF or participate only a little.
I’m wondering how many such people there are. Specifically, how many people (i) don’t participate on LW/AF, (ii) don’t already get paid for AI alignment work, and (iii) do seriously want to spend a significant amount of time working on AI alignment or already do so in their free time? (So I want to exclude researchers at organizations, random people who contact 80,000 Hours for advice on how to get involved, people who attend a MIRI workshop or AI safety camp but then happily go back to doing non-alignment work, etc.) My own feeling before reading your comment was that there are maybe 10-20 such people, but it sounds like there may be many more than that. Do you have a specific number in mind?
if you follow just LW, your understanding of the field of AI safety is likely somewhat distorted
I’m aware of this, and I’ve seen Wei Dai’s post and the comments there. Personally I don’t see an easy way to get access to more private discussions due to a variety of factors (not being invited to workshops, some workshops being too expensive for it to be worth traveling to, not being eligible to apply for certain programs, and so on).
A trend I’ve noticed in the AI safety independent research grants for the past two rounds (April and August) is that most of the grantees have little to no online presence as far as I know (they could be using pseudonyms I am unaware of); I believe Alex Turner and David Manheim are the only exceptions. However, when I think about “who am I most excited to give individual research grants to, if I had that kind of money?”, the names I come up with are people who leave interesting comments and posts on LessWrong about AI safety. (This isn’t surprising because I mostly interact with the AI safety community publicly online, so I don’t have much access to private info.) To give an idea of the kind of people I am thinking of, I would name John Wentworth, Steve Byrnes, Ofer G., Morgan Sinclaire, and Evan Hubinger as examples.
This has me wondering what’s going on. Some possibilities I can think of:
the people who contribute on LW aren’t applying for grants
the private people are higher quality than the online people
the private people have more credentials than the online people (e.g. Hertz Fellowship, math contests experience)
fund managers are more receptive offline than online and it’s easier to network offline
fund managers don’t follow online discussions closely
I would appreciate if the fund managers could weigh in on this so I have a better sense of why my own thinking seems to diverge so much from the actual grant recommendations.
As one of the people you mentioned (I’m flattered!), I’ve also been curious about this.
As for my own anecdata, I basically haven’t applied yet. Technically I did apply and get declined last round, but a) it was a fairly low-effort application since I didn’t really need the money then which b) I said so on the application and c) I didn’t have any public posts until 2 months ago so I wasn’t in your demographic and d) I didn’t have any references because I don’t really know many people in the research community.
I’m about to submit a serious application for this round, where of those only (d) is still true. At least, I haven’t extensively interacted with any high-status researchers for it to make sense to ask anyone for references. And I think maybe there’s a correlation there that explains part of your question: I post/comment online when I’m up to it because it’s one of the best ways for me to get good feedback (this being a great example), even though I’m a slow writer and it’s a laborious process for me to get from “this seems like a coherent, nontrivial idea probably worth writing up” to feeling like I’ve covered the all the inferential gaps, noted all the caveats, taken into account relevant prior writings, and thought of possible objections enough to feel ready to hit the submit button. But anyways, I would guess that maybe online people slightly skew towards being isolated (else they’d get feedback or spread their ideas by just talking to e.g. coworkers), hence not having references. But I don’t think this is a large effect (and I defer to Habryka’s comment). Of the people you mentioned, I believe Evan is currently working with Christiano at OpenAI and has been “clued-in” for a while, and I have no idea about the first 3.
Also, I often wonder how much Alignment research is going on that I’m just not clued into from “merely” reading the Alignment Forum, Alignment Newsletter, papers by OpenAI/DeepMind/CHAI etc. I know that MIRI is nondisclosed-by-default now, and I get that. But they laid out their reasons for that in detail, and that’s on top of the trust they’ve earned from me as an institution for their past research. When I hear about people who are doing their own research but not posting anything, I get pretty skeptical unless they’ve produced good Alignment research in the past (producing other technical research counts for something, my own intuition is that the pre-paradigmatic nature of Alignment research is different enough that the tails come apart), and my system 1 says (especially if they’re getting funded):
(My system 2 would obviously use a different tone than that, but it largely agrees with the substance.)
Also, to echo points made by Jan, LW is not the best place for a broad impression of current research, the Alignment Forum is strictly better. But even the latter is somewhat skewed towards MIRI-esque things over CHAI, OpenAI, and Deepmind’s stuff, here’s another decent comment thread discussing that.
My sense is mostly (1), with maybe some additional disagreement over what online contributions are actually a sign of competence. But usually I am quite excited when an active online contributor applies.
I share your perspective that I am most excited about people who participate in AI Alignment discussion online, but we’ve received relatively few applications from people in that reference class.
Some of the people we’ve given grants to, were the result of Alex Zhu doing a lot of networking with people who are interested in AI alignment, which tends to select on some slightly different things, but given the lack of applications from people with a history of contributing online, that still seems pretty good to me.
This is an excellent question!
The reason may be somewhat simple: most AI alignment researchers do not participate (post or comment) on LW/AF or participate only a little. For more understanding why, check this post of Wei Dai and the discussion under it.
(Also: if you follow just LW, your understanding of the field of AI safety is likely somewhat distorted)
With hypothesis 4.&5. I expect at least Oli to have strong bias of being more enthusiastic in funding people who like to interact with LW (all other research qualities being equal), so I’m pretty sure it’s not the case
2.&3. is somewhat true at least on average: if we operationalize “private people” as “people who do you meet participating in private research retreats or visiting places like MIRI or FHI”, and “online people” as “people posting and commenting on AI safety on LW” than the first group is on average better.
1. is likely true in the sense that best LW contributors are not applying for grants
I’m wondering how many such people there are. Specifically, how many people (i) don’t participate on LW/AF, (ii) don’t already get paid for AI alignment work, and (iii) do seriously want to spend a significant amount of time working on AI alignment or already do so in their free time? (So I want to exclude researchers at organizations, random people who contact 80,000 Hours for advice on how to get involved, people who attend a MIRI workshop or AI safety camp but then happily go back to doing non-alignment work, etc.) My own feeling before reading your comment was that there are maybe 10-20 such people, but it sounds like there may be many more than that. Do you have a specific number in mind?
I’m aware of this, and I’ve seen Wei Dai’s post and the comments there. Personally I don’t see an easy way to get access to more private discussions due to a variety of factors (not being invited to workshops, some workshops being too expensive for it to be worth traveling to, not being eligible to apply for certain programs, and so on).