Sorry, it’s a bummer to be scooped or even just feel scooped.
Most academic fields have, I think, less cross-org coordination than AI governance. I would be hesitant about trying to do generally more cross-org coordination in this space given that it’s a departure from (what I view as) the norm in other fields.
As I am reading applications to hire AI governance researchers, one of my big questions is “has this person done relevant work before, successfully?”. I don’t think it would be much of a mark against that work if it was also similar to other work that was released at the same time, as long as it didn’t seem like there was plagiarism and did seem like there were novel contributions.
Relatedly, multiple researchers teams taking independent stabs at the same question is often useful for reaching a higher quality of overall work, as they sometimes come up with different ideas/emphases/etc.
Some researchers have said (but I’m unsure where I land) that you almost never actually get scooped. Usually projects are a bit different in a way that is important and that you can emphasize in your output. Also you can boostrap from that work to make your project even better (but again, be clear about your original contributions vs. others’).
Thank you, Aaron. This is an important point. I agree that it is worth putting out any work you have made out there. This is exactly the spirit of draft amnesty week! In this case, though, I had only conducted preparatory work to delineate the project. Continuing as if the other paper did not exist was not a realistic path, and my contribution would not have been meaningfully distinct. When I start a research project independently, I do it without much mentorship or institutional feedback, which makes it genuinely hard to assess whether I am working on the right question, using the right methods, or producing something useful to the field.
As someone reviewing applications, how do you evaluate independent research produced outside of a fellowship or academic context? Is there a threshold of rigor or novelty below which it hurts more than it helps to include it? And do you have suggestions for how early-career people in this transition can get lightweight feedback on research directions before investing weeks into a project? I ask because the fellowship application process often requires demonstrating prior work to get the mentorship needed to produce prior work, and I am trying to figure out how to navigate this as efficiently as possible.
As someone reviewing applications, how do you evaluate independent research produced outside of a fellowship or academic context?
I’m not sure how to answer this. I try to evaluate all work based on its quality; whether a project was single-authored also matters a fair amount (and high quality single-author work is an especially strong signal).
Is there a threshold of rigor or novelty below which it hurts more than it helps to include it?
Maybe for rigor, probably not for novelty. Applicants and other researchers should of course be up front about what are their contributions.
And do you have suggestions for how early-career people in this transition can get lightweight feedback on research directions before investing weeks into a project?
Weeks sound like it might be a lot. I encourage people to do Apart Research Sprints or other Hackathon-style things which are shorter. I’m not really sure about getting lightweight feedback. In my experience, when junior people ask me for feedback on a project idea, the project idea is usually too broad or vague for me to know if it’s a good project, and they have usually put less than 30 min of effort into it. So maybe my advice there is something like “if you are going to ask a more established researcher for feedback on your project plan, you should have already put a couple hours into the project, including surveying the relevant literature, coming up with a detailed project plan, and doing a little bit of de-risking”. I’m not sure, maybe that’s more intense than I endorse. Fortunately, even without the goal of getting feedback from somebody else, these are useful steps to begin a project with.
I will also note that prior work does not always have to be extremely relevant. Academia exists and is by far the place where the most people learn research skills.
A couple of my thoughts, written quickly:
Sorry, it’s a bummer to be scooped or even just feel scooped.
Most academic fields have, I think, less cross-org coordination than AI governance. I would be hesitant about trying to do generally more cross-org coordination in this space given that it’s a departure from (what I view as) the norm in other fields.
As I am reading applications to hire AI governance researchers, one of my big questions is “has this person done relevant work before, successfully?”. I don’t think it would be much of a mark against that work if it was also similar to other work that was released at the same time, as long as it didn’t seem like there was plagiarism and did seem like there were novel contributions.
Relatedly, multiple researchers teams taking independent stabs at the same question is often useful for reaching a higher quality of overall work, as they sometimes come up with different ideas/emphases/etc.
Some researchers have said (but I’m unsure where I land) that you almost never actually get scooped. Usually projects are a bit different in a way that is important and that you can emphasize in your output. Also you can boostrap from that work to make your project even better (but again, be clear about your original contributions vs. others’).
Thank you, Aaron. This is an important point. I agree that it is worth putting out any work you have made out there. This is exactly the spirit of draft amnesty week! In this case, though, I had only conducted preparatory work to delineate the project. Continuing as if the other paper did not exist was not a realistic path, and my contribution would not have been meaningfully distinct. When I start a research project independently, I do it without much mentorship or institutional feedback, which makes it genuinely hard to assess whether I am working on the right question, using the right methods, or producing something useful to the field.
As someone reviewing applications, how do you evaluate independent research produced outside of a fellowship or academic context? Is there a threshold of rigor or novelty below which it hurts more than it helps to include it? And do you have suggestions for how early-career people in this transition can get lightweight feedback on research directions before investing weeks into a project? I ask because the fellowship application process often requires demonstrating prior work to get the mentorship needed to produce prior work, and I am trying to figure out how to navigate this as efficiently as possible.
I’m not sure how to answer this. I try to evaluate all work based on its quality; whether a project was single-authored also matters a fair amount (and high quality single-author work is an especially strong signal).
Maybe for rigor, probably not for novelty. Applicants and other researchers should of course be up front about what are their contributions.
Weeks sound like it might be a lot. I encourage people to do Apart Research Sprints or other Hackathon-style things which are shorter. I’m not really sure about getting lightweight feedback. In my experience, when junior people ask me for feedback on a project idea, the project idea is usually too broad or vague for me to know if it’s a good project, and they have usually put less than 30 min of effort into it. So maybe my advice there is something like “if you are going to ask a more established researcher for feedback on your project plan, you should have already put a couple hours into the project, including surveying the relevant literature, coming up with a detailed project plan, and doing a little bit of de-risking”. I’m not sure, maybe that’s more intense than I endorse. Fortunately, even without the goal of getting feedback from somebody else, these are useful steps to begin a project with.
I will also note that prior work does not always have to be extremely relevant. Academia exists and is by far the place where the most people learn research skills.