Advice to junior AI governance researchers

This summer, I’m supervising some research fellows through Cambridge’s ERA AI Fellowship. The program started last week, and I’ve had conversations with about 6 fellows about their research projects & summer goals.

In this post, I’ll highlight a few pieces of advice I’ve found myself regularly giving to research fellows. This post reflects my own opinions and does not necessarily reflect the views of others at ERA.

Prioritize projects that have a clear target audience

Problem: One of the most common reasons why research products fail to add value is that they do not have a target audience. I think it can be easy to find a topic that is interesting/important, spend several months working on it, produce a 20-50 page paper, and then realize that you have no particular stakeholder(s) who find the work action-relevant.

Advice: Try to brainstorm what specific individuals you would want to have affected by your piece. This might be some folks in the AI safety community. This might be government officials at a relevant agency in the US or the UK. Prioritize projects that have a clear target audience and prioritize projects in which you have a way of actually getting your paper/product to that target audience. Ideally, see if you can talk to representative members of your target audience in advance to see if you have a good understanding of what they might find useful.

Caveat #1: Gaining expertise can be a valid reason to do research. Sometimes, the most important target audience is yourself. It may be worthwhile to take on a research project because you want to develop your expertise in a certain area. Even if the end product is not action-relevant for anyone, you might have reason to believe that your expertise will be valuable in the present or future.

Caveat #2: Consider target audiences in the future. Some pieces do not have a target audience in the present, but they could be important in the future. This is particularly relevant when considering Overton Window shifts. It’s quite plausible to me that we get at least one more major Overton Window shift in which governments become much more concerned about AI risks. There may even be critical periods (lasting only a few weeks or a few months) in which policymakers are trying to understand what to do. You probably won’t have time to come up with a good plan in those weeks or months. Therefore, it seems like it could be valuable to do the kind of research now that helps us prepare for such future scenarios.

Be specific about your end products

Problem: A lot of junior researchers find tons of ideas exciting. You might have a junior researcher who is interested in a topic like “compute governance”, “evals”, or “open-sourcing.” That’s a good start. But if the research proposal is to “come up with gaps in the evals space” or “figure out what to do about open-source risks”, there’s a potential to spend several months thinking about high-level ideas and not actually producing anything concrete/specific It’s common for junior researchers to overestimate the feasibility of tackling big/broad research questions.

Advice: Try to be more specific about what you want your final products to look like. If it’s important for you to have a finished research product (either because it would be directly useful or because of the educational/professional benefits of having the experience of completing a project), make sure you prioritize finishing something.

If you’re interested in lots of different projects, prioritize. For example, “I want to spend time on X, Y, and Z. X is the most important end product. I’ll try to focus on finishing X, and I’ll try not to spend much time on Y until X is finished or on track to be finished.”

Caveat #1: You don’t need to aim for a legible end product. Sometimes, it’s very valuable to spend several months examining your high-level thoughts in an area. Deconfusing yourself about a topic (what’s really going on with evals? Are they actually going to help?) can be an important output.

Caveat #2: Priorities can change as you learn more about the topic. If you start on X, and then you realize it’s not actually as valuable as you thought, you should be willing to pivot to Y. The point is to make this an intentional choice– if you intentionally decide to deprioritize X, that’s great! If you blindly just pursue lots of stuff on X and Y and then a few months later you realize you haven’t actually finished X (even though you wanted to), that’s less great.

Caveat #3: Follow your curiosity & do things that energize you. Suppose I think X is important and I want to finish X before starting Y. But one day I wake up and I’m just feeling really fired up to learn more about Y, and I want to put X aside. One strategy is to be like “no! I must work on X! I have made a commitment!” Another strategy is to be like “OK, like, even though Omega would say it’s higher EV to work on X in a world where I were a robot with no preferences, I actually just want to follow my curiosity/energy today and work on Y.” Again, the point is to just be intentional about this.

Take advantage of your network (and others’ networks)

Problem: A lot of people who are attracted to research are introverts who love reading/thinking/writing. Those are essential parts of the process. But I think some of the most high-EV moments often come from talking to people, having your whole theory of change challenged, realizing that other people are working on similar topics, building relationships who can help your work (either on the object-level or by helping you connect to relevant stakeholders), and other things that involve talking to people. A classic failure mode is for someone to spend several months working on something only to have someone else point our a crucial consideration that could’ve significantly shaped the project earlier on.

Advice: Early on, brainstorm a list of experts who you might want to talk to. Having short outlines to share with people can be helpful, here. When I start a new project, I often try to write up a 1-2 page outline that describes my motivation and current thinking on a topic. Then, I share it with ~10 people in my network who I think would offer good object-level feedback or connect me to others who could. I also suggest being explicit about the kind of feedback you’re looking for (e.g., high-level opinions about if the research direction is valuable, feedback on a specific argument, feedback on the writing style/quality, etc.)

If you don’t yet have a super strong network, that’s fine! If you’re in a structured research program, take advantage of the research managers and research mentors. If not, you can still probably message people like me. This can be scary, but in general, I think junior researchers err too much on the side of “not reaching out enough to the Fancy Smancy Scary People.”

Caveat #1: But what if my doc is actually really bad and not ready to be sent to Fancy Smancy Scary Person Who Will Judge me For Being Dumb or Wasting Their Time? Yeah, that’s fair. I do empathize with the fact that this can be hard to assess, especially early on. I think my biggest piece of advice here is to start with Less Scary people, see what they think, and see if they recommend any of the Super Senior people.

Note also that scaryness isn’t just a function of seniority– there are plenty of Super Nice Senior People and also (being honest here) some Scary/Judgey/Harsh non-senior people. Again though, I think junior people tend to err on the side of not reaching out, and I suggest reaching out to research managers if you have an idea and you’re wondering who to share it with.

Miscellaneous

Shallow reviews can help you learn/prioritize. If you’re not sure what you want to focus on, consider spending the first ~2 weeks doing shallow reviews of multiple topics, identifying your favorite topic, and then spending the remaining ~6 weeks diving deeper into that topic.
“One of the most important products of your research is your sustained engagement on the topic. Do not think about summer projects– think about programs of research you could see yourself spending years on.” A quote a senior researcher recently shared with me that I found useful.
You don’t need to produce a paper. I think the “default” assumption for a lot of people is that they need to produce a 20+ page paper that could go on Arxiv or a long EAF/LW post. Consider shorter materials. Examples include policy memos, tools that government stakeholders could use, draft legislative text, and short explainers of important topics.
Remember that policymakers are unlikely to just “stumble upon” your work. In some cases, a research output is so strong or so widely shared that people might stumble upon it “in the wild.” For the most part, I think you should assume that people won’t notice your work– you have to figure out how to get it to them. Examples include “directly emailing relevant people” or “going through someone who has an existing relationship with X person.” I recently heard a little slogan along the lines of “doing the research is step one; getting people to pay attention to it is step two. Don’t skip step two.”