How to generate research proposals
Epistemic status: The advice I give is based on anecdotic experience, and what works well for me might not transfer well to others, but my past self would have found this article quite useful
As part of the application to FHI’s Summer Fellows Program I had to submit a research proposal, and once I finished my original proposal I had to engage again with the process to select my next research project.
In this article I will explain the particular approach I chose for this task, and provide some commentary on what worked well and what did not.
I expect you will get a lot of mileage out of this article if you are an early career researcher struggling to generate research ideas or if you are considering pursuing a career in research and would like to peek into how an important chunk of a researcher’s time is spent.
In short, my top advice for early career researchers is:
Create a pipeline that allows you to note down interesting research questions without committing to them and share them with others to receive feedback
To come up with research questions, read research agendas and talk to other people to find disagreements.
Think about what makes a good research project to develop taste. My personal take is that what matters most is having concrete research questions, that you can devise a good methodology to answer those questions, that the output you intend to produce has a clear audience and goals and that your environment and background are a good fit to the question.
When selecting between ideas, try to fail fast, and don’t be afraid to discard ideas. To do this, try writing up outlines of the projects and schedule conversations with other researchers with overlapping interests.
To avoid spending too much time on the meta level, preallocate time to making the decision. If you have trouble deciding between concrete projects, just pick the one you are most excited about and work on it until you find a roadblock.
Generating and prioritizing research proposals is a critical part of strategic research, and my impression is that systematic approaches are underexplored.
The overall process I used is split up in three chunks: generating research ideas, curating the ideas and operationalizing the ideas. We will cover each of these parts in separate sections of this article.
The outcome of this process is a detailed outline of a research project, which clearly explains the methodology and value of a project and that can be reused as an introduction to a potential publication.
The whole process took ~2 weeks. Progress was uneven, on some days I progressed a lot on the research proposal generation and others I just focused on other projects.
Brainstorming ideas
For the brainstorming phase I set up a brainstorming document where I collected questions that had drawn my attention, together with useful context and notes on what inspired the idea in a bullet point format.
To fill the list I resorted to the following strategies:
Reading research agendas
Talking to other people and finding disagreements
Creating taxonomies of areas of interest
Reading research agendas proved to be quite fruitful. With research agendas I mostly refer to collections of open questions compiled by other researchers. If you already have a topic of interest you can search research agendas focused precisely on that; if not they can provide an excellent introduction to new areas of research.
Some examples of research agendas are Allan Dafoe’s AI Governance Research Agenda [REF], Luke Muehlhauser’s How to study superintelligence strategy [REF] or OpenPhil’s Important unresolved research questions in macroeconomic policy [REF].
Talking to people and finding disagreements was also really useful, albeit it is harder to execute on demand. I had some really good research ideas finding common topics of interest with other researchers, finding out we disagreed about a particular decision we both cared about and finding the root of the disagreement.
The core advantage of this method is that it can lead you to realize some internalized insights that may be useful to share with others, and points of confusion. This also has the upside of leading to finding good project collaborators.
For example, my latest academic article on arXiv can be traced back to a disagreement between my coauthor Pablo Moreno and me on the importance of quantum computing for AI safety reasons.
One downside of the talking method is that sometimes it is tempting to dismiss ideas that your colleagues have criticize. I partially solved this by having quite a low bar to put ideas in the brainstorming doc and treating the viewpoints of others as interesting evidence that I might be missing some important information about the world.
How I originally started brainstorming was creating a taxonomy of areas I was interested in (like AI Safety, Forecasting or Education) and then zooming in to particular topics (like Inverse Reinforcement Learning, how forecasting is used in policy or skill transfer). This was less useful than I expected but it was a good strategy to try overall.
This process ended up with ~30 research questions first time I started, plus I kept adding ideas afterwards. It was very useful to keep the ideas numbered for future reference and discussion.
One thing I would like to try in the future is to keep a log of interesting papers that I can then refer back to to think of possible improvements and questions that they inspire.
After compiling a low-bar list of research ideas, I invited people to my google doc to highlight the ones that seemed interesting to them and put comments about references. My primary goal was to test which ones passed the first test of attractiveness.
Based on the feedback I gathered, I started color coding the ideas, highlighting in green the ones that started to feel more exciting, in yellow the ones I wanted to think more about and in red the ones that I got discouraged from pursuing. I didn’t highlight the ones I wanted to seek more feedback on. The process was in flux, and several ideas changed colors several times.
The end result looked something like this:
Generating the original set of ideas took about ~4 days, and collecting feedback on them took ~4 days as well.
Curating ideas
The ideas that I became excited about by the end of the brainstorming and feedback process were transferred to a new document, the curation doc. Of the original ~30 research ideas, about ~15 made it to the second document.
I proceeded to think about characteristics that good research questions should possess. I came up with the following:
Impact (IMP) - The question is important for the world
Tractability (TRC) - I can devise a good methodology to answer the question
Neglectedness (NEG) - I don’t expect others to independently work on it
Learning value (LV) - I expect to learn useful things from working on it
Actionability (ACT) - The research has a clear audience and can inform action
Respectability (RSP) - The output signals my research competence
Personal fit (PF) - I can provide a differential value for the question
Organization fit (OF) - FHI is a good environment to work on this question
I scored each question on each characteristic, from LOW to MID to HIGH, leaving question marks on places I felt confused about how to score. I added a final column to summarize how I felt about a question after scoring it. This is how the score looked like in the end:
Overall I feel like I was trying to be too comprehensive with the characteristics I chose, and that the exercise would have been more useful if had I restricted myself to fewer categories. Of the categories above, the ones that felt more useful are tractability, actionability and organizational fit.
If I was doing it again I would consider splitting tractability into scope (having concrete and small research questions) and methodology (having clear strategies to tackle these questions); and I would merge personal and organizational fit in a single fit score.
After the table was done I checked again with myself how excited I would be about working on the ideas. I felt I was still excited about ~6 of the projects, so I picked them up and developed them further.
In order to do that I used the following bullet point template:
Context—What is the relevant context around that question, what led me to it, some background claims and assumptions
Research questions—What are the concrete research questions I am hoping to answer. I would often start with general questions and add sub points with more concrete sub questions that split the original question.
Methodology—how I planned to answer the questions above. This was fairly tricky in many situations, and required a lot of creativity.
Upside and downside—why I thought this project was valuable, and what risks did I see associated to it. I would often reference the characteristics from before.
People to talk to—people I know who would be interested in talking about this question
Bibliography—previous and related work on the questions of interest
I was fairly liberal on deciding which parts of the template I expanded on for each question and which ones were kept brief or even empty.
When deciding what parts to expand on and which ones to keep brief it was useful to think in terms of what were my uncertainties around the projects (Maybe this question is really hard to answer?) and zero in on them (Can I outline a methodology in 5 minutes?).
At this point, I sought further feedback from others. This time I was primarily seeking feedback that would allow me to develop the methodology further or discard projects before investing too much effort in them.
I sought feedback via two strategies: sending the document to people and asking for general comments and seeking out people I knew were working on some particular ideas related to mine and questioning them / exposing my ideas to them.
As I received feedback I started color coding them by how excited I was, using the same conventions as before: RED for discouragement, YELLOW for further thought needed, GREEN for excitement and no highlight for further feedback needed.
This is what the ideas looked like at this stage:
Scoring the ideas took <1 day, and writing the first draft of the templates took ~2 days.
Operationalizing the proposals
After seeking feedback, of the ~6 ideas that I developed a bit further I was still excited about pursuing ~3 of them.
For each of these ~3, I scheduled conversations with other fellows and researchers with overlapping interests so we could talk at length about them.
The conversations took different shapes, from them telling me about their projects and reasons to pursue them to having MIRI-style deconfusing sessions where I tried making progress on the questions [REF].
All insights and references I got were used to improve the bullet point template of the previous section.
Every so often I would try to write a formal proposal. The goal of this exercise would be to defend the value of the project to a hypothetical board deciding if they would fund my project.
The proposal would be short (less than one page long), and include in redacted form the important points in the bullet-point template.
If the proposal felt hard to write or wrong in any sense, I would stop to check where that feeling was coming from. Was my methodology incomplete? Did I feel there was low hanging fruit information from scheduling another meeting? Should I try just solving the question with a crude model and see if I was still excited? Or was I simply tired and should take a break?
One unexpected benefit of this exercise is that I was able to copy with very little modification my original research proposal and reuse it as an introduction to the research article I subsequently coathored.
The resulting proposals would look something like this:
In order to make the final decision, it was helpful to allocate time to the process. That is, I explicitly decided I would dedicate two weeks to choosing a good research question. While I did make my decision before the deadline, the commitment pushed me towards seeking progress earlier.
The process of me making the final decision looked like me realizing that I was already making substantial progress on one particular project.
Having the conversations and playing around with the ideas until I wrote a concrete proposal took ~4 days.
Conclusions
In this piece I have exposed how I generated research proposals for the FHI Summer Fellows Program. The process consisted of three phases: brainstorming research ideas, curating them and operationalizing the proposals.
I think that the process of methodically thinking about research idea generation and prioritization has helped me develop a better research taste, and I would recommend it to people who are starting to do research. I intuit that as your research aesthetic matures explicit procedures become less useful.
The most salient benefit of engaging with this project has been creating a pipeline for noting down and developing research ideas that I expect to use and refine going forward. Doing it deliberately and explicitly has also lead to some interesting conversations with the other summer fellows and the rest of FHI’s staff.
Some concrete feedback I would give myself about the research generation pipeline, with the benefit of hindsight:
Thinking in terms of research questions and concrete methodology to answer them instead of in terms of areas I want to work on is probably the biggest learning of my experience doing research over the last year.
Reading research agendas and papers while having a document opened where I could dump research questions was more productive than trying to conjure research ideas from an empty page. Furthermore, the latter method did not lead to substantially more original ideas anyway.
Talking to people was very useful, and I should have done that earlier and more often. The commenting features of google docs are amazing.
Keeping the brainstorming doc and the curation doc separately was useful so I could selectively share them with different people for different levels of feedback.
Overall I think that scoring the ideas on the different characteristics was not that useful out of itself, but the process of thinking how do good ideas look like helped me develop a better taste for good research. Going forward I expect I will not emphasize such a methodical approach to sorting the ideas versus just talking to others. On the other hand, it might have been more useful had I restricted myself to fewer categories.
Allocating explicit time and setting a concrete goal for this process was useful, and made me seek progress earlier.
Hopefully this article has helped you reflect about how to create research questions and you can take away some ideas of things to try. Let me know your thoughts in the comments!
This article was written by Jaime Sevilla. I want to thank Linh Chi Nguyen for useful feedback and editing.
- Some readings & notes on how to do high-quality, efficient research by (17 Nov 2021 17:01 UTC; 69 points)
- My experience on a summer research programme by (22 Sep 2019 9:54 UTC; 50 points)
- 's comment on MichaelA’s Quick takes by (19 May 2021 15:29 UTC; 30 points)
- 's comment on Concrete project lists by (27 Mar 2017 2:23 UTC; 5 points)
In How to generate research proposals I sought to help early career researchers in the daunting task of writing their first research proposal.
Two years after the fact, I think the core of the advice stands very well. The most important points in the post are:
Develop a pipeline to collect ideas as they come to you.
Think about scope (is your question concrete?) and methodology (is your question tractable?).
Devote some time to figuring out what good research looks like.
None of this is particularly original. The value I added is collecting all the advice in a single place and giving an example of how this might shake out in practice.
One thing I would emphasize more is that there are many “obvious” projects lying around, which clearly are valuable, but nobody else has done because they are a lot of work. For example, compiling parameter sizes for Machine Learning models since the 1950s. These projects are great for early career researchers. I think this point is better covered in other similar posts.
In terms of impact, my post has been well received. I keep seeing occassional upvotes, it has been referenced in other articles, and some people have thanked me for writing it.
In terms of writing, it could be better. I use long sentences, and weird phrasing. I do not know what I was thinking titling it “How to generate research proposals” instead of “How to write research proposals”. Still, the summary at the beginning is good, and the use of bold font (Linh Chi Nguyen’s fault) helps focus attention on the most important points.
As further reading I recently came across Research as a Stochastic Decision Process, which discusses another systematic approach to research.
Summary copy pasted from the article:
Summary
Many of our default intuitions about how to pursue uncertain ideas are counterproductive:
We often try easier tasks first, when instead we should try the most informative tasks first.
We often conflate a high-level approach with a low-level instantiation of the approach.
We are often too slow to try to disprove our own ideas.
Building frameworks that reify the research process as a concrete search problem can help unearth these incorrect intuitions and replace them with systematic reasoning.
Thanks for this! I agree that my past self would have found this, or something along those lines, a very useful starting point. That said, I think I would also have found this process a little overwhelming (esp during initial projects, when I was trying to assess personal fit more than anything else). So here’s a quote from the GovAI agenda which I find useful whenever I get lost: _”Ultimately, perhaps the simplest rule of thumb is to just begin with those questions or ideas that most grab you, and start furiously working.”_ (p.13)
Nice, I particularly like the table and bullet-point forms you used for curating your ideas—I often find myself with too many ideas to work on and this seems like a good way to take an objective overview.
During my PhD I read ‘Becoming a successful scientist’ - this presented a strategic approach to scientific discovery and problem selection (Section 3.1) that I haven’t really seen elsewhere. It focused on science, but the ideas of looking for contradictions, paradoxes, new viewpoints or different scales may also be helpful for generating research questions in philosophy/economics.
I have a PDF of the book I’m happy to send by email, PM me.
This is also my impression.
I guess some people will probably be just as interested in pursuing the research ideas of others as pursuing their own. For those people, maybe creating a thread here or on Facebook with a message like “I’m looking for research ideas on Topic X” would work?
Personally, I’ve been noting down research ideas on EA topics that interest me (AI safety and improved institutions would be the two big ones I guess) for quite a while, and I’m pursuing the ideas at a much lower rate than I’m noting them down! So maybe it’d be good for me to connect with people who are hunting for ideas somehow?
Totally second the motion of empowering junior researchers to ask for research ideas here and somewhere else.
Also, I’d encourage you to write down your own research agenda in the form of a blogpost listing some open questions in this forum!
It will be useful for other researchers and you will get interesting feedback on your ideas :)
I’d (very belatedly) second that. You could also then comment about that list of questions in this central directory for open research questions. (Or even just write the whole list as a comment on that post to begin with, I guess.)