This began as a Google Doc of notes to self. Itās still pretty close to that statusāi.e., I donāt explain why each thing is relevant, havenāt spent a long time thinking about the ideal way to organise this, and expect this shortform omits many great readings and tips. But several people indicated finding the doc useful, so Iām now sharing it more widely.
Iāve done ~6 FTE months of academic research (producing one paper) and ~1 FTE year of longtermist research at EA orgs.
I do not have excellent, one-size-fits-all, easy-win answers to how to do high-impact research; I just have various scraps and ideas, and this shortform is merely intend to collect those.
This shortform expresses my personal views only (and is based on a doc created before I started either of my current jobs).
Readingsāmisc
* Asterisks indicate sources I havenāt yet properly read myself.
Iād guess that the best approaches to the first four pointsācoming up with important research questions, picking among them, coming up with a ToC, and assessing impactāwill differ considerably for different topics/āareas. They might be hardest for longtermism, as in that cause area goals are far away and sometimes unclear, and we get limited feedback loops.
On point 6 especially (regarding managing others), but also 1-5, I find it useful to think about the following interrelated points:
How much control/āguidance vs free rein should researchers be given?
Should we value cohesion or not?
Steve Jobs analogy:
A biography of Jobs suggested that, instead of making the product the market wants, he leaned towards making the product he had strong inside-view reasons to think the market will want (even if they donāt know it yet)
Analogously, should research answer the questions decision-makers know they want answered, or questions we expect they would value answers to but that they havenāt yet even noticed exist or understood the significance of?
The latter might have tend to have a lower probability of success (including because you might just overestimate the importance), but might tend to lead to larger successes when it does succeed (because it causes a more fundamental shift and/āor was less likely to have been done soon by someone else anyway).
Again, this was originally written like notes to selfālet me know if I should clarify anything.
Iām grateful to Aleksandr Berezhnoi and Edo Arad for making useful comments on the Doc version of this shortform, and to Kat Woods for encouraging me to make a public post out of the Doc.
Did you consider if this could get more views if it was a normal ālongformā post? Maybe itās not up to your usual standards, but I think itās pretty good.
I did consider that, but felt like maybe itās too much of just a rough, random grab-bag of things for a top-level post. But if the shortform or your comment gets unexpectedly many upvotes, or other people express similar views in comments, I may āpromoteā it.
Readings and notes on how to do high-impact research
This shortform contains some links and notes related to various aspects of how to do high-impact research, including how to:
come up with important research questions
pick which ones to pursue
come up with a ātheory of changeā for your research
assess your impact
be and stay motivated and productive
manage an organisation, staff, or mentees to help them with the above
Iāve also delivered a workshop on the same topics, the slides from which can be found here.
The document has less of an emphasis on object-level things to do with just doing research well (as opposed to doing impactful research), though thatās of course important too. On that, see also Effective Thesisās collection of Resources, Advice for New ResearchersāA collaborative EA doc, Resources to learn how to do research, and various non-EA resources (some are linked to from those links).
Epistemic status
This began as a Google Doc of notes to self. Itās still pretty close to that statusāi.e., I donāt explain why each thing is relevant, havenāt spent a long time thinking about the ideal way to organise this, and expect this shortform omits many great readings and tips. But several people indicated finding the doc useful, so Iām now sharing it more widely.
Iāve done ~6 FTE months of academic research (producing one paper) and ~1 FTE year of longtermist research at EA orgs.
I do not have excellent, one-size-fits-all, easy-win answers to how to do high-impact research; I just have various scraps and ideas, and this shortform is merely intend to collect those.
This shortform expresses my personal views only (and is based on a doc created before I started either of my current jobs).
Readingsāmisc
* Asterisks indicate sources I havenāt yet properly read myself.
Posts tagged Research methods
Posts tagged Org strategy
EA needs consultancies
Ingredients for creating disruptive research teams (Forum post)
Ingredients for building disruptive research teams (EAG talk by the author of the post)
Can we intentionally improve the world? Planners vs. Hayekians
Building collaborative research teams ā Jess Whittlestone
https://āāwww.charityentrepreneurship.com/āāresearch.html (and the pages linked to under āOUR RESEARCH PROCESSā)
What can someone do to become a stronger fit for future Open Philanthropy generalist RA openings?
Tips On Doing Impactful ResearchāEffective Thesis, 2020
Hard problem? Hack away at the edges.
Some of NuƱo Sempereās recent work
Advice from 80,000 Hours: How to do high impact research
Rethink Priorities 2020 Impact and 2021 StrategyāEA Forum
Center on Long-Term Risk: 2021 Plans & 2020 ReviewāEA Forum
EAG talk on Aggregating Knowledge in EA
Literature Review for Academic OutsidersāLessWrong
Scholarship & Learning tagāLessWrong *
ReadingsāPrimarily relevant to generating and picking questions
How to generate research proposalsāEA Forum
Advice from Charity Entrepreneurship: How to do research that matters
Transcript: Karolina Sarek: How to do research that mattersāEA Forum
Research as a stochastic decision process ā Jacob Steinhardt (see also Should marginal longtermist donations support fundamental or intervention research?)
Potential benefits & downsides of making and/āor sharing a research agenda [upcoming post by me, link will be added later]
Should marginal longtermist donations support fundamental or intervention research?
A case for strategy research: what it is and why we need more of it
Why EAs researching mainstream topics can be useful
Research project planning templates/āresources [shared]
Readingsāprimarily relevant to theories of change
Theory of Change in Research [slides] (see also the accompanying worksheet)
Do research organisations make theory of change diagrams? Should they?
https://āālongtermrisk.org/āāidentifying-plausible-paths-to-impact/āā
Modeling the impact of safety agendas
A interesting comment thread debate on pros and cons of using ābackchainingā to decide what research projects to work on and how
Readingsāprimarily relevant to assessing impact
Rethink Priorities Impact SurveyāEA Forum
Should surveys about the quality/āimpact of research outputs be more common?
Posts tagged Impact assessment
Readingsāprimarily relevant to managing an organisation, staff, or mentees
Collection of collections of resources relevant to (research) management, mentorship, training, etc.
Notes
Iād guess that the best approaches to the first four pointsācoming up with important research questions, picking among them, coming up with a ToC, and assessing impactāwill differ considerably for different topics/āareas. They might be hardest for longtermism, as in that cause area goals are far away and sometimes unclear, and we get limited feedback loops.
On point 6 especially (regarding managing others), but also 1-5, I find it useful to think about the following interrelated points:
Should we be more like planners or Hayekians?
Should we be more top-down or bottom-up?
How much control/āguidance vs free rein should researchers be given?
Should we value cohesion or not?
Steve Jobs analogy:
A biography of Jobs suggested that, instead of making the product the market wants, he leaned towards making the product he had strong inside-view reasons to think the market will want (even if they donāt know it yet)
Analogously, should research answer the questions decision-makers know they want answered, or questions we expect they would value answers to but that they havenāt yet even noticed exist or understood the significance of?
The latter might have tend to have a lower probability of success (including because you might just overestimate the importance), but might tend to lead to larger successes when it does succeed (because it causes a more fundamental shift and/āor was less likely to have been done soon by someone else anyway).
Again, this was originally written like notes to selfālet me know if I should clarify anything.
Iām grateful to Aleksandr Berezhnoi and Edo Arad for making useful comments on the Doc version of this shortform, and to Kat Woods for encouraging me to make a public post out of the Doc.
Thanks for posting this! This is a gold mine of resources. This will save the Nonlinear team so much time.
Did you consider if this could get more views if it was a normal ālongformā post? Maybe itās not up to your usual standards, but I think itās pretty good.
Nice to hear you think so!
I did consider that, but felt like maybe itās too much of just a rough, random grab-bag of things for a top-level post. But if the shortform or your comment gets unexpectedly many upvotes, or other people express similar views in comments, I may āpromoteā it.
More concretely, regarding generating and prioritising research questions, one place to start is these lists of question ideas:
Research questions that could have a big social impact, organised by discipline
A central directory for open research questions
Crucial questions for longtermists
Some history topics it might be very valuable to investigate
This is somewhat less noteworthy than the other links
And for concrete tips on things like how to get started, see Notes on EA-related research, writing, testing fit, learning, and the Forum.