Learning to ask action-relevant questions
This is the first of several research skills I wish I’d known about before doing research myself. This post is aimed at students who are trying to come up with research questions for their research projects, and might be helpful for other early-stage researchers. Thanks to Vojta Kovarik and Jaime Sevilla for providing feedback.
(Note: I also published this article on my personal blog).
Content
Introduction
Action-relevance
Where this mindset might fail: incentives and other goals
In short
Exercises
Introduction
One skill I wish I had known about much earlier is to ask myself the question “What would be different in a world where you have answered this question?”.
This is so because, like many generally curious people, I easily get distracted by my wish to understand things at large. Maybe this is partly the fault of my training as a philosopher, but certainly a property of mine that has attracted me to philosophy. I would go around asking questions like “Well, but how does science work?”.
This was actually a question I had a few months ago. At the time, I was working for AI Impacts and visiting the FHI office in Oxford. The question itself had no direct link to my work – which was a lot more hands-on in nature – but, well, I was curious anyway. And then, FHI is a very tempting place for that sort of exploration, given that you can ask exciting researchers what they’re working on over lunch.
Action-relevance
Something I learnt in conversation with people at FHI is to focus more on action-relevance [1] when trying to come up with research questions. A brief working definition for action-relevant research might be “research producing an output that is likely to influence further action in a meaningful way”. In other words: “Is answering this research question actually going to change anything I care about?”.
That said, I don’t mean “action-relevance” in the sense of focusing only on questions that yield a concrete output that you can see the application of. While it’s easy to see the merit of that sort of question, it is sometimes hard to see very specific applications of your research in advance, especially if you’re interested in more foundational questions or if you’re still very early on, exploring your interests, and narrowing down your question. What I have found helpful, instead, was thinking of this as a mindset for guiding my research questions towards what is particularly promising – and given that I very easily get lost in the space of possibilities, having this focus was very reassuring!
Let’s go back to the above question: “How does science work?”. The mindset I was in while asking this was something like “I’ve read some stuff on science as an institution and the kinds of incentives tugging at individual scientists. I am confused, and I wish to not be confused anymore. Wouldn’t it be nice to just understand science?”. I asked this question while making myself tea in the FHI kitchen and, sure enough, another researcher (also preparing a hot drink) said: “Well, what answer to that question would be action-relevant?”. This made me realise that I could spend a lot of time learning about how science works, without then being able to do very much with that knowledge. However, tweaking the original question could easily prevent that. In this case, it might be as simple as asking: “In what ways does science not work?”. If I found an answer to that question, identifying one or several instances where science isn’t working as intended, we’d be one step closer to doing something about it!
You can see in this example how the mindset works even for very broad questions. Indeed, I claim that it’s a useful mindset to have at all levels of granularity, for big and small questions alike.
Here’s an example of a smaller question. Imagine London Fire Prevention Ltd. has commissioned you to study house fires in London. You could take your clipboard, do surveys, and write a report titled: “How scared are Londoners of house fires?”. Compare this to a study where you ask yourself: “Are house fires more frequent in households with one or more smokers?”.
I’m not saying that the first question is useless, but the way it is formulated doesn’t make it clear why its answer might be important. I might find that Londoners are scared of fires, or I might find they’re not, but I’m not sure whether I’d do anything different either way![2] Focusing on my mindset, the most likely scenario in which I come up with the question “How scared are Londoners of house fires?” is where I am just, you know, kind of curious what Londoners say.
The way I have come to formulate this in my head is “If I imagine a world in which I have answered this question, what would look different?”. If my answer is “basically nothing”, then I guess I won’t want to bother answering the question.
Where this mindset might fail: incentives and other goals
This seems painfully obvious, I know. But I think I have in the past fallen prey to really just wanting to get a project under my belt, or succumbed to other incentives.
Imagine being at Uni and writing a research proposal for a thesis or project: you are not necessarily rewarded for asking questions that yield action-relevant answers. In fact, when I tried to bring up action-relevance with tutors, they have most of the time just told me that for a student thesis, I shouldn’t expect to produce interesting insights. But that misses the point that action-relevance is not about grandeur: even very, very small questions can, and should, have a reason to be asked and answered.
To be fair, even in student proposals you have to explain the relevance of your project. What I take issue with is that even while writing the section about the “relevance of my research”, I often felt tempted to not think about the relevance of my research, but instead pattern-match to what I thought would impress my supervisors. So, the question shifted from “Why is this research relevant?” to “What would look good in my proposal?” (Or “What would look good on my CV?”). This has led me to, for example, insert a section on qualitative methods mostly so that I could say I was using a “mixed-methods approach”. I also admit that doing fieldwork sounds more fun than sitting in the library all summer.
I can tell that at least some people in mainstream academia are also succumbing to those incentives. For example, this is apparent when you notice other people using a lot of jargon not to enhance the clarity of what they’re saying, but at the cost of it.
That said, it is probably very human to have motives that aren’t just about producing research that is relevant to the world. To the extent that I have those (I definitely have them), I try to acknowledge them, and then make an explicit decision on what to do. If I know that I need a sexy-looking publication to advance in my career, then I want to acknowledge that and weigh up the extent to which I care about having a great career versus answering relevant questions. I have heard about people in similar situations alternating between “good-looking” papers for their careers and less sexy questions that they thought were more important. As for my excitement about doing fieldwork – I guess I might find that it is important for me to learn qualitative techniques to apply them later on. But if my motivation for doing fieldwork is mostly about talking to interesting people in interesting places, I might consider going on an adventure holiday instead.
In short
I try to always remind myself why and how the answer to my question would be relevant. My go-to prompt for getting into that mindset is “If I imagine a world in which I have answered this question, what would look different?”. Firmly keeping the relevance of my question in mind is also a lot more motivating when I get to the point where I am mulling over the same thoughts again and again and am wondering about how I got there and why I wanted to do this in the first place.
And yes, also small questions can have this quality of relevance to them. To the extent that you have motivations that are not directed at action-relevance, try to acknowledge them and satisfy them in whatever way seems best. Especially early on, it might actually be a good idea to also think about which types of project will help you get skills that will make you a better researcher overall.
Exercises
It seems like exercises are a great way of thinking through and applying new concepts for oneself. Questions 1 and 2 are good for building up an intuition of the concept of action-relevance for yourself, and question 3 lets you apply the concept to your own research.
1 Which of these questions are action-relevant?
I didn’t come up with these questions specifically for this exercise, so I expect the answers to be somewhat messy. Just take up to five minutes to go through the questions to get a feel for what is different about the questions. For each question, try to imagine what would be different in a world where you know the answer to the question.
How likely is civilizational collapse as a result of a pandemic?
Understanding the intersection of conflict and disaster.
Is AI going to take over the world?
Should we work on AI risk? (Should I work on AI risk?)
How do t-tests work?
Is risk socially constructed?
Is climate change causing an increase in rainfall in England?
Epidemics: Compare preparedness to epidemic risk by looking at drug pre-stocking
Is there such a thing as objective knowledge?
Is climate change real?
How predictable is innovation?
What type of democracy should we aim for?
Which different steps could lead to civilizational collapse?
Can I ever really know whether another human feels pain?
Is flood risk increasing in Bangladesh?
“Now what are space and time? Are they actual entities?” (Kant, Critique of Pure Reason, A23/B37-8).)
2 Pick a question you didn’t find relevant.
What would you change about the question to make it more relevant? Write down which adjacent question you could ask that would be more relevant.
3 If you are working on your own research at the moment
If you’re in the early stages of finding a topic: Write down the broad areas you’re interested in. Write down questions associated with these areas. Ask yourself: “If I imagine a world in which I have answered this question, what would look different?” (or another prompt for action-relevance that works for you). Trying to stay in this mindset, tweak your existing questions or come up with new questions.
If you already have a research proposal: Read through it. Check: Is the project’s overall question action-relevant? For any sub-questions that you are asking as part of the project, are they action-relevant for further work? Then, ask yourself: What are your motivations in doing this research? Note down research-related and personal motives for doing this research. Check to what extent your research question is guided by action-relevance, and to what extent it is by other motives.
[1] I’m not sure if “action-relevant” is accepted terminology and how it relates to “decision-relevance”, which has also been floating around (thanks to Vojta for pointing this out).
[2] This might be different if you have reason to believe that how scared someone is correlates with actual risk, or if your intervention is aimed at reassuring people rather than preventing fires.
In the spirit of reversing advice, the very short case for not asking yourself whether something is action-relevant, is that curiosity is an incredibly valuable tool for motivation and directing your learning where there is something important to be learned. Justifying every question on decision-relevance replaces curiosity with (semi) explicit reasoning; it is not clear to me that this is a good trade (many of the best thinkers of the past seem to me to be extremely curious, and in my experience, explicit reasoning is not very powerful).
I don’t have a strong opinion on whether the median EA interested in research should be taking this advice or its opposite.
Yes, I also believe this! And I think the two pieces of advice aren’t necessarily contradictory—I could imagine stages in which curiosity is what you need, and stages where you’d want more focus. I guess I wrote this within the context of already being in touch with my curiosity and needing to rein it in a bit.
Also, I hope it’s sufficiently clear that I’m not trying to claim that action-relevance is *all* you should think about as a fledgling researcher? (If not, I’m happy to make an edit to the post)
If I have the time, I’d like to write several research skills posts for a more nuanced picture, but it seemed good to focus on one concept at a time, so necessarily it might look a bit one-sided.
I didn’t think that you thought that; I think the post is fine as is. I wasn’t trying to critique this post; it’s an important concept and I can certainly think of some people who I think should take this advice.
Another potentially useful heuristic is to pick a research question where the answer is useful whether or not you find what you’d expect. For example, “Are house fires more frequent in households with one or more smokers?” is very decision relevant if the answer is “Far more likely,” but not useful if the answer is “No,” or “A very little bit.” (But if a questions is only relevant if you get an unlikely answer, it’s even less useful. For example, “How scared are Londoners of house fires?” is plausibly very decision relevant if the answer turns out to be “Not at all, and they take no safety measures”—but that’s very unlikely to be the answer.)
A better question might be “Which of the following behaviors or characteristics correlates with increased fire risk; presence of school-aged children, smoking, building age, or income?” Notice that this is more complex than the previous question, but if you’re gathering information about smoking, the other questions are relatively easy to find information about as well—and make the project much more likely to find something useful.
(The decision-theoretic optimal is questions that are decision-relevant in proportion to the likelihood you’ll find each answer. But even if it’s very valuable in expectation, from a career perspective, you don’t want to spend time on questions that have a good chance of being a waste of time, even if they have a small chance of being really useful—but this is a trade-off that requires reflection, because it leads people to take fewer risks, and from a social benefit perspective at least, most people take too few risks already.)
This sounds like the “importance” part of the ITN framework. From EA Concepts:
About a year ago I heard the term “action guiding”, which I guess is the same?
I’ve heard “action relevant” used more often—but both are used.
I guess “action-relevant” has a better noun form, which could be a non-trivial advantage.