Another potentially useful heuristic is to pick a research question where the answer is useful whether or not you find what you’d expect. For example, “Are house fires more frequent in households with one or more smokers?” is very decision relevant if the answer is “Far more likely,” but not useful if the answer is “No,” or “A very little bit.” (But if a questions is only relevant if you get an unlikely answer, it’s even less useful. For example, “How scared are Londoners of house fires?” is plausibly very decision relevant if the answer turns out to be “Not at all, and they take no safety measures”—but that’s very unlikely to be the answer.)
A better question might be “Which of the following behaviors or characteristics correlates with increased fire risk; presence of school-aged children, smoking, building age, or income?” Notice that this is more complex than the previous question, but if you’re gathering information about smoking, the other questions are relatively easy to find information about as well—and make the project much more likely to find something useful.
(The decision-theoretic optimal is questions that are decision-relevant in proportion to the likelihood you’ll find each answer. But even if it’s very valuable in expectation, from a career perspective, you don’t want to spend time on questions that have a good chance of being a waste of time, even if they have a small chance of being really useful—but this is a trade-off that requires reflection, because it leads people to take fewer risks, and from a social benefit perspective at least, most people take too few risks already.)
Another potentially useful heuristic is to pick a research question where the answer is useful whether or not you find what you’d expect. For example, “Are house fires more frequent in households with one or more smokers?” is very decision relevant if the answer is “Far more likely,” but not useful if the answer is “No,” or “A very little bit.” (But if a questions is only relevant if you get an unlikely answer, it’s even less useful. For example, “How scared are Londoners of house fires?” is plausibly very decision relevant if the answer turns out to be “Not at all, and they take no safety measures”—but that’s very unlikely to be the answer.)
A better question might be “Which of the following behaviors or characteristics correlates with increased fire risk; presence of school-aged children, smoking, building age, or income?” Notice that this is more complex than the previous question, but if you’re gathering information about smoking, the other questions are relatively easy to find information about as well—and make the project much more likely to find something useful.
(The decision-theoretic optimal is questions that are decision-relevant in proportion to the likelihood you’ll find each answer. But even if it’s very valuable in expectation, from a career perspective, you don’t want to spend time on questions that have a good chance of being a waste of time, even if they have a small chance of being really useful—but this is a trade-off that requires reflection, because it leads people to take fewer risks, and from a social benefit perspective at least, most people take too few risks already.)