Often proceed gradually toward soliciting forecasts and/or doing expert surveys
tl;dr: I think it’s often good to have a pipeline from untargeted thinking/discussion that stumbles upon important topics, to targeted thinking/discussion of a given important topic, to expert interviews on that topic, to soliciting quantitive forecasts / doing large expert surveys.
I wrote this quickly. I think the core ideas are useful but I imagine they’re already familiar to e.g. many people with experience making surveys.[1] I’m not personally aware of an existing writeup on this and didn’t bother searching for one, but please comment if you know of one!
Introduction
Let’s say you wanna get a better understanding of something. If you know exactly and in detail what it is that you want to get a better understanding of, two tools that can be very useful are forecasts and expert surveys. More specifically, it can be very useful to generate well-operationalized quantitative or fixed-choice questions and then get those questions answered by a large number of people with relevant expertise and/or good forecasting track records.
But it’s probably best to see that as an end point, rather than jumping to it too soon, for two reasons:
Getting responses is costly
Getting a lot of people with relevant expertise and/or good forecasting track records to answer your questions probably requires significant effort or money.
It may also involve substantial opportunity cost, if those people are working on important, net-positive things.
Generating questions is hard
I’ve both done and observed a decent amount of forecasting question writing and survey design. It seems to me that it’s harder to actually do well than most people would probably expect, and that people often don’t realise they haven’t done it well until after they get some feedback or some answers.
One difficulty is having even a rough sense of what it’s best to ask about.
Another difficulty is figuring out what precisely what to ask about and phrasing that very clearly, such that respondents can easily understand your question, they interpret it how you wanted, and the question covers and captures all of the relevant & useful thoughts they have to share.
This often requires/warrants a lot of thought, multiple rounds of feedback, and multiple rounds of testing on an initial batch of respondents.
So if you jump to making forecasting questions or surveys too early, you may:
waste a lot of your or other people’s time/money on unimportant topics/questions
get responses that are confusing or misleading since the question phrasings were unclear
fail to hear a lot of the most interesting things people had to share, since you didn’t ask about those things and your questions had precise scopes
...especially because forecasting questions and surveys are typically “launched” to lots of people at once, so you may not be able to or think to adapt your questions/approach in light of the first few responses, even if the first few give you reason to do so.
The pipeline I propose for mitigating those issues
(Note: The boundaries between these “steps” are fuzzy. It probably often makes sense to jump back and forth to some extent. It probably also often makes sense to be at different stages at the same time for different subtopics/questions within a broad topic.)
Untargeted thinking/discussion
I.e., thinking/discussions/writing/research/whatever that either roams through many topics, or is fairly focused but not focused on the topic that this instance of “the pipeline” will end up focused on
Sometimes this stumbles upon a new (to you) topic, or seems to suggest an already-noticed topic seems worth prioritizing further thought on
Advantage: Very unconstrained; could stumble upon many things that you haven’t already realised are worth prioritizing.
Targeted thinking/discussion of a given important-seeming topic
This is still unconstrained in its precise focus or its method, but now constrained to a particular broad topic.
Advantage: Can go deeper on that topic, while still retaining flexibility regarding what the best scope, most important subquestions, etc. is
Expert interviews on that topic
Similar to the above “step”, but now with a clearer sense of what questions you’re asking and with more active effort to talk to experts specifically.
Within this step, you might want to move through a pipeline with a similar rationale to the overall pipeline, moving from (a) talking in a fairly unstructured way to people with only moderate expertise and opportunity cost to (b) following a specified and carefully considered interview protocol in interviews with the very best experts to talk to on this topic.
(b) could even essentially be a survey delivered verbally but with occasional unplanned follow-up questions based on what respondents said.
Advantage: Get well-founded thoughts on well-considered questions with relatively low cost to these people’s fairly scarce time.
Probably often with some but not complete overlap in the questions and participants.
Some questions are better suited to people with strong forecasting track records and others better suited to people with relevant expertise.
Within this step, you might want to undergo a pipeline with a similar rationale to the overall pipeline, with multiple waves of forecasting-soliciting / surveying that each have more questions, more precise operationalizations of questions, and/or more respondents.
Advantage: Get a large volume of well-founded, easily interpretable thoughts on well-considered questions, with relatively low cost to each person’s fairly scarce time (even if high cost in aggregate).
Misc thoughts
For similar reasons, I think it’s probably usually good for interview protocols and surveys to include at least one “Any other thoughts?” type question and perhaps multiple (e.g., one after each set of questions on a similar theme).
Also for similar reasons, I think it’s probably usually good to allow/encourage forecasters’ to share whatever thoughts they have that they think are worth sharing, rather than solely soliciting their forecasts on the questions asked.
The specific trigger for me writing this was that I mentioned the core idea of this shortform to a colleague it was relevant to, and they said it seemed useful to them.
Another reason I bothered to write it is that in my experience this basic idea has seemed valid and useful, and I think it would’ve been a little useful for me to have read this a couple years ago.
Often proceed gradually toward soliciting forecasts and/or doing expert surveys
tl;dr: I think it’s often good to have a pipeline from untargeted thinking/discussion that stumbles upon important topics, to targeted thinking/discussion of a given important topic, to expert interviews on that topic, to soliciting quantitive forecasts / doing large expert surveys.
I wrote this quickly. I think the core ideas are useful but I imagine they’re already familiar to e.g. many people with experience making surveys.[1] I’m not personally aware of an existing writeup on this and didn’t bother searching for one, but please comment if you know of one!
Introduction
Let’s say you wanna get a better understanding of something. If you know exactly and in detail what it is that you want to get a better understanding of, two tools that can be very useful are forecasts and expert surveys. More specifically, it can be very useful to generate well-operationalized quantitative or fixed-choice questions and then get those questions answered by a large number of people with relevant expertise and/or good forecasting track records.
But it’s probably best to see that as an end point, rather than jumping to it too soon, for two reasons:
Getting responses is costly
Getting a lot of people with relevant expertise and/or good forecasting track records to answer your questions probably requires significant effort or money.
It may also involve substantial opportunity cost, if those people are working on important, net-positive things.
Generating questions is hard
I’ve both done and observed a decent amount of forecasting question writing and survey design. It seems to me that it’s harder to actually do well than most people would probably expect, and that people often don’t realise they haven’t done it well until after they get some feedback or some answers.
One difficulty is having even a rough sense of what it’s best to ask about.
Another difficulty is figuring out what precisely what to ask about and phrasing that very clearly, such that respondents can easily understand your question, they interpret it how you wanted, and the question covers and captures all of the relevant & useful thoughts they have to share.
This often requires/warrants a lot of thought, multiple rounds of feedback, and multiple rounds of testing on an initial batch of respondents.
So if you jump to making forecasting questions or surveys too early, you may:
waste a lot of your or other people’s time/money on unimportant topics/questions
get responses that are confusing or misleading since the question phrasings were unclear
fail to hear a lot of the most interesting things people had to share, since you didn’t ask about those things and your questions had precise scopes
...especially because forecasting questions and surveys are typically “launched” to lots of people at once, so you may not be able to or think to adapt your questions/approach in light of the first few responses, even if the first few give you reason to do so.
The pipeline I propose for mitigating those issues
(Note: The boundaries between these “steps” are fuzzy. It probably often makes sense to jump back and forth to some extent. It probably also often makes sense to be at different stages at the same time for different subtopics/questions within a broad topic.)
Untargeted thinking/discussion
I.e., thinking/discussions/writing/research/whatever that either roams through many topics, or is fairly focused but not focused on the topic that this instance of “the pipeline” will end up focused on
Sometimes this stumbles upon a new (to you) topic, or seems to suggest an already-noticed topic seems worth prioritizing further thought on
Advantage: Very unconstrained; could stumble upon many things that you haven’t already realised are worth prioritizing.
Targeted thinking/discussion of a given important-seeming topic
This is still unconstrained in its precise focus or its method, but now constrained to a particular broad topic.
Advantage: Can go deeper on that topic, while still retaining flexibility regarding what the best scope, most important subquestions, etc. is
Expert interviews on that topic
Similar to the above “step”, but now with a clearer sense of what questions you’re asking and with more active effort to talk to experts specifically.
Within this step, you might want to move through a pipeline with a similar rationale to the overall pipeline, moving from (a) talking in a fairly unstructured way to people with only moderate expertise and opportunity cost to (b) following a specified and carefully considered interview protocol in interviews with the very best experts to talk to on this topic.
(b) could even essentially be a survey delivered verbally but with occasional unplanned follow-up questions based on what respondents said.
Advantage: Get well-founded thoughts on well-considered questions with relatively low cost to these people’s fairly scarce time.
Soliciting quantitative forecasts and/or running expert surveys
It may often be worth doing both.
Probably often with some but not complete overlap in the questions and participants.
Some questions are better suited to people with strong forecasting track records and others better suited to people with relevant expertise.
Within this step, you might want to undergo a pipeline with a similar rationale to the overall pipeline, with multiple waves of forecasting-soliciting / surveying that each have more questions, more precise operationalizations of questions, and/or more respondents.
Advantage: Get a large volume of well-founded, easily interpretable thoughts on well-considered questions, with relatively low cost to each person’s fairly scarce time (even if high cost in aggregate).
Misc thoughts
For similar reasons, I think it’s probably usually good for interview protocols and surveys to include at least one “Any other thoughts?” type question and perhaps multiple (e.g., one after each set of questions on a similar theme).
Also for similar reasons, I think it’s probably usually good to allow/encourage forecasters’ to share whatever thoughts they have that they think are worth sharing, rather than solely soliciting their forecasts on the questions asked.
The specific trigger for me writing this was that I mentioned the core idea of this shortform to a colleague it was relevant to, and they said it seemed useful to them.
Another reason I bothered to write it is that in my experience this basic idea has seemed valid and useful, and I think it would’ve been a little useful for me to have read this a couple years ago.