I worriedly predict that anyone who followed your advice here would just switch to describing whatever they’re doing as “reference class forecasting” since this captures the key dynamic that makes describing what they’re doing as “outside viewing” appealing: namely, they get to pick a choice of “reference class” whose samples yield the answer they want, claim that their point is in the reference class, and then claiming that what they’re doing is what superforecasters do and what Philip Tetlock told them to do and super epistemically virtuous and anyone who argues with them gets all the burden of proof and is probably a bad person but we get to virtuously listen to them and then reject them for having used the “inside view”.
My own take: Rule One of invoking “the outside view” or “reference class forecasting” is that if a point is more dissimilar to examples in your choice of “reference class” than the examples in the “reference class” are dissimilar to each other, what you’re doing is “analogy”, not “outside viewing”.
All those experimental results on people doing well by using the outside view are results on people drawing a new sample from the same bag as previous samples. Not “arguably the same bag” or “well it’s the same bag if you look at this way”, really actually the same bag: how late you’ll be getting Christmas presents this year, based on how late you were in previous years. Superforecasters doing well by extrapolating are extrapolating a time-series over 20 years, which was a straight line over those 20 years, to another 5 years out along the same line with the same error bars, and then using that as the baseline for further adjustments with due epistemic humility about how sometimes straight lines just get interrupted some year. Not by them picking a class of 5 “relevant” historical events that all had the same outcome, and arguing that some 6th historical event goes in the same class and will have that same outcome.
Good point, I’ll add analogy to the list. Much that is called reference class forecasting is really just analogy, and often not even a good analogy.
I really think we should taboo “outside view.” If people are forced to use the term “reference class” to describe what they are doing, it’ll be more obvious when they are doing epistemically shitty things, because the term “reference class” invites the obvious next questions: 1. What reference class? 2. Why is that the best reference class to use?
All those experimental results on people doing well by using the outside view are results on people drawing a new sample from the same bag as previous samples. Not “arguably the same bag” or “well it’s the same bag if you look at this way”, really actually the same bag: how late you’ll be getting Christmas presents this year, based on how late you were in previous years
Hmm, I’m not convinced that this is meaningfully different in kind rather than degree. You aren’t predicting a randomly chosen holdout year, so saying that 2021 is from the same distribution as 2011-2020 is still a take. “X thing I do in the future is from the same distribution of all my attempts in past years*” is still a judgement call, albeit a much easier one than AI timelines.
I agree with (part of) your broader point that incareful applications of the outside view and similar vibes is very susceptible to motivated reasoning (including but not limited to the absurdity heuristic), but I guess my take here is that we should just be more careful individually and more willing to point out bad epistemic moves in others (as you’ve often done a good job of!) as a community.
All our tools are limited and corruptible, and I don’t think on balance reference class forecasting is more susceptible to motivated reasoning than other techniques.
*are you using your last 10 years? since you’ve been an adult? all the years you’ve been alive?
I worriedly predict that anyone who followed your advice here would just switch to describing whatever they’re doing as “reference class forecasting” since this captures the key dynamic that makes describing what they’re doing as “outside viewing” appealing: namely, they get to pick a choice of “reference class” whose samples yield the answer they want, claim that their point is in the reference class, and then claiming that what they’re doing is what superforecasters do and what Philip Tetlock told them to do and super epistemically virtuous and anyone who argues with them gets all the burden of proof and is probably a bad person but we get to virtuously listen to them and then reject them for having used the “inside view”.
My own take: Rule One of invoking “the outside view” or “reference class forecasting” is that if a point is more dissimilar to examples in your choice of “reference class” than the examples in the “reference class” are dissimilar to each other, what you’re doing is “analogy”, not “outside viewing”.
All those experimental results on people doing well by using the outside view are results on people drawing a new sample from the same bag as previous samples. Not “arguably the same bag” or “well it’s the same bag if you look at this way”, really actually the same bag: how late you’ll be getting Christmas presents this year, based on how late you were in previous years. Superforecasters doing well by extrapolating are extrapolating a time-series over 20 years, which was a straight line over those 20 years, to another 5 years out along the same line with the same error bars, and then using that as the baseline for further adjustments with due epistemic humility about how sometimes straight lines just get interrupted some year. Not by them picking a class of 5 “relevant” historical events that all had the same outcome, and arguing that some 6th historical event goes in the same class and will have that same outcome.
Good point, I’ll add analogy to the list. Much that is called reference class forecasting is really just analogy, and often not even a good analogy.
I really think we should taboo “outside view.” If people are forced to use the term “reference class” to describe what they are doing, it’ll be more obvious when they are doing epistemically shitty things, because the term “reference class” invites the obvious next questions: 1. What reference class? 2. Why is that the best reference class to use?
Hmm, I’m not convinced that this is meaningfully different in kind rather than degree. You aren’t predicting a randomly chosen holdout year, so saying that 2021 is from the same distribution as 2011-2020 is still a take. “X thing I do in the future is from the same distribution of all my attempts in past years*” is still a judgement call, albeit a much easier one than AI timelines.
I agree with (part of) your broader point that incareful applications of the outside view and similar vibes is very susceptible to motivated reasoning (including but not limited to the absurdity heuristic), but I guess my take here is that we should just be more careful individually and more willing to point out bad epistemic moves in others (as you’ve often done a good job of!) as a community.
All our tools are limited and corruptible, and I don’t think on balance reference class forecasting is more susceptible to motivated reasoning than other techniques.
*are you using your last 10 years? since you’ve been an adult? all the years you’ve been alive?