Thank so much for the clear and eloquent post. I think a lot of the issues related to lack of expertise and expert bias are stronger than I think you do, and I think it’s both rare and not inordinately difficult to adjust for common biases such that in certain cases a less-informed individual can often beat the expert consensus (because few enough of the experts are doing this, for now). But it was useful to read this detailed and compelling explanation of your view.
The following point seems essential, and I think underemphasized:
Modesty can lead to double-counting, or even groupthink. Suppose in the original example Beatrice does what I suggest and revise their credences to be 0.6, but Adam doesn’t. Now Charlie forms his own view (say 0.4 as well) and does the same procedure as Beatrice, so Charlie now holds a credence of 0.6 as well. The average should be lower: (0.8+0.4+0.4)/3, not (0.8+0.6+0.4)/3, but the results are distorted by using one-and-a-half helpings of Adam’s credence. With larger cases one can imagine people wrongly deferring to hold consensus around a view they should think is implausible, and in general the nigh-intractable challenge from trying to infer cases of double counting from the patterns of ‘all things considered’ evidence.
One can rectify this by distinguishing ‘credence by my lights’ versus ‘credence all things considered’. So one can say “Well, by my lights the credence of P is 0.8, but my actual credence is 0.6, once I account for the views of my epistemic peers etc.” Ironically, one’s personal ‘inside view’ of the evidence is usually the most helpful credence to publicly report (as it helps others modestly aggregate), whilst ones all things considered modest view usually for private consumption.
I rarely see any effort to distinguish between the two outside the rationalist/EA communities, which is one reason I think both over-modesty and overconfident backlash against it are common.
My experience is that most reasonable, intelligent people I know have never explicitly thought of the distinction between the two types of credence. I think many of them have an intuition that something would be lost if they stated their “all things considered” credence only, even though it feels “truer” and “more likely to be right,” though they haven’t formally articulated the problem. And knowing that other people rarely make this distinction, it’s hard for everyone know how to update based on others’ views without double-counting, as you note.
It seems like it’s intuitive for people to state either their inside view, or their all-things-considered view, but not both. To me, stating “both”>”inside view only”>”outside view only”, but I worry that calls for more modest views tend to leak nuance and end up pushing for people to publicly state “outside view only” rather than “both”
Also, I’ve generally heard people call the “credence by my lights” and “credence all things considered” one’s “impressions” and “beliefs,” respectively, which I prefer because they are less clunky. Just fyi.
I agree that this distinction is important and should be used more frequently. I also think good terminology is very important. Clunky terms are unlikely to be used.
Something along the lines of “impressions” or “seemings” may be good for “credence by my lights” (cf optical illusions, where the way certain matter of facts seem or appear to you differs from your beliefs about them). Another possibility is “private signal”.
I don’t think inside vs outside view is a good terminology. E.g., I may have a credence by my lights about X partly because I believe that X falls in a certain reference class. Such reasoning is normally called “outside-view”-reasoning, yet it doesn’t involve deference to epistemic peers.
Thanks for your generous reply, Claire. I agree the ‘double counting’ issue remains challenging, although my thought was that most people, at least in the wider world, are currently pretty immodest, the downsides are not too large in what I take to be common applications where you are trying to weigh up large groups of people/experts. I agree there’s a risk of degrading norms if people mistakenly switch to offering ‘outside view’ credences publicly.
I regret I hadn’t seen the ‘impressions’ versus ‘beliefs’ distinction being used before. ‘Impression’ works very well for ‘credence by my lights’ (I had toyed with using the term ‘image’), but I’m not sure ‘belief’ translates quite so well for those who haven’t seen the way the term is used in the rationalist community. I guess this might just be hard, as there does seem to be a good word (or two) I can find which captures modesty (“being modest, my credence is X”, “modestly, I think it’s Y”, maybe?)
The dichotomy I see the most at MIRI is ‘one’s inside-view model’ v. ‘one’s belief’, where the latter tries to take into account things like model uncertainty, outside-view debiasing for addressing things like the planning fallacy, and deference to epistemic peers. Nate draws this distinction a lot.
As one data point, I did not have this association with “impressions” vs. “beliefs”, even though I do in fact distinguish between these two kinds of credences and often report both (usually with a long clunky explanation since I don’t know of good terminology for it).
The comments on naming beliefs by Hal Finney (2008) appears to be how the consensus around the impressions/beliefs distinction began to form (the commenters include such movers and shakers as Eliezer and Anna Salamon).
Also, impression track records by Katja (September 2017) recent blog post/article circulated in the rationalist community that revived the terminology.
Thanks for drawing our attention to that early Overcoming Bias post. But please note that it was written by Hal Finney, not Robin Hanson. It took me a few minutes to realize this, so it seemed worth highlighting lest others fail to appreciate it.
Incidentally, I’ve been re-reading Finney’s posts over the past couple of days and have been very impressed. What a shame that such a fine thinker is no longer with us.
Explicitly separate “individual impressions” (impressions based only on evidence you’ve verified yourself) from “beliefs” (which include evidence from others’ impressions)
To add to the list of references in this thread, Brian Tomasik talks about this in “Gains from Trade through Compromise” in the section “Epistemic prisoner’s dilemma”.
Thank so much for the clear and eloquent post. I think a lot of the issues related to lack of expertise and expert bias are stronger than I think you do, and I think it’s both rare and not inordinately difficult to adjust for common biases such that in certain cases a less-informed individual can often beat the expert consensus (because few enough of the experts are doing this, for now). But it was useful to read this detailed and compelling explanation of your view.
The following point seems essential, and I think underemphasized:
I rarely see any effort to distinguish between the two outside the rationalist/EA communities, which is one reason I think both over-modesty and overconfident backlash against it are common.
My experience is that most reasonable, intelligent people I know have never explicitly thought of the distinction between the two types of credence. I think many of them have an intuition that something would be lost if they stated their “all things considered” credence only, even though it feels “truer” and “more likely to be right,” though they haven’t formally articulated the problem. And knowing that other people rarely make this distinction, it’s hard for everyone know how to update based on others’ views without double-counting, as you note.
It seems like it’s intuitive for people to state either their inside view, or their all-things-considered view, but not both. To me, stating “both”>”inside view only”>”outside view only”, but I worry that calls for more modest views tend to leak nuance and end up pushing for people to publicly state “outside view only” rather than “both”
Also, I’ve generally heard people call the “credence by my lights” and “credence all things considered” one’s “impressions” and “beliefs,” respectively, which I prefer because they are less clunky. Just fyi.
(views my own, not my employer’s)
I just thought I’d note that this appears similar to the ‘herding’ phenomenon in political polling, which reduces aggregate accuracy: http://www.aapor.org/Education-Resources/Election-Polling-Resources/Herding.aspx
I agree that this distinction is important and should be used more frequently. I also think good terminology is very important. Clunky terms are unlikely to be used.
Something along the lines of “impressions” or “seemings” may be good for “credence by my lights” (cf optical illusions, where the way certain matter of facts seem or appear to you differs from your beliefs about them). Another possibility is “private signal”.
I don’t think inside vs outside view is a good terminology. E.g., I may have a credence by my lights about X partly because I believe that X falls in a certain reference class. Such reasoning is normally called “outside-view”-reasoning, yet it doesn’t involve deference to epistemic peers.
Concur that the distinction between “credence by lights” and “credence all things considered” seems very helpful, possibly deserving of it’s own post.
Thanks for your generous reply, Claire. I agree the ‘double counting’ issue remains challenging, although my thought was that most people, at least in the wider world, are currently pretty immodest, the downsides are not too large in what I take to be common applications where you are trying to weigh up large groups of people/experts. I agree there’s a risk of degrading norms if people mistakenly switch to offering ‘outside view’ credences publicly.
I regret I hadn’t seen the ‘impressions’ versus ‘beliefs’ distinction being used before. ‘Impression’ works very well for ‘credence by my lights’ (I had toyed with using the term ‘image’), but I’m not sure ‘belief’ translates quite so well for those who haven’t seen the way the term is used in the rationalist community. I guess this might just be hard, as there does seem to be a good word (or two) I can find which captures modesty (“being modest, my credence is X”, “modestly, I think it’s Y”, maybe?)
The dichotomy I see the most at MIRI is ‘one’s inside-view model’ v. ‘one’s belief’, where the latter tries to take into account things like model uncertainty, outside-view debiasing for addressing things like the planning fallacy, and deference to epistemic peers. Nate draws this distinction a lot.
I guess you could make a trichotomy:
a) Your inside-view model.
b) Your all-things-considered private signal, where you’ve added outside-view reasoning, taken model uncertainty into account, etc.
c) Your all-things-considered belief, which also takes the views of your epistemic peers into account.
As one data point, I did not have this association with “impressions” vs. “beliefs”, even though I do in fact distinguish between these two kinds of credences and often report both (usually with a long clunky explanation since I don’t know of good terminology for it).
The comments on naming beliefs by Hal Finney (2008) appears to be how the consensus around the impressions/beliefs distinction began to form (the commenters include such movers and shakers as Eliezer and Anna Salamon).
Also, impression track records by Katja (September 2017) recent blog post/article circulated in the rationalist community that revived the terminology.
Thanks for drawing our attention to that early Overcoming Bias post. But please note that it was written by Hal Finney, not Robin Hanson. It took me a few minutes to realize this, so it seemed worth highlighting lest others fail to appreciate it.
Incidentally, I’ve been re-reading Finney’s posts over the past couple of days and have been very impressed. What a shame that such a fine thinker is no longer with us.
ETA: Though one hopes this is temporary.
Somehow I missed your reply originally; I’ve updated my comment to correct the author name of the post.
Thanks! By the way, I found your original comment helpful for writing about the history of the concept of an independent impression.
I’m not sure where I picked it up, though I’m pretty sure it was somewhere in the rationalist community.
E.g. from What epistemic hygiene norms should there be?:
To add to the list of references in this thread, Brian Tomasik talks about this in “Gains from Trade through Compromise” in the section “Epistemic prisoner’s dilemma”.