I’m somewhat sympathetic to the frustration you express. However, I suspect the optimal response isn’t to be more or less epistemically modest indiscriminately. Instead, I suspect the optimal policy is something like:
Always be clear and explicit to what extent a view you’re communicating involves deference to others.
Depending on the purpose of a conversation, prioritize (possibly at different stages) either object-level discussions that ignore others’ views or forming an overall judgment that includes epistemic deference.
E.g. when the purpose is to learn, or to form an independent assessment of something, epistemic deference will often be a distraction.
By contrast, if you make a decision with large-scale and irreversible effects on the world (e.g. “who should get this $5M grant?”) I think it would usually be predictably worse for the world to ignore others’ views.
On the other hand, I think literally everyone using the “average view” among the same set of people is suboptimal even for such purposes: it’s probably better to have less correlated decisions in order to “cover the whole space” of reasonable options. (This might seem odd on some naive models of decision-making, but I think can easily be justified by plausible toy models involving heavy-tailed ex-post returns that are hard to predict ex-ante plus barriers to coordination or undetectable but different biases. I.e. similar to why it would probably be bad if every VC based their investment decisions on the average view of all VCs.)
I think this way one can largely have the best of both worlds.
(I vaguely remember that there is a popular post making a similar recommendation, but couldn’t quickly find it.)
I am not sure whether my error was how much I was deferring in itself. But the decision to defer or not should be made on well defined questions and clearly defined ‘experts’ you might be deferring to. This is not what I was doing. I was deferring on a nebulous question (‘what should I be doing?‘) to an even more nebulous expert audience (a vague sense of what ‘the community’ wanted).
What I should have been doing instead first is to define the question better: Which roles should I be pursuing right now?
This can then be broken down further into subquestions on cause prioritisation, which roles are promising avenues within causes I might be interested in, which roles I might be well suited for, etc, whose information I need to aggregate in a sensible fashion to answer the question which roles I should be pursuing right now.
For each of these subquestions I need to make a separate judgement. For some it makes more sense to defer, for others, less so. Disappointingly, there is no independent expert panel investigating what kind of jobs I might excel at.
But then who to defer to, if I think this is a sensible choice for a particular subquestion, also needs to be clearly defined: for example, I might decide that it makes sense to take 80k at their word about which roles in a particular cause area are particularly promising right now, after reading what they actually say on their website on the subject, perhaps double-checking by asking them via email and polling another couple of people in the field.
‘The community’ is not a well defined expert panel, while the careful aggregation of individual opinions can be, who again, need to be asked well defined questions. Note that this can true even if I gave equal weight to every EA’s opinion: sometimes it can seem like ‘the community’ has an opinion that only few individual EAs hold if actually asked, if any. This is especially true if messaging is distorted and I am not actually asking a well defined question.
if you make a decision with large-scale and irreversible effects on the world (e.g. “who should get this $5M grant?”) I think it would usually be predictably worse for the world to ignore others’ views
Taking into account specific facts or arguments made by other people seems reasonable here. Just writing down e.g. “person X doesn’t like MIRI” in the “cons” column of your spreadsheet seems foolish and wrongheaded.
Framing it as “taking others’ views into account” or “ignoring others’ views” is a big part of the problem, IMO—that language itself directs people towards evaluating the people rather than the arguments, and overall opinions rather than specific facts or claims.
I think we disagree. I’m not sure why you think that even for decisions with large effects one should only or mostly take into account specific facts or arguments, and am curious about your reasoning here.
I do think it will often be even more valuable to understand someone’s specific reasons for having a belief. However, (i) in complex domains achieving a full understanding would be a lot of work, (ii) people usually have incomplete insight into the specific reasons for why they hold a certain belief themselves and instead might appeal to intuition, (iii) in practice you only have so much time and thus can’t fully pursue all disagreements.
So yes, always stopping at “person X thinks that p” and never trying to understand why would be a poor policy. But never stopping at that seems infeasible to me, and I don’t see the benefits from always throwing away the information that X believes p in situations where you don’t fully understand why.
For instance, imagine I pointed a gun to your head and forced you to now choose between two COVID mitigation policies for the US for the next 6 months. I offer you to give you additional information of the type “X thinks that p” with some basic facts on X but no explanation for why they hold this belief. Would you refuse to view that information? If someone else was in that situation, would you pay for me not giving them this information? How much?
There is a somewhat different failure mode where person X’s view isn’t particularly informative compared to the view of other people Y, Z, etc., and so by considerung just X’s view you give it undue weight. But I don’t think you’re talking about that?
I’m partly puzzled by your reaction because the basic phenomenon of deferring to the output of others’ reasoning processes without understanding the underlying facts or arguments strikes me as not unusual at all. For example, I believe that the Earth orbits the Sun rather than the other way around. But I couldn’t give you any very specific argument for this like “on the geocentric hypothesis, the path of this body across the sky would look like this”. Instead, the reason for my belief is that the heliocentric worldview is scientific consensus, i.e. epistemic deference to others without understanding their reasoning.
This also happens when the view in question makes a difference in practice. For instance, as I’m sure you’re aware, hierarchical organizations work (among other things) because managers don’t have to recapitulate every specific argument behind the conclusions of their reports.
To sum up, a very large amount of division of epistemic labor seems like the norm rather than the exception to me, just as for the division of manual labor. The main thing that seems somewhat unusual is making that explicit.
I note that the framing / example case has changed a lot between your original comment / my reply (making a $5m grant and writing “person X is skeptical of MIRI” in the “cons” column) and this parent comment (“imagine I pointed a gun to your head and… offer you to give you additional information;” “never stopping at [person X thinks that p]”). I’m not arguing for entirely refusing to trust other people or dividing labor, as you implied there. I specifically object to giving weight to other people’s top-line views on questions where there’s substantial disagreement, based on your overall assessment of that particular person’s credibility / quality of intuition / whatever, separately from your evaluation of their finer-grained sub-claims.
If you are staking $5m on something, it’s hard for me to imagine a case where it makes sense to end up with an important node in your tree of claims whose justification is “opinions diverge on this but the people I think are smartest tend to believe p.” The reason I think this is usually bad is that (a) it’s actually impossible to know how much weight it’s rational to give someone else’s opinion without inspecting their sub-claims, and (b) it leads to groupthink/herding/information cascades.
As a toy example to illustrate (a): suppose that for MIRI to be the optimal grant recipient, it both needs to be the case that AI risk is high (A) and that MIRI is the Best organization working to mitigate it (B). A and B are independent. The prior is (P(A) = 50, P(B) = 50). Alice and Bob have observed evidence with a 9:1 odds ratio in favor of A, so think (P(A) = 90, P(B) = 50). Carol has observed evidence with a 9:1 odds ratio in favor of B. Alice, Bob and Carol all have the same top-line view of MIRI (P(A and B) = 0.45), but the rational aggregation of Alice and Bob’s “view” is much less positive than the rational aggregation of Bob and Carol’s.
It’s interesting that you mention hierarchical organizations because I think they usually follow a better process for dividing up epistemic labor, which is to assign different sub-problems to different people rather than by averaging a large number of people’s beliefs on a single question. This works better because the sub-problems are more likely to be independent from each other, so they don’t require as much communication / model-sharing to aggregate their results.
In fact, when hierarchical organizations do the other thing—”brute force” aggregate others’ beliefs in situations of disagreement—it usually indicates an organizational failure. My own experience is that I often see people do something a particular way, even though they disagree with it, because they think that’s my preference; but it turns out they had a bad model of my preferences (often because they observed a contextual preference in a different context) and would have been better off using their own judgment.
I think I perceive less of a difference between the examples we’ve been discussing, but after reading your reply I’m also less sure if and where we disagree significantly.
I read your previous claim as essentially saying “it would always be bad to include the information that some person X is skeptical about MIRI when making the decision whether to give MIRI a $5M grant, unless you understand more details about why X has this view”.
I still think this view basically commits you to refusing to see information of that type in the COVID policy thought experiment. This is essentially for the reasons (i)-(iii) I listed above: I think that in practice it will be too costly to understand the views of each such person X in more detail.
(But usually it will be worth it to do this for some people, for instance for the reason spelled out in your toy model. As I said: I do think it will often be even more valuable to understand someone’s specific reasons for having a belief.)
Instead, I suspect you will need to focus on the few highest-priority cases, and in the end you’ll end up with people X1,…,Xl whose views you understand in great detail, people Y1,…,Ym where your understanding stops at other fairly high-level/top-line views (e.g. maybe you know what they think about “will AGI be developed this century?” but not much about why), and people Z1,…,Zn of whom you only know the top-line view of how much funding they’d want to give to MIRI.
At that point, I think you’re basically in a similar situation. There is no gun pointed at your head, but you still want to make a decision right now, and so you can either throw away the information about the views of person Zi or use it without understanding their arguments.
Furthermore, I don’t think your situation with respect to person Yj is that different: if you take their view on “AGI this century?” into account for the decision whether to fund MIRI but have a policy of never using “bare top-level views”, this would commit to to ignoring the same information in a different situation, e.g. the decision whether to place a large bet on whether AGI will be developed this century (purely because what’s a top-level view in one situation will be an argument or “specific” fact in another); this seems odd.
(This is also why I’m not sure I understand the relevance of your point on hierarchical organizations. I agree that usually sub-problems will be assigned to different employees. But e.g. if I assign “AGI this century?” to one employee and “is MIRI well run?” to another employee, why am I justified in believing their conclusions on these fairly high-level questions but not justified in believing anyone’s view on whether MIRI is worth funding?)
Note that thus far I’m mainly arguing against taking into account no-one’s top-level views. Your most recent claim involving “the people I think are smartest” suggests that maybe you mainly object to using a lot of discretion in which particular people’s top-level views to use.
I think my reaction to this is mixed: On one hand, I certainly agree that there is a danger involved here (e.g. in fact I think that many EAs defer too much to others EAs relative to non-EA experts), and that it’s impossible to assess with perfect accuracy how much weight to give to each person. On the other hand, I think it is often possible to assess this with limited but still useful accuracy, both based on subjective and hard-to-justify assessments of how good someone’s judgment seemed in the past (cf. how senior politicians often work with advisors they’ve had a long work relationship with) and on crude objectives proxies (e.g. ‘has a PhD in computer science’).
On the latter, you said that specifically you object to allocating weight to someone’s top-line opinion “separately from your evaluation of their finer-grained sub-claims”. If that means their finer-grained sub-claims on the particular question under consideration, then I disagree for the reasons explained so far. If that means “separately from your evaluation of any finer-grained sub-claim they ever made on anything”, then I agree more with this, though still think this is both common and justified in some cases (e.g. if I learn that I have rare disease A for which specialists universally recommend drug B as treatment, I’ll probably happily take drug B without having ever heard of any specific sub-claim made by any disease-A specialist).
Similarly, I agree that information cascades and groupthink are dangers/downsides, but that they will sometimes be outweighed by the benefits.
If 100 forecasters (who I roughly respect) look at the likelihood of a future event and think it’s ~10% likely, and I look at the same question and think it’s ~33% likely, I think I will be incorrect in my private use of reason for my all-things-considered-view to not update somewhat downwards from 33%.
I think this continues to be true even if we all in theory have access to the same public evidence, etc.
Now, it does depend a bit on the context of what this information is for. For example if I’m asked to give my perspective on a group forecast (and I know that the other 100 forecasters’ predictions will be included anyway), I think it probably makes sense for me to continue to publicly provide ~33% for that question to prevent double-counting and groupthink.
But I think it will be wrong for me to believe 33%, and even more so, wrong to say 33% in a context where somebody else doesn’t have access to the 100 other forecasters.
An additional general concern here to me is computational capacity/kindness—sometimes (often) I just don’t have enough time to evaluate all the object-level arguments! You can maybe argue that until I evaluate all the object-level arguments, I shouldn’t act, yet in practice I feel like I act with lots of uncertainty* all the time! ___
One disagreement I have with Max is whether someone should defer is contingent upon the importance of a decision. I think this begs the question in that it pre-assumes that deference lead to the best outcomes.
Instead, I think you should act such that you all-things-considered-view is that you’re making the best decision. I do think that for many decisions (with the possible exception of creative work), some level of deference leads to better outcomes than zero deference at all, but I don’t think it’s unusually true for important decisions except inasmuch as a) the benefits (and also costs!) of deference are scaled accordingly and b) more people are likely to have thought about important decisions.
__ * Narrow, personal, example that’s basically unrelated to EA: I brush my teeth with fluoride toothpaste. I don’t floss. Why? Cochrane review was fairly equivocal about flossing and fairly certain about toothbrushing. Maybe it’d be more principled if I looked at the data myself and performed my own meta-analysis on the data, or perhaps self-experimented like Gwern, to decide what dental hygiene activities I should take. But in practice I feel like it’s a reasonable decision procedure to just defer to Cochrane review on the empirical facts of the matter, and apply my own value judgments on what activities to take given the facts available.
One disagreement I have with Max is whether someone should defer is contingent upon the importance of a decision. I think this begs the question in that it pre-assumes that deference lead to the best outcomes.
Instead, I think you should act such that you all-things-considered-view is that you’re making the best decision. I do think that for many decisions (with the possible exception of creative work), some level of deference leads to better outcomes than zero deference at all, but I don’t think it’s unusually true for important decisions except inasmuch as a) the benefits (and also costs!) of deference are scaled accordingly and b) more people are likely to have thought about important decisions.
I’m not sure if we have a principled disagreement here, it’s possible that I just described my view badly above.
I agree that one should act such that one’s all-things-considered view is that one is making the best decision (the way I understand that statement it’s basically a tautology).
Then I think there are some heuristics for which features of a decision situation make it more or less likely that deferring more (or at all) leads to decisions with that property. I think on a high level I agree with you that it depends a lot “on the context of what this information is for”, more so than on e.g. importance.
With my example, I was also trying to point less to importance per se but on something like how the costs and benefits are distributed between yourself and others. This is because very loosely speaking I expect not deferring to often be better if the stakes are concentrated on oneself and more deference to be better if one’s own direct stake is small. I used a decision with large effects on others largely because then it’s not plausible that you yourself are affected by a similar amount; but it would also apply to a decision with zero effect on yourself and a small effect on others. Conversely, it would not apply to a decision that is very important to yourself (e.g. something affecting your whole career trajectory).
Apologies for the long delay in response, feel free to not reply if you’re busy.
Hmm I still think we have a substantive rather than framing disagreement (though I think it is likely that our disagreements aren’t large).
This is because very loosely speaking I expect not deferring to often be better if the stakes are concentrated on oneself and more deference to be better if one’s own direct stake is small. I used a decision with large effects on others largely because then it’s not plausible that you yourself are affected by a similar amount; but it would also apply to a decision with zero effect on yourself and a small effect on others. Conversely, it would not apply to a decision that is very important to yourself (e.g. something affecting your whole career trajectory).
Perhaps this heuristic is really useful for a lot of questions you’re considering. I’m reminded of AGB’s great quote:
There are enough individual and practical considerations here (in both directions) that in many situations the actual thing I would advocate for is something like “work out what you would do with both approaches, check against results ‘without fear or favour’, and move towards whatever method is working best for you”.
For me personally and the specific questions I’ve considered, I think considering whether/how much to defer to by dividing into buckets of “how much it affects myself or others” is certainly a pretty useful heuristic in the absence of better heuristics, but it’s mostly superseded by a different decomposition:
Epistemic—In a context-sensitive manner, do we expect greater or lower deference in this particular situation to lead to more accurate beliefs.
Role expectations* -- Whether the explicit and implicit social expectations on the role you’re assuming privilege deference or independence.
So I think a big/main reason it’s bad to defer completely to others (say 80k) on your own career reasons is epistemic: you have so much thought and local knowledge about your own situation that your prior should very strongly be against others having better all-things-considered views on your career choice than you do. I think this is more crux-y for me than how much your career trajectory affects yourself vs others (at any rate hopefully as EAs our career trajectories affect many others anyway!).
On the other hand, I think my Cochrane review example above is a good epistemic example of deference. even though my dental hygiene practices mainly affect myself and not others (perhaps my past and future partners may disagree), I contend it’s better to defer to the meta-analysis over my own independent analysis in this particular facet of my personal life.
The other main (non-epistemic) lens I’d use to privilege greater or lower humility is whether the explicit and implicit social expectations privilege deference or independence. For example, we’d generally** prefer government bureaucrats in most situations to implement policies, rather than making unprincipled exceptions based on private judgements. This will often look superficially similar to “how much this affects myself or others.”
An example of a dissimilarity is when someone filling out a survey. This is a situation where approximately all of the costs and benefits are borne by other people. So if you have a minority opinion on a topic, it may seem like the epistemically humble-and-correct action is to fill out the poll according to what you believe the majority to think (or alternatively, fill it out with the answer that you privately think is on the margin more conducive to advancing your values).
But in all likelihood, such a policy is one-thought-too-many, and in almost all situations it’d be more prudent to fill out public anonymous polls/surveys with what you actually believe.
I agree that one should act such that one’s all-things-considered view is that one is making the best decision (the way I understand that statement it’s basically a tautology).
Agreed, though I mention this because in discussions of epistemic humility-in-practice, it’s very easy to accidentally do double-counting.
*I don’t like this phrase, happy to use a better one.
**I’m aware that there are exceptions, including during the ongoing coronavirus pandemic.
Thanks! I’m not sure if there is a significant difference about how we’d actually make decisions (I mean, on prior there is probably some difference). But I agree that the single heuristics I mentioned above doesn’t by itself do a great job of describing when and how much to defer, and I agree with your “counterexamples”. (Though note that in principle it’s not surprising if there are counterexamples to a “mere heuristics”.)
I particularly appreciate you describing the “Role expectations” point. I agree that something along those lines is important. My guess is that if we would have debated specific decisions I would have implicitly incorporated this consideration, but I don’t think it was clear to me before reading your comment that this is an important property that will often influence my judgment about how much to defer.
I think that in theory Max is right, that there’s some optimal way to have the best of both worlds. But in practice I think that there are pretty strong biases towards conformity, such that it’s probably worthwhile to shift the community as a whole indiscriminately towards being less epistemic modest.
As one example, people might think “I’ll make up my mind on small decisions, and defer on big decisions.” But then they’ll evaluate what feels big to them , rather than to the EA community overall, and thereby the community as a whole will end up being strongly correlated even on relatively small-scale bets. I think your comment itself actually makes this mistake—there’s now enough money in EA that, in my opinion, there should be many $5M grants which aren’t strongly correlated with the views of EA as a whole.
In particular, I note that venture capitalists allocate much larger amounts of money explicitly on anti-conformist principles. Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful, but I’m not confident about that; the hypothesis that we just haven’t internalised the “hits-based” mentality as well as venture capitalists have also seems plausible.
(My best guess is that the average EA defers too much rather than too little. This and other comments on deference is to address specific points made, rather than to push any particular general takes).
Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful
I think this is part of the reason. A plausibly bigger reason is that VC funding can’t result in heavy left-tails. Or rather, left-tails in VC funding are very rarely internalized. Concretely, if you pick your favorite example of “terrible startup for the future of sentient beings,” the VCs in question very rarely get in trouble, and approximately never get punished proportional to the counterfactual harm of their investments. VC funding can be negative for the VC beyond the opportunity cost of money (eg via reputational risk or whatever), but the punishment is quite low relative to the utility costs.
Obviously optimizing for increasing variance is a better deal when you clip the left tail, and optimizing for reducing variance is a better deal when you clip the right tail.
(I also independently think that heavy left tails in the utilitarian sense are probably less common in VC funding than in EA, but I think this is not necessary for my argument to go through).
I agree it’s possible that because of social pressures or similar things the best policy change that’s viable in practice could be an indiscriminate move toward more or less epistemic deference. Though I probably have less of a strong sense that that’s in fact true.
(Note that when implemented well, the “best of both worlds” policy could actually make it easier to express disagreement because it clarifies that there are two types of beliefs/credences to be kept track of separately, and that one of them has to exclude all epistemic deference.
Similarly, to the extent that people think that avoiding ‘bad, unilateral action’ is a key reason in favor of epistemic deference, it could actually “destigmatize” iconoclastic views if it’s common knowledge that an iconoclastic pre-deference view doesn’t imply unusual primarily-other-affecting actions because primarily-other-affecting actions depend on post-deference rather rather than pre-deference views.)
I agree with everything you say about $5M grants and VCs. I’m not sure if you think my mistake was mainly to consider a $5M stake a “large-scale” decision or something else, but if it’s the former I’m happy to concede that this wasn’t the best example to give for a decision where deference should get a lot of weight (though I think we agree that in theory it should get some weight?).
I disagree Max. We can all recall anecdotes of overconfidence because they create well-publicized narratives. With hindsight bias, it seems obvious that overconfidence was the subject. So naturally we overestimate overconfidence risks, just like nuclear power.
The costs of under confidence are invisible and ubiquitous. A grad student fails to submit her paper. An applicant doesn’t apply. A graduate doesn’t write down her NGO idea. Because you can’t see the costs of underconfidence, they could be hundreds or thousands of times the overconfidence costs.
To break apart the question
Should people update based on evidence and not have rigid world-models. Is people disagreeing with you moderate evidence?
Yes to both
Once someone builds the best world-model they can, should they defer to higher-status people’s models
Much much less often than we currently do
How much should we weight disagreement between our models and the models of others?
I’m somewhat sympathetic to the frustration you express. However, I suspect the optimal response isn’t to be more or less epistemically modest indiscriminately. Instead, I suspect the optimal policy is something like:
Always be clear and explicit to what extent a view you’re communicating involves deference to others.
Depending on the purpose of a conversation, prioritize (possibly at different stages) either object-level discussions that ignore others’ views or forming an overall judgment that includes epistemic deference.
E.g. when the purpose is to learn, or to form an independent assessment of something, epistemic deference will often be a distraction.
By contrast, if you make a decision with large-scale and irreversible effects on the world (e.g. “who should get this $5M grant?”) I think it would usually be predictably worse for the world to ignore others’ views.
On the other hand, I think literally everyone using the “average view” among the same set of people is suboptimal even for such purposes: it’s probably better to have less correlated decisions in order to “cover the whole space” of reasonable options. (This might seem odd on some naive models of decision-making, but I think can easily be justified by plausible toy models involving heavy-tailed ex-post returns that are hard to predict ex-ante plus barriers to coordination or undetectable but different biases. I.e. similar to why it would probably be bad if every VC based their investment decisions on the average view of all VCs.)
I think this way one can largely have the best of both worlds.
(I vaguely remember that there is a popular post making a similar recommendation, but couldn’t quickly find it.)
Something I want to add here:
I am not sure whether my error was how much I was deferring in itself. But the decision to defer or not should be made on well defined questions and clearly defined ‘experts’ you might be deferring to. This is not what I was doing. I was deferring on a nebulous question (‘what should I be doing?‘) to an even more nebulous expert audience (a vague sense of what ‘the community’ wanted).
What I should have been doing instead first is to define the question better: Which roles should I be pursuing right now?
This can then be broken down further into subquestions on cause prioritisation, which roles are promising avenues within causes I might be interested in, which roles I might be well suited for, etc, whose information I need to aggregate in a sensible fashion to answer the question which roles I should be pursuing right now.
For each of these subquestions I need to make a separate judgement. For some it makes more sense to defer, for others, less so. Disappointingly, there is no independent expert panel investigating what kind of jobs I might excel at.
But then who to defer to, if I think this is a sensible choice for a particular subquestion, also needs to be clearly defined: for example, I might decide that it makes sense to take 80k at their word about which roles in a particular cause area are particularly promising right now, after reading what they actually say on their website on the subject, perhaps double-checking by asking them via email and polling another couple of people in the field.
‘The community’ is not a well defined expert panel, while the careful aggregation of individual opinions can be, who again, need to be asked well defined questions. Note that this can true even if I gave equal weight to every EA’s opinion: sometimes it can seem like ‘the community’ has an opinion that only few individual EAs hold if actually asked, if any. This is especially true if messaging is distorted and I am not actually asking a well defined question.
Taking into account specific facts or arguments made by other people seems reasonable here. Just writing down e.g. “person X doesn’t like MIRI” in the “cons” column of your spreadsheet seems foolish and wrongheaded.
Framing it as “taking others’ views into account” or “ignoring others’ views” is a big part of the problem, IMO—that language itself directs people towards evaluating the people rather than the arguments, and overall opinions rather than specific facts or claims.
I think we disagree. I’m not sure why you think that even for decisions with large effects one should only or mostly take into account specific facts or arguments, and am curious about your reasoning here.
I do think it will often be even more valuable to understand someone’s specific reasons for having a belief. However, (i) in complex domains achieving a full understanding would be a lot of work, (ii) people usually have incomplete insight into the specific reasons for why they hold a certain belief themselves and instead might appeal to intuition, (iii) in practice you only have so much time and thus can’t fully pursue all disagreements.
So yes, always stopping at “person X thinks that p” and never trying to understand why would be a poor policy. But never stopping at that seems infeasible to me, and I don’t see the benefits from always throwing away the information that X believes p in situations where you don’t fully understand why.
For instance, imagine I pointed a gun to your head and forced you to now choose between two COVID mitigation policies for the US for the next 6 months. I offer you to give you additional information of the type “X thinks that p” with some basic facts on X but no explanation for why they hold this belief. Would you refuse to view that information? If someone else was in that situation, would you pay for me not giving them this information? How much?
There is a somewhat different failure mode where person X’s view isn’t particularly informative compared to the view of other people Y, Z, etc., and so by considerung just X’s view you give it undue weight. But I don’t think you’re talking about that?
I’m partly puzzled by your reaction because the basic phenomenon of deferring to the output of others’ reasoning processes without understanding the underlying facts or arguments strikes me as not unusual at all. For example, I believe that the Earth orbits the Sun rather than the other way around. But I couldn’t give you any very specific argument for this like “on the geocentric hypothesis, the path of this body across the sky would look like this”. Instead, the reason for my belief is that the heliocentric worldview is scientific consensus, i.e. epistemic deference to others without understanding their reasoning.
This also happens when the view in question makes a difference in practice. For instance, as I’m sure you’re aware, hierarchical organizations work (among other things) because managers don’t have to recapitulate every specific argument behind the conclusions of their reports.
To sum up, a very large amount of division of epistemic labor seems like the norm rather than the exception to me, just as for the division of manual labor. The main thing that seems somewhat unusual is making that explicit.
I note that the framing / example case has changed a lot between your original comment / my reply (making a $5m grant and writing “person X is skeptical of MIRI” in the “cons” column) and this parent comment (“imagine I pointed a gun to your head and… offer you to give you additional information;” “never stopping at [person X thinks that p]”). I’m not arguing for entirely refusing to trust other people or dividing labor, as you implied there. I specifically object to giving weight to other people’s top-line views on questions where there’s substantial disagreement, based on your overall assessment of that particular person’s credibility / quality of intuition / whatever, separately from your evaluation of their finer-grained sub-claims.
If you are staking $5m on something, it’s hard for me to imagine a case where it makes sense to end up with an important node in your tree of claims whose justification is “opinions diverge on this but the people I think are smartest tend to believe p.” The reason I think this is usually bad is that (a) it’s actually impossible to know how much weight it’s rational to give someone else’s opinion without inspecting their sub-claims, and (b) it leads to groupthink/herding/information cascades.
As a toy example to illustrate (a): suppose that for MIRI to be the optimal grant recipient, it both needs to be the case that AI risk is high (A) and that MIRI is the Best organization working to mitigate it (B). A and B are independent. The prior is (P(A) = 50, P(B) = 50). Alice and Bob have observed evidence with a 9:1 odds ratio in favor of A, so think (P(A) = 90, P(B) = 50). Carol has observed evidence with a 9:1 odds ratio in favor of B. Alice, Bob and Carol all have the same top-line view of MIRI (P(A and B) = 0.45), but the rational aggregation of Alice and Bob’s “view” is much less positive than the rational aggregation of Bob and Carol’s.
It’s interesting that you mention hierarchical organizations because I think they usually follow a better process for dividing up epistemic labor, which is to assign different sub-problems to different people rather than by averaging a large number of people’s beliefs on a single question. This works better because the sub-problems are more likely to be independent from each other, so they don’t require as much communication / model-sharing to aggregate their results.
In fact, when hierarchical organizations do the other thing—”brute force” aggregate others’ beliefs in situations of disagreement—it usually indicates an organizational failure. My own experience is that I often see people do something a particular way, even though they disagree with it, because they think that’s my preference; but it turns out they had a bad model of my preferences (often because they observed a contextual preference in a different context) and would have been better off using their own judgment.
I think I perceive less of a difference between the examples we’ve been discussing, but after reading your reply I’m also less sure if and where we disagree significantly.
I read your previous claim as essentially saying “it would always be bad to include the information that some person X is skeptical about MIRI when making the decision whether to give MIRI a $5M grant, unless you understand more details about why X has this view”.
I still think this view basically commits you to refusing to see information of that type in the COVID policy thought experiment. This is essentially for the reasons (i)-(iii) I listed above: I think that in practice it will be too costly to understand the views of each such person X in more detail.
(But usually it will be worth it to do this for some people, for instance for the reason spelled out in your toy model. As I said: I do think it will often be even more valuable to understand someone’s specific reasons for having a belief.)
Instead, I suspect you will need to focus on the few highest-priority cases, and in the end you’ll end up with people X1,…,Xl whose views you understand in great detail, people Y1,…,Ym where your understanding stops at other fairly high-level/top-line views (e.g. maybe you know what they think about “will AGI be developed this century?” but not much about why), and people Z1,…,Zn of whom you only know the top-line view of how much funding they’d want to give to MIRI.
(Note that I don’t think this is hypothetical. My impression is that there are in fact long-standing disagreements about MIRI’s work that can’t be fully resolved or even broken down into very precise subclaims/cruxes, despite many people having spent probably hundreds of hours on this. For instance, in the writeups to their first grants to MIRI, Open Phil remark that “We found MIRI’s work especially difficult to evaluate”, and the most recent grant amount was set by a committee that “average[s] individuals’ allocations” . See also this post by Open Phil’s Daniel Dewey and comments.)
At that point, I think you’re basically in a similar situation. There is no gun pointed at your head, but you still want to make a decision right now, and so you can either throw away the information about the views of person Zi or use it without understanding their arguments.
Furthermore, I don’t think your situation with respect to person Yj is that different: if you take their view on “AGI this century?” into account for the decision whether to fund MIRI but have a policy of never using “bare top-level views”, this would commit to to ignoring the same information in a different situation, e.g. the decision whether to place a large bet on whether AGI will be developed this century (purely because what’s a top-level view in one situation will be an argument or “specific” fact in another); this seems odd.
(This is also why I’m not sure I understand the relevance of your point on hierarchical organizations. I agree that usually sub-problems will be assigned to different employees. But e.g. if I assign “AGI this century?” to one employee and “is MIRI well run?” to another employee, why am I justified in believing their conclusions on these fairly high-level questions but not justified in believing anyone’s view on whether MIRI is worth funding?)
Note that thus far I’m mainly arguing against taking into account no-one’s top-level views. Your most recent claim involving “the people I think are smartest” suggests that maybe you mainly object to using a lot of discretion in which particular people’s top-level views to use.
I think my reaction to this is mixed: On one hand, I certainly agree that there is a danger involved here (e.g. in fact I think that many EAs defer too much to others EAs relative to non-EA experts), and that it’s impossible to assess with perfect accuracy how much weight to give to each person. On the other hand, I think it is often possible to assess this with limited but still useful accuracy, both based on subjective and hard-to-justify assessments of how good someone’s judgment seemed in the past (cf. how senior politicians often work with advisors they’ve had a long work relationship with) and on crude objectives proxies (e.g. ‘has a PhD in computer science’).
On the latter, you said that specifically you object to allocating weight to someone’s top-line opinion “separately from your evaluation of their finer-grained sub-claims”. If that means their finer-grained sub-claims on the particular question under consideration, then I disagree for the reasons explained so far. If that means “separately from your evaluation of any finer-grained sub-claim they ever made on anything”, then I agree more with this, though still think this is both common and justified in some cases (e.g. if I learn that I have rare disease A for which specialists universally recommend drug B as treatment, I’ll probably happily take drug B without having ever heard of any specific sub-claim made by any disease-A specialist).
Similarly, I agree that information cascades and groupthink are dangers/downsides, but that they will sometimes be outweighed by the benefits.
If 100 forecasters (who I roughly respect) look at the likelihood of a future event and think it’s ~10% likely, and I look at the same question and think it’s ~33% likely, I think I will be incorrect in my private use of reason for my all-things-considered-view to not update somewhat downwards from 33%.
I think this continues to be true even if we all in theory have access to the same public evidence, etc.
Now, it does depend a bit on the context of what this information is for. For example if I’m asked to give my perspective on a group forecast (and I know that the other 100 forecasters’ predictions will be included anyway), I think it probably makes sense for me to continue to publicly provide ~33% for that question to prevent double-counting and groupthink.
But I think it will be wrong for me to believe 33%, and even more so, wrong to say 33% in a context where somebody else doesn’t have access to the 100 other forecasters.
An additional general concern here to me is computational capacity/kindness—sometimes (often) I just don’t have enough time to evaluate all the object-level arguments! You can maybe argue that until I evaluate all the object-level arguments, I shouldn’t act, yet in practice I feel like I act with lots of uncertainty* all the time!
___
One disagreement I have with Max is whether someone should defer is contingent upon the importance of a decision. I think this begs the question in that it pre-assumes that deference lead to the best outcomes.
Instead, I think you should act such that you all-things-considered-view is that you’re making the best decision. I do think that for many decisions (with the possible exception of creative work), some level of deference leads to better outcomes than zero deference at all, but I don’t think it’s unusually true for important decisions except inasmuch as a) the benefits (and also costs!) of deference are scaled accordingly and b) more people are likely to have thought about important decisions.
__
* Narrow, personal, example that’s basically unrelated to EA: I brush my teeth with fluoride toothpaste. I don’t floss. Why? Cochrane review was fairly equivocal about flossing and fairly certain about toothbrushing. Maybe it’d be more principled if I looked at the data myself and performed my own meta-analysis on the data, or perhaps self-experimented like Gwern, to decide what dental hygiene activities I should take. But in practice I feel like it’s a reasonable decision procedure to just defer to Cochrane review on the empirical facts of the matter, and apply my own value judgments on what activities to take given the facts available.
I’m not sure if we have a principled disagreement here, it’s possible that I just described my view badly above.
I agree that one should act such that one’s all-things-considered view is that one is making the best decision (the way I understand that statement it’s basically a tautology).
Then I think there are some heuristics for which features of a decision situation make it more or less likely that deferring more (or at all) leads to decisions with that property. I think on a high level I agree with you that it depends a lot “on the context of what this information is for”, more so than on e.g. importance.
With my example, I was also trying to point less to importance per se but on something like how the costs and benefits are distributed between yourself and others. This is because very loosely speaking I expect not deferring to often be better if the stakes are concentrated on oneself and more deference to be better if one’s own direct stake is small. I used a decision with large effects on others largely because then it’s not plausible that you yourself are affected by a similar amount; but it would also apply to a decision with zero effect on yourself and a small effect on others. Conversely, it would not apply to a decision that is very important to yourself (e.g. something affecting your whole career trajectory).
Apologies for the long delay in response, feel free to not reply if you’re busy.
Hmm I still think we have a substantive rather than framing disagreement (though I think it is likely that our disagreements aren’t large).
Perhaps this heuristic is really useful for a lot of questions you’re considering. I’m reminded of AGB’s great quote:
For me personally and the specific questions I’ve considered, I think considering whether/how much to defer to by dividing into buckets of “how much it affects myself or others” is certainly a pretty useful heuristic in the absence of better heuristics, but it’s mostly superseded by a different decomposition:
Epistemic—In a context-sensitive manner, do we expect greater or lower deference in this particular situation to lead to more accurate beliefs.
Role expectations* -- Whether the explicit and implicit social expectations on the role you’re assuming privilege deference or independence.
So I think a big/main reason it’s bad to defer completely to others (say 80k) on your own career reasons is epistemic: you have so much thought and local knowledge about your own situation that your prior should very strongly be against others having better all-things-considered views on your career choice than you do. I think this is more crux-y for me than how much your career trajectory affects yourself vs others (at any rate hopefully as EAs our career trajectories affect many others anyway!).
On the other hand, I think my Cochrane review example above is a good epistemic example of deference. even though my dental hygiene practices mainly affect myself and not others (perhaps my past and future partners may disagree), I contend it’s better to defer to the meta-analysis over my own independent analysis in this particular facet of my personal life.
The other main (non-epistemic) lens I’d use to privilege greater or lower humility is whether the explicit and implicit social expectations privilege deference or independence. For example, we’d generally** prefer government bureaucrats in most situations to implement policies, rather than making unprincipled exceptions based on private judgements. This will often look superficially similar to “how much this affects myself or others.”
An example of a dissimilarity is when someone filling out a survey. This is a situation where approximately all of the costs and benefits are borne by other people. So if you have a minority opinion on a topic, it may seem like the epistemically humble-and-correct action is to fill out the poll according to what you believe the majority to think (or alternatively, fill it out with the answer that you privately think is on the margin more conducive to advancing your values).
But in all likelihood, such a policy is one-thought-too-many, and in almost all situations it’d be more prudent to fill out public anonymous polls/surveys with what you actually believe.
Agreed, though I mention this because in discussions of epistemic humility-in-practice, it’s very easy to accidentally do double-counting.
*I don’t like this phrase, happy to use a better one.
**I’m aware that there are exceptions, including during the ongoing coronavirus pandemic.
Thanks! I’m not sure if there is a significant difference about how we’d actually make decisions (I mean, on prior there is probably some difference). But I agree that the single heuristics I mentioned above doesn’t by itself do a great job of describing when and how much to defer, and I agree with your “counterexamples”. (Though note that in principle it’s not surprising if there are counterexamples to a “mere heuristics”.)
I particularly appreciate you describing the “Role expectations” point. I agree that something along those lines is important. My guess is that if we would have debated specific decisions I would have implicitly incorporated this consideration, but I don’t think it was clear to me before reading your comment that this is an important property that will often influence my judgment about how much to defer.
I think that in theory Max is right, that there’s some optimal way to have the best of both worlds. But in practice I think that there are pretty strong biases towards conformity, such that it’s probably worthwhile to shift the community as a whole indiscriminately towards being less epistemic modest.
As one example, people might think “I’ll make up my mind on small decisions, and defer on big decisions.” But then they’ll evaluate what feels big to them , rather than to the EA community overall, and thereby the community as a whole will end up being strongly correlated even on relatively small-scale bets. I think your comment itself actually makes this mistake—there’s now enough money in EA that, in my opinion, there should be many $5M grants which aren’t strongly correlated with the views of EA as a whole.
In particular, I note that venture capitalists allocate much larger amounts of money explicitly on anti-conformist principles. Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful, but I’m not confident about that; the hypothesis that we just haven’t internalised the “hits-based” mentality as well as venture capitalists have also seems plausible.
(My best guess is that the average EA defers too much rather than too little. This and other comments on deference is to address specific points made, rather than to push any particular general takes).
I think this is part of the reason. A plausibly bigger reason is that VC funding can’t result in heavy left-tails. Or rather, left-tails in VC funding are very rarely internalized. Concretely, if you pick your favorite example of “terrible startup for the future of sentient beings,” the VCs in question very rarely get in trouble, and approximately never get punished proportional to the counterfactual harm of their investments. VC funding can be negative for the VC beyond the opportunity cost of money (eg via reputational risk or whatever), but the punishment is quite low relative to the utility costs.
Obviously optimizing for increasing variance is a better deal when you clip the left tail, and optimizing for reducing variance is a better deal when you clip the right tail.
(I also independently think that heavy left tails in the utilitarian sense are probably less common in VC funding than in EA, but I think this is not necessary for my argument to go through).
Good point, I agree this weakens my argument.
I agree it’s possible that because of social pressures or similar things the best policy change that’s viable in practice could be an indiscriminate move toward more or less epistemic deference. Though I probably have less of a strong sense that that’s in fact true.
(Note that when implemented well, the “best of both worlds” policy could actually make it easier to express disagreement because it clarifies that there are two types of beliefs/credences to be kept track of separately, and that one of them has to exclude all epistemic deference.
Similarly, to the extent that people think that avoiding ‘bad, unilateral action’ is a key reason in favor of epistemic deference, it could actually “destigmatize” iconoclastic views if it’s common knowledge that an iconoclastic pre-deference view doesn’t imply unusual primarily-other-affecting actions because primarily-other-affecting actions depend on post-deference rather rather than pre-deference views.)
I agree with everything you say about $5M grants and VCs. I’m not sure if you think my mistake was mainly to consider a $5M stake a “large-scale” decision or something else, but if it’s the former I’m happy to concede that this wasn’t the best example to give for a decision where deference should get a lot of weight (though I think we agree that in theory it should get some weight?).
I strongly agree that “the optimal response isn’t to be more or less epistemically modest indiscriminately”, and with the policy you suggest.
If I recall correctly, somewhat similar recommendations are made in Some thoughts on deference and inside-view models and EA Concepts: Share Impressions Before Credences.
I disagree Max. We can all recall anecdotes of overconfidence because they create well-publicized narratives. With hindsight bias, it seems obvious that overconfidence was the subject. So naturally we overestimate overconfidence risks, just like nuclear power.
The costs of under confidence are invisible and ubiquitous. A grad student fails to submit her paper. An applicant doesn’t apply. A graduate doesn’t write down her NGO idea. Because you can’t see the costs of underconfidence, they could be hundreds or thousands of times the overconfidence costs.
To break apart the question
Should people update based on evidence and not have rigid world-models. Is people disagreeing with you moderate evidence?
Yes to both
Once someone builds the best world-model they can, should they defer to higher-status people’s models
Much much less often than we currently do
How much should we weight disagreement between our models and the models of others?
See Yud’s book: https://www.lesswrong.com/posts/svoD5KLKHyAKEdwPo/against-modest-epistemology