Around 2015-2019 I felt like the main message I got from the EA community was that my judgement was not to be trusted and I should defer, but without explicit instructions how and who to defer to. ... My interpretation was that my judgement generally was not to be trusted, and if it was not good enough to start new projects myself, I should not make generic career decisions myself, even where the possible downsides were very limited.
I also get a lot of this vibe from (parts of) the EA community, and it drives me a little nuts. Examples:
Moral uncertainty, giving other moral systems weight “because other smart people believe them” rather than because they seem object-level reasonable
Lots of emphasis on avoiding accidentally doing harm by being uninformed
People bring up “intelligent people disagree with this” as a reason against something rather than going through the object-level arguments
Being epistemically modest by, say, replacing your own opinions with the average opinion of everyone around you, might improve the epistemics of the majority of people (in fact it almost must by definition), but it is a terrible idea on a group level: it’s a recipe for information cascades, groupthink and herding.
In retrospect, it’s not surprising that this has ended up with numerous people being scarred and seriously demoralized by applying for massively oversubscribed EA jobs.
I guess it’s ironic that 80,000 Hours—one of the most frequent repeaters of the “don’t accidentally cause harm” meme—seems to have accidentally caused you quite a bit of harm with this advice (and/or its misinterpretations being repeated by others)!
That last paragraph is a good observation, and I don’t think it’s entirely coincidental. 80k has a few instances in their history of accidentally causing harm, which has led them (correctly) to be very conservative about it as an organisation.
The thing is, career advice and PR are two areas 80k is very involved in and which have particular likelihood of causing as much harm as good, due to bad advice or distorted messaging. Most decisions individual EAs make are not like this, and it’s a mistake if they treat 80k’s caution as a reflection of how cautious they should be. Or worse, act even more cautiously reasoning the combined intelligence of the 80k staff is greater than their own (likely true, but likely irrelevant).
I don’t think any of 80k’s career advice has caused much harm compared to the counterfactual of not having given that advice at all, so I feel a bit confused how to think about this. Even the grossest misrepresentation of EtG being the only way to do good or something still strikes me as better than the current average experience a college graduate has (which is no guidance, and all career advice comes from companies trying to recruit you).
I think the comparison to “the current average experience a college graduate has” isn’t quite fair, because the group of people who see 80k’s advice and act on is is already quite selected for lots of traits (e.g. altruism). I would be surprised if the average person influenced by 80k’s EtG advice had the average college graduate experience in terms of which careers they consider and hence, where they look for advice, e.g. they might already be more inclined to go into policy, the non-profit sector or research to do good.
(I have no opinion on how your point comes out on the whole. I wasn’t around in 2015, but intuitively it would also surprise me if 80k didn’t do substantially more good during that time than bad, even bracketing out community building effects (, which, admittedly, is hard))
As it happens, there are a couple of examples in this post where poor or distorted versions of 80k advice arguably caused harm relative to no advice; over-focus on working at EA orgs due to ‘talent constraint’ claims probably set Denise’s entire career back by ~2 years for no gain, and a simplistic understanding of replaceability was significantly responsible for her giving up on political work.
Apart from the direct cost, such events leave a sour taste in people’s mouths and so can cause them to dissociate from the community; if we’re going to focus on ‘recruiting’ people while they are young, anything that increases attrition needs to be considered very carefully and skeptically.
I do agree that in general it’s not that hard to beat ‘no advice’, rather a lot of the need for care comes from simplistic advice’s natural tendency to crowd out nuanced advice.
I don’t mean to bash 80k here; when they become aware of these things they try pretty hard to clean it up, they maintain a public list of mistakes (which includes both of the above), and I think they apply way more thought and imagination to the question of how this kind of thing can happen than most other places, even most other EA orgs. I’ve been impressed by the seriousness with which they take this kind of problem over the years.
Yeah, totally agree that we can find individual instances where the advice is bad. Just seems pretty unlikely for that average to be worse, even just by the lights of the person who is given advice (and ignoring altruistic effects, which presumably are more heavy-tailed).
I think I probably agree with the general thrust of this comment, but disagree on various specifics.
‘Intelligent people disagree with this’ is a good reason against being too confident in one’s opinion. At the very least, it should highlight there are opportunities to explore where the disagreement is coming from, which should hopefully help everyone to form better opinions.
I also don’t feel like moral uncertainty is a good example of people deferring too much.
A different way to look at this might be that if ‘good judgement’ is something that lots of people need in their careers, especially if they don’t follow any of the priority paths (as argued here), this is something that needs to be trained—and you don’t train good judgement by always blindly deferring.
Yeah, and besides the training effect there is also the benefit that while one person who disagrees with hundreds is unlikely to be correct, if they are correct, it’s super important that those hundreds of others get to learn from them.
So it may be very important in expectation to notice such disagreements, do a lot of research to understand one’s own and the others’ position as well as possible, and then let them know of the results.
(And yes, the moral uncertainty example doesn’t seem to fit very well, especially for antirealists.)
I’d say that “Intelligent people disagree with this” is a good reason to look into what those people think and why—I agree that it should make you less certain of your current position, but you might actually end up more certain of your original opinion after you’ve understood those disagreements.
See also answers here mentioning that EA feels “intellectually stale”. A friend says he thinks a lot of impressive people have left the EA movement because of this :(
I feel bad, because I think maybe I was one of the first people to push the “avoid accidental harm” thing.
“Stagnation” was also the 5th most often mentioned reason for declining interest in EA, over the last 12 months, when we asked about this in the 2019 EA Survey, accounting for about 7.4% of responses.
There was some discussion about the issue of EA intellectual stagnation in this thread (like I say in my comment, I don’t agree that EA is stagnating).
Yeah, I think it’s very difficult to tell whether the trend which people take themselves be perceiving is explained by there having been a larger amount of low hanging fruit in the earlier years of EA, which led to people encountering a larger number of radical new ideas in the earlier years, or whether there’s actually been a slowdown in EA intellectual productivity. (Similarly, it may be that because people tend to encounter a lot of new ideas when they are first getting involved in EA, people perceive the insights being generated by EA as slowing down). I think it’s hard to tell whether EA is stagnating in a worrying sense in that it is not clear how much intellectual progress we should expect to see now that some of the low hanging fruit is already picked.
That said, I actually think that the positive aspects of EA’s professionalisation (which you point to in your other comment) may explain some of the perceptions described here, which I think are on the whole mistaken. I think in earlier years, there was a lot of amateur, broad speculation for and against various big questions in EA (e.g. big a priori arguments about AI versus animals, much of which was pretty wild and ill-informed). I think, conversely, we now have a much healthier ecosystem, with people making progress on the myriad narrower, technical problems that need to be addressed in order to address those broader questions.
Thanks David, this is more or less what I was trying to express with my response to Stefan in that thread.
I want to add that “making intellectual progress” has two different benefits: One is the obvious one, figuring out more true things so they can influence our actions to do more good. As you say, we may actually be doing better on that one.
The other one is to attract people to the community by it being an intellectually stimulating place. We might be losing the kind of people who answered ‘stagnation’ in the poll above, as they are not able to participate in the professionalised debates, if they happen in public at all.
On the other hand, this might mean that we are not deterring people anymore who may have felt like they need to be into intellectual debates to join the EA community. I don’t know what the right trade-off is, but I suspect it’s actually more important not to put latter group off.
I actually think the principles of deference to expertise and avoiding accidental harm are in principle good and we should continue using them. However, in EA the barrier to being seen as an expert is very low—often its enough to have written a blog or forum post on something, having invested less than 100 hours in total. For me an expert is someone who has spent the better part of his or her career working in a field, for example climate policy. While I think the former is still useful to give an introduction to a field, the latter form of expertise has been somewhat undervalued in EA.
I guess it depends on what topics you’re referring to, but regarding many topics, the bar for being seen as an expert within EA seems substantially higher than 100 hours.
Lots of emphasis on avoiding accidentally doing harm by being uninformed
I gave a talk about this, so I consider myself to be one of the repeaters of that message. But I also think I always tried to add a lot of caveats, like “you should take this advice less seriously if you’re the type of person who listens to advice like this” and similar. It’s a bit hard to calibrate, but I’m definitely in favor of people trying new projects, even at the risk of causing mild accidental harm, and in fact I think that’s something that has helped me grow in the past.
If you think these sorts of framing still miss the mark, I’d be interested in hearing your reasoning about that.
I’m somewhat sympathetic to the frustration you express. However, I suspect the optimal response isn’t to be more or less epistemically modest indiscriminately. Instead, I suspect the optimal policy is something like:
Always be clear and explicit to what extent a view you’re communicating involves deference to others.
Depending on the purpose of a conversation, prioritize (possibly at different stages) either object-level discussions that ignore others’ views or forming an overall judgment that includes epistemic deference.
E.g. when the purpose is to learn, or to form an independent assessment of something, epistemic deference will often be a distraction.
By contrast, if you make a decision with large-scale and irreversible effects on the world (e.g. “who should get this $5M grant?”) I think it would usually be predictably worse for the world to ignore others’ views.
On the other hand, I think literally everyone using the “average view” among the same set of people is suboptimal even for such purposes: it’s probably better to have less correlated decisions in order to “cover the whole space” of reasonable options. (This might seem odd on some naive models of decision-making, but I think can easily be justified by plausible toy models involving heavy-tailed ex-post returns that are hard to predict ex-ante plus barriers to coordination or undetectable but different biases. I.e. similar to why it would probably be bad if every VC based their investment decisions on the average view of all VCs.)
I think this way one can largely have the best of both worlds.
(I vaguely remember that there is a popular post making a similar recommendation, but couldn’t quickly find it.)
I am not sure whether my error was how much I was deferring in itself. But the decision to defer or not should be made on well defined questions and clearly defined ‘experts’ you might be deferring to. This is not what I was doing. I was deferring on a nebulous question (‘what should I be doing?‘) to an even more nebulous expert audience (a vague sense of what ‘the community’ wanted).
What I should have been doing instead first is to define the question better: Which roles should I be pursuing right now?
This can then be broken down further into subquestions on cause prioritisation, which roles are promising avenues within causes I might be interested in, which roles I might be well suited for, etc, whose information I need to aggregate in a sensible fashion to answer the question which roles I should be pursuing right now.
For each of these subquestions I need to make a separate judgement. For some it makes more sense to defer, for others, less so. Disappointingly, there is no independent expert panel investigating what kind of jobs I might excel at.
But then who to defer to, if I think this is a sensible choice for a particular subquestion, also needs to be clearly defined: for example, I might decide that it makes sense to take 80k at their word about which roles in a particular cause area are particularly promising right now, after reading what they actually say on their website on the subject, perhaps double-checking by asking them via email and polling another couple of people in the field.
‘The community’ is not a well defined expert panel, while the careful aggregation of individual opinions can be, who again, need to be asked well defined questions. Note that this can true even if I gave equal weight to every EA’s opinion: sometimes it can seem like ‘the community’ has an opinion that only few individual EAs hold if actually asked, if any. This is especially true if messaging is distorted and I am not actually asking a well defined question.
if you make a decision with large-scale and irreversible effects on the world (e.g. “who should get this $5M grant?”) I think it would usually be predictably worse for the world to ignore others’ views
Taking into account specific facts or arguments made by other people seems reasonable here. Just writing down e.g. “person X doesn’t like MIRI” in the “cons” column of your spreadsheet seems foolish and wrongheaded.
Framing it as “taking others’ views into account” or “ignoring others’ views” is a big part of the problem, IMO—that language itself directs people towards evaluating the people rather than the arguments, and overall opinions rather than specific facts or claims.
I think we disagree. I’m not sure why you think that even for decisions with large effects one should only or mostly take into account specific facts or arguments, and am curious about your reasoning here.
I do think it will often be even more valuable to understand someone’s specific reasons for having a belief. However, (i) in complex domains achieving a full understanding would be a lot of work, (ii) people usually have incomplete insight into the specific reasons for why they hold a certain belief themselves and instead might appeal to intuition, (iii) in practice you only have so much time and thus can’t fully pursue all disagreements.
So yes, always stopping at “person X thinks that p” and never trying to understand why would be a poor policy. But never stopping at that seems infeasible to me, and I don’t see the benefits from always throwing away the information that X believes p in situations where you don’t fully understand why.
For instance, imagine I pointed a gun to your head and forced you to now choose between two COVID mitigation policies for the US for the next 6 months. I offer you to give you additional information of the type “X thinks that p” with some basic facts on X but no explanation for why they hold this belief. Would you refuse to view that information? If someone else was in that situation, would you pay for me not giving them this information? How much?
There is a somewhat different failure mode where person X’s view isn’t particularly informative compared to the view of other people Y, Z, etc., and so by considerung just X’s view you give it undue weight. But I don’t think you’re talking about that?
I’m partly puzzled by your reaction because the basic phenomenon of deferring to the output of others’ reasoning processes without understanding the underlying facts or arguments strikes me as not unusual at all. For example, I believe that the Earth orbits the Sun rather than the other way around. But I couldn’t give you any very specific argument for this like “on the geocentric hypothesis, the path of this body across the sky would look like this”. Instead, the reason for my belief is that the heliocentric worldview is scientific consensus, i.e. epistemic deference to others without understanding their reasoning.
This also happens when the view in question makes a difference in practice. For instance, as I’m sure you’re aware, hierarchical organizations work (among other things) because managers don’t have to recapitulate every specific argument behind the conclusions of their reports.
To sum up, a very large amount of division of epistemic labor seems like the norm rather than the exception to me, just as for the division of manual labor. The main thing that seems somewhat unusual is making that explicit.
I note that the framing / example case has changed a lot between your original comment / my reply (making a $5m grant and writing “person X is skeptical of MIRI” in the “cons” column) and this parent comment (“imagine I pointed a gun to your head and… offer you to give you additional information;” “never stopping at [person X thinks that p]”). I’m not arguing for entirely refusing to trust other people or dividing labor, as you implied there. I specifically object to giving weight to other people’s top-line views on questions where there’s substantial disagreement, based on your overall assessment of that particular person’s credibility / quality of intuition / whatever, separately from your evaluation of their finer-grained sub-claims.
If you are staking $5m on something, it’s hard for me to imagine a case where it makes sense to end up with an important node in your tree of claims whose justification is “opinions diverge on this but the people I think are smartest tend to believe p.” The reason I think this is usually bad is that (a) it’s actually impossible to know how much weight it’s rational to give someone else’s opinion without inspecting their sub-claims, and (b) it leads to groupthink/herding/information cascades.
As a toy example to illustrate (a): suppose that for MIRI to be the optimal grant recipient, it both needs to be the case that AI risk is high (A) and that MIRI is the Best organization working to mitigate it (B). A and B are independent. The prior is (P(A) = 50, P(B) = 50). Alice and Bob have observed evidence with a 9:1 odds ratio in favor of A, so think (P(A) = 90, P(B) = 50). Carol has observed evidence with a 9:1 odds ratio in favor of B. Alice, Bob and Carol all have the same top-line view of MIRI (P(A and B) = 0.45), but the rational aggregation of Alice and Bob’s “view” is much less positive than the rational aggregation of Bob and Carol’s.
It’s interesting that you mention hierarchical organizations because I think they usually follow a better process for dividing up epistemic labor, which is to assign different sub-problems to different people rather than by averaging a large number of people’s beliefs on a single question. This works better because the sub-problems are more likely to be independent from each other, so they don’t require as much communication / model-sharing to aggregate their results.
In fact, when hierarchical organizations do the other thing—”brute force” aggregate others’ beliefs in situations of disagreement—it usually indicates an organizational failure. My own experience is that I often see people do something a particular way, even though they disagree with it, because they think that’s my preference; but it turns out they had a bad model of my preferences (often because they observed a contextual preference in a different context) and would have been better off using their own judgment.
I think I perceive less of a difference between the examples we’ve been discussing, but after reading your reply I’m also less sure if and where we disagree significantly.
I read your previous claim as essentially saying “it would always be bad to include the information that some person X is skeptical about MIRI when making the decision whether to give MIRI a $5M grant, unless you understand more details about why X has this view”.
I still think this view basically commits you to refusing to see information of that type in the COVID policy thought experiment. This is essentially for the reasons (i)-(iii) I listed above: I think that in practice it will be too costly to understand the views of each such person X in more detail.
(But usually it will be worth it to do this for some people, for instance for the reason spelled out in your toy model. As I said: I do think it will often be even more valuable to understand someone’s specific reasons for having a belief.)
Instead, I suspect you will need to focus on the few highest-priority cases, and in the end you’ll end up with people X1,…,Xl whose views you understand in great detail, people Y1,…,Ym where your understanding stops at other fairly high-level/top-line views (e.g. maybe you know what they think about “will AGI be developed this century?” but not much about why), and people Z1,…,Zn of whom you only know the top-line view of how much funding they’d want to give to MIRI.
At that point, I think you’re basically in a similar situation. There is no gun pointed at your head, but you still want to make a decision right now, and so you can either throw away the information about the views of person Zi or use it without understanding their arguments.
Furthermore, I don’t think your situation with respect to person Yj is that different: if you take their view on “AGI this century?” into account for the decision whether to fund MIRI but have a policy of never using “bare top-level views”, this would commit to to ignoring the same information in a different situation, e.g. the decision whether to place a large bet on whether AGI will be developed this century (purely because what’s a top-level view in one situation will be an argument or “specific” fact in another); this seems odd.
(This is also why I’m not sure I understand the relevance of your point on hierarchical organizations. I agree that usually sub-problems will be assigned to different employees. But e.g. if I assign “AGI this century?” to one employee and “is MIRI well run?” to another employee, why am I justified in believing their conclusions on these fairly high-level questions but not justified in believing anyone’s view on whether MIRI is worth funding?)
Note that thus far I’m mainly arguing against taking into account no-one’s top-level views. Your most recent claim involving “the people I think are smartest” suggests that maybe you mainly object to using a lot of discretion in which particular people’s top-level views to use.
I think my reaction to this is mixed: On one hand, I certainly agree that there is a danger involved here (e.g. in fact I think that many EAs defer too much to others EAs relative to non-EA experts), and that it’s impossible to assess with perfect accuracy how much weight to give to each person. On the other hand, I think it is often possible to assess this with limited but still useful accuracy, both based on subjective and hard-to-justify assessments of how good someone’s judgment seemed in the past (cf. how senior politicians often work with advisors they’ve had a long work relationship with) and on crude objectives proxies (e.g. ‘has a PhD in computer science’).
On the latter, you said that specifically you object to allocating weight to someone’s top-line opinion “separately from your evaluation of their finer-grained sub-claims”. If that means their finer-grained sub-claims on the particular question under consideration, then I disagree for the reasons explained so far. If that means “separately from your evaluation of any finer-grained sub-claim they ever made on anything”, then I agree more with this, though still think this is both common and justified in some cases (e.g. if I learn that I have rare disease A for which specialists universally recommend drug B as treatment, I’ll probably happily take drug B without having ever heard of any specific sub-claim made by any disease-A specialist).
Similarly, I agree that information cascades and groupthink are dangers/downsides, but that they will sometimes be outweighed by the benefits.
If 100 forecasters (who I roughly respect) look at the likelihood of a future event and think it’s ~10% likely, and I look at the same question and think it’s ~33% likely, I think I will be incorrect in my private use of reason for my all-things-considered-view to not update somewhat downwards from 33%.
I think this continues to be true even if we all in theory have access to the same public evidence, etc.
Now, it does depend a bit on the context of what this information is for. For example if I’m asked to give my perspective on a group forecast (and I know that the other 100 forecasters’ predictions will be included anyway), I think it probably makes sense for me to continue to publicly provide ~33% for that question to prevent double-counting and groupthink.
But I think it will be wrong for me to believe 33%, and even more so, wrong to say 33% in a context where somebody else doesn’t have access to the 100 other forecasters.
An additional general concern here to me is computational capacity/kindness—sometimes (often) I just don’t have enough time to evaluate all the object-level arguments! You can maybe argue that until I evaluate all the object-level arguments, I shouldn’t act, yet in practice I feel like I act with lots of uncertainty* all the time! ___
One disagreement I have with Max is whether someone should defer is contingent upon the importance of a decision. I think this begs the question in that it pre-assumes that deference lead to the best outcomes.
Instead, I think you should act such that you all-things-considered-view is that you’re making the best decision. I do think that for many decisions (with the possible exception of creative work), some level of deference leads to better outcomes than zero deference at all, but I don’t think it’s unusually true for important decisions except inasmuch as a) the benefits (and also costs!) of deference are scaled accordingly and b) more people are likely to have thought about important decisions.
__ * Narrow, personal, example that’s basically unrelated to EA: I brush my teeth with fluoride toothpaste. I don’t floss. Why? Cochrane review was fairly equivocal about flossing and fairly certain about toothbrushing. Maybe it’d be more principled if I looked at the data myself and performed my own meta-analysis on the data, or perhaps self-experimented like Gwern, to decide what dental hygiene activities I should take. But in practice I feel like it’s a reasonable decision procedure to just defer to Cochrane review on the empirical facts of the matter, and apply my own value judgments on what activities to take given the facts available.
One disagreement I have with Max is whether someone should defer is contingent upon the importance of a decision. I think this begs the question in that it pre-assumes that deference lead to the best outcomes.
Instead, I think you should act such that you all-things-considered-view is that you’re making the best decision. I do think that for many decisions (with the possible exception of creative work), some level of deference leads to better outcomes than zero deference at all, but I don’t think it’s unusually true for important decisions except inasmuch as a) the benefits (and also costs!) of deference are scaled accordingly and b) more people are likely to have thought about important decisions.
I’m not sure if we have a principled disagreement here, it’s possible that I just described my view badly above.
I agree that one should act such that one’s all-things-considered view is that one is making the best decision (the way I understand that statement it’s basically a tautology).
Then I think there are some heuristics for which features of a decision situation make it more or less likely that deferring more (or at all) leads to decisions with that property. I think on a high level I agree with you that it depends a lot “on the context of what this information is for”, more so than on e.g. importance.
With my example, I was also trying to point less to importance per se but on something like how the costs and benefits are distributed between yourself and others. This is because very loosely speaking I expect not deferring to often be better if the stakes are concentrated on oneself and more deference to be better if one’s own direct stake is small. I used a decision with large effects on others largely because then it’s not plausible that you yourself are affected by a similar amount; but it would also apply to a decision with zero effect on yourself and a small effect on others. Conversely, it would not apply to a decision that is very important to yourself (e.g. something affecting your whole career trajectory).
Apologies for the long delay in response, feel free to not reply if you’re busy.
Hmm I still think we have a substantive rather than framing disagreement (though I think it is likely that our disagreements aren’t large).
This is because very loosely speaking I expect not deferring to often be better if the stakes are concentrated on oneself and more deference to be better if one’s own direct stake is small. I used a decision with large effects on others largely because then it’s not plausible that you yourself are affected by a similar amount; but it would also apply to a decision with zero effect on yourself and a small effect on others. Conversely, it would not apply to a decision that is very important to yourself (e.g. something affecting your whole career trajectory).
Perhaps this heuristic is really useful for a lot of questions you’re considering. I’m reminded of AGB’s great quote:
There are enough individual and practical considerations here (in both directions) that in many situations the actual thing I would advocate for is something like “work out what you would do with both approaches, check against results ‘without fear or favour’, and move towards whatever method is working best for you”.
For me personally and the specific questions I’ve considered, I think considering whether/how much to defer to by dividing into buckets of “how much it affects myself or others” is certainly a pretty useful heuristic in the absence of better heuristics, but it’s mostly superseded by a different decomposition:
Epistemic—In a context-sensitive manner, do we expect greater or lower deference in this particular situation to lead to more accurate beliefs.
Role expectations* -- Whether the explicit and implicit social expectations on the role you’re assuming privilege deference or independence.
So I think a big/main reason it’s bad to defer completely to others (say 80k) on your own career reasons is epistemic: you have so much thought and local knowledge about your own situation that your prior should very strongly be against others having better all-things-considered views on your career choice than you do. I think this is more crux-y for me than how much your career trajectory affects yourself vs others (at any rate hopefully as EAs our career trajectories affect many others anyway!).
On the other hand, I think my Cochrane review example above is a good epistemic example of deference. even though my dental hygiene practices mainly affect myself and not others (perhaps my past and future partners may disagree), I contend it’s better to defer to the meta-analysis over my own independent analysis in this particular facet of my personal life.
The other main (non-epistemic) lens I’d use to privilege greater or lower humility is whether the explicit and implicit social expectations privilege deference or independence. For example, we’d generally** prefer government bureaucrats in most situations to implement policies, rather than making unprincipled exceptions based on private judgements. This will often look superficially similar to “how much this affects myself or others.”
An example of a dissimilarity is when someone filling out a survey. This is a situation where approximately all of the costs and benefits are borne by other people. So if you have a minority opinion on a topic, it may seem like the epistemically humble-and-correct action is to fill out the poll according to what you believe the majority to think (or alternatively, fill it out with the answer that you privately think is on the margin more conducive to advancing your values).
But in all likelihood, such a policy is one-thought-too-many, and in almost all situations it’d be more prudent to fill out public anonymous polls/surveys with what you actually believe.
I agree that one should act such that one’s all-things-considered view is that one is making the best decision (the way I understand that statement it’s basically a tautology).
Agreed, though I mention this because in discussions of epistemic humility-in-practice, it’s very easy to accidentally do double-counting.
*I don’t like this phrase, happy to use a better one.
**I’m aware that there are exceptions, including during the ongoing coronavirus pandemic.
Thanks! I’m not sure if there is a significant difference about how we’d actually make decisions (I mean, on prior there is probably some difference). But I agree that the single heuristics I mentioned above doesn’t by itself do a great job of describing when and how much to defer, and I agree with your “counterexamples”. (Though note that in principle it’s not surprising if there are counterexamples to a “mere heuristics”.)
I particularly appreciate you describing the “Role expectations” point. I agree that something along those lines is important. My guess is that if we would have debated specific decisions I would have implicitly incorporated this consideration, but I don’t think it was clear to me before reading your comment that this is an important property that will often influence my judgment about how much to defer.
I think that in theory Max is right, that there’s some optimal way to have the best of both worlds. But in practice I think that there are pretty strong biases towards conformity, such that it’s probably worthwhile to shift the community as a whole indiscriminately towards being less epistemic modest.
As one example, people might think “I’ll make up my mind on small decisions, and defer on big decisions.” But then they’ll evaluate what feels big to them , rather than to the EA community overall, and thereby the community as a whole will end up being strongly correlated even on relatively small-scale bets. I think your comment itself actually makes this mistake—there’s now enough money in EA that, in my opinion, there should be many $5M grants which aren’t strongly correlated with the views of EA as a whole.
In particular, I note that venture capitalists allocate much larger amounts of money explicitly on anti-conformist principles. Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful, but I’m not confident about that; the hypothesis that we just haven’t internalised the “hits-based” mentality as well as venture capitalists have also seems plausible.
(My best guess is that the average EA defers too much rather than too little. This and other comments on deference is to address specific points made, rather than to push any particular general takes).
Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful
I think this is part of the reason. A plausibly bigger reason is that VC funding can’t result in heavy left-tails. Or rather, left-tails in VC funding are very rarely internalized. Concretely, if you pick your favorite example of “terrible startup for the future of sentient beings,” the VCs in question very rarely get in trouble, and approximately never get punished proportional to the counterfactual harm of their investments. VC funding can be negative for the VC beyond the opportunity cost of money (eg via reputational risk or whatever), but the punishment is quite low relative to the utility costs.
Obviously optimizing for increasing variance is a better deal when you clip the left tail, and optimizing for reducing variance is a better deal when you clip the right tail.
(I also independently think that heavy left tails in the utilitarian sense are probably less common in VC funding than in EA, but I think this is not necessary for my argument to go through).
I agree it’s possible that because of social pressures or similar things the best policy change that’s viable in practice could be an indiscriminate move toward more or less epistemic deference. Though I probably have less of a strong sense that that’s in fact true.
(Note that when implemented well, the “best of both worlds” policy could actually make it easier to express disagreement because it clarifies that there are two types of beliefs/credences to be kept track of separately, and that one of them has to exclude all epistemic deference.
Similarly, to the extent that people think that avoiding ‘bad, unilateral action’ is a key reason in favor of epistemic deference, it could actually “destigmatize” iconoclastic views if it’s common knowledge that an iconoclastic pre-deference view doesn’t imply unusual primarily-other-affecting actions because primarily-other-affecting actions depend on post-deference rather rather than pre-deference views.)
I agree with everything you say about $5M grants and VCs. I’m not sure if you think my mistake was mainly to consider a $5M stake a “large-scale” decision or something else, but if it’s the former I’m happy to concede that this wasn’t the best example to give for a decision where deference should get a lot of weight (though I think we agree that in theory it should get some weight?).
I disagree Max. We can all recall anecdotes of overconfidence because they create well-publicized narratives. With hindsight bias, it seems obvious that overconfidence was the subject. So naturally we overestimate overconfidence risks, just like nuclear power.
The costs of under confidence are invisible and ubiquitous. A grad student fails to submit her paper. An applicant doesn’t apply. A graduate doesn’t write down her NGO idea. Because you can’t see the costs of underconfidence, they could be hundreds or thousands of times the overconfidence costs.
To break apart the question
Should people update based on evidence and not have rigid world-models. Is people disagreeing with you moderate evidence?
Yes to both
Once someone builds the best world-model they can, should they defer to higher-status people’s models
Much much less often than we currently do
How much should we weight disagreement between our models and the models of others?
I also get a lot of this vibe from (parts of) the EA community, and it drives me a little nuts. Examples:
Moral uncertainty, giving other moral systems weight “because other smart people believe them” rather than because they seem object-level reasonable
Lots of emphasis on avoiding accidentally doing harm by being uninformed
People bring up “intelligent people disagree with this” as a reason against something rather than going through the object-level arguments
Being epistemically modest by, say, replacing your own opinions with the average opinion of everyone around you, might improve the epistemics of the majority of people (in fact it almost must by definition), but it is a terrible idea on a group level: it’s a recipe for information cascades, groupthink and herding.
In retrospect, it’s not surprising that this has ended up with numerous people being scarred and seriously demoralized by applying for massively oversubscribed EA jobs.
I guess it’s ironic that 80,000 Hours—one of the most frequent repeaters of the “don’t accidentally cause harm” meme—seems to have accidentally caused you quite a bit of harm with this advice (and/or its misinterpretations being repeated by others)!
That last paragraph is a good observation, and I don’t think it’s entirely coincidental. 80k has a few instances in their history of accidentally causing harm, which has led them (correctly) to be very conservative about it as an organisation.
The thing is, career advice and PR are two areas 80k is very involved in and which have particular likelihood of causing as much harm as good, due to bad advice or distorted messaging. Most decisions individual EAs make are not like this, and it’s a mistake if they treat 80k’s caution as a reflection of how cautious they should be. Or worse, act even more cautiously reasoning the combined intelligence of the 80k staff is greater than their own (likely true, but likely irrelevant).
I don’t think any of 80k’s career advice has caused much harm compared to the counterfactual of not having given that advice at all, so I feel a bit confused how to think about this. Even the grossest misrepresentation of EtG being the only way to do good or something still strikes me as better than the current average experience a college graduate has (which is no guidance, and all career advice comes from companies trying to recruit you).
I think the comparison to “the current average experience a college graduate has” isn’t quite fair, because the group of people who see 80k’s advice and act on is is already quite selected for lots of traits (e.g. altruism). I would be surprised if the average person influenced by 80k’s EtG advice had the average college graduate experience in terms of which careers they consider and hence, where they look for advice, e.g. they might already be more inclined to go into policy, the non-profit sector or research to do good.
(I have no opinion on how your point comes out on the whole. I wasn’t around in 2015, but intuitively it would also surprise me if 80k didn’t do substantially more good during that time than bad, even bracketing out community building effects (, which, admittedly, is hard))
(Disclaimer: I am OP’s husband)
As it happens, there are a couple of examples in this post where poor or distorted versions of 80k advice arguably caused harm relative to no advice; over-focus on working at EA orgs due to ‘talent constraint’ claims probably set Denise’s entire career back by ~2 years for no gain, and a simplistic understanding of replaceability was significantly responsible for her giving up on political work.
Apart from the direct cost, such events leave a sour taste in people’s mouths and so can cause them to dissociate from the community; if we’re going to focus on ‘recruiting’ people while they are young, anything that increases attrition needs to be considered very carefully and skeptically.
I do agree that in general it’s not that hard to beat ‘no advice’, rather a lot of the need for care comes from simplistic advice’s natural tendency to crowd out nuanced advice.
I don’t mean to bash 80k here; when they become aware of these things they try pretty hard to clean it up, they maintain a public list of mistakes (which includes both of the above), and I think they apply way more thought and imagination to the question of how this kind of thing can happen than most other places, even most other EA orgs. I’ve been impressed by the seriousness with which they take this kind of problem over the years.
Yeah, totally agree that we can find individual instances where the advice is bad. Just seems pretty unlikely for that average to be worse, even just by the lights of the person who is given advice (and ignoring altruistic effects, which presumably are more heavy-tailed).
I think I probably agree with the general thrust of this comment, but disagree on various specifics.
‘Intelligent people disagree with this’ is a good reason against being too confident in one’s opinion. At the very least, it should highlight there are opportunities to explore where the disagreement is coming from, which should hopefully help everyone to form better opinions.
I also don’t feel like moral uncertainty is a good example of people deferring too much.
A different way to look at this might be that if ‘good judgement’ is something that lots of people need in their careers, especially if they don’t follow any of the priority paths (as argued here), this is something that needs to be trained—and you don’t train good judgement by always blindly deferring.
Yeah, and besides the training effect there is also the benefit that while one person who disagrees with hundreds is unlikely to be correct, if they are correct, it’s super important that those hundreds of others get to learn from them.
So it may be very important in expectation to notice such disagreements, do a lot of research to understand one’s own and the others’ position as well as possible, and then let them know of the results.
(And yes, the moral uncertainty example doesn’t seem to fit very well, especially for antirealists.)
I’d say that “Intelligent people disagree with this” is a good reason to look into what those people think and why—I agree that it should make you less certain of your current position, but you might actually end up more certain of your original opinion after you’ve understood those disagreements.
See also answers here mentioning that EA feels “intellectually stale”. A friend says he thinks a lot of impressive people have left the EA movement because of this :(
I feel bad, because I think maybe I was one of the first people to push the “avoid accidental harm” thing.
“Stagnation” was also the 5th most often mentioned reason for declining interest in EA, over the last 12 months, when we asked about this in the 2019 EA Survey, accounting for about 7.4% of responses.
Thanks, David, for that data.
There was some discussion about the issue of EA intellectual stagnation in this thread (like I say in my comment, I don’t agree that EA is stagnating).
Yeah, I think it’s very difficult to tell whether the trend which people take themselves be perceiving is explained by there having been a larger amount of low hanging fruit in the earlier years of EA, which led to people encountering a larger number of radical new ideas in the earlier years, or whether there’s actually been a slowdown in EA intellectual productivity. (Similarly, it may be that because people tend to encounter a lot of new ideas when they are first getting involved in EA, people perceive the insights being generated by EA as slowing down). I think it’s hard to tell whether EA is stagnating in a worrying sense in that it is not clear how much intellectual progress we should expect to see now that some of the low hanging fruit is already picked.
That said, I actually think that the positive aspects of EA’s professionalisation (which you point to in your other comment) may explain some of the perceptions described here, which I think are on the whole mistaken. I think in earlier years, there was a lot of amateur, broad speculation for and against various big questions in EA (e.g. big a priori arguments about AI versus animals, much of which was pretty wild and ill-informed). I think, conversely, we now have a much healthier ecosystem, with people making progress on the myriad narrower, technical problems that need to be addressed in order to address those broader questions.
Thanks David, this is more or less what I was trying to express with my response to Stefan in that thread.
I want to add that “making intellectual progress” has two different benefits: One is the obvious one, figuring out more true things so they can influence our actions to do more good. As you say, we may actually be doing better on that one.
The other one is to attract people to the community by it being an intellectually stimulating place. We might be losing the kind of people who answered ‘stagnation’ in the poll above, as they are not able to participate in the professionalised debates, if they happen in public at all.
On the other hand, this might mean that we are not deterring people anymore who may have felt like they need to be into intellectual debates to join the EA community. I don’t know what the right trade-off is, but I suspect it’s actually more important not to put latter group off.
I actually think the principles of deference to expertise and avoiding accidental harm are in principle good and we should continue using them. However, in EA the barrier to being seen as an expert is very low—often its enough to have written a blog or forum post on something, having invested less than 100 hours in total. For me an expert is someone who has spent the better part of his or her career working in a field, for example climate policy. While I think the former is still useful to give an introduction to a field, the latter form of expertise has been somewhat undervalued in EA.
I guess it depends on what topics you’re referring to, but regarding many topics, the bar for being seen as an expert within EA seems substantially higher than 100 hours.
I gave a talk about this, so I consider myself to be one of the repeaters of that message. But I also think I always tried to add a lot of caveats, like “you should take this advice less seriously if you’re the type of person who listens to advice like this” and similar. It’s a bit hard to calibrate, but I’m definitely in favor of people trying new projects, even at the risk of causing mild accidental harm, and in fact I think that’s something that has helped me grow in the past.
If you think these sorts of framing still miss the mark, I’d be interested in hearing your reasoning about that.
I’m somewhat sympathetic to the frustration you express. However, I suspect the optimal response isn’t to be more or less epistemically modest indiscriminately. Instead, I suspect the optimal policy is something like:
Always be clear and explicit to what extent a view you’re communicating involves deference to others.
Depending on the purpose of a conversation, prioritize (possibly at different stages) either object-level discussions that ignore others’ views or forming an overall judgment that includes epistemic deference.
E.g. when the purpose is to learn, or to form an independent assessment of something, epistemic deference will often be a distraction.
By contrast, if you make a decision with large-scale and irreversible effects on the world (e.g. “who should get this $5M grant?”) I think it would usually be predictably worse for the world to ignore others’ views.
On the other hand, I think literally everyone using the “average view” among the same set of people is suboptimal even for such purposes: it’s probably better to have less correlated decisions in order to “cover the whole space” of reasonable options. (This might seem odd on some naive models of decision-making, but I think can easily be justified by plausible toy models involving heavy-tailed ex-post returns that are hard to predict ex-ante plus barriers to coordination or undetectable but different biases. I.e. similar to why it would probably be bad if every VC based their investment decisions on the average view of all VCs.)
I think this way one can largely have the best of both worlds.
(I vaguely remember that there is a popular post making a similar recommendation, but couldn’t quickly find it.)
Something I want to add here:
I am not sure whether my error was how much I was deferring in itself. But the decision to defer or not should be made on well defined questions and clearly defined ‘experts’ you might be deferring to. This is not what I was doing. I was deferring on a nebulous question (‘what should I be doing?‘) to an even more nebulous expert audience (a vague sense of what ‘the community’ wanted).
What I should have been doing instead first is to define the question better: Which roles should I be pursuing right now?
This can then be broken down further into subquestions on cause prioritisation, which roles are promising avenues within causes I might be interested in, which roles I might be well suited for, etc, whose information I need to aggregate in a sensible fashion to answer the question which roles I should be pursuing right now.
For each of these subquestions I need to make a separate judgement. For some it makes more sense to defer, for others, less so. Disappointingly, there is no independent expert panel investigating what kind of jobs I might excel at.
But then who to defer to, if I think this is a sensible choice for a particular subquestion, also needs to be clearly defined: for example, I might decide that it makes sense to take 80k at their word about which roles in a particular cause area are particularly promising right now, after reading what they actually say on their website on the subject, perhaps double-checking by asking them via email and polling another couple of people in the field.
‘The community’ is not a well defined expert panel, while the careful aggregation of individual opinions can be, who again, need to be asked well defined questions. Note that this can true even if I gave equal weight to every EA’s opinion: sometimes it can seem like ‘the community’ has an opinion that only few individual EAs hold if actually asked, if any. This is especially true if messaging is distorted and I am not actually asking a well defined question.
Taking into account specific facts or arguments made by other people seems reasonable here. Just writing down e.g. “person X doesn’t like MIRI” in the “cons” column of your spreadsheet seems foolish and wrongheaded.
Framing it as “taking others’ views into account” or “ignoring others’ views” is a big part of the problem, IMO—that language itself directs people towards evaluating the people rather than the arguments, and overall opinions rather than specific facts or claims.
I think we disagree. I’m not sure why you think that even for decisions with large effects one should only or mostly take into account specific facts or arguments, and am curious about your reasoning here.
I do think it will often be even more valuable to understand someone’s specific reasons for having a belief. However, (i) in complex domains achieving a full understanding would be a lot of work, (ii) people usually have incomplete insight into the specific reasons for why they hold a certain belief themselves and instead might appeal to intuition, (iii) in practice you only have so much time and thus can’t fully pursue all disagreements.
So yes, always stopping at “person X thinks that p” and never trying to understand why would be a poor policy. But never stopping at that seems infeasible to me, and I don’t see the benefits from always throwing away the information that X believes p in situations where you don’t fully understand why.
For instance, imagine I pointed a gun to your head and forced you to now choose between two COVID mitigation policies for the US for the next 6 months. I offer you to give you additional information of the type “X thinks that p” with some basic facts on X but no explanation for why they hold this belief. Would you refuse to view that information? If someone else was in that situation, would you pay for me not giving them this information? How much?
There is a somewhat different failure mode where person X’s view isn’t particularly informative compared to the view of other people Y, Z, etc., and so by considerung just X’s view you give it undue weight. But I don’t think you’re talking about that?
I’m partly puzzled by your reaction because the basic phenomenon of deferring to the output of others’ reasoning processes without understanding the underlying facts or arguments strikes me as not unusual at all. For example, I believe that the Earth orbits the Sun rather than the other way around. But I couldn’t give you any very specific argument for this like “on the geocentric hypothesis, the path of this body across the sky would look like this”. Instead, the reason for my belief is that the heliocentric worldview is scientific consensus, i.e. epistemic deference to others without understanding their reasoning.
This also happens when the view in question makes a difference in practice. For instance, as I’m sure you’re aware, hierarchical organizations work (among other things) because managers don’t have to recapitulate every specific argument behind the conclusions of their reports.
To sum up, a very large amount of division of epistemic labor seems like the norm rather than the exception to me, just as for the division of manual labor. The main thing that seems somewhat unusual is making that explicit.
I note that the framing / example case has changed a lot between your original comment / my reply (making a $5m grant and writing “person X is skeptical of MIRI” in the “cons” column) and this parent comment (“imagine I pointed a gun to your head and… offer you to give you additional information;” “never stopping at [person X thinks that p]”). I’m not arguing for entirely refusing to trust other people or dividing labor, as you implied there. I specifically object to giving weight to other people’s top-line views on questions where there’s substantial disagreement, based on your overall assessment of that particular person’s credibility / quality of intuition / whatever, separately from your evaluation of their finer-grained sub-claims.
If you are staking $5m on something, it’s hard for me to imagine a case where it makes sense to end up with an important node in your tree of claims whose justification is “opinions diverge on this but the people I think are smartest tend to believe p.” The reason I think this is usually bad is that (a) it’s actually impossible to know how much weight it’s rational to give someone else’s opinion without inspecting their sub-claims, and (b) it leads to groupthink/herding/information cascades.
As a toy example to illustrate (a): suppose that for MIRI to be the optimal grant recipient, it both needs to be the case that AI risk is high (A) and that MIRI is the Best organization working to mitigate it (B). A and B are independent. The prior is (P(A) = 50, P(B) = 50). Alice and Bob have observed evidence with a 9:1 odds ratio in favor of A, so think (P(A) = 90, P(B) = 50). Carol has observed evidence with a 9:1 odds ratio in favor of B. Alice, Bob and Carol all have the same top-line view of MIRI (P(A and B) = 0.45), but the rational aggregation of Alice and Bob’s “view” is much less positive than the rational aggregation of Bob and Carol’s.
It’s interesting that you mention hierarchical organizations because I think they usually follow a better process for dividing up epistemic labor, which is to assign different sub-problems to different people rather than by averaging a large number of people’s beliefs on a single question. This works better because the sub-problems are more likely to be independent from each other, so they don’t require as much communication / model-sharing to aggregate their results.
In fact, when hierarchical organizations do the other thing—”brute force” aggregate others’ beliefs in situations of disagreement—it usually indicates an organizational failure. My own experience is that I often see people do something a particular way, even though they disagree with it, because they think that’s my preference; but it turns out they had a bad model of my preferences (often because they observed a contextual preference in a different context) and would have been better off using their own judgment.
I think I perceive less of a difference between the examples we’ve been discussing, but after reading your reply I’m also less sure if and where we disagree significantly.
I read your previous claim as essentially saying “it would always be bad to include the information that some person X is skeptical about MIRI when making the decision whether to give MIRI a $5M grant, unless you understand more details about why X has this view”.
I still think this view basically commits you to refusing to see information of that type in the COVID policy thought experiment. This is essentially for the reasons (i)-(iii) I listed above: I think that in practice it will be too costly to understand the views of each such person X in more detail.
(But usually it will be worth it to do this for some people, for instance for the reason spelled out in your toy model. As I said: I do think it will often be even more valuable to understand someone’s specific reasons for having a belief.)
Instead, I suspect you will need to focus on the few highest-priority cases, and in the end you’ll end up with people X1,…,Xl whose views you understand in great detail, people Y1,…,Ym where your understanding stops at other fairly high-level/top-line views (e.g. maybe you know what they think about “will AGI be developed this century?” but not much about why), and people Z1,…,Zn of whom you only know the top-line view of how much funding they’d want to give to MIRI.
(Note that I don’t think this is hypothetical. My impression is that there are in fact long-standing disagreements about MIRI’s work that can’t be fully resolved or even broken down into very precise subclaims/cruxes, despite many people having spent probably hundreds of hours on this. For instance, in the writeups to their first grants to MIRI, Open Phil remark that “We found MIRI’s work especially difficult to evaluate”, and the most recent grant amount was set by a committee that “average[s] individuals’ allocations” . See also this post by Open Phil’s Daniel Dewey and comments.)
At that point, I think you’re basically in a similar situation. There is no gun pointed at your head, but you still want to make a decision right now, and so you can either throw away the information about the views of person Zi or use it without understanding their arguments.
Furthermore, I don’t think your situation with respect to person Yj is that different: if you take their view on “AGI this century?” into account for the decision whether to fund MIRI but have a policy of never using “bare top-level views”, this would commit to to ignoring the same information in a different situation, e.g. the decision whether to place a large bet on whether AGI will be developed this century (purely because what’s a top-level view in one situation will be an argument or “specific” fact in another); this seems odd.
(This is also why I’m not sure I understand the relevance of your point on hierarchical organizations. I agree that usually sub-problems will be assigned to different employees. But e.g. if I assign “AGI this century?” to one employee and “is MIRI well run?” to another employee, why am I justified in believing their conclusions on these fairly high-level questions but not justified in believing anyone’s view on whether MIRI is worth funding?)
Note that thus far I’m mainly arguing against taking into account no-one’s top-level views. Your most recent claim involving “the people I think are smartest” suggests that maybe you mainly object to using a lot of discretion in which particular people’s top-level views to use.
I think my reaction to this is mixed: On one hand, I certainly agree that there is a danger involved here (e.g. in fact I think that many EAs defer too much to others EAs relative to non-EA experts), and that it’s impossible to assess with perfect accuracy how much weight to give to each person. On the other hand, I think it is often possible to assess this with limited but still useful accuracy, both based on subjective and hard-to-justify assessments of how good someone’s judgment seemed in the past (cf. how senior politicians often work with advisors they’ve had a long work relationship with) and on crude objectives proxies (e.g. ‘has a PhD in computer science’).
On the latter, you said that specifically you object to allocating weight to someone’s top-line opinion “separately from your evaluation of their finer-grained sub-claims”. If that means their finer-grained sub-claims on the particular question under consideration, then I disagree for the reasons explained so far. If that means “separately from your evaluation of any finer-grained sub-claim they ever made on anything”, then I agree more with this, though still think this is both common and justified in some cases (e.g. if I learn that I have rare disease A for which specialists universally recommend drug B as treatment, I’ll probably happily take drug B without having ever heard of any specific sub-claim made by any disease-A specialist).
Similarly, I agree that information cascades and groupthink are dangers/downsides, but that they will sometimes be outweighed by the benefits.
If 100 forecasters (who I roughly respect) look at the likelihood of a future event and think it’s ~10% likely, and I look at the same question and think it’s ~33% likely, I think I will be incorrect in my private use of reason for my all-things-considered-view to not update somewhat downwards from 33%.
I think this continues to be true even if we all in theory have access to the same public evidence, etc.
Now, it does depend a bit on the context of what this information is for. For example if I’m asked to give my perspective on a group forecast (and I know that the other 100 forecasters’ predictions will be included anyway), I think it probably makes sense for me to continue to publicly provide ~33% for that question to prevent double-counting and groupthink.
But I think it will be wrong for me to believe 33%, and even more so, wrong to say 33% in a context where somebody else doesn’t have access to the 100 other forecasters.
An additional general concern here to me is computational capacity/kindness—sometimes (often) I just don’t have enough time to evaluate all the object-level arguments! You can maybe argue that until I evaluate all the object-level arguments, I shouldn’t act, yet in practice I feel like I act with lots of uncertainty* all the time!
___
One disagreement I have with Max is whether someone should defer is contingent upon the importance of a decision. I think this begs the question in that it pre-assumes that deference lead to the best outcomes.
Instead, I think you should act such that you all-things-considered-view is that you’re making the best decision. I do think that for many decisions (with the possible exception of creative work), some level of deference leads to better outcomes than zero deference at all, but I don’t think it’s unusually true for important decisions except inasmuch as a) the benefits (and also costs!) of deference are scaled accordingly and b) more people are likely to have thought about important decisions.
__
* Narrow, personal, example that’s basically unrelated to EA: I brush my teeth with fluoride toothpaste. I don’t floss. Why? Cochrane review was fairly equivocal about flossing and fairly certain about toothbrushing. Maybe it’d be more principled if I looked at the data myself and performed my own meta-analysis on the data, or perhaps self-experimented like Gwern, to decide what dental hygiene activities I should take. But in practice I feel like it’s a reasonable decision procedure to just defer to Cochrane review on the empirical facts of the matter, and apply my own value judgments on what activities to take given the facts available.
I’m not sure if we have a principled disagreement here, it’s possible that I just described my view badly above.
I agree that one should act such that one’s all-things-considered view is that one is making the best decision (the way I understand that statement it’s basically a tautology).
Then I think there are some heuristics for which features of a decision situation make it more or less likely that deferring more (or at all) leads to decisions with that property. I think on a high level I agree with you that it depends a lot “on the context of what this information is for”, more so than on e.g. importance.
With my example, I was also trying to point less to importance per se but on something like how the costs and benefits are distributed between yourself and others. This is because very loosely speaking I expect not deferring to often be better if the stakes are concentrated on oneself and more deference to be better if one’s own direct stake is small. I used a decision with large effects on others largely because then it’s not plausible that you yourself are affected by a similar amount; but it would also apply to a decision with zero effect on yourself and a small effect on others. Conversely, it would not apply to a decision that is very important to yourself (e.g. something affecting your whole career trajectory).
Apologies for the long delay in response, feel free to not reply if you’re busy.
Hmm I still think we have a substantive rather than framing disagreement (though I think it is likely that our disagreements aren’t large).
Perhaps this heuristic is really useful for a lot of questions you’re considering. I’m reminded of AGB’s great quote:
For me personally and the specific questions I’ve considered, I think considering whether/how much to defer to by dividing into buckets of “how much it affects myself or others” is certainly a pretty useful heuristic in the absence of better heuristics, but it’s mostly superseded by a different decomposition:
Epistemic—In a context-sensitive manner, do we expect greater or lower deference in this particular situation to lead to more accurate beliefs.
Role expectations* -- Whether the explicit and implicit social expectations on the role you’re assuming privilege deference or independence.
So I think a big/main reason it’s bad to defer completely to others (say 80k) on your own career reasons is epistemic: you have so much thought and local knowledge about your own situation that your prior should very strongly be against others having better all-things-considered views on your career choice than you do. I think this is more crux-y for me than how much your career trajectory affects yourself vs others (at any rate hopefully as EAs our career trajectories affect many others anyway!).
On the other hand, I think my Cochrane review example above is a good epistemic example of deference. even though my dental hygiene practices mainly affect myself and not others (perhaps my past and future partners may disagree), I contend it’s better to defer to the meta-analysis over my own independent analysis in this particular facet of my personal life.
The other main (non-epistemic) lens I’d use to privilege greater or lower humility is whether the explicit and implicit social expectations privilege deference or independence. For example, we’d generally** prefer government bureaucrats in most situations to implement policies, rather than making unprincipled exceptions based on private judgements. This will often look superficially similar to “how much this affects myself or others.”
An example of a dissimilarity is when someone filling out a survey. This is a situation where approximately all of the costs and benefits are borne by other people. So if you have a minority opinion on a topic, it may seem like the epistemically humble-and-correct action is to fill out the poll according to what you believe the majority to think (or alternatively, fill it out with the answer that you privately think is on the margin more conducive to advancing your values).
But in all likelihood, such a policy is one-thought-too-many, and in almost all situations it’d be more prudent to fill out public anonymous polls/surveys with what you actually believe.
Agreed, though I mention this because in discussions of epistemic humility-in-practice, it’s very easy to accidentally do double-counting.
*I don’t like this phrase, happy to use a better one.
**I’m aware that there are exceptions, including during the ongoing coronavirus pandemic.
Thanks! I’m not sure if there is a significant difference about how we’d actually make decisions (I mean, on prior there is probably some difference). But I agree that the single heuristics I mentioned above doesn’t by itself do a great job of describing when and how much to defer, and I agree with your “counterexamples”. (Though note that in principle it’s not surprising if there are counterexamples to a “mere heuristics”.)
I particularly appreciate you describing the “Role expectations” point. I agree that something along those lines is important. My guess is that if we would have debated specific decisions I would have implicitly incorporated this consideration, but I don’t think it was clear to me before reading your comment that this is an important property that will often influence my judgment about how much to defer.
I think that in theory Max is right, that there’s some optimal way to have the best of both worlds. But in practice I think that there are pretty strong biases towards conformity, such that it’s probably worthwhile to shift the community as a whole indiscriminately towards being less epistemic modest.
As one example, people might think “I’ll make up my mind on small decisions, and defer on big decisions.” But then they’ll evaluate what feels big to them , rather than to the EA community overall, and thereby the community as a whole will end up being strongly correlated even on relatively small-scale bets. I think your comment itself actually makes this mistake—there’s now enough money in EA that, in my opinion, there should be many $5M grants which aren’t strongly correlated with the views of EA as a whole.
In particular, I note that venture capitalists allocate much larger amounts of money explicitly on anti-conformist principles. Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful, but I’m not confident about that; the hypothesis that we just haven’t internalised the “hits-based” mentality as well as venture capitalists have also seems plausible.
(My best guess is that the average EA defers too much rather than too little. This and other comments on deference is to address specific points made, rather than to push any particular general takes).
I think this is part of the reason. A plausibly bigger reason is that VC funding can’t result in heavy left-tails. Or rather, left-tails in VC funding are very rarely internalized. Concretely, if you pick your favorite example of “terrible startup for the future of sentient beings,” the VCs in question very rarely get in trouble, and approximately never get punished proportional to the counterfactual harm of their investments. VC funding can be negative for the VC beyond the opportunity cost of money (eg via reputational risk or whatever), but the punishment is quite low relative to the utility costs.
Obviously optimizing for increasing variance is a better deal when you clip the left tail, and optimizing for reducing variance is a better deal when you clip the right tail.
(I also independently think that heavy left tails in the utilitarian sense are probably less common in VC funding than in EA, but I think this is not necessary for my argument to go through).
Good point, I agree this weakens my argument.
I agree it’s possible that because of social pressures or similar things the best policy change that’s viable in practice could be an indiscriminate move toward more or less epistemic deference. Though I probably have less of a strong sense that that’s in fact true.
(Note that when implemented well, the “best of both worlds” policy could actually make it easier to express disagreement because it clarifies that there are two types of beliefs/credences to be kept track of separately, and that one of them has to exclude all epistemic deference.
Similarly, to the extent that people think that avoiding ‘bad, unilateral action’ is a key reason in favor of epistemic deference, it could actually “destigmatize” iconoclastic views if it’s common knowledge that an iconoclastic pre-deference view doesn’t imply unusual primarily-other-affecting actions because primarily-other-affecting actions depend on post-deference rather rather than pre-deference views.)
I agree with everything you say about $5M grants and VCs. I’m not sure if you think my mistake was mainly to consider a $5M stake a “large-scale” decision or something else, but if it’s the former I’m happy to concede that this wasn’t the best example to give for a decision where deference should get a lot of weight (though I think we agree that in theory it should get some weight?).
I strongly agree that “the optimal response isn’t to be more or less epistemically modest indiscriminately”, and with the policy you suggest.
If I recall correctly, somewhat similar recommendations are made in Some thoughts on deference and inside-view models and EA Concepts: Share Impressions Before Credences.
I disagree Max. We can all recall anecdotes of overconfidence because they create well-publicized narratives. With hindsight bias, it seems obvious that overconfidence was the subject. So naturally we overestimate overconfidence risks, just like nuclear power.
The costs of under confidence are invisible and ubiquitous. A grad student fails to submit her paper. An applicant doesn’t apply. A graduate doesn’t write down her NGO idea. Because you can’t see the costs of underconfidence, they could be hundreds or thousands of times the overconfidence costs.
To break apart the question
Should people update based on evidence and not have rigid world-models. Is people disagreeing with you moderate evidence?
Yes to both
Once someone builds the best world-model they can, should they defer to higher-status people’s models
Much much less often than we currently do
How much should we weight disagreement between our models and the models of others?
See Yud’s book: https://www.lesswrong.com/posts/svoD5KLKHyAKEdwPo/against-modest-epistemology