If it informs you that EA beliefs on some question have been unusual from the get-go, it makes sense to update the other way, toward the distribution of beliefs among people not involved in the EA community.
I’m a bit confused by this. Suppose that EA has a good track record on an issue where its beliefs have been unusual from the get-go. For example, I think that by temperament EAs tend to be more open to sci-fi possibilities than others, even before having thought much about them; and that over the last decade or so we’ve increasingly seen sci-fi possibilities arising. Then I should update towards deferring to EAs because it seems like we’re in the sort of world where sci-fi possibilities happen, and it seems like others are (irrationally) dismissing these possibilities.
On a separate note: I currently don’t think that epistemic deference as a concept makes sense, because defying a consensus has two effects that are often roughly the same size: it means you’re more likely to be wrong, and it means you’re creating more value if right.* But if so, then using deferential credences to choose actions will systematically lead you astray, because you’ll neglect the correlation between likelihood of success and value of success.
Toy example: your inside view says your novel plan has 90% chance of working, and if it does it’ll earn $1000; and experts think it has 10% chance of working, and if it does it’ll earn $100. Suppose you place as much weight on your own worldview as experts’. Incorrect calculation: your all-things-considered credence in your plan working is 50%, your all-things-considered estimate of the value of success is $550, your all-things-considered expected value of the plan is $275. Better calculation: your worldview says that the expected value of your plan is $900, the experts think the expected value is $10, average these to get expected value of $455—much more valuable than in the incorrect calculation!
Note that in the latter calculation we never actually calculated any “all-things-considered credences”. For this reason I now only express such credences with a disclaimer like “but this shouldn’t be taken as action-guiding”.
* A third effect which might be bigger than either of them: it motivates you to go out and try stuff, which will give you valuable skills and make you more correct in the future.
I’m a bit confused by this. Suppose that EA has a good track record on an issue where its beliefs have been unusual from the get-go.… Then I should update towards deferring to EAs
I’m defining a way of picking sides in disagreements that makes more sense than giving everyone equal weight, even from a maximally epistemically modest perspective. The way in which the policy “give EAs more weight all around, because they’ve got a good track record on things they’ve been outside the mainstream on” is criticizable on epistemic modesty grounds is that one could object, “Others can see the track record as well as you. Why do you think the right amount to update on it is more than they think the right amount is?” You can salvage a thought along these lines in a epistemic-modesty-criticism-proof way, but it would need some further story about how, say, you have some “inside information” about the fact of EAs’ better track record. Does that help?
Your quote is replying to my attempt at a “gist”, in the introduction—I try to spell this out a bit more further in the middle of the last section, in the bit where I say “More broadly, groups may simply differ in their ability to acquire information, and it may be that a particular group’s ability on this front is difficult to determine without years of close contact.” Let me know if that bit clarifies the point.
Re
I currently don’t think that epistemic deference as a concept makes sense, because defying a consensus has two effects that are often roughly the same size ,
I don’t follow. I get that acting on low-probability scenarios can let you get in on neglected opportunities, but you don’t want to actually get the probabilities wrong, right?
In any event, maybe messing up the epistemics also makes it easier for you to spot neglected opportunities or something, and maybe this benefit sometimes kind of cancels out the cost, but this doesn’t strike me as relevant to the question of whether epistemic deference as a concept makes sense. Startup founders may benefit from overconfidence, but overconfidence as a concept still makes sense.
I don’t follow. I get that acting on low-probability scenarios can let you get in on neglected opportunities, but you don’t want to actually get the probabilities wrong, right?
I reject the idea that all-things-considered probabilities are “right” and inside-view probabilities are “wrong”, because you should very rarely be using all-things-considered probabilities when making decisions, for reasons of simple arithmetic (as per my example). Tell me what you want to use the probability for and I’ll tell you what type of probability you should be using.
You might say: look, even if you never actually use all-things-considered probabilities in the real world, at least in theory they’re still normatively ideal. But I reject that too—see the Anthropic Decision Theory paper for why.
On a separate note: I currently don’t think that epistemic deference as a concept makes sense, because defying a consensus has two effects that are often roughly the same size: it means you’re more likely to be wrong, and it means you’re creating more value if right.
I don’t fully follow this explanation, but if it’s true that defying a consensus has two effects that are the same size, doesn’t that suggest you can choose any consensus-defying action because the EV is the same regardless, since the likelihood of you being wrong is ~cancelled out by the expected value of being right?
Also the “value if right” doesn’t seem likely to be only modulated by the extent to which you are defying the consensus?
Example: If you are flying a plane and considering a new way of landing a plane that goes against what 99% of pilots think is reasonable , the “value if right” might be much smaller than the negative effects of “value if wrong”. It’s also not clear to me that if you now decide to take an landing approach that was against what 99.9% of pilots think was reasonable you will 10x your “value if right” compared to the 99% action.
The probability of success in some project may be correlated with value conditional on success in many domains, not just ones involving deference, and we typically don’t think that gets in the way of using probabilities in the usual way, no? If you’re wondering whether some corner of something sticking out of the ground is a box of treasure or a huge boulder, maybe you think that the probability you can excavate it is higher if it’s the box of treasure, and that there’s only any value to doing so if it is. The expected value of trying to excavate is P(treasure) * P(success|treasure) * value of treasure. All the probabilities are “all-things-considered”.
I respect you a lot, both as a thinker and as a friend, so I really am sorry if this reply seems dismissive. But I think there’s a sort of “LessWrong decision theory black hole” that makes people a bit crazy in ways that are obvious from the outside, and this comment thread isn’t the place to adjudicate all that. I trust that most readers who aren’t in the hole will not see your example as demonstration that you shouldn’t use all-things-considered probabilities when making decisions, so I won’t press the point beyond this comment.
I think there’s a sort of “LessWrong decision theory black hole” that makes people a bit crazy in ways that are obvious from the outside, and this comment thread isn’t the place to adjudicate all that.
From my perspective it’s the opposite: epistemic modesty is an incredibly strong skeptical argument (a type of argument that often gets people very confused), extreme forms of which have been popular in EA despite leading to conclusions which conflict strongly with common sense (like “in most cases, one should pay scarcely any attention to what you find the most persuasive view on an issue”).
In practice, fortunately, even people who endorse strong epistemic modesty don’t actually implement it, and thereby manage to still do useful things. But I haven’t yet seen any supporters of epistemic modesty provide a principled way of deciding when to act on their own judgment, in defiance of the conclusions of (a large majority of) the 8 billion other people on earth.
By contrast, I think that focusing on policies rather than all-things-considered credences (which is the thing I was gesturing at with my toy example) basically dissolves the problem. I don’t expect that you believe me about this, since I haven’t yet written this argument up clearly (although I hope to do so soon). But in some sense I’m not claiming anything new here: I think that an individual’s all-things-considered deferential credences aren’t very useful for almost the exact same reason that it’s not very useful to take a group of people and aggregate their beliefs into a single set of “all-people-considered” credences when trying to get them to make a group decision (at least not using naive methods; doing it using prediction markets is more reasonable).
I’m a bit confused by this. Suppose that EA has a good track record on an issue where its beliefs have been unusual from the get-go. For example, I think that by temperament EAs tend to be more open to sci-fi possibilities than others, even before having thought much about them; and that over the last decade or so we’ve increasingly seen sci-fi possibilities arising. Then I should update towards deferring to EAs because it seems like we’re in the sort of world where sci-fi possibilities happen, and it seems like others are (irrationally) dismissing these possibilities.
On a separate note: I currently don’t think that epistemic deference as a concept makes sense, because defying a consensus has two effects that are often roughly the same size: it means you’re more likely to be wrong, and it means you’re creating more value if right.* But if so, then using deferential credences to choose actions will systematically lead you astray, because you’ll neglect the correlation between likelihood of success and value of success.
Toy example: your inside view says your novel plan has 90% chance of working, and if it does it’ll earn $1000; and experts think it has 10% chance of working, and if it does it’ll earn $100. Suppose you place as much weight on your own worldview as experts’. Incorrect calculation: your all-things-considered credence in your plan working is 50%, your all-things-considered estimate of the value of success is $550, your all-things-considered expected value of the plan is $275. Better calculation: your worldview says that the expected value of your plan is $900, the experts think the expected value is $10, average these to get expected value of $455—much more valuable than in the incorrect calculation!
Note that in the latter calculation we never actually calculated any “all-things-considered credences”. For this reason I now only express such credences with a disclaimer like “but this shouldn’t be taken as action-guiding”.
* A third effect which might be bigger than either of them: it motivates you to go out and try stuff, which will give you valuable skills and make you more correct in the future.
I’m defining a way of picking sides in disagreements that makes more sense than giving everyone equal weight, even from a maximally epistemically modest perspective. The way in which the policy “give EAs more weight all around, because they’ve got a good track record on things they’ve been outside the mainstream on” is criticizable on epistemic modesty grounds is that one could object, “Others can see the track record as well as you. Why do you think the right amount to update on it is more than they think the right amount is?” You can salvage a thought along these lines in a epistemic-modesty-criticism-proof way, but it would need some further story about how, say, you have some “inside information” about the fact of EAs’ better track record. Does that help?
Your quote is replying to my attempt at a “gist”, in the introduction—I try to spell this out a bit more further in the middle of the last section, in the bit where I say “More broadly, groups may simply differ in their ability to acquire information, and it may be that a particular group’s ability on this front is difficult to determine without years of close contact.” Let me know if that bit clarifies the point.
Re
I don’t follow. I get that acting on low-probability scenarios can let you get in on neglected opportunities, but you don’t want to actually get the probabilities wrong, right?
In any event, maybe messing up the epistemics also makes it easier for you to spot neglected opportunities or something, and maybe this benefit sometimes kind of cancels out the cost, but this doesn’t strike me as relevant to the question of whether epistemic deference as a concept makes sense. Startup founders may benefit from overconfidence, but overconfidence as a concept still makes sense.
I reject the idea that all-things-considered probabilities are “right” and inside-view probabilities are “wrong”, because you should very rarely be using all-things-considered probabilities when making decisions, for reasons of simple arithmetic (as per my example). Tell me what you want to use the probability for and I’ll tell you what type of probability you should be using.
You might say: look, even if you never actually use all-things-considered probabilities in the real world, at least in theory they’re still normatively ideal. But I reject that too—see the Anthropic Decision Theory paper for why.
I don’t fully follow this explanation, but if it’s true that defying a consensus has two effects that are the same size, doesn’t that suggest you can choose any consensus-defying action because the EV is the same regardless, since the likelihood of you being wrong is ~cancelled out by the expected value of being right?
Also the “value if right” doesn’t seem likely to be only modulated by the extent to which you are defying the consensus?
Example:
If you are flying a plane and considering a new way of landing a plane that goes against what 99% of pilots think is reasonable , the “value if right” might be much smaller than the negative effects of “value if wrong”. It’s also not clear to me that if you now decide to take an landing approach that was against what 99.9% of pilots think was reasonable you will 10x your “value if right” compared to the 99% action.
The probability of success in some project may be correlated with value conditional on success in many domains, not just ones involving deference, and we typically don’t think that gets in the way of using probabilities in the usual way, no? If you’re wondering whether some corner of something sticking out of the ground is a box of treasure or a huge boulder, maybe you think that the probability you can excavate it is higher if it’s the box of treasure, and that there’s only any value to doing so if it is. The expected value of trying to excavate is P(treasure) * P(success|treasure) * value of treasure. All the probabilities are “all-things-considered”.
I respect you a lot, both as a thinker and as a friend, so I really am sorry if this reply seems dismissive. But I think there’s a sort of “LessWrong decision theory black hole” that makes people a bit crazy in ways that are obvious from the outside, and this comment thread isn’t the place to adjudicate all that. I trust that most readers who aren’t in the hole will not see your example as demonstration that you shouldn’t use all-things-considered probabilities when making decisions, so I won’t press the point beyond this comment.
From my perspective it’s the opposite: epistemic modesty is an incredibly strong skeptical argument (a type of argument that often gets people very confused), extreme forms of which have been popular in EA despite leading to conclusions which conflict strongly with common sense (like “in most cases, one should pay scarcely any attention to what you find the most persuasive view on an issue”).
In practice, fortunately, even people who endorse strong epistemic modesty don’t actually implement it, and thereby manage to still do useful things. But I haven’t yet seen any supporters of epistemic modesty provide a principled way of deciding when to act on their own judgment, in defiance of the conclusions of (a large majority of) the 8 billion other people on earth.
By contrast, I think that focusing on policies rather than all-things-considered credences (which is the thing I was gesturing at with my toy example) basically dissolves the problem. I don’t expect that you believe me about this, since I haven’t yet written this argument up clearly (although I hope to do so soon). But in some sense I’m not claiming anything new here: I think that an individual’s all-things-considered deferential credences aren’t very useful for almost the exact same reason that it’s not very useful to take a group of people and aggregate their beliefs into a single set of “all-people-considered” credences when trying to get them to make a group decision (at least not using naive methods; doing it using prediction markets is more reasonable).
That said, thanks for sharing the Anthropic Decision Theory paper! I’ll check it out.