Although you are right that modesty (or deference) often outperforms one’s own personal judgment, this isn’t always the case. Results below are based on Monte Carlo simulations I haven’t published yet.
Take the case of a crowd estimating a cow’s weight. The members of the crowd announce their guesses sequentially. They adopt a uniform rule of D% deference, so that each person’s guess is a weighted average of a sample from a Normal distribution centered on the cow’s true weight, and of the current crowd average guess:
Under this rule, as deference increases, the crowd converges more slowly on the cow’s true weight: deference is bad for group epistemics. This isn’t the same thing as an information cascade, because the crowd will converge eventually unless they are completely deferent.
Furthermore, the benefits of deference for individual guess accuracy are maximized at about 78% deference except potentially on very long timescales. Beyond this point, the group converges so slowly that it undermines, though doesn’t fully cancel out, the individual benefit of adopting the group average.
Finally, when faced with a choice of whether to make a guess according to the group’s rule for deference or whether to be 100% deferent and simply guess the current group average, you will actually do better to make a partially deferent guess than a 100% deferent guess if the group is more than about 78% deferent. Below that point, it’s better for individual accuracy to 100% defer, which suggests a Prisoner’s Game Dilemma model in which individuals ‘defect’ on the project of obtaining high crowd accuracy by deferring to the crowd rather than contributing their own independent guess, leading to very high and deeply suboptimal but not maximally bad levels of deference.
These results depend on specific modeling assumptions.
We might penalize inaccuracy according to its square. This makes deference less useful for individual accuracy, putting the optimal level closer to 65% rather than 78%.
We can also imagine that instead of using the crowd average, the deferent portion of the guess is sampled from a Normal distribution about the crowd average. In this case, the optimal level of deference is closer to 45%, and beyond about 78% deference, it’s better to ignore the crowd entirely and just do pure direct observation.
I haven’t simulated this yet, but I am curious to know what happens if we assume a fixed 1% of guesses are purely independent.
We are evaluating the whole timeline of guesses and observations. What if we are thinking about a person joining a mature debate, where the crowd average has had more time to converge?
It assumes that making and sharing observations is cost-free. In reality, of course, if every scientist had to redo all the experiments in their field (i.e. never deferred to previous results), science could not progress, and this is true everywhere else as well.
Whether or not the cost of further observations is linked to the current group accuracy. If we imagine a scenario where individual accuracy is a heavy driver of the costs of individual observations, then we might want to prioritize deference to keep costs down and permit more or faster guesses. If instead it is crowd accuracy that controls the costs of observations, then we might want to focus on independent observations.
Overall, I think we need to refocus the debate on epistemic modesty around tradeoffs and modeling assumptions in order to help people make the best choice given their goals:
How much to defer seems to depend on a few key factors:
The cost of making independent observations vs. deferring, and whether or not these costs are linked to current group or individual accuracy
How inaccuracy is penalized
How deferent we think the group is
Whether we are prioritizing our own individual accuracy or the speed with which the group converges on the truth
Basically all the problems with deference can be eliminated if we are able to track the difference between independent observations and deferent guesses.
My main takeaways are that:
Intuition is only good enough to be dangerous for thinking in the abstract about deference
Real-world empirical information is crucial for making the right choice of modeling assumptions to decide on how much to defer
Deference is a common and important topic in the rationalist and EA communities on a number of subjects, and should motivate us to try and take a lot more guesses
It is probably worth trying to figure out how to better track the difference between deferent guesses and independent observations in our discourse
Although you are right that modesty (or deference) often outperforms one’s own personal judgment, this isn’t always the case. Results below are based on Monte Carlo simulations I haven’t published yet.
Take the case of a crowd estimating a cow’s weight. The members of the crowd announce their guesses sequentially. They adopt a uniform rule of D% deference, so that each person’s guess is a weighted average of a sample from a Normal distribution centered on the cow’s true weight, and of the current crowd average guess:
Guess_i = D*(Crowd average) + (1-D)*(Direct observation)
Under this rule, as deference increases, the crowd converges more slowly on the cow’s true weight: deference is bad for group epistemics. This isn’t the same thing as an information cascade, because the crowd will converge eventually unless they are completely deferent.
Furthermore, the benefits of deference for individual guess accuracy are maximized at about 78% deference except potentially on very long timescales. Beyond this point, the group converges so slowly that it undermines, though doesn’t fully cancel out, the individual benefit of adopting the group average.
Finally, when faced with a choice of whether to make a guess according to the group’s rule for deference or whether to be 100% deferent and simply guess the current group average, you will actually do better to make a partially deferent guess than a 100% deferent guess if the group is more than about 78% deferent. Below that point, it’s better for individual accuracy to 100% defer, which suggests a Prisoner’s Game Dilemma model in which individuals ‘defect’ on the project of obtaining high crowd accuracy by deferring to the crowd rather than contributing their own independent guess, leading to very high and deeply suboptimal but not maximally bad levels of deference.
These results depend on specific modeling assumptions.
We might penalize inaccuracy according to its square. This makes deference less useful for individual accuracy, putting the optimal level closer to 65% rather than 78%.
We can also imagine that instead of using the crowd average, the deferent portion of the guess is sampled from a Normal distribution about the crowd average. In this case, the optimal level of deference is closer to 45%, and beyond about 78% deference, it’s better to ignore the crowd entirely and just do pure direct observation.
I haven’t simulated this yet, but I am curious to know what happens if we assume a fixed 1% of guesses are purely independent.
We are evaluating the whole timeline of guesses and observations. What if we are thinking about a person joining a mature debate, where the crowd average has had more time to converge?
It assumes that making and sharing observations is cost-free. In reality, of course, if every scientist had to redo all the experiments in their field (i.e. never deferred to previous results), science could not progress, and this is true everywhere else as well.
Whether or not the cost of further observations is linked to the current group accuracy. If we imagine a scenario where individual accuracy is a heavy driver of the costs of individual observations, then we might want to prioritize deference to keep costs down and permit more or faster guesses. If instead it is crowd accuracy that controls the costs of observations, then we might want to focus on independent observations.
Overall, I think we need to refocus the debate on epistemic modesty around tradeoffs and modeling assumptions in order to help people make the best choice given their goals:
How much to defer seems to depend on a few key factors:
The cost of making independent observations vs. deferring, and whether or not these costs are linked to current group or individual accuracy
How inaccuracy is penalized
How deferent we think the group is
Whether we are prioritizing our own individual accuracy or the speed with which the group converges on the truth
Basically all the problems with deference can be eliminated if we are able to track the difference between independent observations and deferent guesses.
My main takeaways are that:
Intuition is only good enough to be dangerous for thinking in the abstract about deference
Real-world empirical information is crucial for making the right choice of modeling assumptions to decide on how much to defer
Deference is a common and important topic in the rationalist and EA communities on a number of subjects, and should motivate us to try and take a lot more guesses
It is probably worth trying to figure out how to better track the difference between deferent guesses and independent observations in our discourse