Our prior strongly punishes MIRI. While the mean of its evidence distribution is 2,053,690,000 HEWALYs/$10,000, the posterior mean is only 180.8 HEWALYs/$10,000. If we set the prior scale parameter to larger than about 1.09, the posterior estimate for MIRI is greater than 1038 HEWALYs/$10,000, thus beating 80,000 Hours.
This suggests that it might be good in the long run to have a process that learns what prior is appropriate, e.g. by going back and seeing what prior would have best predicted previous years’ impact.
To support a claim that this applies in “virtually all” cases, I’d want to see more engagement with pragmatic problems applying modesty, including:
Identifying experts is far from free epistemically.
Epistemic majoritarianism in practice assumes that no one else is an epistemic majoritarian. Your first guess should be that nearly everyone else is iff you are, in which you should expect information cascades due to the occasional overconfident person. If other people are not majoritarians because they’re too stupid to notice the considerations for it, then it seems a bit silly to defer to them. On the other hand, if they’re not majoritarians because they’re smarter than you are… well, you mention this, but this objection seems to me to be obviously fatal and the only thing left is to explain why the wisdom of the majority disagrees with the epistemically modest.
The vast majority of information available about other people’s opinions does not differentiate clearly between their impressions and their beliefs after adjusting for their knowledge about others’ beliefs.
People lie to maintain socially desirable opinions.
Control over others’ opinions is a valuable social commodity, and apparent expertise gives one some control.
In particular, the last two factors (different sorts of dishonesty) are much bigger deals if most uninformed people copy the opinions of apparently informed people instead of saying “I have no idea”.
Overall, I agree that when you have a verified-independent, verified-honest opinion from a peer, one should weight it equally to one’s own, and defer to one’s verified epistemic superiors—but this has little to do with real life, in which we rarely have that opportunity!