Really? If you’re a rationalist (in the broad Popperian sense and the internet-cult sense), and we share common knowledge of each other’s beliefs, then shouldn’t we be able to argue towards closer agreement? Not if our estimates were totally arbitrary — but clearly they’re not. Again, they’re just especially uncertain.
I think there is an important point here. One of the assumptions in Aumann’s theorem is that both people have the same prior, and I think this is rarely true in the real world.
I roughly think of Bayesian reasoning as starting with a prior, and then adjusting the prior based on observed evidence. If there’s a ton of evidence, and your prior isn’t dumb, the prior doesn’t really matter. But the more speculative the problem, and the less available evidence, the more the prior starts to matter. And your prior bakes in a lot of your assumptions about the world, and I think it’s tricky to resolve disagreements about what your prior should be. At least not in ways that approach being objective.
I think you can make progress on this. Eg, ‘how likely is it that AI could get way better, really fast?’ is a difficult question to answer, and could be baked into a prior either way. And things like AI Impact’s study of discontinuous progress in other technologies can be helpful for getting closer to consensus. But I think choosing a good prior is still a really hard and important problem, and near impossible to be objective about
I think there is an important point here. One of the assumptions in Aumann’s theorem is that both people have the same prior, and I think this is rarely true in the real world.
I roughly think of Bayesian reasoning as starting with a prior, and then adjusting the prior based on observed evidence. If there’s a ton of evidence, and your prior isn’t dumb, the prior doesn’t really matter. But the more speculative the problem, and the less available evidence, the more the prior starts to matter. And your prior bakes in a lot of your assumptions about the world, and I think it’s tricky to resolve disagreements about what your prior should be. At least not in ways that approach being objective.
I think you can make progress on this. Eg, ‘how likely is it that AI could get way better, really fast?’ is a difficult question to answer, and could be baked into a prior either way. And things like AI Impact’s study of discontinuous progress in other technologies can be helpful for getting closer to consensus. But I think choosing a good prior is still a really hard and important problem, and near impossible to be objective about