A property of making directional claims like this is that MacAskill always has 50% confidence in the claim I’m making, since I’m claiming that his best-guess estimate is too high/low.
This isn’t quite right. Conservation of expected evidence means that MacAskill’s current probabilities should match his expectation of the ideal reasoning process. But for probabilities close to 0, this would typically imply that he assigns higher probability to being too high than to being too low. For example: a 3% probability is compatible with 90% probability that the ideal reasoning process would assign probability ~0% and a 10% probability that it would assign 30%. (Related.)
This is especially relevant when the ideal reasoning process is something as competent as 100 people for 1000 years. Those people could make a lot of progress on the important questions (including e.g. themselves working on the relevant research agendas just to predict whether they’ll succeed), so it would be unsurprising for them to end up much closer to 0% or 100% than is justifiable today.
On the point about working on the relevant research agendas, I hadn’t thought about that and kind of want to disallow that from the definition. But I feel the line would then get fuzzy as to what things exactly count as object level work on research agendas.
Edit: After thinking more, I will edit the definition to clarify that the people doing the reasoning can only deliberate about current evidence rather than acquire new evidence. This might still be a bit vague but it seems better than not including.
Nitpicking:
This isn’t quite right. Conservation of expected evidence means that MacAskill’s current probabilities should match his expectation of the ideal reasoning process. But for probabilities close to 0, this would typically imply that he assigns higher probability to being too high than to being too low. For example: a 3% probability is compatible with 90% probability that the ideal reasoning process would assign probability ~0% and a 10% probability that it would assign 30%. (Related.)
This is especially relevant when the ideal reasoning process is something as competent as 100 people for 1000 years. Those people could make a lot of progress on the important questions (including e.g. themselves working on the relevant research agendas just to predict whether they’ll succeed), so it would be unsurprising for them to end up much closer to 0% or 100% than is justifiable today.
Great point, I’m a bit disappointed in myself for not catching this! I’ll strike this out of the post and link to your comment for explanation.
On the point about working on the relevant research agendas, I hadn’t thought about that and kind of want to disallow that from the definition. But I feel the line would then get fuzzy as to what things exactly count as object level work on research agendas.
Edit: After thinking more, I will edit the definition to clarify that the people doing the reasoning can only deliberate about current evidence rather than acquire new evidence. This might still be a bit vague but it seems better than not including.