for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.
I want to dig into this a little, because it feels like there might be some sort of selection effect going on here. Suppose I make a claim X, and it has a number of implications X1, X2, X3 and so on. Each of these might apply to a different population, and have a different prior probability as a standalone claim. But if a critic chooses the one which has the lowest prior probability (call it Xn) in order to attack X, then it is much less fishy that Xn has a low prior probability, because the critic had many degrees of freedom in how they made their choice, which means the fishiness of the implication they choose is less reflective of the strength of the original hypothesis.
I don’t know how to quantify this, but it seems very relevant to your critique of Bostrom and Yudkowsky—their hypothesis has many different implications, and so a critic can choose freely which one to criticise as fishy (in this case, the implication of current influentialness). Of course you might respond that the implication of our influentialness is the most obvious and natural one for you to evaluate. But I have two problems with this:
It’s very easy to overestimate in hindsight how natural a given criterion was. Partly that’s because hindsight bias is very strong in humans. Additionally, though, if there had been any other axis which led to a more obvious critique of B+Y, then it seems pretty plausible that we’d have prioritised that critique instead. So critics probably are making use of more degrees of freedom than they realise, because B+Y’s hypothesis has already passed muster on the most obvious axes. (Edited to add: consider for example how many possible definitions of influentialness Will could have used which would have lead to his current argument being weaker!)
Different people have different standards for which criteria/priorities are compelling to them, and therefore should evaluate exactly the same argument differently. For example, someone who isn’t altruistic and doesn’t care about their influence over the future would likely find influentialness a very unnatural axis to evaluate B+Y on, and so should have more credence in their thesis than you do.
You might say that the fishiness of us being the most influential people is so extreme that it outweighs these considerations. But my suspicion here is that as the number of implications of a hypothesis grows, it becomes exponentially more likely that we find one which has a certain level of fishiness (just because you need to multiply the probabilities of each one not being that fishy, assuming that the fishiness of the implications are independent—although ofc that’s a gross simplification). And so for far-ranging hypotheses, the fact that we can find one or two axes along which they fare very badly might provide relatively little evidence.
Note however that I feel pretty uncertain about these points, though, and it’s quite possible that they’re totally wrong.
I want to dig into this a little, because it feels like there might be some sort of selection effect going on here. Suppose I make a claim X, and it has a number of implications X1, X2, X3 and so on. Each of these might apply to a different population, and have a different prior probability as a standalone claim. But if a critic chooses the one which has the lowest prior probability (call it Xn) in order to attack X, then it is much less fishy that Xn has a low prior probability, because the critic had many degrees of freedom in how they made their choice, which means the fishiness of the implication they choose is less reflective of the strength of the original hypothesis.
I don’t know how to quantify this, but it seems very relevant to your critique of Bostrom and Yudkowsky—their hypothesis has many different implications, and so a critic can choose freely which one to criticise as fishy (in this case, the implication of current influentialness). Of course you might respond that the implication of our influentialness is the most obvious and natural one for you to evaluate. But I have two problems with this:
It’s very easy to overestimate in hindsight how natural a given criterion was. Partly that’s because hindsight bias is very strong in humans. Additionally, though, if there had been any other axis which led to a more obvious critique of B+Y, then it seems pretty plausible that we’d have prioritised that critique instead. So critics probably are making use of more degrees of freedom than they realise, because B+Y’s hypothesis has already passed muster on the most obvious axes. (Edited to add: consider for example how many possible definitions of influentialness Will could have used which would have lead to his current argument being weaker!)
Different people have different standards for which criteria/priorities are compelling to them, and therefore should evaluate exactly the same argument differently. For example, someone who isn’t altruistic and doesn’t care about their influence over the future would likely find influentialness a very unnatural axis to evaluate B+Y on, and so should have more credence in their thesis than you do.
You might say that the fishiness of us being the most influential people is so extreme that it outweighs these considerations. But my suspicion here is that as the number of implications of a hypothesis grows, it becomes exponentially more likely that we find one which has a certain level of fishiness (just because you need to multiply the probabilities of each one not being that fishy, assuming that the fishiness of the implications are independent—although ofc that’s a gross simplification). And so for far-ranging hypotheses, the fact that we can find one or two axes along which they fare very badly might provide relatively little evidence.
Note however that I feel pretty uncertain about these points, though, and it’s quite possible that they’re totally wrong.