I’m not disagreeing with the possibility of a significant impact in expectation. Paying Pascals’ mugger is promising in expectation. The thought is that in order to make a marginal difference to x-risk, there needs to be some threshold for hours/money/etc under which our species will be wiped out and over which our species will survive, and your contributions have to push us over that threshold.
X-risk, at least where the survival of the species is concerned, is an all or nothing thing. (This is different than AI alignment, where your contributions might make things a little better or a little worse.)
I don’t think this is a Pascal’s Mugging situation; the probabilities are of the order of 10^-3 to 10^-8, which is far from infinitesimal. I also don’t think you can necessarily say that there is a threshold for hours/money. Ideas seem to be the bottleneck for AI x-risk at least, and these are not a linear function of time/money invested.
X-risk, at least where the survival of the species is concerned, is an all or nothing thing. (This is different than AI alignment, where your contributions might make things a little better or a little worse.)
It is all-or-nothing in the sense of survival or not, but given that we can never reduce the risk to zero, what is important is reducing the risk to an acceptable level (and this is not all-or-nothing, especially given that it’s hard to know exactly how things will pan out in advance, regardless of our level of effort and perceived progress). Also I don’t understand the comment on AI Alignment—I would say that is all or nothing, as limited global catastrophes seem less likely than extinction (although you can still make things better or worse in expectation); whereas bio is perhaps more likely to have interventions that make things a little better or worse in reality (given that limited global catastrophe is more likely than x-risk with bio).
the probabilities are of the order of 10^-3 to 10^-8, which is far from infinitesimal
I’m not sure what the probabilties are. You’re right that they are far from infinitesimal (just as every number is!): still, the y may be close enough to warrant discounting on whatever basis people discount Pascal’s mugger.
what is important is reducing the risk to an acceptable level
I think the risk is pretty irrelevant. If we lower the risk but still go extinct, we can pat ourselves on the back for fighting the good fight, but I don’t hink we should assign it much value. Our effect on the risk is instrumentally valulable for what it does for the species.
Also I don’t understand the comment on AI Alignment
The thought was that it is possible to make a difference between AI being pretty well and very well aligned, so we might be able to impact whether the future is good or great, and that is worth pursuing regardless of its relation to existential risk.
That would be bad, yes. But lowering the risk (significantly) means that it’s (significantly) less likely that we will go extinct! Say we lower the risk from 1⁄6 (Toby Ord’s all-things-considered estimate for x-risk over the next 100 years) to 1⁄60 this century. We’ve then bought ourselves a lot more time (in expectation) to lower the risk further. If we keep doing this at a high enough rate, we will very likely not go extinct for a very long time.
it is possible to make a difference between AI being pretty well and very well aligned, so we might be able to impact whether the future is good or great
I think “pretty well aligned” basically means we still all die; it has to be very well/perfectly aligned to be compatible with human existence, once you factor in an increase in power level of the AI to superintelligence; so it’s basically all or nothing (I’m with Yudkowsky/MIRI on this).
I’m not disagreeing with the possibility of a significant impact in expectation. Paying Pascals’ mugger is promising in expectation. The thought is that in order to make a marginal difference to x-risk, there needs to be some threshold for hours/money/etc under which our species will be wiped out and over which our species will survive, and your contributions have to push us over that threshold.
X-risk, at least where the survival of the species is concerned, is an all or nothing thing. (This is different than AI alignment, where your contributions might make things a little better or a little worse.)
I don’t think this is a Pascal’s Mugging situation; the probabilities are of the order of 10^-3 to 10^-8, which is far from infinitesimal. I also don’t think you can necessarily say that there is a threshold for hours/money. Ideas seem to be the bottleneck for AI x-risk at least, and these are not a linear function of time/money invested.
It is all-or-nothing in the sense of survival or not, but given that we can never reduce the risk to zero, what is important is reducing the risk to an acceptable level (and this is not all-or-nothing, especially given that it’s hard to know exactly how things will pan out in advance, regardless of our level of effort and perceived progress). Also I don’t understand the comment on AI Alignment—I would say that is all or nothing, as limited global catastrophes seem less likely than extinction (although you can still make things better or worse in expectation); whereas bio is perhaps more likely to have interventions that make things a little better or worse in reality (given that limited global catastrophe is more likely than x-risk with bio).
I’m not sure what the probabilties are. You’re right that they are far from infinitesimal (just as every number is!): still, the y may be close enough to warrant discounting on whatever basis people discount Pascal’s mugger.
I think the risk is pretty irrelevant. If we lower the risk but still go extinct, we can pat ourselves on the back for fighting the good fight, but I don’t hink we should assign it much value. Our effect on the risk is instrumentally valulable for what it does for the species.
The thought was that it is possible to make a difference between AI being pretty well and very well aligned, so we might be able to impact whether the future is good or great, and that is worth pursuing regardless of its relation to existential risk.
That would be bad, yes. But lowering the risk (significantly) means that it’s (significantly) less likely that we will go extinct! Say we lower the risk from 1⁄6 (Toby Ord’s all-things-considered estimate for x-risk over the next 100 years) to 1⁄60 this century. We’ve then bought ourselves a lot more time (in expectation) to lower the risk further. If we keep doing this at a high enough rate, we will very likely not go extinct for a very long time.
I think “pretty well aligned” basically means we still all die; it has to be very well/perfectly aligned to be compatible with human existence, once you factor in an increase in power level of the AI to superintelligence; so it’s basically all or nothing (I’m with Yudkowsky/MIRI on this).