But also, we’re dealing with probabilities that are small but not infinitesimal. This saves us from objections like Pascal’s Mugging—a 1% chance of AI x-risk is not a Pascal’s Mugging.
It seems to me that the relevant probability is not the chance of AI x-risk, but the chance that your efforts could make a marginal difference. That probability is vastly lower, possibly bordering on mugging territory. For x-risk in particular, you make a difference only if your decision to work on x-risk makes a difference to whether or not the species survives. For some of us that may be plausible, but for most, it is very very unlikely.
Let me clarify that I’m not opposed to paying Pascal’s mugger. I think that is probably rational (though I count myself lucky to not be so rational).
But the idea here is that x-risk is all or nothing, which translates into each person having a very small chance of making a very big difference. Climate change can be mitigated, so everyone working on it can make a little difference.
Yep! I think this phenomena of ‘things that are technically all-or-nothing, but it’s most useful to think of them as a continuous thing’ is really common. Eg, if you want to reduce the amount of chickens killed for meat, it helps to stop buying chicken. This lowers demand, which will on average lower chickens killed. But the underlying thing is meat companies noticing and reducing production, which is pretty discrete and chunky and hard to predict well (though not literally all-or-nothing).
Basically any kind of campaign to change minds or achieve social change with some political goal also comes under this. I think AI Safety is about as much a Pascal’s Mugging as any of these other things
I think a huge number of people can contribute meaningfully to x-risk reduction, including pretty-much everyone reading this. You don’t need to be top 0.1% in research skill or intelligence—there are plenty of support roles that could be filled. Just think, by being a PA (or research assistant) to a top researcher or engineer, you might be able to boost their output by 10-30% (and by extension, their impact). I doubt that all the promising researchers have PAs (and RAs). Or consider raising awareness. Helping to recruit just one promising person to the cause is worthy of claiming significant impact (in expectation).
I’m not disagreeing with the possibility of a significant impact in expectation. Paying Pascals’ mugger is promising in expectation. The thought is that in order to make a marginal difference to x-risk, there needs to be some threshold for hours/money/etc under which our species will be wiped out and over which our species will survive, and your contributions have to push us over that threshold.
X-risk, at least where the survival of the species is concerned, is an all or nothing thing. (This is different than AI alignment, where your contributions might make things a little better or a little worse.)
I don’t think this is a Pascal’s Mugging situation; the probabilities are of the order of 10^-3 to 10^-8, which is far from infinitesimal. I also don’t think you can necessarily say that there is a threshold for hours/money. Ideas seem to be the bottleneck for AI x-risk at least, and these are not a linear function of time/money invested.
X-risk, at least where the survival of the species is concerned, is an all or nothing thing. (This is different than AI alignment, where your contributions might make things a little better or a little worse.)
It is all-or-nothing in the sense of survival or not, but given that we can never reduce the risk to zero, what is important is reducing the risk to an acceptable level (and this is not all-or-nothing, especially given that it’s hard to know exactly how things will pan out in advance, regardless of our level of effort and perceived progress). Also I don’t understand the comment on AI Alignment—I would say that is all or nothing, as limited global catastrophes seem less likely than extinction (although you can still make things better or worse in expectation); whereas bio is perhaps more likely to have interventions that make things a little better or worse in reality (given that limited global catastrophe is more likely than x-risk with bio).
the probabilities are of the order of 10^-3 to 10^-8, which is far from infinitesimal
I’m not sure what the probabilties are. You’re right that they are far from infinitesimal (just as every number is!): still, the y may be close enough to warrant discounting on whatever basis people discount Pascal’s mugger.
what is important is reducing the risk to an acceptable level
I think the risk is pretty irrelevant. If we lower the risk but still go extinct, we can pat ourselves on the back for fighting the good fight, but I don’t hink we should assign it much value. Our effect on the risk is instrumentally valulable for what it does for the species.
Also I don’t understand the comment on AI Alignment
The thought was that it is possible to make a difference between AI being pretty well and very well aligned, so we might be able to impact whether the future is good or great, and that is worth pursuing regardless of its relation to existential risk.
That would be bad, yes. But lowering the risk (significantly) means that it’s (significantly) less likely that we will go extinct! Say we lower the risk from 1⁄6 (Toby Ord’s all-things-considered estimate for x-risk over the next 100 years) to 1⁄60 this century. We’ve then bought ourselves a lot more time (in expectation) to lower the risk further. If we keep doing this at a high enough rate, we will very likely not go extinct for a very long time.
it is possible to make a difference between AI being pretty well and very well aligned, so we might be able to impact whether the future is good or great
I think “pretty well aligned” basically means we still all die; it has to be very well/perfectly aligned to be compatible with human existence, once you factor in an increase in power level of the AI to superintelligence; so it’s basically all or nothing (I’m with Yudkowsky/MIRI on this).
It seems to me that the relevant probability is not the chance of AI x-risk, but the chance that your efforts could make a marginal difference. That probability is vastly lower, possibly bordering on mugging territory. For x-risk in particular, you make a difference only if your decision to work on x-risk makes a difference to whether or not the species survives. For some of us that may be plausible, but for most, it is very very unlikely.
Hmm, what would this perspective say to people working on climate change?
Let me clarify that I’m not opposed to paying Pascal’s mugger. I think that is probably rational (though I count myself lucky to not be so rational).
But the idea here is that x-risk is all or nothing, which translates into each person having a very small chance of making a very big difference. Climate change can be mitigated, so everyone working on it can make a little difference.
You could replace working on climate change with ‘working on or voting in elections’, which are also all or nothing.
(Edit: For some previous arguments in this vein, see this post .)
Yep! I think this phenomena of ‘things that are technically all-or-nothing, but it’s most useful to think of them as a continuous thing’ is really common. Eg, if you want to reduce the amount of chickens killed for meat, it helps to stop buying chicken. This lowers demand, which will on average lower chickens killed. But the underlying thing is meat companies noticing and reducing production, which is pretty discrete and chunky and hard to predict well (though not literally all-or-nothing).
Basically any kind of campaign to change minds or achieve social change with some political goal also comes under this. I think AI Safety is about as much a Pascal’s Mugging as any of these other things
I think a huge number of people can contribute meaningfully to x-risk reduction, including pretty-much everyone reading this. You don’t need to be top 0.1% in research skill or intelligence—there are plenty of support roles that could be filled. Just think, by being a PA (or research assistant) to a top researcher or engineer, you might be able to boost their output by 10-30% (and by extension, their impact). I doubt that all the promising researchers have PAs (and RAs). Or consider raising awareness. Helping to recruit just one promising person to the cause is worthy of claiming significant impact (in expectation).
I’m not disagreeing with the possibility of a significant impact in expectation. Paying Pascals’ mugger is promising in expectation. The thought is that in order to make a marginal difference to x-risk, there needs to be some threshold for hours/money/etc under which our species will be wiped out and over which our species will survive, and your contributions have to push us over that threshold.
X-risk, at least where the survival of the species is concerned, is an all or nothing thing. (This is different than AI alignment, where your contributions might make things a little better or a little worse.)
I don’t think this is a Pascal’s Mugging situation; the probabilities are of the order of 10^-3 to 10^-8, which is far from infinitesimal. I also don’t think you can necessarily say that there is a threshold for hours/money. Ideas seem to be the bottleneck for AI x-risk at least, and these are not a linear function of time/money invested.
It is all-or-nothing in the sense of survival or not, but given that we can never reduce the risk to zero, what is important is reducing the risk to an acceptable level (and this is not all-or-nothing, especially given that it’s hard to know exactly how things will pan out in advance, regardless of our level of effort and perceived progress). Also I don’t understand the comment on AI Alignment—I would say that is all or nothing, as limited global catastrophes seem less likely than extinction (although you can still make things better or worse in expectation); whereas bio is perhaps more likely to have interventions that make things a little better or worse in reality (given that limited global catastrophe is more likely than x-risk with bio).
I’m not sure what the probabilties are. You’re right that they are far from infinitesimal (just as every number is!): still, the y may be close enough to warrant discounting on whatever basis people discount Pascal’s mugger.
I think the risk is pretty irrelevant. If we lower the risk but still go extinct, we can pat ourselves on the back for fighting the good fight, but I don’t hink we should assign it much value. Our effect on the risk is instrumentally valulable for what it does for the species.
The thought was that it is possible to make a difference between AI being pretty well and very well aligned, so we might be able to impact whether the future is good or great, and that is worth pursuing regardless of its relation to existential risk.
That would be bad, yes. But lowering the risk (significantly) means that it’s (significantly) less likely that we will go extinct! Say we lower the risk from 1⁄6 (Toby Ord’s all-things-considered estimate for x-risk over the next 100 years) to 1⁄60 this century. We’ve then bought ourselves a lot more time (in expectation) to lower the risk further. If we keep doing this at a high enough rate, we will very likely not go extinct for a very long time.
I think “pretty well aligned” basically means we still all die; it has to be very well/perfectly aligned to be compatible with human existence, once you factor in an increase in power level of the AI to superintelligence; so it’s basically all or nothing (I’m with Yudkowsky/MIRI on this).