I think the impulse to call AGI safety a Pascal’s Mugging does not stem from extremely low probabilities. In fact, I don’t think extremely low probabilities are necessary or sufficient for a Pascal’s Mugging.
Instead, I think Pascal’s Mugging is about epistemic helplessness in evaluating reasonably low priors. Even if I have no hope of evaluating the mugger’s claim, at least until it’s too late, I’m mathematically prohibited from assigning his promises a probability of zero. This bug lets the mugger increase the size of his promises or threats until I give in.
AGI safety in particular suffers a similar problem. How is the layperson to evaluate the arguments of AGI safety researchers?
There’s no consensus in the field of AI that AGI poses a real risk.
Laypeople can’t understand the math.
AGI safety researchers tend to “keep the problems” while “rejecting the solutions” for analogous problems in the human world.
What evidence exists resembles the ordinary weirdness of buggy software, and has typically been correctable by a patch.
Even if they design and implement an AGI safety program to avert FOOM, humanity will never get direct evidence that it’s working, because it will prevent them from ever experiencing the problem in the first place.
I personally think AGI safety is a realistic concern, and I support the research program continuing. But I also think it does intrinsically suffer from being extremely hard for laypeople to evaluate. This, to me, makes it fit what I think is people’s true objection when they complain about Pascal’s Mugging. They have to start with their prior on whether or not AGI safety researchers are bullshitting them, frankly, and “inevaluable” claims are a classic bullshitter’s tool. I guess AGI safety researchers are Pascalian saints?
There’s no consensus in the field of AI that AGI poses a real risk.
I’m not sure what the threshold is for consensus but a new survey of ML researchers finds:
“Support for AI safety research is up: 69% of respondents believe society should prioritize AI safety research “more” or “much more” than it is currently prioritized, up from 49% in 2016. …
The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. … Many respondents put the chance substantially higher: 48% of respondents gave at least 10% chance of an extremely bad outcome. Though another 25% put it at 0%.”
I’m a little skeptical about this survey due to its 17% response rate. I also worry about conflict of interest. AI Impacts is lead by people associated with the rationalist community, and the rationalist community has its inception in trying to figure out ways to convince people of the threat of AGI.
However, I think it’s great that these surveys are being created and support further efforts to make the state of expert opinion on this subject more legible.
I think the impulse to call AGI safety a Pascal’s Mugging does not stem from extremely low probabilities. In fact, I don’t think extremely low probabilities are necessary or sufficient for a Pascal’s Mugging.
Instead, I think Pascal’s Mugging is about epistemic helplessness in evaluating reasonably low priors. Even if I have no hope of evaluating the mugger’s claim, at least until it’s too late, I’m mathematically prohibited from assigning his promises a probability of zero. This bug lets the mugger increase the size of his promises or threats until I give in.
AGI safety in particular suffers a similar problem. How is the layperson to evaluate the arguments of AGI safety researchers?
There’s no consensus in the field of AI that AGI poses a real risk.
Laypeople can’t understand the math.
AGI safety researchers tend to “keep the problems” while “rejecting the solutions” for analogous problems in the human world.
What evidence exists resembles the ordinary weirdness of buggy software, and has typically been correctable by a patch.
Even if they design and implement an AGI safety program to avert FOOM, humanity will never get direct evidence that it’s working, because it will prevent them from ever experiencing the problem in the first place.
I personally think AGI safety is a realistic concern, and I support the research program continuing. But I also think it does intrinsically suffer from being extremely hard for laypeople to evaluate. This, to me, makes it fit what I think is people’s true objection when they complain about Pascal’s Mugging. They have to start with their prior on whether or not AGI safety researchers are bullshitting them, frankly, and “inevaluable” claims are a classic bullshitter’s tool. I guess AGI safety researchers are Pascalian saints?
I’m not sure what the threshold is for consensus but a new survey of ML researchers finds:
“Support for AI safety research is up: 69% of respondents believe society should prioritize AI safety research “more” or “much more” than it is currently prioritized, up from 49% in 2016. …
The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. … Many respondents put the chance substantially higher: 48% of respondents gave at least 10% chance of an extremely bad outcome. Though another 25% put it at 0%.”
Thank you for bringing the data!
I’m a little skeptical about this survey due to its 17% response rate. I also worry about conflict of interest. AI Impacts is lead by people associated with the rationalist community, and the rationalist community has its inception in trying to figure out ways to convince people of the threat of AGI.
However, I think it’s great that these surveys are being created and support further efforts to make the state of expert opinion on this subject more legible.