Summary: We routinely act to prevent, mitigate, or insure against risks with P = ‘one-in-a-million’. Risks similarly or more probable than this should not prompt concerns about ‘pascal’s mugging’ etc.
Motivation
Reckless appeals to astronomical stakes often prompt worries about pascal’s mugging or similar. Sure, a 10^-20 chance of 10^40 has the same expected value as 10^20 with P = 1, but treating them as equivalent when making decisions is counter-intuitive. Thus one can (perhaps should) be wary of lines which amount to “The scale of the longterm future is so vast we can basically ignore the probability—so long as it is greater than 10^-lots—to see x-risk reduction is the greatest priority.”
Most folks who work on (e.g.) AI safety do not think the risks they are trying to reduce are extremely (nor astronomically) remote. Pascalian worries are unlikely to apply to attempts to reduce a baseline risk of 1⁄10 or 1⁄100. They also are unlikely to apply if the risk is a few orders of magnitude less (or a few orders of magnitude less tractable to reduce) than some suppose.
Despite this, I sometimes hear remarks along the lines of “I only think this risk is 1/1000 (or 1⁄10 000, or even ‘a bit less than 1%’) so me working on this is me falling for Pascal’s wager.” This is mistaken: an orders-of-magnitude lower risk (or likelihood of success) makes, all else equal, something orders of magnitude less promising, but it does not mean it can be dismissed out-of-hand.
Exactly where the boundary should be drawn for pascalian probabilities is up for grabs (10^-10 seems reasonably pascalian, 10^-2 definitely not). I suggest a very conservative threshold at ‘1 in a million’: human activity in general (and our own in particular) is routinely addressed to reduce, mitigate, or insure against risks between 1/1000 and 1⁄1 000 000, and we typically consider these activities ‘reasonable prudence’ rather than ‘getting mugged by mere possibility’.
Illustrations
Among many other things:
Aviation and other ‘safety critical’ activities
One thing which can go wrong when flying an airliner is an engine stops working. Besides all the engineering and maintenance to make engines reliable, airlines take many measures to mitigate this risk:
Airliners have more than one engine, and are designed and operated so that they are able to fly and land at a nearby airport ‘on the other engine’ should one fail at any point in the flight.
Pilots practice in initial and refresher simulator training how to respond to emergencies like an engine failure (apparently engine failure just after take-off is the riskiest)
Pilots also make a plan before each flight what to do ‘just in case’ an engine fails whilst they are taking off.
This risk is very remote: the rate of (jet) engine failure is something like 1 per 400 000 flight hours. So for a typical flight, maybe the risk is something like 10^-4 to 10^-5. The risk of an engine failure resulting in a fatal crash is even more remote: the most recent examples I could find happened in the 90s. Given the millions of airline flights a year, ‘1 in a million flights’ is comfortable upper bound.
Similarly, the individual risk-reduction measures mentioned above are unlikely to be averting that many micro(/nano?) crashes. A pilot who (somehow) manages to skive off their recurrency training or skip the pre-flight briefing may still muddle through if the risk they failed to prepare for realises. I suspect most consider the diligent practice by pilots for events they are are unlikely to ever see in their career admirable rather than getting suckered by Pascal’s mugging.
Aviation is the poster child of safety engineering, but it is not unique. Civil engineering disasters (think building collapses) share similar properties to aviation ones: the absolute rate is very low (plausibly of the order of ‘1 per million structure years’ or lower); this low risk did not happen by magic, but rather through concerted efforts across design, manufacture, maintenance, and operation; by design, a failure in one element should not lead to disaster (cf. the famous ‘swiss cheese model’); it also means the marginal effect of each contribution to avoiding disaster is very small. A sloppy inspection or shoddy design does not guarantee disaster, yet thorough engineers and inspectors are lauded rather than ridiculed. The same points can be made can be made across most safety and many security fields.
Voting (and similar collective actions)
There’s a well worn discussion about the rationality of voting, given the remote likelihood a marginal vote is decisive. A line popular in EA-land is considering voting as rational charity: although the expected value to you if your vote is decisive to get the better party in government might only be cents, if one is voting for party one believes best improves overall welfare across the electorate, this expected value (roughly) gets multiplied by population, and so climbs into the thousands of dollars.
Yet this would not be rational if in fact individuals should dismiss motivating reasons based on 1⁄1 000 000 probabilities as ‘Pascal’s mugging’. Some initial work making this case suggested the key probability of being the decisive vote was k/ n(voters in election), with k between 1 and 10 depending how close the election was expected to be. So voting in larger elections (>10M voters) could not be justified by this rationale.
The same style of problem applies to other collective activity: it is similarly unlikely you will make the decisive signature for a petition to be heeded, the decisive attendance for a protest to gain traction, nor the decisive vegan which removes a granule of animal product production. It is doubtful they are irrational in virtue of Pascal’s mugging.
Asteroid defence
Perhaps the OG x-risk reducers are those who work on planetary defence: trying to look for asteroids which could hit Earth; and planning how, if one was headed our way, how a collision could be avoided. These efforts have been ongoing for decades, and have included steadily improvements in observation and detection, and recently included tests of particular collision avoidance methods.
The track record of asteroid impacts (especially the most devastating) indicate this risk is minute, and still lower when conditioned on diligent efforts to track near-earth objects and finding none of the big ones are on a collision course. Once again, the rate of a ‘planet killer’ collision is somewhere around 10^-6 to 10^-7 per century. Also once again, I think most are glad this risk is being addressed rather than ignored.
Conclusion
Three minor points, then one major one.
One is that Pascal’s mugging looks wrong at 1 in a million does not make the worry generally misguided; even if ‘standard rules apply’ at 10^-6, maybe something different is called for at (say) 10^-60. I only argue pascalian worries are inapposite at or above the 10^-6 level.
Two is you can play with interval or aggregation to multiply up or down the risk. Even if ‘not looking both ways before you cross the road’ once incurs only a minute risk of serious injury (as a conservative BOTEC: ~2000 injuries/year in the UK, ~70M population, cross a road 1/day on average, RR = 100 for not looking both ways ~~ 8/ million per event?) following this as a policy across one’s lifetime increases the risk a few orders of magnitude—and everyone following this policy would significantly increase the number of pedestrians struck and killed by cars each year.
This highlights a common challenge for naive (and some not so naive) efforts to ‘discount low probabilities’: we can slice up some composite risk reduction measures such that individually every one should be rejected (“the risk of getting hit by a car but-for looking both ways as you cross the road is minuscule enough to the subject of Pascal’s mugging”), yet we endorse the efforts as a whole given their aggregate impact. Perhaps the main upshot was 1⁄1 000 000 risk as the ‘don’t worry about Pascal’s mugging’ was too conservative—maybe more ‘at least 1⁄1 000 000 on at least one reasonable way to slice it’
Three is we might dispute how reasonable the above illustrations are. Maybe we err risk-intolerant or over-insure ourselves; maybe the cost savings of slightly laxer airline safety would be worth an aviation disaster or two a year; maybe the optimal level of political engagement should be lower than the status quo; maybe asteroid impacts are so remote its practitioners should spend their laudable endeavour elsewhere. Yet even if they are unreasonable, they are unreasonable because the numbers are not adding up (/multiplying together) per orthodox expected utility theory, and not because they should have been ruled out in principle by a ‘pascal’s mugging’ style objection.
Finally, returning to x-risk. The examples above were also chosen to illustrate a different ‘vibe’ that could apply to x-risk besides ‘impeding disaster and heroic drama’. Safety engineering is non-heroic by design: a saviour snatching affairs from the jaws of disaster indicates an intolerable single point of failure. Rather, success is a team effort which is resilient to an individual’s mistake, and their excellence only slightly notches down the risk even further. Yet this work remains both laudable and worthwhile: a career spent investigating not-so-near misses to tease out human factors to make them even more distant has much to celebrate, even if not much of a highlight reel.
‘Existential safety’ could be something similar. Risks of AI, nukes, pandemics etc. should be at least as remote of those of a building collapsing, a plane crashing, or a nuclear power plant melting down. Hopefully these risks are similarly remote, and hopefully one’s contribution amounts to a slight incremental reduction. Only the vainglorious would wish otherwise.
Not all hopes are expectations, and facts don’t care about how well we vibe with them. Most people working on x-risk (including myself) think the risks they work on are much more likely than an airliner crashing. Yet although the scale of the future may be inadequate stakes for pascalian gambles, its enormity is sufficient to justify most non-pascalian values. Asteroid impacts, although extremely remote, still warrant some attention. If it transpired all other risks were similarly unlikely, I’d still work on mine.
Whether x-risk reduction is the best thing one can do if the risks are more ‘bridge collapse’ than ‘Russian roulette’ turns on questions like how should one price the value of the future, and how it stacks up versus other contributions to other causes. If you like multiplying things at approximate face value as much as I do, the answer plausibly still remains ‘yes’. But if ‘no’, pascal’s mugging should not be the reason why.
Most* small probabilities aren’t pascalian
Summary: We routinely act to prevent, mitigate, or insure against risks with P = ‘one-in-a-million’. Risks similarly or more probable than this should not prompt concerns about ‘pascal’s mugging’ etc.
Motivation
Reckless appeals to astronomical stakes often prompt worries about pascal’s mugging or similar. Sure, a 10^-20 chance of 10^40 has the same expected value as 10^20 with P = 1, but treating them as equivalent when making decisions is counter-intuitive. Thus one can (perhaps should) be wary of lines which amount to “The scale of the longterm future is so vast we can basically ignore the probability—so long as it is greater than 10^-lots—to see x-risk reduction is the greatest priority.”
Most folks who work on (e.g.) AI safety do not think the risks they are trying to reduce are extremely (nor astronomically) remote. Pascalian worries are unlikely to apply to attempts to reduce a baseline risk of 1⁄10 or 1⁄100. They also are unlikely to apply if the risk is a few orders of magnitude less (or a few orders of magnitude less tractable to reduce) than some suppose.
Despite this, I sometimes hear remarks along the lines of “I only think this risk is 1/1000 (or 1⁄10 000, or even ‘a bit less than 1%’) so me working on this is me falling for Pascal’s wager.” This is mistaken: an orders-of-magnitude lower risk (or likelihood of success) makes, all else equal, something orders of magnitude less promising, but it does not mean it can be dismissed out-of-hand.
Exactly where the boundary should be drawn for pascalian probabilities is up for grabs (10^-10 seems reasonably pascalian, 10^-2 definitely not). I suggest a very conservative threshold at ‘1 in a million’: human activity in general (and our own in particular) is routinely addressed to reduce, mitigate, or insure against risks between 1/1000 and 1⁄1 000 000, and we typically consider these activities ‘reasonable prudence’ rather than ‘getting mugged by mere possibility’.
Illustrations
Among many other things:
Aviation and other ‘safety critical’ activities
One thing which can go wrong when flying an airliner is an engine stops working. Besides all the engineering and maintenance to make engines reliable, airlines take many measures to mitigate this risk:
Airliners have more than one engine, and are designed and operated so that they are able to fly and land at a nearby airport ‘on the other engine’ should one fail at any point in the flight.
Pilots practice in initial and refresher simulator training how to respond to emergencies like an engine failure (apparently engine failure just after take-off is the riskiest)
Pilots also make a plan before each flight what to do ‘just in case’ an engine fails whilst they are taking off.
This risk is very remote: the rate of (jet) engine failure is something like 1 per 400 000 flight hours. So for a typical flight, maybe the risk is something like 10^-4 to 10^-5. The risk of an engine failure resulting in a fatal crash is even more remote: the most recent examples I could find happened in the 90s. Given the millions of airline flights a year, ‘1 in a million flights’ is comfortable upper bound.
Similarly, the individual risk-reduction measures mentioned above are unlikely to be averting that many micro(/nano?) crashes. A pilot who (somehow) manages to skive off their recurrency training or skip the pre-flight briefing may still muddle through if the risk they failed to prepare for realises. I suspect most consider the diligent practice by pilots for events they are are unlikely to ever see in their career admirable rather than getting suckered by Pascal’s mugging.
Aviation is the poster child of safety engineering, but it is not unique. Civil engineering disasters (think building collapses) share similar properties to aviation ones: the absolute rate is very low (plausibly of the order of ‘1 per million structure years’ or lower); this low risk did not happen by magic, but rather through concerted efforts across design, manufacture, maintenance, and operation; by design, a failure in one element should not lead to disaster (cf. the famous ‘swiss cheese model’); it also means the marginal effect of each contribution to avoiding disaster is very small. A sloppy inspection or shoddy design does not guarantee disaster, yet thorough engineers and inspectors are lauded rather than ridiculed. The same points can be made can be made across most safety and many security fields.
Voting (and similar collective actions)
There’s a well worn discussion about the rationality of voting, given the remote likelihood a marginal vote is decisive. A line popular in EA-land is considering voting as rational charity: although the expected value to you if your vote is decisive to get the better party in government might only be cents, if one is voting for party one believes best improves overall welfare across the electorate, this expected value (roughly) gets multiplied by population, and so climbs into the thousands of dollars.
Yet this would not be rational if in fact individuals should dismiss motivating reasons based on 1⁄1 000 000 probabilities as ‘Pascal’s mugging’. Some initial work making this case suggested the key probability of being the decisive vote was k/ n(voters in election), with k between 1 and 10 depending how close the election was expected to be. So voting in larger elections (>10M voters) could not be justified by this rationale.
The same style of problem applies to other collective activity: it is similarly unlikely you will make the decisive signature for a petition to be heeded, the decisive attendance for a protest to gain traction, nor the decisive vegan which removes a granule of animal product production. It is doubtful they are irrational in virtue of Pascal’s mugging.
Asteroid defence
Perhaps the OG x-risk reducers are those who work on planetary defence: trying to look for asteroids which could hit Earth; and planning how, if one was headed our way, how a collision could be avoided. These efforts have been ongoing for decades, and have included steadily improvements in observation and detection, and recently included tests of particular collision avoidance methods.
The track record of asteroid impacts (especially the most devastating) indicate this risk is minute, and still lower when conditioned on diligent efforts to track near-earth objects and finding none of the big ones are on a collision course. Once again, the rate of a ‘planet killer’ collision is somewhere around 10^-6 to 10^-7 per century. Also once again, I think most are glad this risk is being addressed rather than ignored.
Conclusion
Three minor points, then one major one.
One is that Pascal’s mugging looks wrong at 1 in a million does not make the worry generally misguided; even if ‘standard rules apply’ at 10^-6, maybe something different is called for at (say) 10^-60. I only argue pascalian worries are inapposite at or above the 10^-6 level.
Two is you can play with interval or aggregation to multiply up or down the risk. Even if ‘not looking both ways before you cross the road’ once incurs only a minute risk of serious injury (as a conservative BOTEC: ~2000 injuries/year in the UK, ~70M population, cross a road 1/day on average, RR = 100 for not looking both ways ~~ 8/ million per event?) following this as a policy across one’s lifetime increases the risk a few orders of magnitude—and everyone following this policy would significantly increase the number of pedestrians struck and killed by cars each year.
This highlights a common challenge for naive (and some not so naive) efforts to ‘discount low probabilities’: we can slice up some composite risk reduction measures such that individually every one should be rejected (“the risk of getting hit by a car but-for looking both ways as you cross the road is minuscule enough to the subject of Pascal’s mugging”), yet we endorse the efforts as a whole given their aggregate impact. Perhaps the main upshot was 1⁄1 000 000 risk as the ‘don’t worry about Pascal’s mugging’ was too conservative—maybe more ‘at least 1⁄1 000 000 on at least one reasonable way to slice it’
Three is we might dispute how reasonable the above illustrations are. Maybe we err risk-intolerant or over-insure ourselves; maybe the cost savings of slightly laxer airline safety would be worth an aviation disaster or two a year; maybe the optimal level of political engagement should be lower than the status quo; maybe asteroid impacts are so remote its practitioners should spend their laudable endeavour elsewhere. Yet even if they are unreasonable, they are unreasonable because the numbers are not adding up (/multiplying together) per orthodox expected utility theory, and not because they should have been ruled out in principle by a ‘pascal’s mugging’ style objection.
Finally, returning to x-risk. The examples above were also chosen to illustrate a different ‘vibe’ that could apply to x-risk besides ‘impeding disaster and heroic drama’. Safety engineering is non-heroic by design: a saviour snatching affairs from the jaws of disaster indicates an intolerable single point of failure. Rather, success is a team effort which is resilient to an individual’s mistake, and their excellence only slightly notches down the risk even further. Yet this work remains both laudable and worthwhile: a career spent investigating not-so-near misses to tease out human factors to make them even more distant has much to celebrate, even if not much of a highlight reel.
‘Existential safety’ could be something similar. Risks of AI, nukes, pandemics etc. should be at least as remote of those of a building collapsing, a plane crashing, or a nuclear power plant melting down. Hopefully these risks are similarly remote, and hopefully one’s contribution amounts to a slight incremental reduction. Only the vainglorious would wish otherwise.
Not all hopes are expectations, and facts don’t care about how well we vibe with them. Most people working on x-risk (including myself) think the risks they work on are much more likely than an airliner crashing. Yet although the scale of the future may be inadequate stakes for pascalian gambles, its enormity is sufficient to justify most non-pascalian values. Asteroid impacts, although extremely remote, still warrant some attention. If it transpired all other risks were similarly unlikely, I’d still work on mine.
Whether x-risk reduction is the best thing one can do if the risks are more ‘bridge collapse’ than ‘Russian roulette’ turns on questions like how should one price the value of the future, and how it stacks up versus other contributions to other causes. If you like multiplying things at approximate face value as much as I do, the answer plausibly still remains ‘yes’. But if ‘no’, pascal’s mugging should not be the reason why.