As an aside, I’m not sure I agree that reducing safety-related externalities is largely an engineering problem, unless we include social engineering. Things like organizational culture, checklists, maintenance policies, risk assessments, etc., also seem quite important to me. (Or in the nuclear policy example even things like arms control, geopolitics, …)
I think this depends a bit what class of safety issues we’re thinking about. For example, a properly functioning nuke is meant to explode and kills loads of people. A lot of nuclear safety issues are then borderline misuse issues: people deciding to use them when really they shouldn’t, for instance due to misinterpretations of others’ actions. Many other technological ‘accident risks’ are less social, although never entirely non-social (e.g. even in the case of bridge safety, you still need to trust some organization to do maintenance/testing properly.)
That seems correct all else equal. However, it can be outweighed by actors seeking relative gains or other competitive pressures.
I definitely don’t want to deny that actors can sometimes have incentives to use badly world-worseningly unsafe technologies. But you do need the right balance of conditions to hold: individual units of the technology need to offer their users large enough benefits and small enough personal safety risks, need to create large enough external safety risks, and need to have safety levels that increase slowly enough over time.
Weapons of mass destruction are sort of special in this regard. They can in some cases have exceptionally high value to their users (deterring or preventing invasion), which makes them willing to bear unusually high risks. Since their purpose is to kill huge numbers of people on very short notice, there’s naturally a risk of them killing huge numbers of people (but under the wrong circumstances). This risk is also unusually hard to reduce over time, since it’s often more about people making bad decisions than it is about the technology ‘misbehaving’ per se; there is also a natural trade-off between increasing readiness and decreasing the risk of bad usage decisions being made. The risk also naturally falls very heavily on other actors (since the technology is meant to harm other actors).
I do generally find it easiest to understand how AI safety issues could make the world permanently worse when I imagine superweapon/WMD-like systems (of the sort that also seem to be imagined in work like “Racing to the Precicipe”). I think existential safety risks become a much harder sell, though, if we’re primarily imagining non-superweapon applications and distributed/gradual/what-failure-looks-like-style scenarios.
I also think it’s worth noting that, on an annual basis, even nukes don’t have a super high chance of producing global catastrophes through accidental use; if you have a high enough discount rate, and you buy the theory that they substantially reduce the risk of great power war, then it’s even possible (maybe not likely) that their existence is currently positive EV by non-longtermist lights.
But you do need the right balance of conditions to hold: individual units of the technology need to offer their users large enough benefits and small enough personal safety risks, need to create large enough external safety risks, and need to have safety levels that increase slowly enough over time.
Weapons of mass destruction are sort of special in this regard. [...]
[...] I think existential safety risks become a much harder sell, though, if we’re primarily imagining non-superweapon applications and distributed/gradual/what-failure-looks-like-style scenarios.
Yes, my guess is we broadly agree about all of this.
I also think it’s worth noting that, on an annual basis, even nukes don’t have a super high chance of producing global catastrophes through accidental use; if you have a high enough discount rate, and you buy the theory that they substantially reduce the risk of great power war, then it’s even possible (maybe not likely) that their existence is currently positive EV by non-longtermist lights.
This also sounds right to me. FWIW, it’s not even obvious to me if nukes are negative-EV by longtermist lights. Since nuclear winter seems unlikely to cause immediate extinction this depends on messy questions such as how the EV of trajectory changes from conventional great power war compares to the EV of trajectory changes from nuclear winter scenarios.
I think this depends a bit what class of safety issues we’re thinking about. [...] Many other technological ‘accident risks’ are less social, although never entirely non-social (e.g. even in the case of bridge safety, you still need to trust some organization to do maintenance/testing properly.)
I’m not sure I agree with this. While they haven’t been selected to be representative, the sense I got from the accident case studies I’ve read (e.g. Chernobyl, nuclear weapons accidents, and various cases from the books Flirting With Disaster and Warnings) is that the social component was quite substantial. It seems to me that usually either better engineering (though sometimes this wasn’t possible) or better social management of dealing with engineering limitations (usually possible) could have avoided these accidents. It makes a lot of sense to me that some people prefer to talk of “sociotechnical systems”.
I think this depends a bit what class of safety issues we’re thinking about. For example, a properly functioning nuke is meant to explode and kills loads of people. A lot of nuclear safety issues are then borderline misuse issues: people deciding to use them when really they shouldn’t, for instance due to misinterpretations of others’ actions. Many other technological ‘accident risks’ are less social, although never entirely non-social (e.g. even in the case of bridge safety, you still need to trust some organization to do maintenance/testing properly.)
I definitely don’t want to deny that actors can sometimes have incentives to use badly world-worseningly unsafe technologies. But you do need the right balance of conditions to hold: individual units of the technology need to offer their users large enough benefits and small enough personal safety risks, need to create large enough external safety risks, and need to have safety levels that increase slowly enough over time.
Weapons of mass destruction are sort of special in this regard. They can in some cases have exceptionally high value to their users (deterring or preventing invasion), which makes them willing to bear unusually high risks. Since their purpose is to kill huge numbers of people on very short notice, there’s naturally a risk of them killing huge numbers of people (but under the wrong circumstances). This risk is also unusually hard to reduce over time, since it’s often more about people making bad decisions than it is about the technology ‘misbehaving’ per se; there is also a natural trade-off between increasing readiness and decreasing the risk of bad usage decisions being made. The risk also naturally falls very heavily on other actors (since the technology is meant to harm other actors).
I do generally find it easiest to understand how AI safety issues could make the world permanently worse when I imagine superweapon/WMD-like systems (of the sort that also seem to be imagined in work like “Racing to the Precicipe”). I think existential safety risks become a much harder sell, though, if we’re primarily imagining non-superweapon applications and distributed/gradual/what-failure-looks-like-style scenarios.
I also think it’s worth noting that, on an annual basis, even nukes don’t have a super high chance of producing global catastrophes through accidental use; if you have a high enough discount rate, and you buy the theory that they substantially reduce the risk of great power war, then it’s even possible (maybe not likely) that their existence is currently positive EV by non-longtermist lights.
Yes, my guess is we broadly agree about all of this.
This also sounds right to me. FWIW, it’s not even obvious to me if nukes are negative-EV by longtermist lights. Since nuclear winter seems unlikely to cause immediate extinction this depends on messy questions such as how the EV of trajectory changes from conventional great power war compares to the EV of trajectory changes from nuclear winter scenarios.
I’m not sure I agree with this. While they haven’t been selected to be representative, the sense I got from the accident case studies I’ve read (e.g. Chernobyl, nuclear weapons accidents, and various cases from the books Flirting With Disaster and Warnings) is that the social component was quite substantial. It seems to me that usually either better engineering (though sometimes this wasn’t possible) or better social management of dealing with engineering limitations (usually possible) could have avoided these accidents. It makes a lot of sense to me that some people prefer to talk of “sociotechnical systems”.