But you do need the right balance of conditions to hold: individual units of the technology need to offer their users large enough benefits and small enough personal safety risks, need to create large enough external safety risks, and need to have safety levels that increase slowly enough over time.
Weapons of mass destruction are sort of special in this regard. [...]
[...] I think existential safety risks become a much harder sell, though, if we’re primarily imagining non-superweapon applications and distributed/gradual/what-failure-looks-like-style scenarios.
Yes, my guess is we broadly agree about all of this.
I also think it’s worth noting that, on an annual basis, even nukes don’t have a super high chance of producing global catastrophes through accidental use; if you have a high enough discount rate, and you buy the theory that they substantially reduce the risk of great power war, then it’s even possible (maybe not likely) that their existence is currently positive EV by non-longtermist lights.
This also sounds right to me. FWIW, it’s not even obvious to me if nukes are negative-EV by longtermist lights. Since nuclear winter seems unlikely to cause immediate extinction this depends on messy questions such as how the EV of trajectory changes from conventional great power war compares to the EV of trajectory changes from nuclear winter scenarios.
Yes, my guess is we broadly agree about all of this.
This also sounds right to me. FWIW, it’s not even obvious to me if nukes are negative-EV by longtermist lights. Since nuclear winter seems unlikely to cause immediate extinction this depends on messy questions such as how the EV of trajectory changes from conventional great power war compares to the EV of trajectory changes from nuclear winter scenarios.