those doing so should caveat that they are designed to mitigate the possibility (and not certainty) of catastrophic outcomes. This should be obvious, but given that people will be waiting in the wings to weaponise anything that could be called a regulatory overreaction, I think it’s worth doing.
I think to a lot of people, it matters just how much of a possibility there is. From what I’ve seen, many people are just (irrationally imo!) willing to bite the bullet on yolo-ing ASI if there is “only a 10%” chance of extinction. For this reason I counter with my actual assessment: doom is the default outcome of AGI/ASI (~90% likely). Only very few people are willing to bite that bullet! (And much more common is for people to fall back on dismissing the risk as “low”—e.g. experts saying “only” 1-25%).
Of course, if we do somehow survive all this, people will accuse me and others like me of crying wolf. But 1⁄10 outcomes aren’t that uncommon! I’m willing to take the reputation hit though, whether justified or not.
I think in general a big problem with AI x-risk discourse is that there are a lot of innumerate people around, who just don’t understand what probability means (or at least act like they don’t, and count everything as a confident statement even if appropriately hedged).
I think this might not be irrationality, but a genuine difference in values.
In particular, I think something like a discount rate disagreement is at the core of a lot of disagreements on AI safety, and to be blunt, you shouldn’t expect convergence unless you successfully persuade them of this.
I don’t think it’s discount rate (esp given short timelines); I think it’s more that people haven’t really thought about why their p(doom|ASI) is low. But people seem remarkably resistant to actually tackle the cruxes of the object level arguments, or fully extrapolate the implications of what they do agree on. When they do, they invariably come up short.
I think to a lot of people, it matters just how much of a possibility there is. From what I’ve seen, many people are just (irrationally imo!) willing to bite the bullet on yolo-ing ASI if there is “only a 10%” chance of extinction. For this reason I counter with my actual assessment: doom is the default outcome of AGI/ASI (~90% likely). Only very few people are willing to bite that bullet! (And much more common is for people to fall back on dismissing the risk as “low”—e.g. experts saying “only” 1-25%).
Of course, if we do somehow survive all this, people will accuse me and others like me of crying wolf. But 1⁄10 outcomes aren’t that uncommon! I’m willing to take the reputation hit though, whether justified or not.
I think in general a big problem with AI x-risk discourse is that there are a lot of innumerate people around, who just don’t understand what probability means (or at least act like they don’t, and count everything as a confident statement even if appropriately hedged).
I think this might not be irrationality, but a genuine difference in values.
In particular, I think something like a discount rate disagreement is at the core of a lot of disagreements on AI safety, and to be blunt, you shouldn’t expect convergence unless you successfully persuade them of this.
I don’t think it’s discount rate (esp given short timelines); I think it’s more that people haven’t really thought about why their p(doom|ASI) is low. But people seem remarkably resistant to actually tackle the cruxes of the object level arguments, or fully extrapolate the implications of what they do agree on. When they do, they invariably come up short.