“So, for x-risk to be high, many people (e.g. lab employees, politicians, advisors) have to catastrophically fail at pursuing their own self-interest.”
I don’t think this obviously follows.
Firstly, because the effect of not doing unsafe AI things yourself is seldom that no one else does them, it’s more of a tragedy of the commons type situation right? Especially if there is one leading lab that is irrationally optimistic about safety, which doesn’t seem to require that low a view of human rationality in general.
Secondly, someone like Musk might have a value system where they care a lot about personally capturing the upside of getting to personally aligned superintelligence first, and then they might do dangerous things for the same reason that a risk neutral person will take a 90% chance of instant death and a 10% chance of living to be 10 million over the status quo.
“So, for x-risk to be high, many people (e.g. lab employees, politicians, advisors) have to catastrophically fail at pursuing their own self-interest.”
I don’t think this obviously follows.
Firstly, because the effect of not doing unsafe AI things yourself is seldom that no one else does them, it’s more of a tragedy of the commons type situation right? Especially if there is one leading lab that is irrationally optimistic about safety, which doesn’t seem to require that low a view of human rationality in general.
Secondly, someone like Musk might have a value system where they care a lot about personally capturing the upside of getting to personally aligned superintelligence first, and then they might do dangerous things for the same reason that a risk neutral person will take a 90% chance of instant death and a 10% chance of living to be 10 million over the status quo.