I just want to remind that simply having some of the company budget allocated to pay people to spend their time thinking about and studying the potential impacts of the technology the company is developing, is in itself a good thing.
About the possibility that they would come to the conclusion the most rational thing would be to stop the development—I think the concern here is moot anyway because of the many player dilemma in the AI space (if one stops the others don’t have to), which is (I think) impossible to solve from inside any single company anyway.
Having some of the OpenAI company budget allocated to ‘AI safety’ could just be safety-washing—essentially, part of the OpenAI PR/marketing budget, rather than an actual safety effort.
If the safety people don’t actually have any power to slow or stop the rush towards AGI, I don’t see their utility.
As for the arms race dilemma, imagine if OpenAI announced one day ‘Oh no, we’ve made a horrible mistake; AGI would be way too risky; we are stopped all AGI-related research to protect humanity; he’s how to audit us to make sure we follow through on this promise’. I think the other major players in the AI space would be under considerable pressure from investors, employees, media, politicians, and the public to also stop their AGI research.
It’s just not that hard to coordinate on the ‘no-AGI-research’ focal point if enough serious people decide to do so, and there’s enough public support.
I just want to remind that simply having some of the company budget allocated to pay people to spend their time thinking about and studying the potential impacts of the technology the company is developing, is in itself a good thing.
About the possibility that they would come to the conclusion the most rational thing would be to stop the development—I think the concern here is moot anyway because of the many player dilemma in the AI space (if one stops the others don’t have to), which is (I think) impossible to solve from inside any single company anyway.
Having some of the OpenAI company budget allocated to ‘AI safety’ could just be safety-washing—essentially, part of the OpenAI PR/marketing budget, rather than an actual safety effort.
If the safety people don’t actually have any power to slow or stop the rush towards AGI, I don’t see their utility.
As for the arms race dilemma, imagine if OpenAI announced one day ‘Oh no, we’ve made a horrible mistake; AGI would be way too risky; we are stopped all AGI-related research to protect humanity; he’s how to audit us to make sure we follow through on this promise’. I think the other major players in the AI space would be under considerable pressure from investors, employees, media, politicians, and the public to also stop their AGI research.
It’s just not that hard to coordinate on the ‘no-AGI-research’ focal point if enough serious people decide to do so, and there’s enough public support.