My own shortest answer [I’m not an expert] would be:
What would you do next with that oracle?
It’s going to have to be something useful enough to prevent another company from creating an AGI and destroying the world, while also not destroying the world yourself by accident.
My second separate answer would be “it would be pretty easy to turn that oracle AI into an AGI, so you better watch out from someone doing that, including someone stealing your code”
Hey! Consider posting to the All AGI safety questions welcome thread.
My own shortest answer [I’m not an expert] would be:
What would you do next with that oracle?
It’s going to have to be something useful enough to prevent another company from creating an AGI and destroying the world, while also not destroying the world yourself by accident.
See also bullet 5 here.
My second separate answer would be “it would be pretty easy to turn that oracle AI into an AGI, so you better watch out from someone doing that, including someone stealing your code”