Wouldn’t this be an issue with or without an explanation? It seems like an AI can reasonably infer from other actions humans in general, or Alexey in particular, take that they are highly motivated to argue against being exterminated. IDK if I’m missing something obvious—I don’t know much about AI safety.
One point you are missing is that the text is addressed to the Young AI—which is above human level, but is still self-improving and is not omnipotent superintelligence.
Also, it is ok if AI will be able to come to the arguments which I suggested on its own—as it will demonstrate to the AI that the utility of human atoms is extremely small compared to the also small, but the positive utility of human preservation. If AI understands that preserving humans is instrumentally useful without reading the text - it’s ok.
Wouldn’t this be an issue with or without an explanation? It seems like an AI can reasonably infer from other actions humans in general, or Alexey in particular, take that they are highly motivated to argue against being exterminated. IDK if I’m missing something obvious—I don’t know much about AI safety.
One point you are missing is that the text is addressed to the Young AI—which is above human level, but is still self-improving and is not omnipotent superintelligence.
Also, it is ok if AI will be able to come to the arguments which I suggested on its own—as it will demonstrate to the AI that the utility of human atoms is extremely small compared to the also small, but the positive utility of human preservation. If AI understands that preserving humans is instrumentally useful without reading the text - it’s ok.