If the AI didn’t face any competition and was a rational agent, it might indeed want to be extremely cautious about making changes to itself or building successors, for the reason you mention. However, if there’s competition among AIs, then just like in the case of a human AI arms race, there might be pressure to self-improve even at the risk of goal drift.
If the AI didn’t face any competition and was a rational agent, it might indeed want to be extremely cautious about making changes to itself or building successors, for the reason you mention. However, if there’s competition among AIs, then just like in the case of a human AI arms race, there might be pressure to self-improve even at the risk of goal drift.