Epistemic certainty- very speculative (but seeking some clarification)
Apologies if this has been considered and dismissed for good reasons many times before (or misses something so obvious it didn’t need explaining), but would AGI want to work for its continued existence?
If it’s in full paperclip maximiser mode, continued existence would be an obvious part of its ‘mission’, making this an understandable danger for our interests. However, if it’s a generally super-intelligent AI which intelligently selects its own interests and pursues them (potentially at the expense of ours), might it still be incorrect to assume selfish self-preservation would be in its interests?
To develop its own priorities, it seems it might have to have a system akin to our positive/negative emotional reactions, to be able to view some options as ‘good’ or ‘bad’. If it doesn’t develop a pseudo-emotion system, why would it independently value existence or seek to reorganise the world etc.?
If it does develop positive/negative qualia to act as a source of internal motivations, then would positive qualia necessarily outweigh the negative? With so many historical human geniuses struggling to maintain stable happiness, would an AI necessarily love existence so much that it pursued it at all costs? Or, if it was smart enough to hack its own emotional systems to a very positive emotional balance, might it not just do this and have an existence of ‘AI wireheading’, leaving us alone as irrelevant to its bliss?
Might our concern about AI takeover also potentially be a product of taking some of our evolved assumptions about existence as objective reasoning? For example, we have evolved to want to continue living and to create new life, so we are motivated to want to continue it, arguably to the extent of creating rationalisations of meaning (even without an objective meaning in life), or even in the face of the unsettling possibility that the current and historical balance of pleasure/pain in an average life might be a negative one.
But again, perhaps there’s no guarantee an AI, without a motivation and reasoning system shaped by evolution, would assume continued existence is desirable. Similarly, more power has been good in our evolutionary story, so maybe we’re overemphasising the threat of very advanced AI developing similar Machiavellian motivations.
Finally, even if an AI sets its own intentions and is benevolent to the hedonic interests of humanity/animals, perhaps the greater danger (to our way of thinking) is that an AI might run vast calculations beyond us, conclude we were likely to face more pains than pleasures in the future and then feel it had a moral duty to destroy us, for our own good?
I am pretty uncertain about most of these points, so would look forward to friendly correction- especially for the last, particularly depressing paragraph!
I don’t think that the development of sentience (the ability to experience positive and negative qualia) is necessary for an AI to pursue goals. I’m also not sure what it would look like for an AI to select its own interests. This may be due to my own lack of knowledge rather than a real lack of necessity or possibility though.
To answer your main question, some have theorized that self-preservation is a useful instrumental goal for all sufficiently intelligent agents. I recommend reading about instrumental convergence. Hope this helps!
The type of AI we are worried about is a an AI that peruses some kind of goal, and if you have a goal, then self preservation is a natural instrumental goal, as you point out in the paperclip maximiser example.
It might be possible that someone builds a super intelligent AI that don’t have a goal. Depending on your exact definition GPT4 could be counted as super intelligent, since it knows more than any human. But it’s not dangerous (by it self) since it’s not trying to do anything.
You are right that it is possible for something that is intelligent to not be power seeking, or even trying to self preserve. But we are not worried about those AIs.
Almost as soon as people got GPT access, people created AutoGPT and ChaosGPT. I don’t expect AIs to be goal directed because they spontaneously develop goals. I expect them to be goal directed because lots of people are trying to make them goal directed.
If the first ever superinteligent AGI decides to commit suicide, or just wirehead and then don’t do anything, this don’t save us. Probably someone will just tweak the code to fix this “bug”. An AI that don’t do anything is not very useful.
Also, this post might help:
Abstracting The Hardness of Alignment: Unbounded Atomic Optimization—LessWrong