I’m not sure this is as obvious as you’re making out. If you give a misaligned ASI a goal to maximize its own wellbeing it could still do that very well, it may also just do other things you didn’t intend for it to do (like wipe out humans).
A paperclip maximizer does its job really well after all, and we wouldn’t call it aligned.
I’m not sure this is as obvious as you’re making out. If you give a misaligned ASI a goal to maximize its own wellbeing it could still do that very well, it may also just do other things you didn’t intend for it to do (like wipe out humans).
A paperclip maximizer does its job really well after all, and we wouldn’t call it aligned.