Successionism is not a good argument for building ASI. If your goal is to build a sentient ASI that’s maximally flourishing (which is a good thing to do if you ask me*), then you still need to solve the alignment problem. A misaligned ASI will not optimize for well-being, whether that’s human well-being or AI well-being. It will optimize for whatever arbitrary inscrutable goals it has.
*but like, surely there is a way to do this without also wiping out humanity?
I’m not sure this is as obvious as you’re making out. If you give a misaligned ASI a goal to maximize its own wellbeing it could still do that very well, it may also just do other things you didn’t intend for it to do (like wipe out humans).
A paperclip maximizer does its job really well after all, and we wouldn’t call it aligned.
Successionism is not a good argument for building ASI. If your goal is to build a sentient ASI that’s maximally flourishing (which is a good thing to do if you ask me*), then you still need to solve the alignment problem. A misaligned ASI will not optimize for well-being, whether that’s human well-being or AI well-being. It will optimize for whatever arbitrary inscrutable goals it has.
*but like, surely there is a way to do this without also wiping out humanity?
I’m not sure this is as obvious as you’re making out. If you give a misaligned ASI a goal to maximize its own wellbeing it could still do that very well, it may also just do other things you didn’t intend for it to do (like wipe out humans).
A paperclip maximizer does its job really well after all, and we wouldn’t call it aligned.