I do not agree that it is absolutely clear that the default goal of an AGI is for it to kill literally everyone, as the OP asserts.
The OP says
goals that entailkilling literally everyone (which is the default)
[my emphasis in bold]. This is a key distinction. No one is saying that the default goal will be killing humans; the whole issue is one of collateral damage—it will end up with (to us) arbitrary goals that result in convergent intstrumental goals that lead to us all dead as collateral damage (e.g. turning the planet into “computronium”, or dismantling the Sun for energy).
Sure, I understand that it’s a supposed default instrumental goal and not a terminal goal. Sorry that my wording didn’t make that distinction clear. I’ve now edited it to do so, but I think my overall points stand.
It’s not even (necessarily) a default instrumental goal. It’s collateral damage as the result of other instrumental goals. It may just go straight for dismantling the Sun, knowing that we won’t be able to stop it. Or straight for ripping apart the planet with nanobots (no need for a poison everyone simultaneously step).
The OP says
[my emphasis in bold]. This is a key distinction. No one is saying that the default goal will be killing humans; the whole issue is one of collateral damage—it will end up with (to us) arbitrary goals that result in convergent intstrumental goals that lead to us all dead as collateral damage (e.g. turning the planet into “computronium”, or dismantling the Sun for energy).
Sure, I understand that it’s a supposed default instrumental goal and not a terminal goal. Sorry that my wording didn’t make that distinction clear. I’ve now edited it to do so, but I think my overall points stand.
It’s not even (necessarily) a default instrumental goal. It’s collateral damage as the result of other instrumental goals. It may just go straight for dismantling the Sun, knowing that we won’t be able to stop it. Or straight for ripping apart the planet with nanobots (no need for a poison everyone simultaneously step).
Fair enough, I edited it again. I still think the larger points stand unchanged.