Interesting that you give significant weight to non-extinction existential catastrophes (such as the AI leaving us the Milky Way). By what mechanism would that happen? Naively, all or (especially) nothing seem much more likely. It doesn’t seem like we’d have much bargaining power with not perfectly-aligned ASI. If it’s something analogous to us preserving other species, then I’m not optimistic that we’d get anything close to a flourishing civilisation confined to one galaxy. A small population in a “zoo”; or grossly distorted “pet” versions of humans; or merely being kept, overwhelmingly inactive, in digital storage, seem more likely.
So I’m imagining, for instance, AGIs with some shards of caring about human ~autonomy, but also other (stronger) shards that are for caring about (say) paperclips (also this was just meant as an example). I was also thinking that this might be what “a small population in a ‘zoo’” would look like – the Milky Way is small compared to the reachable universe! (Though before writing out my response, I almost wrote it as “our solar system” instead of “the Milky Way,” so I was imagining a relatively expansive set within this category; I’m not sure if distorted “pet” versions of humans would qualify or not.)
Interesting that you give significant weight to non-extinction existential catastrophes (such as the AI leaving us the Milky Way). By what mechanism would that happen? Naively, all or (especially) nothing seem much more likely. It doesn’t seem like we’d have much bargaining power with not perfectly-aligned ASI. If it’s something analogous to us preserving other species, then I’m not optimistic that we’d get anything close to a flourishing civilisation confined to one galaxy. A small population in a “zoo”; or grossly distorted “pet” versions of humans; or merely being kept, overwhelmingly inactive, in digital storage, seem more likely.
So I’m imagining, for instance, AGIs with some shards of caring about human ~autonomy, but also other (stronger) shards that are for caring about (say) paperclips (also this was just meant as an example). I was also thinking that this might be what “a small population in a ‘zoo’” would look like – the Milky Way is small compared to the reachable universe! (Though before writing out my response, I almost wrote it as “our solar system” instead of “the Milky Way,” so I was imagining a relatively expansive set within this category; I’m not sure if distorted “pet” versions of humans would qualify or not.)
Why wouldn’t the stronger shards just overpower the weaker shards?