That sounds reasonable to me, and I’m also surprised I haven’t seen that argument elsewhere. The most plausible counterarguments off the top of my head are: 1) Maybe evolution just can’t produce beings with that strong of a proximal objective of life-maximization, so the emergence of values that aren’t proximally about life-maximization (as with humans) is convergent. 2) Singletons about non-life-maximizing values are also convergent, perhaps because intelligence produces optimization power so it’s easier for such values to gain sway even though they aren’t life-maximizing. 3) Even if your conclusion is correct, this might not speak in favor of human space colonization anyway for the reason Michael St. Jules mentions in another comment, that more suffering would result from fighting those aliens.
I completely agree with 3 and it’s indeed worth clarifying. Even ignoring this, the possibility of humans being more compassionate than pro-life grabby aliens might actually be an argument against human-driven space colonization, since compassion—especially when combined with scope sensitivity—might increase agential s-risks related to potential catastrophic cooperation failure between AIs (see e.g., Baumann and Harris 2021, 46:24), which are the most worrying s-risks according to Jesse Clifton’s preface of CLR’s agenda. A space filled with life-maximizing aliens who don’t give a crap about welfare might be better than one filled with compassionate humans who create AGIs that might do the exact opposite of what they want (because of escalating conflicts and stuff). Obviously, uncertainty stays huge here.
Besides, 1 and 2 seem to be good counter-considerations, thanks! :)
I’m not sure I get why “Singletons about non-life-maximizing values are also convergent”, though. Do you—or anyone else reading this—can point at any reference that would help me understand this?
I’m not sure I get why “Singletons about non-life-maximizing values are also convergent”, though.
Sorry, I wrote that point lazily because that whole list was supposed to be rather speculative. It should be “Singletons about non-life-maximizing values could also be convergent.” I think that if some technologically advanced species doesn’t go extinct, the same sorts of forces that allow some human institutions to persist for millennia (religions are the best example, I guess) combined with goal-preserving AIs would make the emergence of a singleton fairly likely—not very confident in this, though, and I think #2 is the weakest argument. Bostrom’s “The Future of Human Evolution” touches on similar points.
That sounds reasonable to me, and I’m also surprised I haven’t seen that argument elsewhere. The most plausible counterarguments off the top of my head are: 1) Maybe evolution just can’t produce beings with that strong of a proximal objective of life-maximization, so the emergence of values that aren’t proximally about life-maximization (as with humans) is convergent. 2) Singletons about non-life-maximizing values are also convergent, perhaps because intelligence produces optimization power so it’s easier for such values to gain sway even though they aren’t life-maximizing. 3) Even if your conclusion is correct, this might not speak in favor of human space colonization anyway for the reason Michael St. Jules mentions in another comment, that more suffering would result from fighting those aliens.
I completely agree with 3 and it’s indeed worth clarifying. Even ignoring this, the possibility of humans being more compassionate than pro-life grabby aliens might actually be an argument against human-driven space colonization, since compassion—especially when combined with scope sensitivity—might increase agential s-risks related to potential catastrophic cooperation failure between AIs (see e.g., Baumann and Harris 2021, 46:24), which are the most worrying s-risks according to Jesse Clifton’s preface of CLR’s agenda. A space filled with life-maximizing aliens who don’t give a crap about welfare might be better than one filled with compassionate humans who create AGIs that might do the exact opposite of what they want (because of escalating conflicts and stuff). Obviously, uncertainty stays huge here.
Besides, 1 and 2 seem to be good counter-considerations, thanks! :)
I’m not sure I get why “Singletons about non-life-maximizing values are also convergent”, though. Do you—or anyone else reading this—can point at any reference that would help me understand this?
Sorry, I wrote that point lazily because that whole list was supposed to be rather speculative. It should be “Singletons about non-life-maximizing values could also be convergent.” I think that if some technologically advanced species doesn’t go extinct, the same sorts of forces that allow some human institutions to persist for millennia (religions are the best example, I guess) combined with goal-preserving AIs would make the emergence of a singleton fairly likely—not very confident in this, though, and I think #2 is the weakest argument. Bostrom’s “The Future of Human Evolution” touches on similar points.