cosmopolitanism: I have a roundup of links here. I think your concerns are best discussed in the Arbital article on generalized cosmopolitanism:
“We are cosmopolitans! We also grew up reading science fiction about aliens that turned out to have their own perspectives, and AIs willing to extend a hand in friendship but being mistreated by carbon chaunivists! We’d be fine with a weird and wonderful intergalactic civilization full of non-organic beings appreciating their own daily life in ways we wouldn’t understand. But paperclip maximizersdon’t do that! We predict that if you got to see the use a paperclip maximizer would make of the cosmic endowment, if you really understood what was going on inside that universe, you’d be as horrified as we are. You and I have a difference of empirical predictions about the consequences of running a paperclip maximizer, not a values difference about how far to widen the circle of concern.”
Re the fragility/complexity link:
I am not familiar with the posts you link to, but it looks like they focus on human values (emphasis mine):
My view after reading is that “human” is a shorthand that isn’t doubled down on throughout. Fun theory especially characterizes a sense of what’s at stake when we have values that can at least entertain the idea of pluralism (as opposed to values that don’t hesitate to create extreme lock-in scenarios), and “human value” is sort of a first term approximate proxy of a detailed understanding of that.
Most people would consider a universe filled with paperclips pretty bad, but I think it can plausibly be good as long as the superintelligent AI is having a good time turning everything into paperclips.
This is a niche and extreme flavor of utilitarianism and I wouldn’t expect it’s conclusions to be robust to moral uncertainty.
But it’s a nice question that identifies cruxes in metaethics.
Why would an unaligned AI have lower option value than an aligned one?
I think this is just a combination of taking lock-in actions that we can’t undo seriously along with forecasts about how aggressively a random draw from values can be expected to lock in.
I think this is just a combination of taking lock-in actions that we can’t undo seriously along with forecasts about how aggressively a random draw from values can be expected to lock in.
I have just remembered the post AGI and lock-in is relevant to this discussion.
cosmopolitanism: I have a roundup of links here. I think your concerns are best discussed in the Arbital article on generalized cosmopolitanism:
Re the fragility/complexity link:
My view after reading is that “human” is a shorthand that isn’t doubled down on throughout. Fun theory especially characterizes a sense of what’s at stake when we have values that can at least entertain the idea of pluralism (as opposed to values that don’t hesitate to create extreme lock-in scenarios), and “human value” is sort of a first term approximate proxy of a detailed understanding of that.
This is a niche and extreme flavor of utilitarianism and I wouldn’t expect it’s conclusions to be robust to moral uncertainty.
But it’s a nice question that identifies cruxes in metaethics.
I think this is just a combination of taking lock-in actions that we can’t undo seriously along with forecasts about how aggressively a random draw from values can be expected to lock in.
Thanks!
I have just remembered the post AGI and lock-in is relevant to this discussion.