You say “one would need to provide concrete evidence about what kinds of objectives advanced AIs are actually expected to develop”—but Eliezer has done that quite explicitly.
What concrete evidence has Eliezer provided about the objectives advanced AIs are actually expected to develop?
Are you willing to posit that advanced systems are coherent, with at least one non-satiable component? Because that’s pretty minimal as an assumption, but implies with probability one that they prefer paperclipping of some sort.
Are humans coherent with at least one non-satiable component? If so, then I don’t understand the distinction you’re making that would justify positing AI values to be worse than human values from a utilitarian perspective.
If not, then I’m additionally unclear on why you believe AIs will be unlike humans in this respect, to the extent that they would become “paperclippers.” That term itself seems ambiguous to me (do you mean AIs will literally terminally value accumulating certain configurations of matter?). I would really appreciate a clearer explanation of your argument. As it stands, I don’t fully understand what point you’re trying to make.
Humans are neither coherent, nor do they necessarily have a nonsatiable goal—though some might. But they have both to a far greater extent than less intelligent creatures.
What concrete evidence has Eliezer provided about the objectives advanced AIs are actually expected to develop?
Are you willing to posit that advanced systems are coherent, with at least one non-satiable component? Because that’s pretty minimal as an assumption, but implies with probability one that they prefer paperclipping of some sort.
Are humans coherent with at least one non-satiable component? If so, then I don’t understand the distinction you’re making that would justify positing AI values to be worse than human values from a utilitarian perspective.
If not, then I’m additionally unclear on why you believe AIs will be unlike humans in this respect, to the extent that they would become “paperclippers.” That term itself seems ambiguous to me (do you mean AIs will literally terminally value accumulating certain configurations of matter?). I would really appreciate a clearer explanation of your argument. As it stands, I don’t fully understand what point you’re trying to make.
Humans are neither coherent, nor do they necessarily have a nonsatiable goal—though some might. But they have both to a far greater extent than less intelligent creatures.
What is coherence here? Perfect coherence sounds like a very strong assumption to me, not a minimal one.
Close enough not to have any cyclic components that would lead to infinite cycles for the nonsatiable component of their utility.