But it seems that it would be very bad if everyone took this advice literally.
Fortunately, not everyone does take this advice literally :).
This is very similar to the tragedy of the commons. If everyone acts out of their own self motivated interests, then everyone will be worse off. However, the situation as you described does not fully reflect reality because none of the groups you mentioned are actually trying to influence AI researchers at the moment. Therefore, MCE has a decisive advantage. Of course, this is always subject to change.
In contrast, preventing the extinction of humanity seems to occupy a privileged position
I find that it is often the case that people will dismiss any specific moral recommendation for AI except this one. Personally I don’t see a reason to think that there are certain universal principles of minimal alignment. You may argue that human extinction is something that almost everyone agrees is bad—but now the principle of minimal alignment has shifted to “have the AI prevent things that almost everyone agrees is bad” which is another privileged moral judgement that I see no intrinsic reason to hold.
In truth, I see no neutral assumptions to ground AI alignment theory in. I think this is made even more difficult because even relatively small differences in moral theory from the point of view of information theoretic descriptions of moral values can lead to drastically different outcomes. However, I do find hope in moral compromise.
Fortunately, not everyone does take this advice literally :).
This is very similar to the tragedy of the commons. If everyone acts out of their own self motivated interests, then everyone will be worse off. However, the situation as you described does not fully reflect reality because none of the groups you mentioned are actually trying to influence AI researchers at the moment. Therefore, MCE has a decisive advantage. Of course, this is always subject to change.
I find that it is often the case that people will dismiss any specific moral recommendation for AI except this one. Personally I don’t see a reason to think that there are certain universal principles of minimal alignment. You may argue that human extinction is something that almost everyone agrees is bad—but now the principle of minimal alignment has shifted to “have the AI prevent things that almost everyone agrees is bad” which is another privileged moral judgement that I see no intrinsic reason to hold.
In truth, I see no neutral assumptions to ground AI alignment theory in. I think this is made even more difficult because even relatively small differences in moral theory from the point of view of information theoretic descriptions of moral values can lead to drastically different outcomes. However, I do find hope in moral compromise.