Thanks for writing this, I thought it was a good article. And thanks to Greg for funding it.
My pushback would be on the cooperation and coordination point. It seems that a lot of other people, with other moral values, could make a very similar argument: that they need to promote their values now, as the stakes as very high with possible upcoming value lock-in. To people with those values, these arguments should seem roughly as important as the above argument is to you.
Christians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with sinners who will go to hell.
Egalitarians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with wider and wider diversities of wealth.
Libertarians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with property rights violations.
Naturalists could argue that, if the singularity is approaching, it is vitally important that we ensure the beauty of nature won’t be bespoiled all over the universe.
Nationalists could argue that, if the singularity is approaching, it is vitally important that we ensure the universe will be filled with people who respect the flag.
But it seems that it would be very bad if everyone took this advice literally. We would all end up spending a lot of time and effort on propaganda, which would probably be great for advertising companies but not much else, as so much of it is zero sum. Even though it might make sense, by their values, for expanding-moral-circle people and pro-abortion people to have a big propaganda war over whether foetuses deserve moral consideration, it seems plausible we’d be better off if they both decided to spend the money on anti-malaria bednets.
In contrast, preventing the extinction of humanity seems to occupy a privileged position—not exactly comparable with the above agendas, though I can’t exactly cache out why it seems this way to me. Perhaps to devout Confucians a pre-occupation with preventing extinction seems to be just another distraction from the important task of expressing filial piety – though I doubt this.
(Moral Realists, of course, could argue that the situation is not really symmetric, because promoting the true values is distinctly different from promoting any other values.)
Yeah, I think that’s basically right. I think moral circle expansion (MCE) is closer to your list items than extinction risk reduction (ERR) is because MCE mostly competes in the values space, while ERR mostly competes in the technology space.
However, MCE is competing in a narrower space than just values. It’s in the MC space, which is just the space of advocacy on what our moral circle should look like. So I think it’s fairly distinct from the list items in that sense, though you could still say they’re in the same space because all advocacy competes for news coverage, ad buys, recruiting advocacy-oriented people, etc. (Technology projects could also compete for these things, though there are separations, e.g. journalists with a social beat versus journalists with a tech beat.)
I think the comparably narrow space of ERR is ER, which also includes people who don’t want extinction risk reduced (or even want it increased), such as some hardcore environmentalists, antinatalists, and negative utilitarians.
I think these are legitimate cooperation/coordination perspectives, and it’s not really clear to me how they add up. But in general, I think this matters mostly in situations where you actually can coordinate. For example, in the US general election when Democrats and Republicans come together and agree not to give to their respective campaigns (in exchange for their counterpart also not doing so). Or if there were anti-MCE EAs with whom MCE EAs could coordinate (which I think is basically what you’re saying with “we’d be better off if they both decided to spend the money on anti-malaria bednets”).
But it seems that it would be very bad if everyone took this advice literally.
Fortunately, not everyone does take this advice literally :).
This is very similar to the tragedy of the commons. If everyone acts out of their own self motivated interests, then everyone will be worse off. However, the situation as you described does not fully reflect reality because none of the groups you mentioned are actually trying to influence AI researchers at the moment. Therefore, MCE has a decisive advantage. Of course, this is always subject to change.
In contrast, preventing the extinction of humanity seems to occupy a privileged position
I find that it is often the case that people will dismiss any specific moral recommendation for AI except this one. Personally I don’t see a reason to think that there are certain universal principles of minimal alignment. You may argue that human extinction is something that almost everyone agrees is bad—but now the principle of minimal alignment has shifted to “have the AI prevent things that almost everyone agrees is bad” which is another privileged moral judgement that I see no intrinsic reason to hold.
In truth, I see no neutral assumptions to ground AI alignment theory in. I think this is made even more difficult because even relatively small differences in moral theory from the point of view of information theoretic descriptions of moral values can lead to drastically different outcomes. However, I do find hope in moral compromise.
As EA as a movement has grown so far, the community appears to converge upon a rationalization process whereby most of us have realized what is centrally morally important is the experiences of well-being of a relatively wide breadth of moral patients, and the relatively equal moral weight assigned to well-being of each moral patient. The difference between SI and those who focus on AIA is primarily their differing estimates of the expected value of far-future in terms of average or total well-being. Among the examples you provided, it seems some worldviews are more amenable to the rationalization process which lends itself to consequentialism and EA. Many community members were egalitarians and libertarians who find common cause now in trying to figure out if to focus on AIA or MCE. I think your point is important in that ultimately advocating for this type of values spreading could be bad. However what appears to be an extreme amount of diversity could end up looking less fraught in a competition among values as divergent worldviews converge on similar goals.
Since different types of worldviews, like any amenable to aggregate consequentialist frameworks, can collate around a single goal of something like MCE. The relevance of your point, then, would hinge upon how universal MCE really is or can be across worldviews, relative to other types of values, such that it wouldn’t clash with many worldviews in a values-spreading contest. This is a matter of debate I haven’t thought of. It seems an important way to frame solutions to the challenge to Jacy’s point you raise.
Thanks for writing this, I thought it was a good article. And thanks to Greg for funding it.
My pushback would be on the cooperation and coordination point. It seems that a lot of other people, with other moral values, could make a very similar argument: that they need to promote their values now, as the stakes as very high with possible upcoming value lock-in. To people with those values, these arguments should seem roughly as important as the above argument is to you.
Christians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with sinners who will go to hell.
Egalitarians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with wider and wider diversities of wealth.
Libertarians could argue that, if the singularity is approaching, it is vitally important that we ensure the universe won’t be filled with property rights violations.
Naturalists could argue that, if the singularity is approaching, it is vitally important that we ensure the beauty of nature won’t be bespoiled all over the universe.
Nationalists could argue that, if the singularity is approaching, it is vitally important that we ensure the universe will be filled with people who respect the flag.
But it seems that it would be very bad if everyone took this advice literally. We would all end up spending a lot of time and effort on propaganda, which would probably be great for advertising companies but not much else, as so much of it is zero sum. Even though it might make sense, by their values, for expanding-moral-circle people and pro-abortion people to have a big propaganda war over whether foetuses deserve moral consideration, it seems plausible we’d be better off if they both decided to spend the money on anti-malaria bednets.
In contrast, preventing the extinction of humanity seems to occupy a privileged position—not exactly comparable with the above agendas, though I can’t exactly cache out why it seems this way to me. Perhaps to devout Confucians a pre-occupation with preventing extinction seems to be just another distraction from the important task of expressing filial piety – though I doubt this.
(Moral Realists, of course, could argue that the situation is not really symmetric, because promoting the true values is distinctly different from promoting any other values.)
Yeah, I think that’s basically right. I think moral circle expansion (MCE) is closer to your list items than extinction risk reduction (ERR) is because MCE mostly competes in the values space, while ERR mostly competes in the technology space.
However, MCE is competing in a narrower space than just values. It’s in the MC space, which is just the space of advocacy on what our moral circle should look like. So I think it’s fairly distinct from the list items in that sense, though you could still say they’re in the same space because all advocacy competes for news coverage, ad buys, recruiting advocacy-oriented people, etc. (Technology projects could also compete for these things, though there are separations, e.g. journalists with a social beat versus journalists with a tech beat.)
I think the comparably narrow space of ERR is ER, which also includes people who don’t want extinction risk reduced (or even want it increased), such as some hardcore environmentalists, antinatalists, and negative utilitarians.
I think these are legitimate cooperation/coordination perspectives, and it’s not really clear to me how they add up. But in general, I think this matters mostly in situations where you actually can coordinate. For example, in the US general election when Democrats and Republicans come together and agree not to give to their respective campaigns (in exchange for their counterpart also not doing so). Or if there were anti-MCE EAs with whom MCE EAs could coordinate (which I think is basically what you’re saying with “we’d be better off if they both decided to spend the money on anti-malaria bednets”).
Fortunately, not everyone does take this advice literally :).
This is very similar to the tragedy of the commons. If everyone acts out of their own self motivated interests, then everyone will be worse off. However, the situation as you described does not fully reflect reality because none of the groups you mentioned are actually trying to influence AI researchers at the moment. Therefore, MCE has a decisive advantage. Of course, this is always subject to change.
I find that it is often the case that people will dismiss any specific moral recommendation for AI except this one. Personally I don’t see a reason to think that there are certain universal principles of minimal alignment. You may argue that human extinction is something that almost everyone agrees is bad—but now the principle of minimal alignment has shifted to “have the AI prevent things that almost everyone agrees is bad” which is another privileged moral judgement that I see no intrinsic reason to hold.
In truth, I see no neutral assumptions to ground AI alignment theory in. I think this is made even more difficult because even relatively small differences in moral theory from the point of view of information theoretic descriptions of moral values can lead to drastically different outcomes. However, I do find hope in moral compromise.
As EA as a movement has grown so far, the community appears to converge upon a rationalization process whereby most of us have realized what is centrally morally important is the experiences of well-being of a relatively wide breadth of moral patients, and the relatively equal moral weight assigned to well-being of each moral patient. The difference between SI and those who focus on AIA is primarily their differing estimates of the expected value of far-future in terms of average or total well-being. Among the examples you provided, it seems some worldviews are more amenable to the rationalization process which lends itself to consequentialism and EA. Many community members were egalitarians and libertarians who find common cause now in trying to figure out if to focus on AIA or MCE. I think your point is important in that ultimately advocating for this type of values spreading could be bad. However what appears to be an extreme amount of diversity could end up looking less fraught in a competition among values as divergent worldviews converge on similar goals.
Since different types of worldviews, like any amenable to aggregate consequentialist frameworks, can collate around a single goal of something like MCE. The relevance of your point, then, would hinge upon how universal MCE really is or can be across worldviews, relative to other types of values, such that it wouldn’t clash with many worldviews in a values-spreading contest. This is a matter of debate I haven’t thought of. It seems an important way to frame solutions to the challenge to Jacy’s point you raise.