Discussion topic: People vary a lot in the extent to which, and how likely it is, that post-AGI, different people will converge on the same moral views. I feel fairly sceptical about having a high likelihood of convergence; I certainly don’t think we should bank on it.
[See my response to Andreas below. Here I meant “convergence” as shorthand to refer to “fully accurate, motivational convergence”.]
I’m also pretty skeptical of convergence, largely because I’m a moral anti-realist. I don’t see why we would converge to any view in particular, except by coincidence or mechanisms that don’t track stance-independent moral truths (because there are none). It’s just people weighing their own particular moral intuitions. Humans differ in our most basic moral intuitions and leanings.
Barring value lock-in, I suspect there would be convergence towards the recognition that unnecessary suffering is bad and worth preventing (when cheap enough) because this seems pretty widely held and something societies move towards, but I’d guess there will still be disagreement on some of these:
population ethics
hedonism vs preference views vs others
whether non-sentient/non-conscious things matter terminally, e.g. preserving nature
whether groups of moral patients have special status beyond their aggregates, e.g. ethnic groups, species
deontology vs consequentialism vs virtue ethics
what counts as conscious/sentient (I think this is partly normative, not just empirical)
decision theory, attitudes towards risk and ambiguity, fanaticism
Or more basic things like religion, nationalism. People will want to shape their utopias in the image of their religious concept of heaven, and the idealised versions of their countries.
Could you clarify what you mean by ‘converge’? One thing that seems somewhat tricky to square is believing that convergence is unlikely, but that value lock-in is likely. Should we understand convergence as involving agreement in views facilitated by broadly rational processes, or something along those lines, to be contrasted with general agreement in values that might be facilitated by irrational or arational forces, of the kind that might ensure uniformity of views following a lock-in scenario?
Yeah, thanks for pushing me to be clearer: I meant “convergence” as shorthand to refer to “fully accurate, motivational convergence”. So I mean a scenario where people have the correct moral views, on everything that matters significantly, and are motivated to act on those moral views. I’ll try to say FAM-convergence from now on.
Factory farming, and to a lesser extent global poverty, persist because there are some costs to ending them, and the rich aren’t altruistic enough (or the altruists aren’t rich enough) to end them. Importantly, it will not just be that factory farming itself ends, but due to cognitive dissonance, people’s moral views towards nonhumans will likely change a lot too once ~no-one is eating animals. So there will predictably be convergence on viewing c2025 treatment of animals as terrible.
There is an ongoing homogenization of global culture which will probably continue. As the educational and cultural inputs to people converge, it seems likely their beliefs (including moral beliefs) will also converge at least somewhat.
Some fraction of current disagreements about economic/political/moral questions are caused just by people not being sufficiently informed/rational. So those disagreements would go away when we have ~ideal post-human reasoners.
A more ambitious version of the above is that perhaps post-humans will take epistemic humility very seriously, and they will know that all their peers are also very rational, so they will treat their own moral intuitions as little evidence of what the true/best/idealised-upon-reflection moral beliefs are. Then, everyone just defers very heavily to the annual survey of all of (post)humanity’s views on e.g. population axiology rather than backing their own intuition.
(Arguably this doesn’t count as convergence if people’s intuitions still differ, but I think if people’s all-things-considered beliefs, and therefore their actions, converge that is enough.)
I’m wondering whether we should expect worlds which converge on moral views to converge on bad moral views.
From the space of world religions—we’ve seen a trend where we converge over time (at least from a high level of abstraction where we can refer to “Christianity” and “Islam” rather than “mega-church Christians” or whatever). Is this because the religions that succeed are exclusive and expansionary? Of all religions that have existed, I know that many of them don’t much care if you also worship other gods. My empirical (ish) question is whether we should expect world in which a sizable fraction of the population follows the same religion to be one where the religion they follow is exclusive (you can’t follow others) and expansionary (other people should also follow this religion). PS- I know that not all Christians or Muslims are exclusionary about other religions, this is over-simplified.
This is relevant because, if this is a mechanism, we might expect the same thing of morality or political organisation—beliefs which demand you don’t follow others, and that others follow the same beliefs as you, rather than tolerant beliefs. Perhaps this would make it more likely that futures which converge have converged on something extreme and closed, rather than exploratory and open.
This is pretty vague—just wondering if others a) know more than me about the religion question and can speak to that or b) have had similar thoughts, or c) think that the existence of exclusive and expansionary (and wrong) ideologies might make convergence more likely.
Maybe another way to think about this (dropping the religion stuff—don’t want to cast aspersions on any particular religions) is that we could think of black-ball and white-ball ideologies (like the Bostrom thought experiment where black-balls = technologies which can cause extinction). Perhaps certain ideologies are just much more exclusive and expansion focused than others—black-balls. You can pick out as many white-balls as you like, but picking out a black-ball means you have to get rid of your white-balls. Even if there are few black-balls in the bag, you’d always end up holding one.
Interesting, is this the sort of thing you have in mind? It at least seems similar to me, and I remember thinking that post got at something important.
Yes, this is yet another reason for a moratorium on further-AGI development imo. If everyone has a genie with unlimited wishes, and are all pushing the world in different directions, the result will be chaos. Yampolskiy’s solution to this is everyone having their own private solipsistic universe simulations...
Discussion topic: People vary a lot in the extent to which, and how likely it is, that post-AGI, different people will converge on the same moral views. I feel fairly sceptical about having a high likelihood of convergence; I certainly don’t think we should bank on it.
[See my response to Andreas below. Here I meant “convergence” as shorthand to refer to “fully accurate, motivational convergence”.]
I’m also pretty skeptical of convergence, largely because I’m a moral anti-realist. I don’t see why we would converge to any view in particular, except by coincidence or mechanisms that don’t track stance-independent moral truths (because there are none). It’s just people weighing their own particular moral intuitions. Humans differ in our most basic moral intuitions and leanings.
Barring value lock-in, I suspect there would be convergence towards the recognition that unnecessary suffering is bad and worth preventing (when cheap enough) because this seems pretty widely held and something societies move towards, but I’d guess there will still be disagreement on some of these:
population ethics
hedonism vs preference views vs others
whether non-sentient/non-conscious things matter terminally, e.g. preserving nature
whether groups of moral patients have special status beyond their aggregates, e.g. ethnic groups, species
deontology vs consequentialism vs virtue ethics
what counts as conscious/sentient (I think this is partly normative, not just empirical)
decision theory, attitudes towards risk and ambiguity, fanaticism
Or more basic things like religion, nationalism. People will want to shape their utopias in the image of their religious concept of heaven, and the idealised versions of their countries.
Could you clarify what you mean by ‘converge’? One thing that seems somewhat tricky to square is believing that convergence is unlikely, but that value lock-in is likely. Should we understand convergence as involving agreement in views facilitated by broadly rational processes, or something along those lines, to be contrasted with general agreement in values that might be facilitated by irrational or arational forces, of the kind that might ensure uniformity of views following a lock-in scenario?
Yeah, thanks for pushing me to be clearer: I meant “convergence” as shorthand to refer to “fully accurate, motivational convergence”. So I mean a scenario where people have the correct moral views, on everything that matters significantly, and are motivated to act on those moral views. I’ll try to say FAM-convergence from now on.
A bull case for convergence:
Factory farming, and to a lesser extent global poverty, persist because there are some costs to ending them, and the rich aren’t altruistic enough (or the altruists aren’t rich enough) to end them. Importantly, it will not just be that factory farming itself ends, but due to cognitive dissonance, people’s moral views towards nonhumans will likely change a lot too once ~no-one is eating animals. So there will predictably be convergence on viewing c2025 treatment of animals as terrible.
There is an ongoing homogenization of global culture which will probably continue. As the educational and cultural inputs to people converge, it seems likely their beliefs (including moral beliefs) will also converge at least somewhat.
Some fraction of current disagreements about economic/political/moral questions are caused just by people not being sufficiently informed/rational. So those disagreements would go away when we have ~ideal post-human reasoners.
A more ambitious version of the above is that perhaps post-humans will take epistemic humility very seriously, and they will know that all their peers are also very rational, so they will treat their own moral intuitions as little evidence of what the true/best/idealised-upon-reflection moral beliefs are. Then, everyone just defers very heavily to the annual survey of all of (post)humanity’s views on e.g. population axiology rather than backing their own intuition.
(Arguably this doesn’t count as convergence if people’s intuitions still differ, but I think if people’s all-things-considered beliefs, and therefore their actions, converge that is enough.)
But I agree we shouldn’t bank on convergence!
I’m wondering whether we should expect worlds which converge on moral views to converge on bad moral views.
From the space of world religions—we’ve seen a trend where we converge over time (at least from a high level of abstraction where we can refer to “Christianity” and “Islam” rather than “mega-church Christians” or whatever). Is this because the religions that succeed are exclusive and expansionary? Of all religions that have existed, I know that many of them don’t much care if you also worship other gods. My empirical (ish) question is whether we should expect world in which a sizable fraction of the population follows the same religion to be one where the religion they follow is exclusive (you can’t follow others) and expansionary (other people should also follow this religion). PS- I know that not all Christians or Muslims are exclusionary about other religions, this is over-simplified.
This is relevant because, if this is a mechanism, we might expect the same thing of morality or political organisation—beliefs which demand you don’t follow others, and that others follow the same beliefs as you, rather than tolerant beliefs. Perhaps this would make it more likely that futures which converge have converged on something extreme and closed, rather than exploratory and open.
This is pretty vague—just wondering if others a) know more than me about the religion question and can speak to that or b) have had similar thoughts, or c) think that the existence of exclusive and expansionary (and wrong) ideologies might make convergence more likely.
Maybe another way to think about this (dropping the religion stuff—don’t want to cast aspersions on any particular religions) is that we could think of black-ball and white-ball ideologies (like the Bostrom thought experiment where black-balls = technologies which can cause extinction). Perhaps certain ideologies are just much more exclusive and expansion focused than others—black-balls. You can pick out as many white-balls as you like, but picking out a black-ball means you have to get rid of your white-balls. Even if there are few black-balls in the bag, you’d always end up holding one.
Interesting, is this the sort of thing you have in mind? It at least seems similar to me, and I remember thinking that post got at something important.
Hopefully they do not just converge on the same moral views, but also good ones!
A negotiated paretotopian future could create lots of moral value regardless of values not converging on their own.
Yes, this is yet another reason for a moratorium on further-AGI development imo. If everyone has a genie with unlimited wishes, and are all pushing the world in different directions, the result will be chaos. Yampolskiy’s solution to this is everyone having their own private solipsistic universe simulations...