The main risk for AIA seems to be that the technical research done to better understand how to build an aligned AI will increase AI capabilities generally, meaning it’s also easier for humanity to produce an unaligned AI.
This doesn’t seem like a big consideration to me. Even if unfriendly AI comes sooner by an entire decade, this matters little on a cosmic timescale. An argument I find more compelling: If we plot the expected utility of an AGI as a function of the amount of effort put into aligning it, there might be a “valley of bad alignment” that is worse than no attempt at alignment at all. (A paperclip maximizer will quickly kill us and not generate much long-term suffering, whereas an AI that understands the importance of human survival but doesn’t understand any other values will imprison us for all eternity. Something like that.)
I’d like to know more about why people think that our moral circles have expanded. I suspect activism plays a smaller role than you think. Steven Pinker talks about possible reasons for declining violence in his book The Better Angels of Our Nature. I’m guessing this is highly related to moral circle expansion.
One theory I haven’t seen elsewhere is that self-interest plays a big role in moral circle expansion. Consider the example of slavery. The BBC writes:
It becomes clear that humanitarianism and imperial muscling were able bedfellows...
One can be certain that the high ideals of abolition and the promotion of legitimate trade were equally matched by economic and territorial ambitions, impulses which brought forward partition and colonial rule in Africa in the late 19th century.
You’ll note that the villains of the slave story are the slavers—people with an interest in slavery. The heroes seem to have been Britons who would not lose much if slavery was outlawed (though I guess boycotting sugar would go against their self-interest?) Similarly, I think I remember reading that poor northern whites were motivated to fight in the US Civil War because they were worried their labor would be displaced by slave labor.
According to this story, the expanding circle is a side effect of the world growing wealthier. As lower levels of Maslow’s hierarchy are met, people care more about humanitarian issues. (I’m assuming that genetic relatedness predicts where on the hierarchy another being falls.) Conquest is less common now because it’s more profitable to control a multinational company than control lots of territory. Slavery is less common because unskilled laborers are less of an asset & more of a liability, and it’s hard to coerce skilled labor. Violence has declined because sub-replacement fertility means we’re no longer in a zero-sum competition for resources. (Note that the bloodiest war in recent memory happened in the Democratic Republic of Congo, a country where women average six children each—source. Congo has a lot of mineral wealth, which seems to incentivize conflict. Probably this wealth doesn’t diminish as much in the presence of conflict as much as e.g. manufacturing wealth would.)
I suppose a quick test for the Maslow’s hierarchy story is to check whether wealthy people are more likely to be vegan (controlling for meat calories being as expensive as non-meat calories).
I don’t think everyone is completely self-interested all the time, but I think people are self-interested enough that it makes sense for activists to apply leverage strategically.
Re: a computer program used to mine asteroids, I’d expect certain AI alignment work to be useful here. If we understand AI algorithms more deeply, an asteroid miner can be simpler and less likely sentient. Contrast with the scenario where AI progress is slow, brain emulations come before AGI, and the asteroid miner is piloted using an emulation of someone’s brain.
I’m not comfortable relying on innate human goodness to deal with moral dilemmas. I’d rather eliminate incentives for immoral behavior. In the presence of bad incentives, I worry about activism backfiring as people come up with rationalizations for their immoral behavior. See e.g. biblical justifications for slavery in the antebellum south. Instead of seeing the EA movement as something that will sweep the globe and make everyone altruistic, I’m more inclined to see it as a team of special forces working to adjust the incentives that everyone else operates under in order to create good outcomes as a side effect of everyone else working towards their incentives.
Singer and Pinker talk a lot about the importance of reason and empathy to the expanding moral circle. This might be achieved through better online discussion platforms, widespread adoption of meditation, etc.
Anyway, I think that if we take a broad view of moral circle expansion, the best way to achieve it might be some unexpected thing: improving the happiness of voters who control nuclear weapons, helping workers deal with technological job displacement, and so on. IMO, more EAs should work on worldpeace.
Thanks for this post. Some scattered thoughts:
This doesn’t seem like a big consideration to me. Even if unfriendly AI comes sooner by an entire decade, this matters little on a cosmic timescale. An argument I find more compelling: If we plot the expected utility of an AGI as a function of the amount of effort put into aligning it, there might be a “valley of bad alignment” that is worse than no attempt at alignment at all. (A paperclip maximizer will quickly kill us and not generate much long-term suffering, whereas an AI that understands the importance of human survival but doesn’t understand any other values will imprison us for all eternity. Something like that.)
I’d like to know more about why people think that our moral circles have expanded. I suspect activism plays a smaller role than you think. Steven Pinker talks about possible reasons for declining violence in his book The Better Angels of Our Nature. I’m guessing this is highly related to moral circle expansion.
One theory I haven’t seen elsewhere is that self-interest plays a big role in moral circle expansion. Consider the example of slavery. The BBC writes:
You’ll note that the villains of the slave story are the slavers—people with an interest in slavery. The heroes seem to have been Britons who would not lose much if slavery was outlawed (though I guess boycotting sugar would go against their self-interest?) Similarly, I think I remember reading that poor northern whites were motivated to fight in the US Civil War because they were worried their labor would be displaced by slave labor.
According to this story, the expanding circle is a side effect of the world growing wealthier. As lower levels of Maslow’s hierarchy are met, people care more about humanitarian issues. (I’m assuming that genetic relatedness predicts where on the hierarchy another being falls.) Conquest is less common now because it’s more profitable to control a multinational company than control lots of territory. Slavery is less common because unskilled laborers are less of an asset & more of a liability, and it’s hard to coerce skilled labor. Violence has declined because sub-replacement fertility means we’re no longer in a zero-sum competition for resources. (Note that the bloodiest war in recent memory happened in the Democratic Republic of Congo, a country where women average six children each—source. Congo has a lot of mineral wealth, which seems to incentivize conflict. Probably this wealth doesn’t diminish as much in the presence of conflict as much as e.g. manufacturing wealth would.)
I suppose a quick test for the Maslow’s hierarchy story is to check whether wealthy people are more likely to be vegan (controlling for meat calories being as expensive as non-meat calories).
I don’t think everyone is completely self-interested all the time, but I think people are self-interested enough that it makes sense for activists to apply leverage strategically.
Re: a computer program used to mine asteroids, I’d expect certain AI alignment work to be useful here. If we understand AI algorithms more deeply, an asteroid miner can be simpler and less likely sentient. Contrast with the scenario where AI progress is slow, brain emulations come before AGI, and the asteroid miner is piloted using an emulation of someone’s brain.
I’m not comfortable relying on innate human goodness to deal with moral dilemmas. I’d rather eliminate incentives for immoral behavior. In the presence of bad incentives, I worry about activism backfiring as people come up with rationalizations for their immoral behavior. See e.g. biblical justifications for slavery in the antebellum south. Instead of seeing the EA movement as something that will sweep the globe and make everyone altruistic, I’m more inclined to see it as a team of special forces working to adjust the incentives that everyone else operates under in order to create good outcomes as a side effect of everyone else working towards their incentives.
Another moral circle expansion story involves improved hygiene. See also.
Singer and Pinker talk a lot about the importance of reason and empathy to the expanding moral circle. This might be achieved through better online discussion platforms, widespread adoption of meditation, etc.
Anyway, I think that if we take a broad view of moral circle expansion, the best way to achieve it might be some unexpected thing: improving the happiness of voters who control nuclear weapons, helping workers deal with technological job displacement, and so on. IMO, more EAs should work on world peace.