I’m curious why there hasn’t been more work exploring a pro-AI or pro-AI-acceleration position from an effective altruist perspective. Some points:
Unlike existential risk from other sources (e.g. an asteroid) AI x-risk is unique because humans would be replaced by other beings, rather than completely dying out. This means you can’t simply apply a naive argument that AI threatens total extinction of value to make the case that AI safety is astronomically important, in the sense that you can for other x-risks. You generally need additional assumptions.
Total utilitarianism is generally seen as non-speciesist, and therefore has no intrinsic preference for human values over unaligned AI values. If AIs are conscious, there don’t appear to be strong prima facie reasons for preferring humans to AIs under hedonistic utilitarianism. Under preference utilitarianism, it doesn’t necessarily matter whether AIs are conscious.
Total utilitarianism generally recommends large population sizes. Accelerating AI can be modeled as a kind of “population accelerationism”. Extremely large AI populations could be preferable under utilitarianism compared to small human populations, even those with high per-capita incomes. Indeed, humans populations have recently stagnated via low population growth rates, and AI promises to lift this bottleneck.
Therefore, AI accelerationism seems straightforwardly recommended by total utilitarianism under some plausible theories.
Here’s a non-exhaustive list of guesses for why I think EAs haven’t historically been sympathetic to arguments like the one above, and have instead generally advocated AI safety over AI acceleration (at least when these two values conflict):
A belief that AIs won’t be conscious, and therefore won’t have much moral value compared to humans.
But why would we assume AIs won’t be conscious? For example, if Brian Tomasik is right, consciousness is somewhat universal, rather than being restricted to humans or members of the animal kingdom.
I also haven’t actually seen much EA literature defend this assumption explicitly, which would be odd if this belief is the primary reason EAs have for focusing on AI safety over AI acceleration.
A presumption in favor of human values over unaligned AI values for some reasons that aren’t based on strict impartial utilitarian arguments. These could include the beliefs that: (1) Humans are more likely to have “interesting” values compared to AIs, and (2) Humans are more likely to be motivated by moral arguments than AIs, and are more likely to reach a deliberative equilibrium of something like “ideal moral values” compared to AIs.
Why would humans be more likely to have “interesting” values than AIs? It seems very plausible that AIs will have interesting values even if their motives seem alien to us. AIs might have even more “interesting” values than humans.
It seems to me like wishful thinking to assume that humans are strongly motivated by moral arguments and would settle upon something like “ideal moral values”
A belief that population growth is inevitable, so it is better to focus on AI safety.
But a central question here is why pushing for AI safety—in the sense of AI research that enhances human interests—is better than the alternative on the margin. What reason is there to think AI safety now is better than pushing for greater AI population growth now? (Potential responses to this question are outlined in other bullet points above and below.)
AI safety has lasting effects due to a future value lock-in event, whereas accelerationism would have, at best, temporary effects.
Are you sure there will ever actually be a “value lock-in event”?
Even if there is at some point a value lock-in event, wouldn’t pushing for accelerationism also plausibly affect the values that are locked in? For example, the value of “population growth is good” seems more likely to be locked in, if you advocate for that now.
A belief that humans would be kinder and more benevolent than unaligned AIs
Humans seem pretty bad already. For example, humans are responsible for factory farming. It’s plausible that AIs could be even more callous and morally indifferent than humans, but the bar already seems low.
I’m also not convinced that moral values will be a major force shaping “what happens to the cosmic endowment”. It seems to me that the forces shaping economic consumption matter more than moral values.
A bedrock heuristic that it would be extraordinarily bad if “we all died from AI”, and therefore we should pursue AI safety over AI accelerationism.
But it would also be bad if we all died from old age while waiting for AI, and missed out on all the benefits that AI offers to humans, which is a point in favor of acceleration. Why would this heuristic be weaker?
An adherence to person-affecting views in which the values of currently-existing humans are what matter most; and a belief that AI threatens to kill existing humans.
But in this view, AI accelerationism could easily be favored since AIs could greatly benefit existing humans by extending our lifespans and enriching our lives with advanced technology.
An implicit acceptance of human supremacism, i.e. the idea that what matters is propagating the interests of the human species, or preserving the human species, even at the expense of individual interests (either within humanity or outside humanity) or the interests of other species.
But isn’t EA known for being unusually anti-speciesist compared to other communities? Peter Singer is often seen as a “founding father” of the movement, and a huge part of his ethical philosophy was about how we shouldn’t be human supremacists.
More generally, it seems wrong to care about preserving the “human species” in an abstract sense relative to preserving the current generation of actually living humans.
A belief that most humans are biased towards acceleration over safety, and therefore it is better for EAs to focus on safety as a useful correction mechanism for society.
But was an anti-safety bias common for previous technologies? I think something closer to the opposite is probably true: most humans seem, if anything, biased towards being overly cautious about new technologies rather than overly optimistic.
A belief that society is massively underrating the potential for AI, which favors extra work on AI safety, since it’s so neglected.
But if society is massively underrating AI, then this should also favor accelerating AI too? There doesn’t seem to be an obvious asymmetry between these two values.
An adherence to negative utilitarianism, which would favor obstructing AI, along with any other technology that could enable the population of conscious minds to expand.
This seems like a plausible moral argument to me, but it doesn’t seem like a very popular position among EAs.
A heuristic that “change is generally bad” and AI represents a gigantic change.
I don’t think many EAs would defend this heuristic explicitly.
Added: AI represents a large change to the world. Delaying AI therefore preserves option value.
This heuristic seems like it would have favored advocating delaying the industrial revolution, and all sorts of moral, social, and technological changes to the world in the past. Is that a position that EAs would be willing to bite the bullet on?
I suspect that if transformative AI is 20 or even 30 years away, AI will still be doing really big, impressive things in 2033, and people at that time will get a sense that even more impressive things are soon to come. In that case, I don’t think many people will think that AI safety advocates in 2023 were crying wolf, since one decade is not very long, and the importance of the technology will have only become more obvious in the meantime.