I agree EAs often discuss the importance of “getting alignment right” and then subtly frame this in terms of ensuring that AIs either care about consciousness or possess consciousness themselves. However, the most common explicit justification for delaying AI development is the argument that doing so increases the likelihood that AIs will be aligned with human interests. This distinction is crucial because aligning AI with human interests is not the same as ensuring that AI maximizes utilitarian value—human interests and utilitarian value are not equivalent.
Currently, we lack strong empirical evidence to determine whether AIs will ultimately generate more or less value than humans from a utilitarian point of view. Because we do not yet know which is the case, there is no clear justification for defaulting to delaying AI development rather than accelerating it. If AIs turn out to generate more moral value than humans, then delaying AI would mean we are actively making a mistake—we would be increasing the probability of future human dominance, since by assumption, the main effect from delaying AI is to increase the probability that AIs will be aligned with human interests. This would risk entrenching a suboptimal future.
On the other hand, if AIs end up generating less value, as many effective altruists currently believe, then delaying AI would indeed be the right decision. However, since we do not yet have enough evidence to determine which scenario is correct, we should recognize this uncertainty rather than assume that delaying AI is the obviously preferable, or default course of action.
Your argument appears to assume that, in the absence of evidence about what goals future AI systems will have, delaying AI development should be the default position to mitigate risk. But why should we accept this assumption? Why not consider acceleration just as reasonable a default? If we lack meaningful evidence about the values AI will develop, then we have no more justification for assuming that delay is preferable than we do for assuming that acceleration is.
In fact, one could just as easily argue the opposite: that AI might develop moral values superior to those of humans. This claim appears to have about as much empirical support as the assumption that AI values will be worse. This argument could then justify accelerating AI rather than delaying it. Using the same logic that you just applied, one could make a symmetrical counterargument against your position: that accelerating AI is actually the correct course of action, since any minor harms caused by moving forward are vastly outweighed by the long-term risk of locking in suboptimal values through unnecessary delay. Delaying AI development would, in this context, risk entrenching human values, which are suboptimal to the default AI values that we would get through accelerating.
You might think that even weak evidence in favor of delaying AI is sufficient to support this strategy as the default course of action. But this would seem to assume a “knife’s edge” scenario, where even a slight epistemic advantage—such as a 51% chance that delay is beneficial versus a 49% chance that acceleration is beneficial—should be enough to justify committing to a pause. If we adopted this kind of reasoning in other domains, we would quickly fall into epistemic paralysis, constantly shifting strategies based on fragile, easily reversible analysis.
Given this high level of uncertainty about AI’s future trajectory, I think the best approach is to focus on the most immediate and concrete tradeoffs that we can analyze with some degree of confidence. This includes whether delaying or accelerating AI is likely to be more beneficial to the current generation of humans. However, based on the available evidence, I believe that accelerating AI—rather than delaying it—is likely the better choice, as I highlight in my post.