I think Bostrom’s argument merely compares a pure x-risk (such as a huge asteroid hurtling towards Earth) relative to technological acceleration, and then concludes that reducing the probability of a pure x-risk is more important because the x-risk threatens the eventual colonization of the universe. I agree with this argument in the case of a pure x-risk, but as I noted in my original comment, I don’t think that AI risk is a pure x-risk.
If, by contrast, all we’re doing by doing AI safety research is influencing something like “the values of the agents in society in the future” (and not actually influencing the probability of eventual colonization), then this action seems to plausibly just wash out in the long-term. In this case, it seems very appropriate to compare the short-term effects of AI safety to the short-term effects of acceleration.
Let me put it another way. We can think about two (potentially competing) strategies for making the future better, along with their relevant short and possible long-term effects:
Doing AI safety research
Short-term effects: makes it more likely that AIs are kind to current or near-future humans
Possible long-term effect: makes it more likely that AIs in the very long-run will share the values of the human species, relative to some unaligned alternative
Accelerating AI
Short-term effect: helps current humans by hastening the arrival of advanced technology
Possible long-term effect: makes it more likely that we have a large population size at low per capita incomes, relative to a low population size with high per capita income
My opinion is that both of these long-term effects are very speculative, so it’s generally better to focus on a heuristic of doing what’s better in the short-term, while keeping the long-term consequences in mind. And when I do that, I do not come to a strong conclusion that AI safety research “beats” AI acceleration, from a total utilitarian perspective.
I think Bostrom’s argument merely compares a pure x-risk (such as a huge asteroid hurtling towards Earth) relative to technological acceleration, and then concludes that reducing the probability of a pure x-risk is more important because the x-risk threatens the eventual colonization of the universe. I agree with this argument in the case of a pure x-risk, but as I noted in my original comment, I don’t think that AI risk is a pure x-risk.
If, by contrast, all we’re doing by doing AI safety research is influencing something like “the values of the agents in society in the future” (and not actually influencing the probability of eventual colonization), then this action seems to plausibly just wash out in the long-term. In this case, it seems very appropriate to compare the short-term effects of AI safety to the short-term effects of acceleration.
Let me put it another way. We can think about two (potentially competing) strategies for making the future better, along with their relevant short and possible long-term effects:
Doing AI safety research
Short-term effects: makes it more likely that AIs are kind to current or near-future humans
Possible long-term effect: makes it more likely that AIs in the very long-run will share the values of the human species, relative to some unaligned alternative
Accelerating AI
Short-term effect: helps current humans by hastening the arrival of advanced technology
Possible long-term effect: makes it more likely that we have a large population size at low per capita incomes, relative to a low population size with high per capita income
My opinion is that both of these long-term effects are very speculative, so it’s generally better to focus on a heuristic of doing what’s better in the short-term, while keeping the long-term consequences in mind. And when I do that, I do not come to a strong conclusion that AI safety research “beats” AI acceleration, from a total utilitarian perspective.