One reason is that longtermists are largely philosophers, who have no particular expertise on the details of aligning AI.
Another reason worth taking into consideration is if the true moral view is “fussy”, rather than “easygoing”. If you’re “easygoing” in what you consider utopia, then, conditional on survival, most achievable value gets realized by default (we get great human lives), and extinction is the one really action-relevant lock-in event. But if you’re fussy about realizing the best possible utopia, then, conditional on survival, we’re still likely to miss most achievable value across a huge swathe of futures (we don’t tile the universe with happy digital minds, say, or whatever crazy future might be the best utopia). The space of “didn’t go extinct but missed most of the value” turns out to be enormous, and some of the features determining where in that space we land (early decisions about digital minds, population-ethics, allocation of resources during space settlement, which value-systems get amplified during the AI transition) are themselves plausibly locked-in, even if they don’t feel as salient as extinction.
(But then again, maybe I’m recency biased because I just re-read Better Futures for the discussion week here on the Forum)
That doesn’t settle the prioritization, and, like, the people are Forethought and 80.000 Hours are directly and explicitly working on the AI transition? So it’s not like x-risk is off the table. My vibe for highly-engaged EAs is that perhaps it just feels that the main arguments about x-risk have already been made.
I agree that in deciding how much to prioritise averting extinction vs improving worlds in which we persist, it’s important to think about the difference in value between (non-existence)(default survival)(actual utopia). But that argument has been around a long while. I think Ben Garfinkel was advancing the idea that (actual utopia) - (default survival) might be much larger than (default survival) - (non-existence) in the late 2010s. I’m interested in what’s changed that’s affected discourse. It’s possible the answer is ‘more people have read arguments of this form’. But in that case people who had already read those arguments should update less than if the change is eg us getting more info about how difficult alignment is.
One reason is that longtermists are largely philosophers, who have no particular expertise on the details of aligning AI.
Another reason worth taking into consideration is if the true moral view is “fussy”, rather than “easygoing”. If you’re “easygoing” in what you consider utopia, then, conditional on survival, most achievable value gets realized by default (we get great human lives), and extinction is the one really action-relevant lock-in event. But if you’re fussy about realizing the best possible utopia, then, conditional on survival, we’re still likely to miss most achievable value across a huge swathe of futures (we don’t tile the universe with happy digital minds, say, or whatever crazy future might be the best utopia). The space of “didn’t go extinct but missed most of the value” turns out to be enormous, and some of the features determining where in that space we land (early decisions about digital minds, population-ethics, allocation of resources during space settlement, which value-systems get amplified during the AI transition) are themselves plausibly locked-in, even if they don’t feel as salient as extinction.
(But then again, maybe I’m recency biased because I just re-read Better Futures for the discussion week here on the Forum)
That doesn’t settle the prioritization, and, like, the people are Forethought and 80.000 Hours are directly and explicitly working on the AI transition? So it’s not like x-risk is off the table. My vibe for highly-engaged EAs is that perhaps it just feels that the main arguments about x-risk have already been made.
I agree that in deciding how much to prioritise averting extinction vs improving worlds in which we persist, it’s important to think about the difference in value between (non-existence)(default survival)(actual utopia). But that argument has been around a long while. I think Ben Garfinkel was advancing the idea that (actual utopia) - (default survival) might be much larger than (default survival) - (non-existence) in the late 2010s. I’m interested in what’s changed that’s affected discourse. It’s possible the answer is ‘more people have read arguments of this form’. But in that case people who had already read those arguments should update less than if the change is eg us getting more info about how difficult alignment is.