It would be surprising to me if making the transfer of power more voluntary/careful led to worse outcomes (or only led to slightly better outcomes such that the downsides of slowing down a bit made things worse).
Two questions here:
Why would accelerating AI make the transition less voluntary? (In my own mind, I’d be inclined to reverse this sentiment a bit: delaying AI by regulation generally involves forcibly stopping people from adopting AI. Force might be justified if it brings about a greater good, but that’s not the argument here.)
I can understand being “careful”. Being careful does seem like a good thing. But “being careful” generally trades off against other values in almost every domain I can think of, and there is such a thing as too much of a good thing. What reason is there to think that pushing for “more caution” is better on the margin compared to acceleration, especially considering society’s default response to AI in the absence of intervention?
So in the multi-agent slowly-replacing case, I’d argue that individual decisions don’t necessarily represent a voluntary decision on behalf of society (I’m imagining something like this scenario). In the misaligned power-seeking case, it seems obvious to me that this is involuntary. I agree that it technically could be a collective voluntary decision to hand over power more quickly, though (and in that case I’d be somewhat less against it).
I think emre’s comment lays out the intuitive case for being careful / taking your time, as does Ryan’s. I think the empirics are a bit messy once you take into account benefits of preventing other risks but I’d guess they come out in favor of delaying by at least a few years.
Two questions here:
Why would accelerating AI make the transition less voluntary? (In my own mind, I’d be inclined to reverse this sentiment a bit: delaying AI by regulation generally involves forcibly stopping people from adopting AI. Force might be justified if it brings about a greater good, but that’s not the argument here.)
I can understand being “careful”. Being careful does seem like a good thing. But “being careful” generally trades off against other values in almost every domain I can think of, and there is such a thing as too much of a good thing. What reason is there to think that pushing for “more caution” is better on the margin compared to acceleration, especially considering society’s default response to AI in the absence of intervention?
So in the multi-agent slowly-replacing case, I’d argue that individual decisions don’t necessarily represent a voluntary decision on behalf of society (I’m imagining something like this scenario). In the misaligned power-seeking case, it seems obvious to me that this is involuntary. I agree that it technically could be a collective voluntary decision to hand over power more quickly, though (and in that case I’d be somewhat less against it).
I think emre’s comment lays out the intuitive case for being careful / taking your time, as does Ryan’s. I think the empirics are a bit messy once you take into account benefits of preventing other risks but I’d guess they come out in favor of delaying by at least a few years.