Great point, thanks Matthew, upon reflection I agree that the section you quoted isn’t quite accurate. I guess I would restate it as something like “This confluence of factors creates incredibly powerful incentives to not think too hard about potential downsides of this new technology we’re racing towards. Motivated reasoning is much more likely when the motivations are so strong and the risks (while large) diffuse, distant (perhaps) and uncertain.” Does that make more sense?
I guess I do still think there are aspects of the situation that I would still call a coordination problem though. Imagine a situation where there are two actors, each agree that pushing a button has a 10% chance of killing them both, if it doesn’t kill them then the button-pusher gets $1 million, and each time it gets pushed the probability of the next push killing them both goes up by 10%. They each agree on the facts of the situation, but there is an incentive to defect first if you believe the other actor might defect, right? See this o3 analysis of the situation for more math than I can summon at this time of night 😅
There are psychological pressures that can lead to motivated reasoning on both sides of this issue. On the pro-acceleration side, individuals may be motivated to downplay or dismiss the potential risks and downsides of rapid AI development. On the other side, those advocating for slowing or pausing AI progress may be motivated to dismiss or undervalue the possible benefits and upsides. Because both the risks and the potential rewards of AI are substantial, I don’t see a compelling reason to assume that one side must be much more prone to denial or bias than the other.
At most, I see a simple selection effect: the people most actively pushing for faster AI development are likely those who are least worried about the risks. This could lead to a unilateralist curse, where the least concerned actors push capabilities forward despite a high risk of disaster. But the opposite scenario could also happen, if the most concerned actors are able to slow down progress for everyone else, delaying the benefits of AI unacceptably. Whether you should care more about the first or second scenario depends on your judgement of whether rapid AI progress is good or bad overall.
Ultimately, I think it’s more productive to frame the issue around empirical facts and value judgments: specifically, how much risk rapid AI development actually introduces, and how much value we ought to place on the potential benefits of rapid development. I find this framing more helpful, not only because it identifies the core disagreement between accelerationists and pause advocates, but also because I think it better accounts for the pace of AI development we actually observe in the real world.
Great point, thanks Matthew, upon reflection I agree that the section you quoted isn’t quite accurate. I guess I would restate it as something like “This confluence of factors creates incredibly powerful incentives to not think too hard about potential downsides of this new technology we’re racing towards. Motivated reasoning is much more likely when the motivations are so strong and the risks (while large) diffuse, distant (perhaps) and uncertain.” Does that make more sense?
I guess I do still think there are aspects of the situation that I would still call a coordination problem though. Imagine a situation where there are two actors, each agree that pushing a button has a 10% chance of killing them both, if it doesn’t kill them then the button-pusher gets $1 million, and each time it gets pushed the probability of the next push killing them both goes up by 10%. They each agree on the facts of the situation, but there is an incentive to defect first if you believe the other actor might defect, right? See this o3 analysis of the situation for more math than I can summon at this time of night 😅
There are psychological pressures that can lead to motivated reasoning on both sides of this issue. On the pro-acceleration side, individuals may be motivated to downplay or dismiss the potential risks and downsides of rapid AI development. On the other side, those advocating for slowing or pausing AI progress may be motivated to dismiss or undervalue the possible benefits and upsides. Because both the risks and the potential rewards of AI are substantial, I don’t see a compelling reason to assume that one side must be much more prone to denial or bias than the other.
At most, I see a simple selection effect: the people most actively pushing for faster AI development are likely those who are least worried about the risks. This could lead to a unilateralist curse, where the least concerned actors push capabilities forward despite a high risk of disaster. But the opposite scenario could also happen, if the most concerned actors are able to slow down progress for everyone else, delaying the benefits of AI unacceptably. Whether you should care more about the first or second scenario depends on your judgement of whether rapid AI progress is good or bad overall.
Ultimately, I think it’s more productive to frame the issue around empirical facts and value judgments: specifically, how much risk rapid AI development actually introduces, and how much value we ought to place on the potential benefits of rapid development. I find this framing more helpful, not only because it identifies the core disagreement between accelerationists and pause advocates, but also because I think it better accounts for the pace of AI development we actually observe in the real world.
I agree that it seems like a valuable framing, thanks Matthew.