We are dealing with severely misaligned incentives. And when incentives point the wrong way, awareness does not fix much.
Anyone working at a frontier lab is presented with an extraordinary asymmetry: enormous and immediate personal rewards from pushing capabilities forward, against a probabilistic risk borne by everyone else. People are extremely good at persuading themselves that what benefits them is also the right thing to do, especially when the rewards are this high.
The US government, even if it understands the risks, is locked into strategic competition with China that makes slowing down impossible. A serious warning confirms the danger, but can make frontier AI seem even more important, increasing the pressure to get there first.
And internationally, the institutions that might once have helped manage this kind of competition are all being torched just as they are most needed. Communication, verification, restraint and collective action that helped prevent WWIII are all being made harder, leaving fewer ways to slow or manage the race.
So we are heading to a situation where the labs, governments and even the public understand the danger quite well, but the incentives of those who control development still push them toward acceleration, and the mechanisms for managing this are gone.
“So we are heading to a situation where the labs, governments and even the public understand the danger quite well”, I don’t think this is true? I would suspect even most AI researchers would give a poor answer if asked in detail what it would mean to have superintelligent AI systems, what kinds of behaviours such systems may be expected to show, and what dangers this poses. I could be wrong, but I feel like few people have really envisioned what a worst case scenario actually looks like.
If people did, statements like “strategic competition with China” would appear silly, a bit like saying I’m locked in a strategic competition to shoot myself in the face before my enemy does.
Most of the behaviour in the labs and government is driven by ego and power-seeking heuristics, not rational strategy.
I agree with much of what you’re saying, but disagree that slowing down is impossible. General ASI is not a tool, it’s an adversary. As many have pointed out, ASI is also a threat to whoever “gets there first” first.
Furthermore, these’s some evidence that labs in China are advancing on the coattails of US based labs, so slowing work in the US, even unilaterally, would slow labs in China.
The speed of action required and adverse economic incentives make this challenging, but we have precedent as humans overcoming great challenges in the past, including nuclear weapons, and terrible diseases. We developed international governance, and non-proliferation treaties. It is achievable—there are only a few labs capable of doing this kind of work, and the supply chain is long and relatively easy to control. It is a solvable problem, if we work together.
We are dealing with severely misaligned incentives. And when incentives point the wrong way, awareness does not fix much.
Anyone working at a frontier lab is presented with an extraordinary asymmetry: enormous and immediate personal rewards from pushing capabilities forward, against a probabilistic risk borne by everyone else. People are extremely good at persuading themselves that what benefits them is also the right thing to do, especially when the rewards are this high.
The US government, even if it understands the risks, is locked into strategic competition with China that makes slowing down impossible. A serious warning confirms the danger, but can make frontier AI seem even more important, increasing the pressure to get there first.
And internationally, the institutions that might once have helped manage this kind of competition are all being torched just as they are most needed. Communication, verification, restraint and collective action that helped prevent WWIII are all being made harder, leaving fewer ways to slow or manage the race.
So we are heading to a situation where the labs, governments and even the public understand the danger quite well, but the incentives of those who control development still push them toward acceleration, and the mechanisms for managing this are gone.
“So we are heading to a situation where the labs, governments and even the public understand the danger quite well”, I don’t think this is true? I would suspect even most AI researchers would give a poor answer if asked in detail what it would mean to have superintelligent AI systems, what kinds of behaviours such systems may be expected to show, and what dangers this poses. I could be wrong, but I feel like few people have really envisioned what a worst case scenario actually looks like.
If people did, statements like “strategic competition with China” would appear silly, a bit like saying I’m locked in a strategic competition to shoot myself in the face before my enemy does.
Most of the behaviour in the labs and government is driven by ego and power-seeking heuristics, not rational strategy.
I agree with much of what you’re saying, but disagree that slowing down is impossible. General ASI is not a tool, it’s an adversary. As many have pointed out, ASI is also a threat to whoever “gets there first” first.
Furthermore, these’s some evidence that labs in China are advancing on the coattails of US based labs, so slowing work in the US, even unilaterally, would slow labs in China.
The speed of action required and adverse economic incentives make this challenging, but we have precedent as humans overcoming great challenges in the past, including nuclear weapons, and terrible diseases. We developed international governance, and non-proliferation treaties. It is achievable—there are only a few labs capable of doing this kind of work, and the supply chain is long and relatively easy to control. It is a solvable problem, if we work together.