Thanks for this analysis. I think your post deserves more attention, so I upvoted it.
We need more game-theory analyses like this, of geopolitical arms race scenarios.
Way too often, people just assume that the US-China rivalry can be modelled simply as a one-shot Prisoner’s Dilemma, in which the only equilibrium is mutual defection (from humanity’s general interests) through both sides trying to build ASI as soon as possible.
As your post indicates, the relevant game theory must include incomplete and asymmetric information, possible mixed-strategy equilibria, iterated play that depends strongly on what the other player has been doing, etc.
I would also encourage more development of game theory scenarios that explicitly model the creation of ASI as the introduction of a new player with its own rules, strategies, and payoffs.
Building an ASI isn’t just giving existing players a new tool for ‘winning the game’. It’s introducing a new player with its own interests (unless the ASI is 100% totally, reliably controlled & aligned with one existing player—which is probably impossible.)
Thanks for the comment! I agree that more game-theory analysis of arms race scenarios could be useful. I haven’t been able to find much other analysis, but if you know of any sources where I can learn more, that would be great.
As for the ASI being “another player”, my naive initial reaction is that it feels like an ASI that isn’t 100% controlled/aligned probably just results in everyone dying really quickly, so it feels somewhat pointless to model our conflicting interests with it using game theory. However, maybe there are worlds such as this one where the playing field is even enough such that complex interactions between humans and the ASI could be interesting to try to model. If you have any initial thoughts on this I would love to hear them.
I think the main benefit of explicitly modeling ASI as being a ‘new player’ in the geopolitical game is that it highlights precisely the idea that the ASI will NOT just automatically be a tool used by China or the US—but rather than it will have its own distinctive payoffs, interests, strategies, and agendas. That’s the key issue that many current political leaders (e.g. AI Czar David Sacks) do not seem to understand—if America builds an ASI, it won’t be ‘America’s ASI’, it will be the ASI’s ASI, so to speak.
ASI being unaligned doesn’t necessarily mean that it will kill all humans quickly—there are many, many possible outcomes other than immediate extinction that might be in the ASI’s interests.
The more seriously we model the possible divergences of ASI interests from the interests of current nation-states, the more persuasively we can make the argument that any nation building an ASI is not just flipping a coin between ‘geopolitical dominance forever’ and ‘human extinction forever’—rather, it’s introducing a whole new set of ASI interests that need to be taken into account.
Having given this a bit more thought, I think the starting point for something like this might be to generalize and assume the ASI just has “different” interests (we don’t know what those interests are right now both because we don’t know how ASI will be developed and because we haven’t solved alignment yet), and then also to assume that the ASI has just enough power to make it interesting to model (not because this assumption is realistic, but because if the ASI was too weak or too strong relative to humans, the modeling exercise would be uninformative).
I don’t know where to go from here, however. Maybe Buterin’s def/acc world that I linked in my earlier comment would be a good scenario to start with.
Thanks for this analysis. I think your post deserves more attention, so I upvoted it.
We need more game-theory analyses like this, of geopolitical arms race scenarios.
Way too often, people just assume that the US-China rivalry can be modelled simply as a one-shot Prisoner’s Dilemma, in which the only equilibrium is mutual defection (from humanity’s general interests) through both sides trying to build ASI as soon as possible.
As your post indicates, the relevant game theory must include incomplete and asymmetric information, possible mixed-strategy equilibria, iterated play that depends strongly on what the other player has been doing, etc.
I would also encourage more development of game theory scenarios that explicitly model the creation of ASI as the introduction of a new player with its own rules, strategies, and payoffs.
Building an ASI isn’t just giving existing players a new tool for ‘winning the game’. It’s introducing a new player with its own interests (unless the ASI is 100% totally, reliably controlled & aligned with one existing player—which is probably impossible.)
Thanks for the comment! I agree that more game-theory analysis of arms race scenarios could be useful. I haven’t been able to find much other analysis, but if you know of any sources where I can learn more, that would be great.
As for the ASI being “another player”, my naive initial reaction is that it feels like an ASI that isn’t 100% controlled/aligned probably just results in everyone dying really quickly, so it feels somewhat pointless to model our conflicting interests with it using game theory. However, maybe there are worlds such as this one where the playing field is even enough such that complex interactions between humans and the ASI could be interesting to try to model. If you have any initial thoughts on this I would love to hear them.
Matt—thanks for the quick and helpful reply.
I think the main benefit of explicitly modeling ASI as being a ‘new player’ in the geopolitical game is that it highlights precisely the idea that the ASI will NOT just automatically be a tool used by China or the US—but rather than it will have its own distinctive payoffs, interests, strategies, and agendas. That’s the key issue that many current political leaders (e.g. AI Czar David Sacks) do not seem to understand—if America builds an ASI, it won’t be ‘America’s ASI’, it will be the ASI’s ASI, so to speak.
ASI being unaligned doesn’t necessarily mean that it will kill all humans quickly—there are many, many possible outcomes other than immediate extinction that might be in the ASI’s interests.
The more seriously we model the possible divergences of ASI interests from the interests of current nation-states, the more persuasively we can make the argument that any nation building an ASI is not just flipping a coin between ‘geopolitical dominance forever’ and ‘human extinction forever’—rather, it’s introducing a whole new set of ASI interests that need to be taken into account.
Having given this a bit more thought, I think the starting point for something like this might be to generalize and assume the ASI just has “different” interests (we don’t know what those interests are right now both because we don’t know how ASI will be developed and because we haven’t solved alignment yet), and then also to assume that the ASI has just enough power to make it interesting to model (not because this assumption is realistic, but because if the ASI was too weak or too strong relative to humans, the modeling exercise would be uninformative).
I don’t know where to go from here, however. Maybe Buterin’s def/acc world that I linked in my earlier comment would be a good scenario to start with.