Thanks for the comment! I agree that more game-theory analysis of arms race scenarios could be useful. I haven’t been able to find much other analysis, but if you know of any sources where I can learn more, that would be great.
As for the ASI being “another player”, my naive initial reaction is that it feels like an ASI that isn’t 100% controlled/aligned probably just results in everyone dying really quickly, so it feels somewhat pointless to model our conflicting interests with it using game theory. However, maybe there are worlds such as this one where the playing field is even enough such that complex interactions between humans and the ASI could be interesting to try to model. If you have any initial thoughts on this I would love to hear them.
enterthewoods
Thanks for writing this! I think you touch upon a few facts that I’ve been thinking about a lot recently:
1. If the future consists of an ASI-enabled singleton (whether US-led, China-led, jointly-led, etc.), the moral values and posture of the singleton towards the rest of the world matter a lot. Given enough concentration of power, no outside force could realistically impose better morality on the singleton.
2. The present-day moral failures of current powerful people will not necessarily carry over in an ASI-enabled-abundance world. Dictator’s hands are often forced into oppressing their people. Dictator games in the experimental economics literature suggest unconditional altruism is frequent, although I don’t know what would occur if these games were run with extremely large sums of money. I suspect, however, that with extreme abundance, giving away a small fraction of this abundance to make the world a very good place to live (compared to where we are now, at least) seems like something almost anyone would do. Perhaps I have a naively optimistic view of human morality, however.
Taken together, these suggest that the future we live in might be one that is essentially reliant on the sheer kindness of a singleton. This is more of a hope than a solution. I would rather not risk the singleton being absolutely terrible. I’m currently trying to think of ways in which a singleton could be organized to maximize the chances of things like moral innovation and a genuinely positive posture towards the rest of the world being present. If anyone knows of any work that has already been done on this, pointing this out to me would be much appreciated!
Having given this a bit more thought, I think the starting point for something like this might be to generalize and assume the ASI just has “different” interests (we don’t know what those interests are right now both because we don’t know how ASI will be developed and because we haven’t solved alignment yet), and then also to assume that the ASI has just enough power to make it interesting to model (not because this assumption is realistic, but because if the ASI was too weak or too strong relative to humans, the modeling exercise would be uninformative).
I don’t know where to go from here, however. Maybe Buterin’s def/acc world that I linked in my earlier comment would be a good scenario to start with.