I have this game theory thing I’ve been working on that involves modifying the Iterated Prisoner’s Dilemma to include death, asymmetric power, and aggressor reputation. Agents’ points are their “power” that dynamically impacts their payoff matrix.
The basic takeaway is that this simple simulation seems to make a case for cooperating with weaker agents, by showing how the cooperative strategies outcompete the aggressive ones in the long run. I think, before I can make a proper post about it, I’ll need to run some analysis to graph out how, for instance, having a higher percentage of cooperative agents increases the odds of survival, which implies a kind of Veil of Ignorance logic towards being cooperative.
Note that I mean cooperative in the sense that you don’t defect first except against aggressors that have defected first against non-aggressors.
With default settings, the most common result of any given run is that a significant number of the cooperative strategies survive and almost all of the aggressive ones die out. Very occasionally, particularly if you adjust the settings are certain way, a single “Opportunist” strategy, that Tit-For-Tats against stronger agents and Defects against weaker ones, will be the only survivor. This seems to imply, at least, to me, that being a cooperative strategy significantly increases your odds of survival, as the alternative is to hope to win a “Highlander” scenario.
I think this is relevant to AI alignment as a variation on Anthropic Capture, the “Hail Mary” approach that Bostrom mentions in Superintelligence. It could work as part of a defence-in-depth, a kind of “infoblessing” that could persuade some AGI to spare us as a kind of Superrational Signalling. While you might assume this only works if aliens are probable, it also functions in a multi-agent scenario where there are several AGI at near peer levels of power. It also potentially could be a way to align a previously unaligned AGI even after it is deployed. If enough AGIs are aligned in this way, their alliance could defeat the unaligned AGIs.
I realize that a very obvious critique of this work is that the simulation is probably too simple. I intentionally tried to keep it an MVP in its first iteration. I also should, as mentioned earlier, complete a more thorough and rigorous analysis of the apparent results. I’m also keenly aware that it seems like this is a “neglected” path towards alignment, and I’m uncertain whether this is because the idea is a bad one that’s already been discarded by others who are more competent. I know that there are related ideas around Decision Theory, Acausal Trade, and Superrationality, but I’ve never seen this particular kind of effort, which confuses me, because it seems obvious and trivial to try.
My main question to ask is simply, does this seem like something worth pursuing and expanding further, or am I wasting my time on a foolish endeavour?
Is the main idea here that if “have you ever defected against a non-agressor” is the agreed way to decide who to not cooperate with, then cooperative strategies win and then stay cooperative when power imbalances are large? Is there a reason would agents choose to use that instead of any other decision criteria? Also, I wouldn’t say that the obvious critique is that your model is too simple, I would say it’s that your model can easily be unrealistic if you don’t try very hard to check it. Good game theory models are sometimes very simple.
That is sorta the idea yes. Agents would choose this decision criteria mostly because it vastly increases their odds of survival, which allows them to further whatever goals they have. I would hope that this result is obvious enough that many agents will be able to converge on it, increasing the proportion using it, and thus increasing the overall survival rate.
The other takeaway is that, given that humans will be weaker than AGI/ASI, any game theoretic reason for such entities to still cooperate with us can potentially help reduce the existential risk.
I agree that the model requires further scrutiny to determine if it is realistic enough to matter.
I have this game theory thing I’ve been working on that involves modifying the Iterated Prisoner’s Dilemma to include death, asymmetric power, and aggressor reputation. Agents’ points are their “power” that dynamically impacts their payoff matrix.
The basic takeaway is that this simple simulation seems to make a case for cooperating with weaker agents, by showing how the cooperative strategies outcompete the aggressive ones in the long run. I think, before I can make a proper post about it, I’ll need to run some analysis to graph out how, for instance, having a higher percentage of cooperative agents increases the odds of survival, which implies a kind of Veil of Ignorance logic towards being cooperative.
Note that I mean cooperative in the sense that you don’t defect first except against aggressors that have defected first against non-aggressors.
With default settings, the most common result of any given run is that a significant number of the cooperative strategies survive and almost all of the aggressive ones die out. Very occasionally, particularly if you adjust the settings are certain way, a single “Opportunist” strategy, that Tit-For-Tats against stronger agents and Defects against weaker ones, will be the only survivor. This seems to imply, at least, to me, that being a cooperative strategy significantly increases your odds of survival, as the alternative is to hope to win a “Highlander” scenario.
I think this is relevant to AI alignment as a variation on Anthropic Capture, the “Hail Mary” approach that Bostrom mentions in Superintelligence. It could work as part of a defence-in-depth, a kind of “infoblessing” that could persuade some AGI to spare us as a kind of Superrational Signalling. While you might assume this only works if aliens are probable, it also functions in a multi-agent scenario where there are several AGI at near peer levels of power. It also potentially could be a way to align a previously unaligned AGI even after it is deployed. If enough AGIs are aligned in this way, their alliance could defeat the unaligned AGIs.
You can run the simulation yourself here: https://paxscientia.com/power/
I have the code and initial analysis here: https://github.com/josephius/power
I realize that a very obvious critique of this work is that the simulation is probably too simple. I intentionally tried to keep it an MVP in its first iteration. I also should, as mentioned earlier, complete a more thorough and rigorous analysis of the apparent results. I’m also keenly aware that it seems like this is a “neglected” path towards alignment, and I’m uncertain whether this is because the idea is a bad one that’s already been discarded by others who are more competent. I know that there are related ideas around Decision Theory, Acausal Trade, and Superrationality, but I’ve never seen this particular kind of effort, which confuses me, because it seems obvious and trivial to try.
My main question to ask is simply, does this seem like something worth pursuing and expanding further, or am I wasting my time on a foolish endeavour?
Is the main idea here that if “have you ever defected against a non-agressor” is the agreed way to decide who to not cooperate with, then cooperative strategies win and then stay cooperative when power imbalances are large? Is there a reason would agents choose to use that instead of any other decision criteria? Also, I wouldn’t say that the obvious critique is that your model is too simple, I would say it’s that your model can easily be unrealistic if you don’t try very hard to check it. Good game theory models are sometimes very simple.
That is sorta the idea yes. Agents would choose this decision criteria mostly because it vastly increases their odds of survival, which allows them to further whatever goals they have. I would hope that this result is obvious enough that many agents will be able to converge on it, increasing the proportion using it, and thus increasing the overall survival rate.
The other takeaway is that, given that humans will be weaker than AGI/ASI, any game theoretic reason for such entities to still cooperate with us can potentially help reduce the existential risk.
I agree that the model requires further scrutiny to determine if it is realistic enough to matter.