Is the main idea here that if “have you ever defected against a non-agressor” is the agreed way to decide who to not cooperate with, then cooperative strategies win and then stay cooperative when power imbalances are large? Is there a reason would agents choose to use that instead of any other decision criteria? Also, I wouldn’t say that the obvious critique is that your model is too simple, I would say it’s that your model can easily be unrealistic if you don’t try very hard to check it. Good game theory models are sometimes very simple.
That is sorta the idea yes. Agents would choose this decision criteria mostly because it vastly increases their odds of survival, which allows them to further whatever goals they have. I would hope that this result is obvious enough that many agents will be able to converge on it, increasing the proportion using it, and thus increasing the overall survival rate.
The other takeaway is that, given that humans will be weaker than AGI/ASI, any game theoretic reason for such entities to still cooperate with us can potentially help reduce the existential risk.
I agree that the model requires further scrutiny to determine if it is realistic enough to matter.
Is the main idea here that if “have you ever defected against a non-agressor” is the agreed way to decide who to not cooperate with, then cooperative strategies win and then stay cooperative when power imbalances are large? Is there a reason would agents choose to use that instead of any other decision criteria? Also, I wouldn’t say that the obvious critique is that your model is too simple, I would say it’s that your model can easily be unrealistic if you don’t try very hard to check it. Good game theory models are sometimes very simple.
That is sorta the idea yes. Agents would choose this decision criteria mostly because it vastly increases their odds of survival, which allows them to further whatever goals they have. I would hope that this result is obvious enough that many agents will be able to converge on it, increasing the proportion using it, and thus increasing the overall survival rate.
The other takeaway is that, given that humans will be weaker than AGI/ASI, any game theoretic reason for such entities to still cooperate with us can potentially help reduce the existential risk.
I agree that the model requires further scrutiny to determine if it is realistic enough to matter.