I think the type of early deal that would be most valuable is where the US and China both agree to produce a joint āconsensusā ASI aligned to āthe goodā. In more detail:
The US and China, as you note, are unsure who will win, and would be better off making a deal to preserve some minimum amount of future influence. But I think I am more worried than you about the costs of continued multipolarity into space colonisation. You write āEven having two alternative systems might open up the possibility for comparison, healthy competition, and moral trade.ā War, threats, and unhealthy (e.g., burning the cosmic commons) competition also seem like important possibilities here.
Instead, I think having a joint superintelligence that coordinates using our cosmic endowment would be better, with some amount of influence within the āmoral parliamentā of the ASI for each of the US and China.
Just that would be preferable to dividing up the universe into two camps I thinkāit is easier to do moral trades within one agent acting under moral uncertainty than coordinating between two agents.
A better version, though, could involve the US and China agreeing on some core moral precepts, or just a moral reflection process, and then jointly designing a moral curriculum for the proto-ASI including plenty of Western and Chinese texts, and letting the ASI do as it sees fit. Presumably both sides genuinely believe they are right and that an appropriate moral training process for the AI will lead to liberalism/āSocialism with Chinese characteristics. So this exploits the two sides having different credences (where as you note your proposed deals are possible even if both sides have the same credences). This creates a larger surplus for posisble agreements.
Of course, agreeing to create a joint ASI could also have big nearer term benefits, e.g. avoiding racing and slowing down AI progress and investing more in safety.
This proposal is clearly very far outside the overton window currently, but I donāt think this is that much worse on feasibility than your proposed great power resource-sharing deals. It also solves the enforcement challenge as well which is convenient since we might have needed to create such a consensus AI to enforce a different sort of deal.
I am tentatively excited about this proposal, but I expect there isnāt much to do to further it until the relevant parties are taking things more seriously.
Nod. Plus, another advantage of your āconsensus ASIā approachāwhich is essentially a values handshakeāover the deal types outlined by OP is that combined-US-China presents a unified front if and when third-party alien civilizations are encountered.
(A āunified frontā is an advantage if military power, and thus bargaining power, scales superlinearly in the deep future. Which seems >50% likely to me.)
If the US and China are in a state where theyāre willing to cooperate on ASI, I would much prefer that they agree not to build ASI (until thereās a broad consensus that we know how to make it safely).
If they agree to that, and we do eventually figure out how to build aligned ASI, then it would be good to have a global agreement on what that ASI should do. But if weāre going to do work today to work toward some sort of international cooperation on ASI, then the objective of that cooperation should be to not build ASI.
I think the type of early deal that would be most valuable is where the US and China both agree to produce a joint āconsensusā ASI aligned to āthe goodā. In more detail:
The US and China, as you note, are unsure who will win, and would be better off making a deal to preserve some minimum amount of future influence. But I think I am more worried than you about the costs of continued multipolarity into space colonisation. You write āEven having two alternative systems might open up the possibility for comparison, healthy competition, and moral trade.ā War, threats, and unhealthy (e.g., burning the cosmic commons) competition also seem like important possibilities here.
Instead, I think having a joint superintelligence that coordinates using our cosmic endowment would be better, with some amount of influence within the āmoral parliamentā of the ASI for each of the US and China.
Just that would be preferable to dividing up the universe into two camps I thinkāit is easier to do moral trades within one agent acting under moral uncertainty than coordinating between two agents.
A better version, though, could involve the US and China agreeing on some core moral precepts, or just a moral reflection process, and then jointly designing a moral curriculum for the proto-ASI including plenty of Western and Chinese texts, and letting the ASI do as it sees fit. Presumably both sides genuinely believe they are right and that an appropriate moral training process for the AI will lead to liberalism/āSocialism with Chinese characteristics. So this exploits the two sides having different credences (where as you note your proposed deals are possible even if both sides have the same credences). This creates a larger surplus for posisble agreements.
Of course, agreeing to create a joint ASI could also have big nearer term benefits, e.g. avoiding racing and slowing down AI progress and investing more in safety.
This proposal is clearly very far outside the overton window currently, but I donāt think this is that much worse on feasibility than your proposed great power resource-sharing deals. It also solves the enforcement challenge as well which is convenient since we might have needed to create such a consensus AI to enforce a different sort of deal.
I am tentatively excited about this proposal, but I expect there isnāt much to do to further it until the relevant parties are taking things more seriously.
Nod. Plus, another advantage of your āconsensus ASIā approachāwhich is essentially a values handshakeāover the deal types outlined by OP is that combined-US-China presents a unified front if and when third-party alien civilizations are encountered.
(A āunified frontā is an advantage if military power, and thus bargaining power, scales superlinearly in the deep future. Which seems >50% likely to me.)
If the US and China are in a state where theyāre willing to cooperate on ASI, I would much prefer that they agree not to build ASI (until thereās a broad consensus that we know how to make it safely).
If they agree to that, and we do eventually figure out how to build aligned ASI, then it would be good to have a global agreement on what that ASI should do. But if weāre going to do work today to work toward some sort of international cooperation on ASI, then the objective of that cooperation should be to not build ASI.