Claim: Credible plans for a “pivotal act” may drive AI race dynamics
(Epistemic status: I had mathematica do all the grunt work and did not check the results carefully)
Consider a simple normal-form game with two equally capable agents A and B, each of which is deciding whether to aggressively pursue AI development, and three free parameters:
the probability that accelerating AI development results in an existential catastrophe (with utility −1 for both agents, versus a utility 0 status quo).
the utility of developing the first friendly AI
the utility of the other agent developing friendly AI
We’ll first assume the coin only gets flipped once: developing a friendly AI lets you immediately control all other AI development.
Since our choice of parameterization was in retrospect one that requires a lot of typing, we’ll define , and then rescale to get something more readable
(Accelerate, Accelerate) is always a Nash equilibrium, no matter how trivial the differences captures are.
(Don’t, Don’t) is a Nash equilibrium when , as you would expect
(Don’t, Don’t) is never a trembling-hand equilibrium, since (Don’t) does not weakly dominate (Accelerate) for either player.
When (Accelerate) weakly dominates (Don’t) and (Accelerate, Accelerate) is a trembling-hand equilibrium.
Now consider the case where (Accelerate, Accelerate) instead flips two coins.
This is potentially a much safer situation:
(Accelerate, Accelerate) is only a Nash equilibrium when
(Don’t, Don’t) is still a Nash equilibrium when
(Don’t, Don’t) is a trembling-hand equilibrium if it’s a Nash equilibrium and (Accelerate, Accelerate) is not.
It’s an artifact. What’s really being measured here is EA’s skew towards educated urban professionals, for whom identifying as “left” or “center left” is a signal of group membership. The actual dominant ideology is, broadly speaking, technocratic liberalism: the same cluster as “center-right” Rockefeller Republicans, “centrist” New Democrats, or “center-left” LibDems, but qualitatively different from the “center-right” CDU or “center-left” New Dealers.
Getting Christian conservatives to allocate their donations more effectively would be an extremely good thing but it will never ever happen under the aegis of EA. You would need a totally separate cultural infrastructure, built for (and probably by) communitarians instead of liberal universalists.