Thanks for all the questions and comments! I’ll answer this one in stages.
On your first question:
Do you agree that the goodness of this analogy is roughly proportional to how slow our AI takeoff is? For instance if the first AGI ever created becomes more powerful than the rest of the world, then it seems that anyone who influenced the properties of this AGI would have a huge impact on the future.
I agree with this.
To take the fairly extreme case of the Neolithic Revolution, I think that there are at least a few reasons why groups at the time would have had trouble steering the future. One key reason is what the world was highly “anarchic,” in the international relations sense of the term: there were many different political communities, with divergent interests and a limited ability to either coerce one another or form credible commitments. One result of anarchy is that, if the adoption of some technology or cultural/institutional practice would give some group an edge, then it’s almost bound to be adopted by some group at some point: other groups will need to either lose influence or adopt the technology/innovation to avoid subjugation. This explains why the emergence and gradual spread of agricultural civilization was close to inevitable, even though (there’s some evidence) people often preferred the hunter-gatherer way of life. There was an element of technological or economic determinism that put the course of history outside of any individual group’s control (at least to a significant degree).
Another issue, in the context of the Neolithic Revolution, is that norms, institutions, etc., tend to shift over time, even in there aren’t very strong selection pressures. This was even more true before the advent of writing. So we do have a few examples of religious or philosophical traditions that have stuck around, at least in mutated forms, for a couple thousand years; but this is unlikely, in any individual case, and would have been even more unlikely 10,000 years ago. At least so far, we also don’t have examples of more formal political institutions (e.g. constitutions) that have largely stuck around for more than few thousand years either.
There are a couple reasons why AI could be different. The first reason is that—under certain scenarios, especially ones with highly discontinuous and centralized progress—it’s perhaps more likely that one political community will become much more powerful than all others and thereby make the world less “anarchic.” Another is that, especially if the world is non-anarchic, values and institutions might naturally be more stable in a heavily AI-based world. It seems plausible that humans will eventually step almost completely out of the loop, even if they don’t do this immediately after extremely high levels of automation are achieved. At this point, if one particular group has disproportionate influence over the design/use of existing AI systems, then that one group might indeed have a ton of influence over the long-run future.
Goal preservation is the idea that an agent or civilization might eventually prevent goal drift over time, except perhaps in cases where its current goals approve of goal changes. While consequentialist agents have strong incentive to work toward goal preservation, implementing it in non-trivial, and especially in chaotic, systems seems very difficult. It’s unclear to me how likely a future superintelligent civilization is to ultimately preserve its goals. Even if it does so, there may be significant goal drift between the values of present-day humans and the ultimate goals that a future advanced civilization locks in.
In set theory, a singleton is a set with only one member, but as I introduced the notion, the term refers to a world order in which there is a single decision-making agency at the highest level.[1] Among its powers would be (1) the ability to prevent any threats (internal or external) to its own existence and supremacy, and (2) the ability to exert effective control over major features of its domain (including taxation and territorial allocation).
Hi Elliot,
Thanks for all the questions and comments! I’ll answer this one in stages.
On your first question:
I agree with this.
To take the fairly extreme case of the Neolithic Revolution, I think that there are at least a few reasons why groups at the time would have had trouble steering the future. One key reason is what the world was highly “anarchic,” in the international relations sense of the term: there were many different political communities, with divergent interests and a limited ability to either coerce one another or form credible commitments. One result of anarchy is that, if the adoption of some technology or cultural/institutional practice would give some group an edge, then it’s almost bound to be adopted by some group at some point: other groups will need to either lose influence or adopt the technology/innovation to avoid subjugation. This explains why the emergence and gradual spread of agricultural civilization was close to inevitable, even though (there’s some evidence) people often preferred the hunter-gatherer way of life. There was an element of technological or economic determinism that put the course of history outside of any individual group’s control (at least to a significant degree).
Another issue, in the context of the Neolithic Revolution, is that norms, institutions, etc., tend to shift over time, even in there aren’t very strong selection pressures. This was even more true before the advent of writing. So we do have a few examples of religious or philosophical traditions that have stuck around, at least in mutated forms, for a couple thousand years; but this is unlikely, in any individual case, and would have been even more unlikely 10,000 years ago. At least so far, we also don’t have examples of more formal political institutions (e.g. constitutions) that have largely stuck around for more than few thousand years either.
There are a couple reasons why AI could be different. The first reason is that—under certain scenarios, especially ones with highly discontinuous and centralized progress—it’s perhaps more likely that one political community will become much more powerful than all others and thereby make the world less “anarchic.” Another is that, especially if the world is non-anarchic, values and institutions might naturally be more stable in a heavily AI-based world. It seems plausible that humans will eventually step almost completely out of the loop, even if they don’t do this immediately after extremely high levels of automation are achieved. At this point, if one particular group has disproportionate influence over the design/use of existing AI systems, then that one group might indeed have a ton of influence over the long-run future.
Thanks to Ben for doing this AMA, and to Elliot for this interesting set of questions!
Just wanted to mention two links that readers might find interesting in this context. Firstly, Tomasik’s Will Future Civilization Eventually Achieve Goal Preservation? Here’s the summary:
Secondly, Bostrom’s What is a Singleton? Here’s a quote: