Goal preservation is the idea that an agent or civilization might eventually prevent goal drift over time, except perhaps in cases where its current goals approve of goal changes. While consequentialist agents have strong incentive to work toward goal preservation, implementing it in non-trivial, and especially in chaotic, systems seems very difficult. It’s unclear to me how likely a future superintelligent civilization is to ultimately preserve its goals. Even if it does so, there may be significant goal drift between the values of present-day humans and the ultimate goals that a future advanced civilization locks in.
In set theory, a singleton is a set with only one member, but as I introduced the notion, the term refers to a world order in which there is a single decision-making agency at the highest level.[1] Among its powers would be (1) the ability to prevent any threats (internal or external) to its own existence and supremacy, and (2) the ability to exert effective control over major features of its domain (including taxation and territorial allocation).
Thanks to Ben for doing this AMA, and to Elliot for this interesting set of questions!
Just wanted to mention two links that readers might find interesting in this context. Firstly, Tomasik’s Will Future Civilization Eventually Achieve Goal Preservation? Here’s the summary:
Secondly, Bostrom’s What is a Singleton? Here’s a quote: