I’m curating this (although I wish it had a more skimmable summary).
It’s an important topic (and a weak point in the classic most important century discussion) and a lot of the considerations[1] seem important and new (at least to me!). I like that the post and document make a serious attempt at clarifying what isn’t being said (like some claims about likelihood), flag different levels of uncertainty in the various claims, and clarify what is meant by “AGI”[2].
Here’s a quick attempt at a restructured/slightly paraphrased summary — please correct me if I got something wrong:
Assuming AGI, it’s relatively possible[3] to stabilize/lock in many features of society — both good and bad — for a long time (millions or trillions of years). This is because:
AGIs can be faithful to a specific goal or a set of goals for a long time
with sufficient resources, institutions can be created that will pursue these values until an external source (like foreign intervention, the death of an authoritarian leader, or internal rebellion) stops them
current economic and military powers could come together and use AGI to make an institution of this kind, which would be able to defend itself against external sources
Meaning that such an institution could pursue its agenda for millions or trillions of years.
Why this matters:
Stabilizing features like this can be bad (an existential risk) if their values or goals are poorly chosen or insufficiently flexible.
Stable institutions could be important to ensuring good values do persist.
The feasibility of the above is evidence that “significant influence over the long-run future is possible.”
Thanks Lizka. I think about section 0.0 as being a ~1-page summary (in between the 1-paragraph summary and the 6-page summary) but I could have better flagged that it can be read that way. And your bullet point summary is definitely even punchier.
I’m curating this (although I wish it had a more skimmable summary).
It’s an important topic (and a weak point in the classic most important century discussion) and a lot of the considerations[1] seem important and new (at least to me!). I like that the post and document make a serious attempt at clarifying what isn’t being said (like some claims about likelihood), flag different levels of uncertainty in the various claims, and clarify what is meant by “AGI”[2].
Here’s a quick attempt at a restructured/slightly paraphrased summary — please correct me if I got something wrong:
Assuming AGI, it’s relatively possible[3] to stabilize/lock in many features of society — both good and bad — for a long time (millions or trillions of years). This is because:
AGIs can be faithful to a specific goal or a set of goals for a long time
with sufficient resources, institutions can be created that will pursue these values until an external source (like foreign intervention, the death of an authoritarian leader, or internal rebellion) stops them
current economic and military powers could come together and use AGI to make an institution of this kind, which would be able to defend itself against external sources
Meaning that such an institution could pursue its agenda for millions or trillions of years.
Why this matters:
Stabilizing features like this can be bad (an existential risk) if their values or goals are poorly chosen or insufficiently flexible.
Stable institutions could be important to ensuring good values do persist.
The feasibility of the above is evidence that “significant influence over the long-run future is possible.”
e.g. what kinds of information/values could be stored for a long time and how
See here.
For the different levels of confidence the authors have in these arguments, you can look at this section in the document.
Thanks Lizka. I think about section 0.0 as being a ~1-page summary (in between the 1-paragraph summary and the 6-page summary) but I could have better flagged that it can be read that way. And your bullet point summary is definitely even punchier.