That said, fwiw, since Iâm recommending Holdenâs doc, I should also flag that I think the breakdown of possible outcomes that Holden sketches there isnât a good one, because:
He defines utopia, dystopia, and âmiddling worldsâ solely by how good they are, whereas âpaperclippingâ is awkwardly squeezed in with a definition based on how it comes about (namely, that itâs a world run by misaligned AI). This leads two two issues in my view:
I think the classic paperclipping scenario would itself be a âmiddlingâ world, yet Holden frames âpaperclippingâ as a distinct concept from âmiddlingâ worlds.
Misaligned AI actually need not lead to something approx. as good/âbad as paperclipping; it could instead lead to dystopia, or could maybe lead to utopia, depending on how we define âalignmentâ and depending on metaethics.
Thereâs no explicit mention of extinction.
I think Holden is seeing âpaperclippingâ as synonymous with extinction?
But misaligned AI need not lead to extinction.
And extinction is âmiddlingâ relative to utopia and dystopia.
And extinction is also very different from some other âmiddlingâ worlds according to many ethical theories (though probably not total utilitarianism).
The author or readers might also find the following interesting:
Flourishing futures (a list of resources on that topic)
Holden Karnofskyâs call for people to think about âHow should we value various possible long-run outcomes relative to each other?â and his notes on why and how to do so[1]
That said, fwiw, since Iâm recommending Holdenâs doc, I should also flag that I think the breakdown of possible outcomes that Holden sketches there isnât a good one, because:
He defines utopia, dystopia, and âmiddling worldsâ solely by how good they are, whereas âpaperclippingâ is awkwardly squeezed in with a definition based on how it comes about (namely, that itâs a world run by misaligned AI). This leads two two issues in my view:
I think the classic paperclipping scenario would itself be a âmiddlingâ world, yet Holden frames âpaperclippingâ as a distinct concept from âmiddlingâ worlds.
Misaligned AI actually need not lead to something approx. as good/âbad as paperclipping; it could instead lead to dystopia, or could maybe lead to utopia, depending on how we define âalignmentâ and depending on metaethics.
Thereâs no explicit mention of extinction.
I think Holden is seeing âpaperclippingâ as synonymous with extinction?
But misaligned AI need not lead to extinction.
And extinction is âmiddlingâ relative to utopia and dystopia.
And extinction is also very different from some other âmiddlingâ worlds according to many ethical theories (though probably not total utilitarianism).