Unfortunately I worry that paperclips might exist at an un-de-throne-able optimum point of memetic sharability vs educationalness.
The great thing about the paperclip meme is that it is an AI safety Koan—it helps people learn about orthogonality, because until you until you understand orthogonality the story sounds totally insane, but then once you make the leap then everything fits perfectly. (Similarly, IMO Koans are supposed to be stories that make a kind of sense to a spiritually attained person but sound ridiculous to anyone else, thus helping people become attained.)
By contrast, other more realistic stories like this one, hew closer to existing Hollywood stories about evil robots, mad scientists, and genies granting tricky wishes. Even though this story is written with perfect understanding of everything that “paperclips” is about, the very plausibility of this story detracts from the Koan-like ability to teach people “AI safety isn’t about fear of evil overlords, it is literally a technical problem”.
IMO, instead of trying to dethrone paperclips as the snappiest and most memetic story, what the AI safety movement could use is more immersive, detailed, and well-written stories basically all along the pareto frontier of memability vs realism that we can point people to after they’ve heard about paperclips. Paul Christiano’s detailed future-histories of AI slowly derailing humanity’s future might exist at the far end of realism, but it’s long and very sophisticated… but it would be sweet to have some more-memeable, medium-realistic, medium-length, well-produced tales in between that and “paperclips”.
Anyways, I don’t mean to pick on your story at all, it just prompted some musings. You might enjoy the horror stories of Zero Hp Lovecraft, including “The Gig Economy” and his more recent “Don’t Make Me Think”, about BCIs.
Pick away, I appreciate the engagement! It’s interesting to hear what this prompts from you. Orthoganality is a curious thing, and I would hope to get better at such things because one insecurity I had while writing this piece was that I don’t want to be so ‘Black Mirror’esque as to be ‘following the zeitgeist’; this was a thought experiment of my own that isn’t supposed to be technodisasterist, rather a cautionary tale.
Although I read a book in Japanese about Zen (Zen and Star Wars), I never internalised the idea of ko-an (a new word for me), so thank you for that.
New EA Cause Area: “dethrone paperclips”!
Unfortunately I worry that paperclips might exist at an un-de-throne-able optimum point of memetic sharability vs educationalness.
The great thing about the paperclip meme is that it is an AI safety Koan—it helps people learn about orthogonality, because until you until you understand orthogonality the story sounds totally insane, but then once you make the leap then everything fits perfectly. (Similarly, IMO Koans are supposed to be stories that make a kind of sense to a spiritually attained person but sound ridiculous to anyone else, thus helping people become attained.)
By contrast, other more realistic stories like this one, hew closer to existing Hollywood stories about evil robots, mad scientists, and genies granting tricky wishes. Even though this story is written with perfect understanding of everything that “paperclips” is about, the very plausibility of this story detracts from the Koan-like ability to teach people “AI safety isn’t about fear of evil overlords, it is literally a technical problem”.
IMO, instead of trying to dethrone paperclips as the snappiest and most memetic story, what the AI safety movement could use is more immersive, detailed, and well-written stories basically all along the pareto frontier of memability vs realism that we can point people to after they’ve heard about paperclips. Paul Christiano’s detailed future-histories of AI slowly derailing humanity’s future might exist at the far end of realism, but it’s long and very sophisticated… but it would be sweet to have some more-memeable, medium-realistic, medium-length, well-produced tales in between that and “paperclips”.
Anyways, I don’t mean to pick on your story at all, it just prompted some musings. You might enjoy the horror stories of Zero Hp Lovecraft, including “The Gig Economy” and his more recent “Don’t Make Me Think”, about BCIs.
Pick away, I appreciate the engagement! It’s interesting to hear what this prompts from you. Orthoganality is a curious thing, and I would hope to get better at such things because one insecurity I had while writing this piece was that I don’t want to be so ‘Black Mirror’esque as to be ‘following the zeitgeist’; this was a thought experiment of my own that isn’t supposed to be technodisasterist, rather a cautionary tale.
Although I read a book in Japanese about Zen (Zen and Star Wars), I never internalised the idea of ko-an (a new word for me), so thank you for that.