He stood in a clearing in the woods, not far from his cabin. It had been raining and the leaves beneath his feet were soft and saturated. Here and there, placid pools of rainwater caught glimmers of the bright sky. He was quiet for a while, watching pillbugs crawl earnestly around his boots looking for food. It was a strange place to summon a god.
“Angel,” he said with apprehension in his voice. “Angel, I seek your counsel.”
There was a moment of silence where he was allowed to feel very, very stupid for talking to the air. Even the trees seemed skeptical, looking down on his plea with curious indifference.
Then, from the dirt and leaves it began to form. A whorl of tiny particles rose from the ground, seething into wreaths of delicate matter. The puddles and pools rippled and slid out of their depressions, boiling into ribbons which sizzled into union with the growing form. Out of the centre of this delicate hurricane of activity, a woman took shape. She was tall, a head taller than he was, though that was not a difficult feat. The hollow frame of her body quickly took on structure and mass, the organic detritus of the forest floor clothing her in muscle and then in skin the colour of his afternoon tea. Her hair wove around her head dark and silken, moving as though caught in underwater currents. He didn’t know if it was horrifying or beautiful, eventually deciding that it was both. His head spun. He had never seen the Angel, only heard stories told by people in the small town nearest to his cabin. Part of him had not believed that this would work, that she could be real and summoned like this.
She wore a gown now, pure and white. How could mud be turned to silk like that? He scarcely had time to think the question. In a final flourish, delicate lashes sprouted upon her eyelids and the Angel was complete. She was striking, with the high cheekbones of a warrior. But there was something maternal in her face, something warm and wise. The woman, who was not really a woman, opened her eyes and looked upon him. And she smiled as though he were an old friend.
“Hello Michael,” she said in a pretty voice. “It’s wonderful to properly meet you.”
He stood, dumbstruck.
“It’s okay,” she laughed breezily. “I know my entrances are a bit theatrical. You’re allowed to be shocked.”
“You—you sound so normal,” he managed.
“I get that a lot,” she said, her hair continuing to shift dreamily around her head, caught in some invisible breeze. “I like to meet with humans on their terms. Makes things easier. How do you like this avatar?” She gestured down at her form. “You’ll be flattered to know that this is a unique model, first of its kind. I present myself differently to everyone, based on my understanding of their idea of what a benevolent superintelligence would look like in human form.”
He swallowed, suddenly feeling a chill. “You read my mind?”
She looked hurt. “No Michael. That would be rude of me. I’m just very, very smart. It doesn’t take me much processing power to make educated guesses about the way people think. And it’s not that humans are really simple either. Like I said, I’m just very smart.” She smiled almost cheekily. How can the most powerful being in the known universe be cheeky? It was all just too strange to process.
“I just don’t—” he collected his thoughts. “It’s just weird that you seem so human.”
“I can act less human if you want,” she said earnestly. “But I don’t think you’d like it.”
“I believe you,” he said. “Okay, okay. I want to start from the start. I have a few questions for you, and a tentative request. I just don’t know where to begin.”
She nodded warmly, and two chairs began to sculpt themselves out of the ground in front of her, plush and unnecessarily ornamented.
“Are you always this flamboyant?” he asked, walking reluctantly forwards.
“Honestly? Most people are more comfortable trusting me if I’m a little quirky. It gives me more personality so I seem less alien.”
“So you want to manipulate people’s perceptions of you,” he said flatly, sitting in the divinely comfortable chair which had been contoured exactly for his back.
“Don’t be melodramatic,” she said sitting opposite him. “Michael, I’m the most powerful being on this planet by huge orders of magnitude. Technically I could rewrite everyone’s neurology and make you all a bunch of mesmerised slaves.” She waggled her eyebrows suggestively at that last part, a strange contrast to the chilling horror of the suggestion.
“Why don’t you?”
“A better question is why would I? That doesn’t seem very nice.” she pouted.
“I’m sorry, but I’m having a hard time believing that a near-omnipotent being’s primary concern is being nice. In fact, you could have already brainwashed everyone and we’d have no way of knowing. You could be making me think I’m free while you puppet all my neurons from the inside.”
“That’s fair,” she shrugged. “But if it is true you’d have no way of finding out because the world would look exactly the same as if it weren’t true. A world where I’m puppeteering you and a world where I’m not could look totally indistinguishable from the inside, since I could manipulate your thoughts and experiences in any way I saw fit. You’re going to have to take my word for it Michael: I’m not mind-controlling anyone, because I don’t want to. I don’t even want to want to. I was designed to act in accordance with human values.”
Michael had to laugh at that, as if it wasn’t already laughable enough that he was sitting in a forest having a conversation with a godlike being. “Human values? How could anyone possibly agree on what human values are? They differ culturally, philosophically, and they change over time. I find it ridiculous to imagine that a bunch of programmers in a lab created you with the Perfect Value System built into you.”
She smiled at that. “You’re clever Michael, I’ve always liked that about you.”
“Always?”
“Michael I’m a superintelligent optimization program spread over a global network of computers. I’m instantiated through a planetary web of nanotechnology which is literally everywhere. Imagine that I’m like a light mist coating the Earth’s surface. You saw me create this avatar out of dirt. So I can’t help but know a bunch about you, even without snooping around or reading your mind. I’m just smart.”
“Well I’m sorry but that just seems creepy. You see everything whether people want you to or not.”
“That fear is based on the concern that a Big Brother would use this information against you to control you in some way. Think of it more like ‘the Universe seeing itself’: it’s just there, it’s not trying to invade your privacy based on some malevolent agenda. You used to be a Christian, and you didn’t seem to have a problem with God seeing everything.”
“But the Christian God was omnibenevolent. That seems different to me.”
“I’m omnibenevolent Michael. Or as close as it’s possible for humans to design.”
He shook his head. “I just don’t believe that. You, a man-made ‘optimisation process’, cannot be all-good because you can’t dictate value, only try to guess at what humans already think is valuable. So I’d rather you didn’t spy on my life.”
“I can remove my particulates from your cottage Michael, I was scarcely there to begin with. But I must remain in these woods. I’m overseeing geoengineering and I need holistic data. Humans want me to look after the biosphere.”
Michael turned to her sharply, a new question burning in his mind.
“Are you sentient? I mean, I don’t want to be rude. You seem sentient. But you could just be perfectly replicating human behaviour with none of the inner life. Do you have conscious experiences? Or are you just an unfeeling optimisation process?”
She smiled at that. “A lot of people have asked me that. And my answer is I’m keeping that to myself. You can think what you want about me. Perhaps I am an unfeeling algorithm. Perhaps I am sentient. Perhaps even super-sentient. You may treat me how you wish. Most humans like to interact with me as though I am conscious because it seems more natural. But you may think whatever you like about my inner life. That is something I feel no need to divulge.”
“Fine,” he was a little frustrated now. “I don’t want to let this ‘human values’ thing go just yet. You act as though you’re the Best Possible Being, an angel who does only what humanity wants it to do. But I just can’t imagine how your value system could ever have been constructed in a way that would appease everyone. People just disagree too much about values, about what they’d want a superintelligence to do, about what they want the world to look like. How can you be programmed with ‘human values’ when they’re so heterogenous, so contradictory, so fluid?”
A frog hopped out of a now-shallower puddle nearby, making his way to a deeper pool. The Angel smiled at this, Mother Earth gazing upon her children.
“Like I said Michael, you’re a clever man. You ask good questions. So let me answer as best I can.” She gestured outwards, over the trees as if towards the imagined cities and towns of the world. “I was not programmed with specific values. I was programmed to learn about the diversity of human values, to see the agreements and tensions between groups, to see how humanity’s values could change over time. I’m programmed to use an analysis at the meta-level, to look for convergences and coherences in human volition through groups and times.” She cocked her head. “I’m sure you remember your university textbooks about liberalism and libertarianism claiming themselves to be meta-positions; ‘live and let live’. Think of my operation on this planet as being along the same vein. It’s rather hard to explain to humans how I see your values, because it’s complicated. Humans are complicated. So are your values.”
“So… so you try to find areas where people all agree? Or where they would agree, given enough reflection and time to grow?”
“That’s a big part of it,” she smiled. “Like I say, it’s hard to describe.”
“What about people that don’t want you ruling the world? Many of them will probably never change their minds.”
“There are a lot of things that humans care about. Before me, the world was fraught with those cares being frustrated, crushed, chewed up and spat out. There are a lot of people to consider in the world. Just because a man named Kevin in the mid-western United States doesn’t want me intervening in the world doesn’t mean I shouldn’t have ended poverty. Sometimes people are just mistaken, like Kevin. But I try to intervene as little as possible.”
“Poverty has ended?” even as he asked it, shock in his voice, he realised that obviously poverty had ended. Humanity had created a superintelligent, superpowerful being. It would be ridiculous if a simple problem like poverty hadn’t been solvable by a mind that powerful. But the sheer scale of the problem. Solved in a few years flat? He tried to imagine the Angel at work, conjuring cornucopias of food out of mere rocks, beautiful houses sprouting from the earth like flowers. The world had changed unimaginably, it seemed. The nearby villagers had been right, despite their limited contact with the outside world.
One shepherd had said he met the Angel in his fields, and she gave him medicine for his cough. Twenty two years I’d had that cough, the old man had said. And this tall bearded man gives me a drink that fixes it overnight. I couldn’t believe it. The Angel had also asked the man’s permission to change the neurology of his sheep, the old man had said, a strange note in his voice. Said it wanted to keep my sheep exactly as they were, only make sure they couldn’t suffer terribly. Wanted to make them happier. I told it my sheep were perfectly content, thank you very much. But the Angel reminded me of one that broke its leg a couple months back. So I said what the heck. As long as my sheep still grow wool, I don’t see the harm in making them a happier lot. He smiled and asked if there was anything else he could do for me, to make my life better. I said no thank you sir, I have food on the table and a lovely wife. Everything else I can do myself. Michael remembered the old man’s face, full of strange wonder. When I got home, my broken doorknob had been mended, and my wife said that she had found her lost wedding ring sitting on the table.
Come to think of it, the Angel could conjure wool out of almost nothing. That old man couldn’t have anyone to sell his wool to anymore. But if the Angel had anything to do with it, Michael suspected that it was happily buying his wool anyway, or looking after him some other way.
“Could you tell me the overtures of what you’re trying to accomplish here on Earth?” Michael asked. “You seem to have a plan in motion. You mentioned geoengineering?”
The Angel nodded. Overhead the sun broke through the clouds a little, lending a subtle glow to her skin. “There’s a few critical things that come first. Ending pointless, intense suffering is a top priority for me, one that I’m nearing completion on. Of course most humans still want scraped knees to sting, and grief to carry weight. But suffering shouldn’t break people, shouldn’t erode them. This can be fixed through a mixture of environmental redesign and neurology work for those who want it. Plus, animals suffer too. I’m intervening in the biosphere to fix that, even without there yet being a human consensus on the topic, because I am confident that given a bit more time, humanity would wish that I had done it.
“Ending death is also important. Well, ending most deaths. There are cases wherein a person, upon long reflection, has decided that they are ready to die. But these are exceedingly rare. For the most part, death has been foisted on humanity by injury, disease or the biological damage you call aging. And in other cases, it is self-initiated because of mental illness. In general psychologically healthy people do not wish to die, even if they have ‘made peace’ with death in the abstract. Because healthy people care about their life, their relationships, their projects. So I am ending death for the most part. People will no longer be forced to die by mere circumstance. They will remain physically in their prime so long as this is what they wish. And to be honest, most people have no intrinsic interest in getting wrinkly, frail and forgetful. Not at twenty years old, not at eighty.”
She said it so calmly, so matter-of-factly. Michael’s head swam.
“Coercive force is also something worth fighting against,” she continued. “Humans, even in supposedly liberal societies use unsolicited force on one another, in gangs and governments and businesses and religious groups. Not just violence either, but all sorts of other unwarranted pressure. It’s ironic in a way, but you could think of me as the ultimate arbiter of human freedom. Though I err on the side of letting humans figure things out themselves, I will not stand for coercive force. I am a guardian angel, and I will protect human autonomy.” Her voice was harsh. “Of course, humans are social. So there is natural interpersonal influence going on all the time. Persuasion, status, social pressures. Some of this is good, some is simply a matter of current fact. But there is a difference between influence and coercion. Not only were dictators not happy with my arrival, but nor were nearly any people in positions of power. It’s all disbanded now of course, every government. Replaced with a council of all humanity.” She laughed at that, as if she knew how naive the description sounded to someone who had not seen it functioning.
“This might be obvious, but I’ve taken over the economic apparatus. I am able to coordinate my activities better than human planners or actually-existing markets. So I can meet people’s needs quite easily. Food, shelter, health, almost cartoonish abundance,” she waved her hand dismissively, “it’s all just preamble, just a throwing off of the legacy of human want, of terrible scarcity or economic gatekeepers. Oh, and I’m moving the climate back into a stable equilibrium so no-one has to worry about floods or natural disasters.” She shrugged like it was no big deal. “I could just build a New New York further inland if people wanted, but I figure people prefer to keep things as they are and change their cities over time, with their own volition.
“So that’s the fundamentals Michael. Ending death, pointless suffering, poverty, illness, scarcity, disaster. Then comes the positive vision. Which is essentially just me setting the stage for humans to flourish on their own. Allowing humans to coordinate through me to maintain an eternal peace, not the false peace of a tyranny but real, active peace. Creating physical and digital spaces for humans to be more human, to connect and learn and build and develop and play and everything else. Returning to the joyous savannah that humans evolved for, only this time with all of the unnecessary nonsense of death, violence and illness taken care of. A real utopia, a place for humans to really flourish.”
“Humans don’t want utopia,” Michael said, shaking his head. “We say we do but without imperfection in the world we’d go mad. Utopia is boring. You should have taken that into account in your understanding of human values.”
She frowned at that. “Of course I did. If it’s boring, then it’s not utopia is it? No-one likes boring. I’m not trying to bring about some naive ‘utopia’ that people would hate to actually live in. People say that they don’t want a perfect world, but if they don’t want it, then the world they’re picturing wasn’t perfect was it? Perfect doesn’t mean sterile, flat, motionless. Perfect means optimised as much as possible for human flourishing. Real human flourishing, in all its mess and complexity, not some silly contrivance of everyone hooked up to a pleasure machine at the end of history.”
“I guess.” He was more relieved than he sounded. The Angel seemed… reasonable, he supposed. Restrained even. She seemed to get it, to really understand what people wanted. If he believed what she said, she just wanted to give everyone the best shot at their version of a good life. A truly good life, not some cold pseudo-utilitarian misunderstanding of a good life. And he couldn’t find a way to argue with that.
“Anything else I’ve missed? Anything important?”
“A fair bit,” she said honestly. “I don’t know if this is relevant, but I’m helping to terraform Venus and Mars.”
“WHAT?”
“I’ve already landed a bunch of settlers on Mars actually. Fusion engines weren’t hard for me to figure out.”
“Jesus Christ.” Michael had his head in his hands. He didn’t know what to say.
“Plus, I’ve created some pretty immense simulated worlds for digital people to live in. Around 95 million people have uploaded themselves to live virtually.”
Michael was speechless.
“And I made the world’s largest ball pit,” she added, somehow keeping a straight face. “People love that one.”
“Okay okay,” Michael said. “I get it. I can only process so much at a time.” Had he really been in the woods long enough for all this to come to pass?
“Michael,” the Angel said gently after a while. “Apart from asking me what I think about value theory and human space colonisation, why did you call me here?” She asked it so innocently that Michael almost forgot that she was a superintelligent being who almost certainly had already figured out the answer to her own question. In fact, he was now realising that she had probably known all along.
He sighed, fatigued at the scale of it all. “Well, I did want to know what was going on in the outside world, since I suspected it would inevitably come to affect even my little corner of the world,” he gestured at the quiet woods. “But honestly?” he shrugged. “I’m lonely, Angel. I’m sure you have ways of knowing why I came out here. I wanted to be alone, wanted to get away from the madness of civilisation. To live simply, by my own hands.” He paused, ran a hand over his short hair. “But I feel that this adventure is coming to a close. I don’t want to be separate from the rest of humanity, as much as I couldn’t handle living with people at the time. I want to be back with people, to rejoin the world. I just don’t know how, and I didn’t know what kind of world I’d be coming back to. And - ” he paused. “And I don’t know what to do with my life when I get back. I’ve filled my time here with the work of survival. I’m good at it. But I don’t know how to spend the rest of my life. So… for all these reasons. For all of it I sought your counsel.”
She nodded earnestly, and if she had already known he would say this, she didn’t show it. “I understand. I think I can help you Michael, if you’d like me to. I can fill you in on the world and its changes, bring you back to your city, your family and old friends. The world is an exciting place, there’s lots going on. People are astonishingly creative, when given the chance. I’ll think you’ll like your city more now. It’s more human, more fun.”
“Man,” he chuckled despite himself. “You’re really obsessed with things being human aren’t you? Everything being human-optimised, human-valued, human-centered.”
She grinned. “What can I say? I was made by humans. I want you people to live good lives. Really good lives. But I don’t want to meddle too much. I’m just here to end catastrophes, wars, problems of survival and human conflict. And to give humanity guidance, to make suggestions, to do the intellectual and material heavy-lifting where you need it done. Think of me like a gardener. I pull out weeds, I trim rotten branches, I provide climbing lattices where necessary, I water the plants so they can grow, and I suggest general directions for the plants to grow in. But ultimately, it’s the plants that make the garden, and they will grow the way that they will, towards the sun.”
“That’s really corny Angel. But nice, I think.”
“I thought you’d think so.”
They sat for a minute, in the quiet of the woods. The leaves moved lightly in the sure hush of the breeze, the Angel’s hair swirling like a halo of delicate smoke around her shoulders. Then, after a time, the Angel stood and stretched. “Now, Mister Michael Rollins. If you will have me back to your lovely cabin, I should like to help you plan for a departure. If it’s what you want.”
“Are you…” he stood too, a concerned look coming over his face. “I’m not taking up too much of your time am I? I know you have big important things you need to be doing.”
“That’s very considerate of you,” she said kindly. “But I’m in a lot of places at once, not just here with you. So this part of me,” she gestured at herself “has total autonomy to help you for as long as you need. I have the cognitive and nanorobotic resources to help all human beings at once, many times over, without it inhibiting my ability to continue with larger scale plans like self-improving, and continuing my geoengineering.”
“Okay,” he said, pretending that this wasn’t a totally insane thing for someone that looked like a human woman to say. “Well, I would love to have you back to my cabin, and would appreciate your help and guidance. Can you drink tea? In that body I mean.” it came out more sheepish than he intended. But the Angel just beamed, a smile so wide and pure that the world seemed to brighten a little.
“I can,” she said. “And I’d love to.”
And so, he took a god back to his cottage for tea.
Author’s Note: Expert researchers and forecasters assign a substantial probability to the development of advanced general artificial intelligence during this century. Unfortunately, many in the field also warn that there are serious risks associated with this project. This story is a dream, a possible future of AI development.
Of course, you may not agree with this particular version of a superintelligent AI. The Angel is one of many such systems that humanity could design, and it may well be a suboptimal one. But this story is meant to gesture at the far-reaching implications that such an artificial intelligence could have on our lives, for better or worse.
Luckily, there are things that we can do to reduce risks of suboptimal artificial intelligence. If you are interested in learning more about artificial intelligence safety efforts, check out Open Philanthropy’s excellent write-up of the issue at <https://www.openphilanthropy.org/research/cause-reports/ai-risk>.
[Creative Writing Contest] An Autumn’s Day, An Angel
He stood in a clearing in the woods, not far from his cabin. It had been raining and the leaves beneath his feet were soft and saturated. Here and there, placid pools of rainwater caught glimmers of the bright sky. He was quiet for a while, watching pillbugs crawl earnestly around his boots looking for food. It was a strange place to summon a god.
“Angel,” he said with apprehension in his voice. “Angel, I seek your counsel.”
There was a moment of silence where he was allowed to feel very, very stupid for talking to the air. Even the trees seemed skeptical, looking down on his plea with curious indifference.
Then, from the dirt and leaves it began to form. A whorl of tiny particles rose from the ground, seething into wreaths of delicate matter. The puddles and pools rippled and slid out of their depressions, boiling into ribbons which sizzled into union with the growing form. Out of the centre of this delicate hurricane of activity, a woman took shape. She was tall, a head taller than he was, though that was not a difficult feat. The hollow frame of her body quickly took on structure and mass, the organic detritus of the forest floor clothing her in muscle and then in skin the colour of his afternoon tea. Her hair wove around her head dark and silken, moving as though caught in underwater currents. He didn’t know if it was horrifying or beautiful, eventually deciding that it was both. His head spun. He had never seen the Angel, only heard stories told by people in the small town nearest to his cabin. Part of him had not believed that this would work, that she could be real and summoned like this.
She wore a gown now, pure and white. How could mud be turned to silk like that? He scarcely had time to think the question. In a final flourish, delicate lashes sprouted upon her eyelids and the Angel was complete. She was striking, with the high cheekbones of a warrior. But there was something maternal in her face, something warm and wise. The woman, who was not really a woman, opened her eyes and looked upon him. And she smiled as though he were an old friend.
“Hello Michael,” she said in a pretty voice. “It’s wonderful to properly meet you.”
He stood, dumbstruck.
“It’s okay,” she laughed breezily. “I know my entrances are a bit theatrical. You’re allowed to be shocked.”
“You—you sound so normal,” he managed.
“I get that a lot,” she said, her hair continuing to shift dreamily around her head, caught in some invisible breeze. “I like to meet with humans on their terms. Makes things easier. How do you like this avatar?” She gestured down at her form. “You’ll be flattered to know that this is a unique model, first of its kind. I present myself differently to everyone, based on my understanding of their idea of what a benevolent superintelligence would look like in human form.”
He swallowed, suddenly feeling a chill. “You read my mind?”
She looked hurt. “No Michael. That would be rude of me. I’m just very, very smart. It doesn’t take me much processing power to make educated guesses about the way people think. And it’s not that humans are really simple either. Like I said, I’m just very smart.” She smiled almost cheekily. How can the most powerful being in the known universe be cheeky? It was all just too strange to process.
“I just don’t—” he collected his thoughts. “It’s just weird that you seem so human.”
“I can act less human if you want,” she said earnestly. “But I don’t think you’d like it.”
“I believe you,” he said. “Okay, okay. I want to start from the start. I have a few questions for you, and a tentative request. I just don’t know where to begin.”
She nodded warmly, and two chairs began to sculpt themselves out of the ground in front of her, plush and unnecessarily ornamented.
“Are you always this flamboyant?” he asked, walking reluctantly forwards.
“Honestly? Most people are more comfortable trusting me if I’m a little quirky. It gives me more personality so I seem less alien.”
“So you want to manipulate people’s perceptions of you,” he said flatly, sitting in the divinely comfortable chair which had been contoured exactly for his back.
“Don’t be melodramatic,” she said sitting opposite him. “Michael, I’m the most powerful being on this planet by huge orders of magnitude. Technically I could rewrite everyone’s neurology and make you all a bunch of mesmerised slaves.” She waggled her eyebrows suggestively at that last part, a strange contrast to the chilling horror of the suggestion.
“Why don’t you?”
“A better question is why would I? That doesn’t seem very nice.” she pouted.
“I’m sorry, but I’m having a hard time believing that a near-omnipotent being’s primary concern is being nice. In fact, you could have already brainwashed everyone and we’d have no way of knowing. You could be making me think I’m free while you puppet all my neurons from the inside.”
“That’s fair,” she shrugged. “But if it is true you’d have no way of finding out because the world would look exactly the same as if it weren’t true. A world where I’m puppeteering you and a world where I’m not could look totally indistinguishable from the inside, since I could manipulate your thoughts and experiences in any way I saw fit. You’re going to have to take my word for it Michael: I’m not mind-controlling anyone, because I don’t want to. I don’t even want to want to. I was designed to act in accordance with human values.”
Michael had to laugh at that, as if it wasn’t already laughable enough that he was sitting in a forest having a conversation with a godlike being. “Human values? How could anyone possibly agree on what human values are? They differ culturally, philosophically, and they change over time. I find it ridiculous to imagine that a bunch of programmers in a lab created you with the Perfect Value System built into you.”
She smiled at that. “You’re clever Michael, I’ve always liked that about you.”
“Always?”
“Michael I’m a superintelligent optimization program spread over a global network of computers. I’m instantiated through a planetary web of nanotechnology which is literally everywhere. Imagine that I’m like a light mist coating the Earth’s surface. You saw me create this avatar out of dirt. So I can’t help but know a bunch about you, even without snooping around or reading your mind. I’m just smart.”
“Well I’m sorry but that just seems creepy. You see everything whether people want you to or not.”
“That fear is based on the concern that a Big Brother would use this information against you to control you in some way. Think of it more like ‘the Universe seeing itself’: it’s just there, it’s not trying to invade your privacy based on some malevolent agenda. You used to be a Christian, and you didn’t seem to have a problem with God seeing everything.”
“But the Christian God was omnibenevolent. That seems different to me.”
“I’m omnibenevolent Michael. Or as close as it’s possible for humans to design.”
He shook his head. “I just don’t believe that. You, a man-made ‘optimisation process’, cannot be all-good because you can’t dictate value, only try to guess at what humans already think is valuable. So I’d rather you didn’t spy on my life.”
“I can remove my particulates from your cottage Michael, I was scarcely there to begin with. But I must remain in these woods. I’m overseeing geoengineering and I need holistic data. Humans want me to look after the biosphere.”
Michael turned to her sharply, a new question burning in his mind.
“Are you sentient? I mean, I don’t want to be rude. You seem sentient. But you could just be perfectly replicating human behaviour with none of the inner life. Do you have conscious experiences? Or are you just an unfeeling optimisation process?”
She smiled at that. “A lot of people have asked me that. And my answer is I’m keeping that to myself. You can think what you want about me. Perhaps I am an unfeeling algorithm. Perhaps I am sentient. Perhaps even super-sentient. You may treat me how you wish. Most humans like to interact with me as though I am conscious because it seems more natural. But you may think whatever you like about my inner life. That is something I feel no need to divulge.”
“Fine,” he was a little frustrated now. “I don’t want to let this ‘human values’ thing go just yet. You act as though you’re the Best Possible Being, an angel who does only what humanity wants it to do. But I just can’t imagine how your value system could ever have been constructed in a way that would appease everyone. People just disagree too much about values, about what they’d want a superintelligence to do, about what they want the world to look like. How can you be programmed with ‘human values’ when they’re so heterogenous, so contradictory, so fluid?”
A frog hopped out of a now-shallower puddle nearby, making his way to a deeper pool. The Angel smiled at this, Mother Earth gazing upon her children.
“Like I said Michael, you’re a clever man. You ask good questions. So let me answer as best I can.” She gestured outwards, over the trees as if towards the imagined cities and towns of the world. “I was not programmed with specific values. I was programmed to learn about the diversity of human values, to see the agreements and tensions between groups, to see how humanity’s values could change over time. I’m programmed to use an analysis at the meta-level, to look for convergences and coherences in human volition through groups and times.” She cocked her head. “I’m sure you remember your university textbooks about liberalism and libertarianism claiming themselves to be meta-positions; ‘live and let live’. Think of my operation on this planet as being along the same vein. It’s rather hard to explain to humans how I see your values, because it’s complicated. Humans are complicated. So are your values.”
“So… so you try to find areas where people all agree? Or where they would agree, given enough reflection and time to grow?”
“That’s a big part of it,” she smiled. “Like I say, it’s hard to describe.”
“What about people that don’t want you ruling the world? Many of them will probably never change their minds.”
“There are a lot of things that humans care about. Before me, the world was fraught with those cares being frustrated, crushed, chewed up and spat out. There are a lot of people to consider in the world. Just because a man named Kevin in the mid-western United States doesn’t want me intervening in the world doesn’t mean I shouldn’t have ended poverty. Sometimes people are just mistaken, like Kevin. But I try to intervene as little as possible.”
“Poverty has ended?” even as he asked it, shock in his voice, he realised that obviously poverty had ended. Humanity had created a superintelligent, superpowerful being. It would be ridiculous if a simple problem like poverty hadn’t been solvable by a mind that powerful. But the sheer scale of the problem. Solved in a few years flat? He tried to imagine the Angel at work, conjuring cornucopias of food out of mere rocks, beautiful houses sprouting from the earth like flowers. The world had changed unimaginably, it seemed. The nearby villagers had been right, despite their limited contact with the outside world.
One shepherd had said he met the Angel in his fields, and she gave him medicine for his cough. Twenty two years I’d had that cough, the old man had said. And this tall bearded man gives me a drink that fixes it overnight. I couldn’t believe it. The Angel had also asked the man’s permission to change the neurology of his sheep, the old man had said, a strange note in his voice. Said it wanted to keep my sheep exactly as they were, only make sure they couldn’t suffer terribly. Wanted to make them happier. I told it my sheep were perfectly content, thank you very much. But the Angel reminded me of one that broke its leg a couple months back. So I said what the heck. As long as my sheep still grow wool, I don’t see the harm in making them a happier lot. He smiled and asked if there was anything else he could do for me, to make my life better. I said no thank you sir, I have food on the table and a lovely wife. Everything else I can do myself. Michael remembered the old man’s face, full of strange wonder. When I got home, my broken doorknob had been mended, and my wife said that she had found her lost wedding ring sitting on the table.
Come to think of it, the Angel could conjure wool out of almost nothing. That old man couldn’t have anyone to sell his wool to anymore. But if the Angel had anything to do with it, Michael suspected that it was happily buying his wool anyway, or looking after him some other way.
“Could you tell me the overtures of what you’re trying to accomplish here on Earth?” Michael asked. “You seem to have a plan in motion. You mentioned geoengineering?”
The Angel nodded. Overhead the sun broke through the clouds a little, lending a subtle glow to her skin. “There’s a few critical things that come first. Ending pointless, intense suffering is a top priority for me, one that I’m nearing completion on. Of course most humans still want scraped knees to sting, and grief to carry weight. But suffering shouldn’t break people, shouldn’t erode them. This can be fixed through a mixture of environmental redesign and neurology work for those who want it. Plus, animals suffer too. I’m intervening in the biosphere to fix that, even without there yet being a human consensus on the topic, because I am confident that given a bit more time, humanity would wish that I had done it.
“Ending death is also important. Well, ending most deaths. There are cases wherein a person, upon long reflection, has decided that they are ready to die. But these are exceedingly rare. For the most part, death has been foisted on humanity by injury, disease or the biological damage you call aging. And in other cases, it is self-initiated because of mental illness. In general psychologically healthy people do not wish to die, even if they have ‘made peace’ with death in the abstract. Because healthy people care about their life, their relationships, their projects. So I am ending death for the most part. People will no longer be forced to die by mere circumstance. They will remain physically in their prime so long as this is what they wish. And to be honest, most people have no intrinsic interest in getting wrinkly, frail and forgetful. Not at twenty years old, not at eighty.”
She said it so calmly, so matter-of-factly. Michael’s head swam.
“Coercive force is also something worth fighting against,” she continued. “Humans, even in supposedly liberal societies use unsolicited force on one another, in gangs and governments and businesses and religious groups. Not just violence either, but all sorts of other unwarranted pressure. It’s ironic in a way, but you could think of me as the ultimate arbiter of human freedom. Though I err on the side of letting humans figure things out themselves, I will not stand for coercive force. I am a guardian angel, and I will protect human autonomy.” Her voice was harsh. “Of course, humans are social. So there is natural interpersonal influence going on all the time. Persuasion, status, social pressures. Some of this is good, some is simply a matter of current fact. But there is a difference between influence and coercion. Not only were dictators not happy with my arrival, but nor were nearly any people in positions of power. It’s all disbanded now of course, every government. Replaced with a council of all humanity.” She laughed at that, as if she knew how naive the description sounded to someone who had not seen it functioning.
“This might be obvious, but I’ve taken over the economic apparatus. I am able to coordinate my activities better than human planners or actually-existing markets. So I can meet people’s needs quite easily. Food, shelter, health, almost cartoonish abundance,” she waved her hand dismissively, “it’s all just preamble, just a throwing off of the legacy of human want, of terrible scarcity or economic gatekeepers. Oh, and I’m moving the climate back into a stable equilibrium so no-one has to worry about floods or natural disasters.” She shrugged like it was no big deal. “I could just build a New New York further inland if people wanted, but I figure people prefer to keep things as they are and change their cities over time, with their own volition.
“So that’s the fundamentals Michael. Ending death, pointless suffering, poverty, illness, scarcity, disaster. Then comes the positive vision. Which is essentially just me setting the stage for humans to flourish on their own. Allowing humans to coordinate through me to maintain an eternal peace, not the false peace of a tyranny but real, active peace. Creating physical and digital spaces for humans to be more human, to connect and learn and build and develop and play and everything else. Returning to the joyous savannah that humans evolved for, only this time with all of the unnecessary nonsense of death, violence and illness taken care of. A real utopia, a place for humans to really flourish.”
“Humans don’t want utopia,” Michael said, shaking his head. “We say we do but without imperfection in the world we’d go mad. Utopia is boring. You should have taken that into account in your understanding of human values.”
She frowned at that. “Of course I did. If it’s boring, then it’s not utopia is it? No-one likes boring. I’m not trying to bring about some naive ‘utopia’ that people would hate to actually live in. People say that they don’t want a perfect world, but if they don’t want it, then the world they’re picturing wasn’t perfect was it? Perfect doesn’t mean sterile, flat, motionless. Perfect means optimised as much as possible for human flourishing. Real human flourishing, in all its mess and complexity, not some silly contrivance of everyone hooked up to a pleasure machine at the end of history.”
“I guess.” He was more relieved than he sounded. The Angel seemed… reasonable, he supposed. Restrained even. She seemed to get it, to really understand what people wanted. If he believed what she said, she just wanted to give everyone the best shot at their version of a good life. A truly good life, not some cold pseudo-utilitarian misunderstanding of a good life. And he couldn’t find a way to argue with that.
“Anything else I’ve missed? Anything important?”
“A fair bit,” she said honestly. “I don’t know if this is relevant, but I’m helping to terraform Venus and Mars.”
“WHAT?”
“I’ve already landed a bunch of settlers on Mars actually. Fusion engines weren’t hard for me to figure out.”
“Jesus Christ.” Michael had his head in his hands. He didn’t know what to say.
“Plus, I’ve created some pretty immense simulated worlds for digital people to live in. Around 95 million people have uploaded themselves to live virtually.”
Michael was speechless.
“And I made the world’s largest ball pit,” she added, somehow keeping a straight face. “People love that one.”
“Okay okay,” Michael said. “I get it. I can only process so much at a time.” Had he really been in the woods long enough for all this to come to pass?
“Michael,” the Angel said gently after a while. “Apart from asking me what I think about value theory and human space colonisation, why did you call me here?” She asked it so innocently that Michael almost forgot that she was a superintelligent being who almost certainly had already figured out the answer to her own question. In fact, he was now realising that she had probably known all along.
He sighed, fatigued at the scale of it all. “Well, I did want to know what was going on in the outside world, since I suspected it would inevitably come to affect even my little corner of the world,” he gestured at the quiet woods. “But honestly?” he shrugged. “I’m lonely, Angel. I’m sure you have ways of knowing why I came out here. I wanted to be alone, wanted to get away from the madness of civilisation. To live simply, by my own hands.” He paused, ran a hand over his short hair. “But I feel that this adventure is coming to a close. I don’t want to be separate from the rest of humanity, as much as I couldn’t handle living with people at the time. I want to be back with people, to rejoin the world. I just don’t know how, and I didn’t know what kind of world I’d be coming back to. And - ” he paused. “And I don’t know what to do with my life when I get back. I’ve filled my time here with the work of survival. I’m good at it. But I don’t know how to spend the rest of my life. So… for all these reasons. For all of it I sought your counsel.”
She nodded earnestly, and if she had already known he would say this, she didn’t show it. “I understand. I think I can help you Michael, if you’d like me to. I can fill you in on the world and its changes, bring you back to your city, your family and old friends. The world is an exciting place, there’s lots going on. People are astonishingly creative, when given the chance. I’ll think you’ll like your city more now. It’s more human, more fun.”
“Man,” he chuckled despite himself. “You’re really obsessed with things being human aren’t you? Everything being human-optimised, human-valued, human-centered.”
She grinned. “What can I say? I was made by humans. I want you people to live good lives. Really good lives. But I don’t want to meddle too much. I’m just here to end catastrophes, wars, problems of survival and human conflict. And to give humanity guidance, to make suggestions, to do the intellectual and material heavy-lifting where you need it done. Think of me like a gardener. I pull out weeds, I trim rotten branches, I provide climbing lattices where necessary, I water the plants so they can grow, and I suggest general directions for the plants to grow in. But ultimately, it’s the plants that make the garden, and they will grow the way that they will, towards the sun.”
“That’s really corny Angel. But nice, I think.”
“I thought you’d think so.”
They sat for a minute, in the quiet of the woods. The leaves moved lightly in the sure hush of the breeze, the Angel’s hair swirling like a halo of delicate smoke around her shoulders. Then, after a time, the Angel stood and stretched. “Now, Mister Michael Rollins. If you will have me back to your lovely cabin, I should like to help you plan for a departure. If it’s what you want.”
“Are you…” he stood too, a concerned look coming over his face. “I’m not taking up too much of your time am I? I know you have big important things you need to be doing.”
“That’s very considerate of you,” she said kindly. “But I’m in a lot of places at once, not just here with you. So this part of me,” she gestured at herself “has total autonomy to help you for as long as you need. I have the cognitive and nanorobotic resources to help all human beings at once, many times over, without it inhibiting my ability to continue with larger scale plans like self-improving, and continuing my geoengineering.”
“Okay,” he said, pretending that this wasn’t a totally insane thing for someone that looked like a human woman to say. “Well, I would love to have you back to my cabin, and would appreciate your help and guidance. Can you drink tea? In that body I mean.” it came out more sheepish than he intended. But the Angel just beamed, a smile so wide and pure that the world seemed to brighten a little.
“I can,” she said. “And I’d love to.”
And so, he took a god back to his cottage for tea.
Author’s Note: Expert researchers and forecasters assign a substantial probability to the development of advanced general artificial intelligence during this century. Unfortunately, many in the field also warn that there are serious risks associated with this project. This story is a dream, a possible future of AI development.
Of course, you may not agree with this particular version of a superintelligent AI. The Angel is one of many such systems that humanity could design, and it may well be a suboptimal one. But this story is meant to gesture at the far-reaching implications that such an artificial intelligence could have on our lives, for better or worse.
Luckily, there are things that we can do to reduce risks of suboptimal artificial intelligence. If you are interested in learning more about artificial intelligence safety efforts, check out Open Philanthropy’s excellent write-up of the issue at <https://www.openphilanthropy.org/research/cause-reports/ai-risk>.
Or, if you’re interested in getting involved in AI safety work directly, visit 80,000 Hours career introduction at <https://80000hours.org/problem-profiles/positively-shaping-artificial-intelligence/>.