Once upon a time, very long ago, there was a rumour that spread across the desert kingdom of Tralburg. The rumour described a network of ancient caverns lost under the shifting desert sands, and within them, two magical bottles. These vessels, it was said, each contained an old and powerful spirit which could grant a person any wish they desired.
Of all the kingdoms bordering this mysterious desert, the kingdom of Tralburg was the wealthiest and most powerful, having vast arable land, profitable gold mines, and many of the finest tradespeople in the land. King Tralburg III, it was said, was a good leader, a kind and thoughtful man. When a messenger whispered to him the rumours of two magical bottles hidden deep in the desert, his eyes lit up and he called a meeting of his most trusted advisors.
Looking out over the royal balcony at the great urban sprawl of his kingdom, he spoke to the few men and women he had gathered.
“I trust you have heard the rumours,” he began, “of a great and mysterious treasure buried in the desert sands that border our kingdom—rumours of ancient spirits that can grant a person any wish. I know not whether these rumours are true. But if they are… even if there is a small chance… then I believe we must pursue the discovery of these vessels.”
He turned to his advisors, a blazing excitement in his eyes. “Think of what we could do with this power my friends. We could create wheat out of sand, gold out of rubble. We could cure every disease that plagues us, create a world of peace and prosperity, a shining city where no-one dies for want of bread or medicine. The possibilities outpace our mind’s abilities to fathom them.”
His many advisors could see that he believed in the good that he spoke of. Lesser men may have coveted the vessels for selfish reasons: control, authority, power. But King Tralburg was a good man. Nevertheless, some of his advisors grew worried.
“Your Majesty,” said Luren, a tall dark-haired man nearly as old as the King himself, “do the people of our kingdom know of this rumour? I know that you have only good intentions for seeking these spirits, but others may not be so benevolent. They may seek the spirits to wish for terrible things: to become a ruler themselves, to enslave our people, to create monstrous weapons, or any number of unthinkable horrors. Some men are good, of course. But others are selfish and will wish only for things that benefit themselves, perhaps at great expense to everyone else. If they know of this rumour, then time is of the essence. We must beat them to the vessels.”
The King’s brow furrowed with worry. “I had not considered that. I fear that you may be right. We ought to proceed with great haste in discovering the location of these caverns, to protect the world from wish-makers with bad intentions.”
But then, another advisor spoke up. Her name was Sara, and she had proved herself a noble and clever advisor. “My King,” she said gently. “I worry about the risk of rushing. Think: if each person believes the others to be a threat, then we may all begin a dangerous race. We will each rush to the vessels, each hoping to win the race and wish destruction or dominion over the others. What is more, in our haste we may not take the time to consider the best wish. What if we wish for something trivial or dangerous out of haste?”
The King’s eyes widened at this, and he saw the reason in it. And so, he decided to create a peacekeeping force, which would patrol the dunes day and night to ensure that no person, group, or kingdom could sneak into the desert to claim the vessels before the others. With these patrols in place, they could search for the caverns at a reasonable pace, taking the time to formulate the wish that would be the most beneficial to the world.
However, it was not long before a group of the King’s explorers, upon discovering a strangely shaped boulder half-buried under the dunes, found an entrance, beautifully ornate in its architecture. Scouts reported that it indeed led down into a maze of dark caverns.
“Let us make haste!” Luren exclaimed to the King. “A better world awaits. Every moment we delay is a moment that people suffer unnecessarily.”
Sara tried to protest, to caution against such haste. But the King was convinced, and that afternoon they found themselves being led on dromedaries out across the blistering sands. They reached the entrance at sundown and were escorted down into a musty series of tunnels hewn masterfully from the rock. Torchlight danced strangely over ancient carvings so that they seemed to silently sway to unheard music. Sara shivered as the air grew colder.
After some time, the tunnel opened suddenly into a gaping cavern. In its centre was a series of stairs leading to a raised platform. And there, on the stone dais, sat two glass bottles: one dark green, one clear.
The King climbed the stairs slowly, reverentially. Sara could see his hands shaking with anticipation. He reached for the dark green bottle. Suddenly, she thought of something.
“Wait!” she said, clambering the stairs after him. “Your Majesty, look.” She pointed at the bottles. “There are two spirits here. One, we cannot see.”
Indeed, the dark green bottle was frosted and opaque. It was impossible to see the spirit inside.
“The other, we can see.”
Inside the clear bottle, a lavender mist swirled and formed into the strange shape of the spirit, wide gentle eyes peering through the glass at the group.
“We do not know how these spirits think,” Sara said, placing a hand on the King’s shoulder. “They are not human like us. For all we know, the spirit in the dark green bottle could be poorly made, a creature that would take any wish you made and corrupt it. It may not even be evil, just different to us. We cannot assume it shares our ideas, our values, our understanding of context.”
The King brushed his fingers across the glass of the dark green bottle and was stunned by a flash of visions. He saw a King wish for a golden touch, only to have his own daughter turn to a golden statue beneath his hands. He saw a Queen wish for her subjects to be happy, and the spirit’s thousand ghostly arms reaching across the city, contorting people’s faces into permanent smiles with fish-hook fingers.
“Yes,” he said in a shaky voice. “That spirit is indeed poorly made. No matter what I wish, the spirit will surely fail to understand the intent behind my words.”
He touched the clear bottle, seeing the spirit inside gazing back at him with placid eyes. He was filled with a different vision, a vision of shining cities and beautiful lives, a billion people flourishing together, a bright future worth wishing for. Yes, this spirit was well-made, designed to grant wishes safely, and to pay attention to the intent behind a wish. It would know not to turn your daughter to a statue, or to force people to smile. A wiser spirit indeed.
And yet, the King paused, a soft smile coming over his face. When he spoke, he said words so good and wise that they would be written in every history book to come.
“Let us wait a while before we make our wish. I’d like to get it right.”
Author’s Note: Expert researchers and forecasters assign a substantial probability to the development of advanced general artificial intelligence during this century. Unfortunately, many in the field also warn that there are serious risks associated with this project. This story is a parable, meant to illustrate the various risks that we may face from advanced artificial intelligence.
First, there is the risk of competing groups or nations racing to develop advanced AI capabilities. Such haste, like people rushing to find the genie’s caverns, could lead to crucial oversights in making sure that such systems are safe.
Second, there is the risk of people using advanced AI systems for suboptimal ends. Like a King wishing for magical weapons to help win a war, groups or nations could use even more rudimentary AI systems to cause harm.
Third, there is the risk of poorly designed AI systems. Like the spirit in the dark green bottle, we must remember that artificial systems may not think like us. Often when researchers program an AI to complete a task, it completes it in surprising and undesirable ways, like a chess-playing program hacking the system to win rather than learning to play well.
Fourth, there is the risk of badly formulated goals. Even in reliable artificial systems, we may fail to specify our goals correctly, leading an artificial intelligence to pursue bad outcomes.
A genie is a good analogy for an advanced general intelligence, because it could be a powerful force for good or bad in the world, a new form of intelligence with promises and risks associated. Like with King Tralburg III, good intentions are not enough. We must focus on making sure that artificial intelligence is developed safely, and for the good of everyone.
[Creative Writing Contest] The Dark Green Bottle
Once upon a time, very long ago, there was a rumour that spread across the desert kingdom of Tralburg. The rumour described a network of ancient caverns lost under the shifting desert sands, and within them, two magical bottles. These vessels, it was said, each contained an old and powerful spirit which could grant a person any wish they desired.
Of all the kingdoms bordering this mysterious desert, the kingdom of Tralburg was the wealthiest and most powerful, having vast arable land, profitable gold mines, and many of the finest tradespeople in the land. King Tralburg III, it was said, was a good leader, a kind and thoughtful man. When a messenger whispered to him the rumours of two magical bottles hidden deep in the desert, his eyes lit up and he called a meeting of his most trusted advisors.
Looking out over the royal balcony at the great urban sprawl of his kingdom, he spoke to the few men and women he had gathered.
“I trust you have heard the rumours,” he began, “of a great and mysterious treasure buried in the desert sands that border our kingdom—rumours of ancient spirits that can grant a person any wish. I know not whether these rumours are true. But if they are… even if there is a small chance… then I believe we must pursue the discovery of these vessels.”
He turned to his advisors, a blazing excitement in his eyes. “Think of what we could do with this power my friends. We could create wheat out of sand, gold out of rubble. We could cure every disease that plagues us, create a world of peace and prosperity, a shining city where no-one dies for want of bread or medicine. The possibilities outpace our mind’s abilities to fathom them.”
His many advisors could see that he believed in the good that he spoke of. Lesser men may have coveted the vessels for selfish reasons: control, authority, power. But King Tralburg was a good man. Nevertheless, some of his advisors grew worried.
“Your Majesty,” said Luren, a tall dark-haired man nearly as old as the King himself, “do the people of our kingdom know of this rumour? I know that you have only good intentions for seeking these spirits, but others may not be so benevolent. They may seek the spirits to wish for terrible things: to become a ruler themselves, to enslave our people, to create monstrous weapons, or any number of unthinkable horrors. Some men are good, of course. But others are selfish and will wish only for things that benefit themselves, perhaps at great expense to everyone else. If they know of this rumour, then time is of the essence. We must beat them to the vessels.”
The King’s brow furrowed with worry. “I had not considered that. I fear that you may be right. We ought to proceed with great haste in discovering the location of these caverns, to protect the world from wish-makers with bad intentions.”
But then, another advisor spoke up. Her name was Sara, and she had proved herself a noble and clever advisor. “My King,” she said gently. “I worry about the risk of rushing. Think: if each person believes the others to be a threat, then we may all begin a dangerous race. We will each rush to the vessels, each hoping to win the race and wish destruction or dominion over the others. What is more, in our haste we may not take the time to consider the best wish. What if we wish for something trivial or dangerous out of haste?”
The King’s eyes widened at this, and he saw the reason in it. And so, he decided to create a peacekeeping force, which would patrol the dunes day and night to ensure that no person, group, or kingdom could sneak into the desert to claim the vessels before the others. With these patrols in place, they could search for the caverns at a reasonable pace, taking the time to formulate the wish that would be the most beneficial to the world.
However, it was not long before a group of the King’s explorers, upon discovering a strangely shaped boulder half-buried under the dunes, found an entrance, beautifully ornate in its architecture. Scouts reported that it indeed led down into a maze of dark caverns.
“Let us make haste!” Luren exclaimed to the King. “A better world awaits. Every moment we delay is a moment that people suffer unnecessarily.”
Sara tried to protest, to caution against such haste. But the King was convinced, and that afternoon they found themselves being led on dromedaries out across the blistering sands. They reached the entrance at sundown and were escorted down into a musty series of tunnels hewn masterfully from the rock. Torchlight danced strangely over ancient carvings so that they seemed to silently sway to unheard music. Sara shivered as the air grew colder.
After some time, the tunnel opened suddenly into a gaping cavern. In its centre was a series of stairs leading to a raised platform. And there, on the stone dais, sat two glass bottles: one dark green, one clear.
The King climbed the stairs slowly, reverentially. Sara could see his hands shaking with anticipation. He reached for the dark green bottle. Suddenly, she thought of something.
“Wait!” she said, clambering the stairs after him. “Your Majesty, look.” She pointed at the bottles. “There are two spirits here. One, we cannot see.”
Indeed, the dark green bottle was frosted and opaque. It was impossible to see the spirit inside.
“The other, we can see.”
Inside the clear bottle, a lavender mist swirled and formed into the strange shape of the spirit, wide gentle eyes peering through the glass at the group.
“We do not know how these spirits think,” Sara said, placing a hand on the King’s shoulder. “They are not human like us. For all we know, the spirit in the dark green bottle could be poorly made, a creature that would take any wish you made and corrupt it. It may not even be evil, just different to us. We cannot assume it shares our ideas, our values, our understanding of context.”
The King brushed his fingers across the glass of the dark green bottle and was stunned by a flash of visions. He saw a King wish for a golden touch, only to have his own daughter turn to a golden statue beneath his hands. He saw a Queen wish for her subjects to be happy, and the spirit’s thousand ghostly arms reaching across the city, contorting people’s faces into permanent smiles with fish-hook fingers.
“Yes,” he said in a shaky voice. “That spirit is indeed poorly made. No matter what I wish, the spirit will surely fail to understand the intent behind my words.”
He touched the clear bottle, seeing the spirit inside gazing back at him with placid eyes. He was filled with a different vision, a vision of shining cities and beautiful lives, a billion people flourishing together, a bright future worth wishing for. Yes, this spirit was well-made, designed to grant wishes safely, and to pay attention to the intent behind a wish. It would know not to turn your daughter to a statue, or to force people to smile. A wiser spirit indeed.
And yet, the King paused, a soft smile coming over his face. When he spoke, he said words so good and wise that they would be written in every history book to come.
“Let us wait a while before we make our wish. I’d like to get it right.”
Author’s Note: Expert researchers and forecasters assign a substantial probability to the development of advanced general artificial intelligence during this century. Unfortunately, many in the field also warn that there are serious risks associated with this project. This story is a parable, meant to illustrate the various risks that we may face from advanced artificial intelligence.
First, there is the risk of competing groups or nations racing to develop advanced AI capabilities. Such haste, like people rushing to find the genie’s caverns, could lead to crucial oversights in making sure that such systems are safe.
Second, there is the risk of people using advanced AI systems for suboptimal ends. Like a King wishing for magical weapons to help win a war, groups or nations could use even more rudimentary AI systems to cause harm.
Third, there is the risk of poorly designed AI systems. Like the spirit in the dark green bottle, we must remember that artificial systems may not think like us. Often when researchers program an AI to complete a task, it completes it in surprising and undesirable ways, like a chess-playing program hacking the system to win rather than learning to play well.
Fourth, there is the risk of badly formulated goals. Even in reliable artificial systems, we may fail to specify our goals correctly, leading an artificial intelligence to pursue bad outcomes.
A genie is a good analogy for an advanced general intelligence, because it could be a powerful force for good or bad in the world, a new form of intelligence with promises and risks associated. Like with King Tralburg III, good intentions are not enough. We must focus on making sure that artificial intelligence is developed safely, and for the good of everyone.
If you are interested in learning more about artificial intelligence safety efforts, check out Open Philanthropy’s excellent write-up of the issue at <https://www.openphilanthropy.org/research/cause-reports/ai-risk>.
Or, if you’re interested in getting involved in AI safety work directly, visit 80,000 Hours career introduction at <https://80000hours.org/problem-profiles/positively-shaping-artificial-intelligence/>.