I did not understand what the story was trying to say, very well. It just seemed to me to be ‘a series of bad things happened because of failures of effective communication and understanding?’ I can read it as a criticism of overconfidence, but I feel as if there have already been a lot of criticisms of overconfidence, and at this point I’m kind of worried we need more criticisms of underconfidence? I did not end up with very strong opinions about the story in any particular way, and I suspect it was a failure of my understanding at least as much as a failure of the story.
As tagged, this story strikes me as a fable intended to explain one of the mechanisms behind so-called “S-risks”, hellish scenarios that might be a fate worse than the “death” represented by X-risks. It’s thought that one of the ways a far-future civilization (or AI or whatever) might end up creating vast amounts of suffering, could be as the result of a cycle of threats and bargaining with rival civilizations: https://centerforreducingsuffering.org/research/intro/
Of course it’s a little confusing to have the twist with the sentient birds—I think rather than a literal “farmed animal welfare” thing, this is intended to showcase a situation where two different civilizations have very different values (because one civilization values the sentient birds extremely highly and the other, thinking they are just ordinary birds, values them not at all). Meanwhile, the other civilization wants to keep the gold in the ground, vs the Kunus find it valuable only after it’s mined. (In a similar way, someone who is pro-choice could threaten to kill a fetus, which to them is only a ball of cells, and this might be a very effective threat against a pro-life person who thinks every fetus has a fully human soul.)
I don’t really understand why the story is a frame story, or why the main purpose of the ritual is for all the Kunus to feel “collective guilt”… EA is usually trying to steer away from giving the impression that we want everyone to feel guilty all the time.
Totally unrelated point, but I thought the economics of this story were a little wacky… having some gold rocks in your pocket doesn’t do anything to help you have a better quality of life or cause there to be more food in the kingdom… it’s just a form of currency. Unless the kingdom of the Kunus is suffering from deflation or tight monentary policy or whatever, I think the Goldseeker is barking up the wrong tree and isn’t actually doing anything to improve the overall prosperity of the kingdom. (Of course if you give the gold evenly to everyone or disproportionately to the poor, that would have a helpful redistributive effect. But you’re not growing the economic pie at all by digging up yellow rocks.)
Overall, I don’t think this story works that well because the reframing of S-risk from far-future galactic-civilizations to fantasy-parable doesn’t really make the idea clearer or more intuitive. Although I do like the twist where the Goldseeker was a good person trying to improve the world and hatched a clever, harmless-seeming plan to do that, only to unwittingly end up torturing countless sentient beings. This is a good part of the parable—if S-risks ever occur, the civilizations that commit those galactic war crimes will probably be convinced of their righteousness, and indeed probably won’t even recognize that they are committing a wrong.
“As tagged, this story strikes me as a fable intended to explain one of the mechanisms behind so-called “S-risks”, hellish scenarios that might be a fate worse than the “death” represented by X-risks.”
That’s what I was going for, although I’m aware that I didn’t make this as clear as I should have.
”Of course it’s a little confusing to have the twist with the sentient birds—I think rather than a literal “farmed animal welfare” thing, this is intended to showcase a situation where two different civilizations have very different values.”S
ame thing here. This is what I was trying to get at, but couldn’t think of many other scenarios involving suffering agents where one group of people cares and another doesn’t.”
I don’t really understand why the story is a frame story, or why the main purpose of the ritual is for all the Kunus to feel “collective guilt”… EA is usually trying to steer away from giving the impression that we want everyone to feel guilty all the time.”T
his is really helpful feedback—I didn’t realize that “collective guilt” came across as the point of the story, and I definitely agree that making people feel guilty is counterproductive. I can’t remember why I threw in that phrase (probably because I couldn’t think of anything else), but I’ll change it now.
Totally unrelated point, but I thought the economics of this story were a little wacky.
Yup, definitely more than a “little” wacky :) Maybe using another resource like food or water or land would be better—but then it would have been harder to make the point that each country thought were doing the right thing
.This is a good part of the parable—if S-risks ever occur, the civilizations that commit those galactic war crimes will probably be convinced of their righteousness, and indeed probably won’t even recognize that they are committing a wrong.
This is the central point that I wanted to get across. Whether we’re considering a civilization or an advanced AI, s-risks need not result from intentional malevolence. I’m glad it didn’t get too distorted, but it seems like there are better ways to build a story around this point.
Another side-note: a lot of the ideas behind this story are discussed in the Center on Long-Term Risk’s research agenda. I don’t know whether they would agree with my presentation or conceptualization of those ideas.
Thanks for the feedback! I think this is probably a failure of the story more than a failure of your understanding—after all, a story that’s hard to understand isn’t fulfilling its purpose very well. Jackson Wagner’s comment below is a good summary of the main points I was intending to get across.
Next time I write, I’ll try to be more clear about the points I’m trying to convey.
I did not understand what the story was trying to say, very well. It just seemed to me to be ‘a series of bad things happened because of failures of effective communication and understanding?’ I can read it as a criticism of overconfidence, but I feel as if there have already been a lot of criticisms of overconfidence, and at this point I’m kind of worried we need more criticisms of underconfidence? I did not end up with very strong opinions about the story in any particular way, and I suspect it was a failure of my understanding at least as much as a failure of the story.
As tagged, this story strikes me as a fable intended to explain one of the mechanisms behind so-called “S-risks”, hellish scenarios that might be a fate worse than the “death” represented by X-risks. It’s thought that one of the ways a far-future civilization (or AI or whatever) might end up creating vast amounts of suffering, could be as the result of a cycle of threats and bargaining with rival civilizations: https://centerforreducingsuffering.org/research/intro/
Of course it’s a little confusing to have the twist with the sentient birds—I think rather than a literal “farmed animal welfare” thing, this is intended to showcase a situation where two different civilizations have very different values (because one civilization values the sentient birds extremely highly and the other, thinking they are just ordinary birds, values them not at all). Meanwhile, the other civilization wants to keep the gold in the ground, vs the Kunus find it valuable only after it’s mined. (In a similar way, someone who is pro-choice could threaten to kill a fetus, which to them is only a ball of cells, and this might be a very effective threat against a pro-life person who thinks every fetus has a fully human soul.)
I don’t really understand why the story is a frame story, or why the main purpose of the ritual is for all the Kunus to feel “collective guilt”… EA is usually trying to steer away from giving the impression that we want everyone to feel guilty all the time.
Totally unrelated point, but I thought the economics of this story were a little wacky… having some gold rocks in your pocket doesn’t do anything to help you have a better quality of life or cause there to be more food in the kingdom… it’s just a form of currency. Unless the kingdom of the Kunus is suffering from deflation or tight monentary policy or whatever, I think the Goldseeker is barking up the wrong tree and isn’t actually doing anything to improve the overall prosperity of the kingdom. (Of course if you give the gold evenly to everyone or disproportionately to the poor, that would have a helpful redistributive effect. But you’re not growing the economic pie at all by digging up yellow rocks.)
Overall, I don’t think this story works that well because the reframing of S-risk from far-future galactic-civilizations to fantasy-parable doesn’t really make the idea clearer or more intuitive. Although I do like the twist where the Goldseeker was a good person trying to improve the world and hatched a clever, harmless-seeming plan to do that, only to unwittingly end up torturing countless sentient beings. This is a good part of the parable—if S-risks ever occur, the civilizations that commit those galactic war crimes will probably be convinced of their righteousness, and indeed probably won’t even recognize that they are committing a wrong.
Changed “guilt” to “responsibility,” but I’m not sure if that’s much better.
“As tagged, this story strikes me as a fable intended to explain one of the mechanisms behind so-called “S-risks”, hellish scenarios that might be a fate worse than the “death” represented by X-risks.”
That’s what I was going for, although I’m aware that I didn’t make this as clear as I should have.
”Of course it’s a little confusing to have the twist with the sentient birds—I think rather than a literal “farmed animal welfare” thing, this is intended to showcase a situation where two different civilizations have very different values.”S
ame thing here. This is what I was trying to get at, but couldn’t think of many other scenarios involving suffering agents where one group of people cares and another doesn’t.”
I don’t really understand why the story is a frame story, or why the main purpose of the ritual is for all the Kunus to feel “collective guilt”… EA is usually trying to steer away from giving the impression that we want everyone to feel guilty all the time.”T
his is really helpful feedback—I didn’t realize that “collective guilt” came across as the point of the story, and I definitely agree that making people feel guilty is counterproductive. I can’t remember why I threw in that phrase (probably because I couldn’t think of anything else), but I’ll change it now.
Totally unrelated point, but I thought the economics of this story were a little wacky.
Yup, definitely more than a “little” wacky :) Maybe using another resource like food or water or land would be better—but then it would have been harder to make the point that each country thought were doing the right thing
.This is a good part of the parable—if S-risks ever occur, the civilizations that commit those galactic war crimes will probably be convinced of their righteousness, and indeed probably won’t even recognize that they are committing a wrong.
This is the central point that I wanted to get across. Whether we’re considering a civilization or an advanced AI, s-risks need not result from intentional malevolence. I’m glad it didn’t get too distorted, but it seems like there are better ways to build a story around this point.
Another side-note: a lot of the ideas behind this story are discussed in the Center on Long-Term Risk’s research agenda. I don’t know whether they would agree with my presentation or conceptualization of those ideas.
Thank you so much for the feedback!
Thanks for the feedback! I think this is probably a failure of the story more than a failure of your understanding—after all, a story that’s hard to understand isn’t fulfilling its purpose very well. Jackson Wagner’s comment below is a good summary of the main points I was intending to get across.
Next time I write, I’ll try to be more clear about the points I’m trying to convey.