To be completely honest, I think that “making people reading it think more kindly of effective altruism” is a good goal for creative nonfiction, but not a very helpful goal for fiction. My experience with writing fiction (mostly plays) is that fiction is a really poor platform for convincing people of ideas (I almost always zone out if I feel like a playwright is trying to convince me to believe something), but it’s a really good platform for raising difficult questions that readers have to think through themselves. I suppose my hope with this villain is to confront people somewhat graphically with questions that are important and answers to those questions that are terrible, in the hopes of sparking further thought rather than coming to a specific answer.
I upvoted your comment because what you said was interesting even if I disagree with the overall sentiment.
I agree that fiction is good for raising difficult questions and exploring nuance. I also agree that didactic fiction can be a bit off-putting. However, most fiction expresses a writer’s beliefs about the world and what they value (most writers will take inspiration from what they already know and what they think). When the writing is good, this tends not to be off-putting because the writer handled this with enough nuance. An author’s best guess of what is good or bad (or true or false) can still come through pretty clearly without the reader feeling like they have to agree with them because there’s enough going on, there is enough complexity and enough nuance, in the story for readers to be able to put different emphasis on different elements (and therefore for readers to feel free enough to come to their own conclusions for the author’s point of view to not be off-putting).
Simple messages in fiction, dealt with no nuance, make for bad writing. Fiction that makes readers think and confront difficult questions can make for much better fiction. It doesn’t follow that fiction can’t change people’s impressions of what is good or bad (or true or false). If you can change people’s impressions of what is good or bad (or true or false), it seems worth taking that into account. Is it a good goal for fiction to confront a reader with any difficult question without considering whether this difficult question is worth thinking about over other questions? My guess is, in full EA spirit, that it is better to choose which difficult questions to ask (especially if some difficult questions will lead people to missing questions/considerations that you think are way more important).
I think maybe a good goal for fiction could be to leave the reader with a better understanding of your point of view/the way you see the world than if they hadn’t read your piece whilst also having them see the nuances/complexities involved (because there generally aren’t ever simple answers and fiction can be a great way to explore that).
I second everything Sophia said, but would like to raise a few other points to clarify:
First, I also dislike works-that-fit-in-my-brain’s-internal-category-labeled-propaganda. (As some evidence of this, I’ll offer my comment on ‘blue bird and black bird’.) Nonetheless I feel that there is an enormous range for stories in which “make people reading it think more kindly of X” is a clear goal that do not fit in my brain’s internal category of propaganda. It’s quite clear that Lois McMaster Bujold is opposed to eternal smouldering guerilla wars of resistance and in favor of artificial wombs, and none of that makes the Vorkosigan Saga non-amazing. I can also offer basically any culture-clash story; usually those have the objective of making the audience say “both cultures have their good points and their bad points and their inhabitants are both human”, but a lot of those still work effectively without making the audience zone out. None of those flip my “I am being propagandized to” switch.
Two possible explanations for what’s going on.
First is the worldview emphasis. If the author “presents a world in which X is true,” the audience can ask themselves, “is this world consistent? Does it resemble our world?” If it is straightforwardly true that, in the Lord of the Rings, power inherently corrupts, reading the Lord of the Rings gives us an opportunity to look at a world in which power inherently corrupts. Insofar as it seems coherent and non-self-contradictory, the audience has an example of “what could be” to compare that fictional world to their real world. Maybe it doesn’t give you any useful real-world experience; maybe Tolkien is relying on factors that don’t exist in his world, or are too weak to have the effects he describes. But by expanding the audience’s worldview, the author gives them a new model to use to analyze reality.
Second is Yudkowsky’s line: “Nonfiction conveys knowledge. Fiction conveys experience.” We don’t, most of us, have experience with trying to use logic and math to do good. A work of fiction in which a character tries to use logic and math to figure out how to do the most good, even if it is in a fantasy world in which you can use magic to heal people, gives us a starting model of “how to use logic and math to do the most good,” lets us know that this is a thing you can do (at least in an apparently-but-not-necessarily self-consistent alternate universe), gives you a model (as previous paragraph) and thereby gets us a start on doing it in our universe. By a character giving you a model of one potential way to behave, the audience can learn that this is a potential way to behave, and thereby start wondering if it is worth trying to adapt any elements of it to their own lives, without the author ever needing to preach.
You both raise very good points, and I think you’ve convinced me there are ways to do this that don’t come across as propaganda.
At the same time, I would still stand by my stance that having more EA villains in fiction would overall be a good thing for EA. Good villains are thought-provoking even though their actions are evil—Killmonger in Black Panther and Karli in the Falcon/Winter Soldier series come to mind as pop culture characters who’ve made me think much more than the heroes in their respective films/shows.
I think that the rationalist/EA fiction I’ve seen always falls into a very propaganda-adjacent territory rather than the versions you’ve described—things like HPMOR, which I’ve never heard good feedback on from anybody who wasn’t essentially already in the rationalist community. (I’m sure such feedback exists, but the response in my friend groups has been overwhelmingly negative.) It feels to me like the goal of attracting people outside the community by portraying EA/rationality as positively as possible is self-defeating, because it results as stories that just aren’t very interesting to people who are here for a good story rather than an exposition of the author’s philosophy.
I would much prefer a story that works as a story, even if it’s from a perspective of a villain and doesn’t give you a clear authorial point of view on any of the relevant questions. (Whether or not this works as a story is of course a separate question I’m too biased to judged.) My general sense from my test readers has been that the questions (was what Mr. Green was doing in fact wrong? What’s wrong with the speaker’s super-harsh utilitarianism?) are capable of starting interesting EA-type conversations, and that we can trust readers to have interesting and ethical thoughts on their own.
Also I want to do a completely separate post in response to one of your short comments:
”What’s wrong with the speaker’s super-harsh utilitarianism?”
My immediate response, just automatic on reflex without engaging my brain’s slow mode, is “planning fallacy / arrogance / sin of pride.” What’s wrong is that he assumes he’s in a sufficiently strong level of knowledge, self-discipline, and self-control that he actually can pull off his ubermensch act, instead of it all going horribly wrong and blowing up in his face. That’s always what’s wrong with characters doing that. That’s why so much EA thought focuses on the question of how to ‘first, do no harm’. That’s why EA takes the High Modernists so seriously as an object lesson, that’s why rationalist circles are the only ones I’ve ever been in where you can just say “Chesterton’s Fence!” and the burden of proving an argument automatically switches over to the party arguing for a reform.
Writers have been asking that question and giving basically that answer for, what, a hundred and fifty years? Since “Crime and Punishment,” I think, which is apparently from 1866. The most recent modern artwork that I found memorable that said it was Fate/zero, back in, IIRC, 2012. I’m not going to say you can’t update timeless and eternal themes for a modern audience, that’s a perfectly reasonable thing to do, but I didn’t actually find that the story started interesting EA-type conversations (though, again, I thought it was very well-written—your voice and prose style are excellent), and the specific reason for that is because I didn’t really see that it was doing anything new or exciting, philosophically speaking. Insofar as you intended it to have a different moral answer, that… didn’t really come across? It was just asking the same question that had been answered so many times before.
(For clarification: Step One is that the story has to work as a story, I agree with that completely, if I’m passing that over in my response, it isn’t because I disagree, but because I agree too much to have anything interesting to say on it.)
The odd thing is, I would use HPMOR as a model of how to do it right. Its main failing is that it fails to make it clear that the protagonist isn’t a perfect propaganda figure, and shouldn’t be emulated—that the audience ends up thinking that the protagonist is making giant mistakes throughout the story because the author thinks those are the correct decisions, not because he’s an eleven-year-old in over his head. But the author agrees with him enough that if you aren’t paying careful attention, he comes off as an arrogant jerk the author endorses, instead of a person with many virtues and the vice of being an arrogant jerk.
(Clarification: I tried to read HPMOR twice and disliked it before I fell in love the third time. I think it is genuinely Great Art, but flawed in the way most Great Art is flawed—that of aiming for being twice as good as the best art that previously existed and ending up being an uneven mix of 160% as good and 60% as good, which may well make you throw the book at the wall during the 60% parts.)
But that side-note aside: I agree that making all EA-style figures into saints is a risk. People may well get turned off at being preached to; I know I do. Again, see my comment on “Blue Bird and Black Bird.” But...
If you’ve ever read Scott Alexander’s Unsong—I don’t know if you have—the central figure in the story is the Comet King. He isn’t the hero; the hero is a not-very-interesting nebbish stuck in the world the Comet King made. But the Comet King is EA, is really presented as purely good, and is a genuinely psychologically fascinating character. Every page he’s on blazes with light and life and joy, and yet he isn’t a Mary Sue, because by the time the main story starts, he has already lost. The book is about the aftermath of a perfectly good hero failing to save the world, the book is unquestionably pro-effective-altruist, and where the book fails, it isn’t because it’s being too pro-effective-altruist, it’s because the protagonist if the main story is really kind of dull and uninteresting.
So I think it can work. As good examples of other fiction that work despite being ideological, I’d recommend Bujold’s “Shards of Honor,” Terry Pratchett’s “Night Watch,” and, as I said, “The Lord of the Rings.”
But although I I think there’s problems with EA-heroes if badly written, I also think there’s an equal and equivalent problem with EA-style villains, even if they are well-written. It can work I agree, artistically speaking, to put a viewpoint you sympathize with but disagree with in the mouth of your villain, then make him take it to evil ends. That’s something I’ve written and it’s something I enjoy writing, because it allows you to have a strong ideological conflict between two good ends while still having a hero, and that has a lot of potential to work well.
But I’m not wholly comfortable with it, and this is why:
I think our subconscious or semi-conscious mind has bins labeled “traits of villains” and “traits of heroes,” and when we see something in real life that we are used to thinking of as always being labeled in fiction with “trait of villain” or “trait of heroes,” we apply that label in real life. When an artwork gives ‘uses reason and logic to try to maximize good’ as a trait of villains, especially in a culture in which everyone else is also using it as a trait of villain, it reinforces that as being part of the ‘villain trait’ box, since it’s usually (not in your case!) paralleled with a conservative, chauvinistic anti-intellectual badass hero. I think this will genuinely cause people to immediately round off, in real life, ‘tries to use reason and logic to maximize good’ as something that will go horribly wrong in real life, and ‘is a badass cowboy who plays by his own rules but accomplishes good things anyway because he’s a FUNDAMENTALLY GOOD PERSON’ as being something that will go right in real life. In this case, if you want me to say I have less respect for people than you do, that’s entirely possible; I don’t think people rationally feel that they do this, I don’t think people who slow down to think do this, but I think that people who operate on automatic do this; that they treat the Terminator movies as a model for how AI will go wrong in real life, and I think that this has bad consequences if everyone is saying the same thing about how ‘trying to maximize good’ will go horribly wrong.
Now, if you believe that EA will go horribly wrong in real life, if you really think that Engels ought to be treated as an early member of EA and as a model of how badly EA will go wrong in the future, it makes sense to write that. But I don’t. And from your comments, I don’t think you do.
Now, again, this isn’t a reason why I think you should never do the ‘charismatic villain arguing for an underappreciated cause’ bit. Plenty of good stories have done it. I’ve done it. But if it isn’t a cause you disapprove of, and most of the other writers writing about the cause have the same take you have on it (and if it’s a thriving cause and the take is one of the first ones you thought of, they probably will) you’re (a) doing something that is individually totally reasonable, and (b) contributing to an unreasonable aggregate. And that’s not something I really approve of, and it’s not something you… ex ante ought to expect members of the subculture to approve of? Or subsidize? Like, Larks’ second point wasn’t one I said explicitly, because it’s attributing malice where malice isn’t the most likely option, but I totally thought it before the obvious realization kicked in that it was statistically unlikely.
These thoughts are super helpful for understanding where you’re coming from, so thank you!! I really appreciate you taking the time to write them all out—my thoughts will be much shorter because I don’t have much to add, not because they weren’t thought-provoking and interesting!I
think we have somewhat different beliefs about what makes the speaker’s actions wrong—I think for me it lands very far to one side of the “clearly evil” to “clearly good” trolley problem spectrum and it’s wrongness is a) very clear to me, and b) very hard for me to pin down a reason for: I don’t really find any of the answers I can think of satisfying (including the ones in the works you mentioned—the fact that plans can fail and change the planner in unforeseen ways is a beautiful and important observation, but in this case the plan more-or-less succeeds and it still feels evil to me.) I find this combination fascinating, but I can see how this comes across rather differently if you don’t share this dissatisfaction, which it sounds like is less universal than I believed.
Unsong sounds like a very interesting piece of writing, I will have to check it out!
I’m glad you aren’t offended! I get easily worried that I might be saying things in an offensive manner and I appreciate you reassuring me that I didn’t! I am always very happy to write long and elaborate reviews of fiction and I am glad you appreciated it.
And I would agree that the protagonist is evil (indeed, he admits he is evil—he’s quite clear that he enjoyed what he did) and also took a set of actions which may have had net-positive utility. I don’t think we know that it did; it’s possible that some vague combination of making people distrust EA-style arguments, imposing costs on people both directly (his victims) and indirectly (court costs, prison costs, stress to everyone vaguely associated, costs of additional security precautions taken because his existence is evidence of the world being less safe than you thought) and so forth and so on made it net-negative.
But I will confidently deny that he was in an epistemic position to expect his actions would be positive, let alone the optimal decision. I could theoretically imagine a world in which this was false, and he genuinely did have the knowledge required for his actions to actually be both ex post and ex ante optimal, but I don’t actually think I can actually imagine a world in which I was in the epistemic state of knowing that he knew his actions were ex post and ex ante optimal; my mental state in such a world would be sufficiently different that I’m not sure I’d be the same person. So I’m really quite comfortable condemning him, though I’ll admit I’d vote for life imprisonment instead of execution.
And Unsong is very interesting! It doesn’t always succeed at what it’s doing, as I mentioned I find the protagonist kind of boring, but it’s trying such fascinating things and it succeeds sufficiently often to be worth reading.
Reply-edit for clarification to expand my response to one of your points: I think it is worth, in a lot of situations, judging based on “should it have worked,” instead of “did it work.” That your model predicted it shouldn’t work and it did work is evidence your model is seriously flawed, just to be clear, I’m not arguing we should completely throw out the experiment and just go with our previous model, but, also, we shouldn’t say “the one guy who won the lottery was right and everyone else was wrong,” because everyone who bought a ticket had the same chance of winning, and ex ante the lottery was a losing bet for all of them.
(Unless the lottery was crooked but that’s a side note.)
So, even if it worked, I still think the protagonist’s motive was unreasonable; even if it worked, I don’t feel it should have worked, statistically speaking, as opposed to him getting immediately spotted, arrested, and spending the next five years of his life in jail in which he can do no good at all. Or someone’s angry brother taking a shot at him with a firearm, causing him to die instantly after he’d donated only $8000 to Givewell’s top charities, as opposed to if he’d peacefully sat back and worked a high-paying job he would have donated $800,000 over the course of his life. Or someone successfully suing to get all the donated money back as a class-action suit, causing the Against Malaria Foundation to go bankrupt because it already spent it all on bed nets and couldn’t get a refund. Not that all of those are equally likely, but there are a lot of ways for his kind of plan to fail at levels of badness approaching these, and if they fail this way he definitely killed people, and I don’t find the assumption that he knew none of them would happen very persuasive.
To be completely honest, I think that “making people reading it think more kindly of effective altruism” is a good goal for creative nonfiction, but not a very helpful goal for fiction. My experience with writing fiction (mostly plays) is that fiction is a really poor platform for convincing people of ideas (I almost always zone out if I feel like a playwright is trying to convince me to believe something), but it’s a really good platform for raising difficult questions that readers have to think through themselves. I suppose my hope with this villain is to confront people somewhat graphically with questions that are important and answers to those questions that are terrible, in the hopes of sparking further thought rather than coming to a specific answer.
I upvoted your comment because what you said was interesting even if I disagree with the overall sentiment.
I agree that fiction is good for raising difficult questions and exploring nuance. I also agree that didactic fiction can be a bit off-putting. However, most fiction expresses a writer’s beliefs about the world and what they value (most writers will take inspiration from what they already know and what they think). When the writing is good, this tends not to be off-putting because the writer handled this with enough nuance. An author’s best guess of what is good or bad (or true or false) can still come through pretty clearly without the reader feeling like they have to agree with them because there’s enough going on, there is enough complexity and enough nuance, in the story for readers to be able to put different emphasis on different elements (and therefore for readers to feel free enough to come to their own conclusions for the author’s point of view to not be off-putting).
Simple messages in fiction, dealt with no nuance, make for bad writing. Fiction that makes readers think and confront difficult questions can make for much better fiction. It doesn’t follow that fiction can’t change people’s impressions of what is good or bad (or true or false). If you can change people’s impressions of what is good or bad (or true or false), it seems worth taking that into account. Is it a good goal for fiction to confront a reader with any difficult question without considering whether this difficult question is worth thinking about over other questions? My guess is, in full EA spirit, that it is better to choose which difficult questions to ask (especially if some difficult questions will lead people to missing questions/considerations that you think are way more important).
I think maybe a good goal for fiction could be to leave the reader with a better understanding of your point of view/the way you see the world than if they hadn’t read your piece whilst also having them see the nuances/complexities involved (because there generally aren’t ever simple answers and fiction can be a great way to explore that).
I second everything Sophia said, but would like to raise a few other points to clarify:
First, I also dislike works-that-fit-in-my-brain’s-internal-category-labeled-propaganda. (As some evidence of this, I’ll offer my comment on ‘blue bird and black bird’.) Nonetheless I feel that there is an enormous range for stories in which “make people reading it think more kindly of X” is a clear goal that do not fit in my brain’s internal category of propaganda. It’s quite clear that Lois McMaster Bujold is opposed to eternal smouldering guerilla wars of resistance and in favor of artificial wombs, and none of that makes the Vorkosigan Saga non-amazing. I can also offer basically any culture-clash story; usually those have the objective of making the audience say “both cultures have their good points and their bad points and their inhabitants are both human”, but a lot of those still work effectively without making the audience zone out. None of those flip my “I am being propagandized to” switch.
Two possible explanations for what’s going on.
First is the worldview emphasis. If the author “presents a world in which X is true,” the audience can ask themselves, “is this world consistent? Does it resemble our world?” If it is straightforwardly true that, in the Lord of the Rings, power inherently corrupts, reading the Lord of the Rings gives us an opportunity to look at a world in which power inherently corrupts. Insofar as it seems coherent and non-self-contradictory, the audience has an example of “what could be” to compare that fictional world to their real world. Maybe it doesn’t give you any useful real-world experience; maybe Tolkien is relying on factors that don’t exist in his world, or are too weak to have the effects he describes. But by expanding the audience’s worldview, the author gives them a new model to use to analyze reality.
Second is Yudkowsky’s line: “Nonfiction conveys knowledge. Fiction conveys experience.” We don’t, most of us, have experience with trying to use logic and math to do good. A work of fiction in which a character tries to use logic and math to figure out how to do the most good, even if it is in a fantasy world in which you can use magic to heal people, gives us a starting model of “how to use logic and math to do the most good,” lets us know that this is a thing you can do (at least in an apparently-but-not-necessarily self-consistent alternate universe), gives you a model (as previous paragraph) and thereby gets us a start on doing it in our universe. By a character giving you a model of one potential way to behave, the audience can learn that this is a potential way to behave, and thereby start wondering if it is worth trying to adapt any elements of it to their own lives, without the author ever needing to preach.
You both raise very good points, and I think you’ve convinced me there are ways to do this that don’t come across as propaganda.
At the same time, I would still stand by my stance that having more EA villains in fiction would overall be a good thing for EA. Good villains are thought-provoking even though their actions are evil—Killmonger in Black Panther and Karli in the Falcon/Winter Soldier series come to mind as pop culture characters who’ve made me think much more than the heroes in their respective films/shows.
I think that the rationalist/EA fiction I’ve seen always falls into a very propaganda-adjacent territory rather than the versions you’ve described—things like HPMOR, which I’ve never heard good feedback on from anybody who wasn’t essentially already in the rationalist community. (I’m sure such feedback exists, but the response in my friend groups has been overwhelmingly negative.) It feels to me like the goal of attracting people outside the community by portraying EA/rationality as positively as possible is self-defeating, because it results as stories that just aren’t very interesting to people who are here for a good story rather than an exposition of the author’s philosophy.
I would much prefer a story that works as a story, even if it’s from a perspective of a villain and doesn’t give you a clear authorial point of view on any of the relevant questions. (Whether or not this works as a story is of course a separate question I’m too biased to judged.) My general sense from my test readers has been that the questions (was what Mr. Green was doing in fact wrong? What’s wrong with the speaker’s super-harsh utilitarianism?) are capable of starting interesting EA-type conversations, and that we can trust readers to have interesting and ethical thoughts on their own.
Also I want to do a completely separate post in response to one of your short comments:
”What’s wrong with the speaker’s super-harsh utilitarianism?”
My immediate response, just automatic on reflex without engaging my brain’s slow mode, is “planning fallacy / arrogance / sin of pride.” What’s wrong is that he assumes he’s in a sufficiently strong level of knowledge, self-discipline, and self-control that he actually can pull off his ubermensch act, instead of it all going horribly wrong and blowing up in his face. That’s always what’s wrong with characters doing that. That’s why so much EA thought focuses on the question of how to ‘first, do no harm’. That’s why EA takes the High Modernists so seriously as an object lesson, that’s why rationalist circles are the only ones I’ve ever been in where you can just say “Chesterton’s Fence!” and the burden of proving an argument automatically switches over to the party arguing for a reform.
Writers have been asking that question and giving basically that answer for, what, a hundred and fifty years? Since “Crime and Punishment,” I think, which is apparently from 1866. The most recent modern artwork that I found memorable that said it was Fate/zero, back in, IIRC, 2012. I’m not going to say you can’t update timeless and eternal themes for a modern audience, that’s a perfectly reasonable thing to do, but I didn’t actually find that the story started interesting EA-type conversations (though, again, I thought it was very well-written—your voice and prose style are excellent), and the specific reason for that is because I didn’t really see that it was doing anything new or exciting, philosophically speaking. Insofar as you intended it to have a different moral answer, that… didn’t really come across? It was just asking the same question that had been answered so many times before.
(For clarification: Step One is that the story has to work as a story, I agree with that completely, if I’m passing that over in my response, it isn’t because I disagree, but because I agree too much to have anything interesting to say on it.)
The odd thing is, I would use HPMOR as a model of how to do it right. Its main failing is that it fails to make it clear that the protagonist isn’t a perfect propaganda figure, and shouldn’t be emulated—that the audience ends up thinking that the protagonist is making giant mistakes throughout the story because the author thinks those are the correct decisions, not because he’s an eleven-year-old in over his head. But the author agrees with him enough that if you aren’t paying careful attention, he comes off as an arrogant jerk the author endorses, instead of a person with many virtues and the vice of being an arrogant jerk.
(Clarification: I tried to read HPMOR twice and disliked it before I fell in love the third time. I think it is genuinely Great Art, but flawed in the way most Great Art is flawed—that of aiming for being twice as good as the best art that previously existed and ending up being an uneven mix of 160% as good and 60% as good, which may well make you throw the book at the wall during the 60% parts.)
But that side-note aside: I agree that making all EA-style figures into saints is a risk. People may well get turned off at being preached to; I know I do. Again, see my comment on “Blue Bird and Black Bird.” But...
If you’ve ever read Scott Alexander’s Unsong—I don’t know if you have—the central figure in the story is the Comet King. He isn’t the hero; the hero is a not-very-interesting nebbish stuck in the world the Comet King made. But the Comet King is EA, is really presented as purely good, and is a genuinely psychologically fascinating character. Every page he’s on blazes with light and life and joy, and yet he isn’t a Mary Sue, because by the time the main story starts, he has already lost. The book is about the aftermath of a perfectly good hero failing to save the world, the book is unquestionably pro-effective-altruist, and where the book fails, it isn’t because it’s being too pro-effective-altruist, it’s because the protagonist if the main story is really kind of dull and uninteresting.
So I think it can work. As good examples of other fiction that work despite being ideological, I’d recommend Bujold’s “Shards of Honor,” Terry Pratchett’s “Night Watch,” and, as I said, “The Lord of the Rings.”
But although I I think there’s problems with EA-heroes if badly written, I also think there’s an equal and equivalent problem with EA-style villains, even if they are well-written. It can work I agree, artistically speaking, to put a viewpoint you sympathize with but disagree with in the mouth of your villain, then make him take it to evil ends. That’s something I’ve written and it’s something I enjoy writing, because it allows you to have a strong ideological conflict between two good ends while still having a hero, and that has a lot of potential to work well.
But I’m not wholly comfortable with it, and this is why:
I think our subconscious or semi-conscious mind has bins labeled “traits of villains” and “traits of heroes,” and when we see something in real life that we are used to thinking of as always being labeled in fiction with “trait of villain” or “trait of heroes,” we apply that label in real life. When an artwork gives ‘uses reason and logic to try to maximize good’ as a trait of villains, especially in a culture in which everyone else is also using it as a trait of villain, it reinforces that as being part of the ‘villain trait’ box, since it’s usually (not in your case!) paralleled with a conservative, chauvinistic anti-intellectual badass hero. I think this will genuinely cause people to immediately round off, in real life, ‘tries to use reason and logic to maximize good’ as something that will go horribly wrong in real life, and ‘is a badass cowboy who plays by his own rules but accomplishes good things anyway because he’s a FUNDAMENTALLY GOOD PERSON’ as being something that will go right in real life. In this case, if you want me to say I have less respect for people than you do, that’s entirely possible; I don’t think people rationally feel that they do this, I don’t think people who slow down to think do this, but I think that people who operate on automatic do this; that they treat the Terminator movies as a model for how AI will go wrong in real life, and I think that this has bad consequences if everyone is saying the same thing about how ‘trying to maximize good’ will go horribly wrong.
Now, if you believe that EA will go horribly wrong in real life, if you really think that Engels ought to be treated as an early member of EA and as a model of how badly EA will go wrong in the future, it makes sense to write that. But I don’t. And from your comments, I don’t think you do.
Now, again, this isn’t a reason why I think you should never do the ‘charismatic villain arguing for an underappreciated cause’ bit. Plenty of good stories have done it. I’ve done it. But if it isn’t a cause you disapprove of, and most of the other writers writing about the cause have the same take you have on it (and if it’s a thriving cause and the take is one of the first ones you thought of, they probably will) you’re (a) doing something that is individually totally reasonable, and (b) contributing to an unreasonable aggregate. And that’s not something I really approve of, and it’s not something you… ex ante ought to expect members of the subculture to approve of? Or subsidize? Like, Larks’ second point wasn’t one I said explicitly, because it’s attributing malice where malice isn’t the most likely option, but I totally thought it before the obvious realization kicked in that it was statistically unlikely.
These thoughts are super helpful for understanding where you’re coming from, so thank you!! I really appreciate you taking the time to write them all out—my thoughts will be much shorter because I don’t have much to add, not because they weren’t thought-provoking and interesting!I
think we have somewhat different beliefs about what makes the speaker’s actions wrong—I think for me it lands very far to one side of the “clearly evil” to “clearly good” trolley problem spectrum and it’s wrongness is a) very clear to me, and b) very hard for me to pin down a reason for: I don’t really find any of the answers I can think of satisfying (including the ones in the works you mentioned—the fact that plans can fail and change the planner in unforeseen ways is a beautiful and important observation, but in this case the plan more-or-less succeeds and it still feels evil to me.) I find this combination fascinating, but I can see how this comes across rather differently if you don’t share this dissatisfaction, which it sounds like is less universal than I believed.
Unsong sounds like a very interesting piece of writing, I will have to check it out!
I’m glad you aren’t offended! I get easily worried that I might be saying things in an offensive manner and I appreciate you reassuring me that I didn’t! I am always very happy to write long and elaborate reviews of fiction and I am glad you appreciated it.
And I would agree that the protagonist is evil (indeed, he admits he is evil—he’s quite clear that he enjoyed what he did) and also took a set of actions which may have had net-positive utility. I don’t think we know that it did; it’s possible that some vague combination of making people distrust EA-style arguments, imposing costs on people both directly (his victims) and indirectly (court costs, prison costs, stress to everyone vaguely associated, costs of additional security precautions taken because his existence is evidence of the world being less safe than you thought) and so forth and so on made it net-negative.
But I will confidently deny that he was in an epistemic position to expect his actions would be positive, let alone the optimal decision. I could theoretically imagine a world in which this was false, and he genuinely did have the knowledge required for his actions to actually be both ex post and ex ante optimal, but I don’t actually think I can actually imagine a world in which I was in the epistemic state of knowing that he knew his actions were ex post and ex ante optimal; my mental state in such a world would be sufficiently different that I’m not sure I’d be the same person. So I’m really quite comfortable condemning him, though I’ll admit I’d vote for life imprisonment instead of execution.
And Unsong is very interesting! It doesn’t always succeed at what it’s doing, as I mentioned I find the protagonist kind of boring, but it’s trying such fascinating things and it succeeds sufficiently often to be worth reading.
Reply-edit for clarification to expand my response to one of your points: I think it is worth, in a lot of situations, judging based on “should it have worked,” instead of “did it work.” That your model predicted it shouldn’t work and it did work is evidence your model is seriously flawed, just to be clear, I’m not arguing we should completely throw out the experiment and just go with our previous model, but, also, we shouldn’t say “the one guy who won the lottery was right and everyone else was wrong,” because everyone who bought a ticket had the same chance of winning, and ex ante the lottery was a losing bet for all of them.
(Unless the lottery was crooked but that’s a side note.)
So, even if it worked, I still think the protagonist’s motive was unreasonable; even if it worked, I don’t feel it should have worked, statistically speaking, as opposed to him getting immediately spotted, arrested, and spending the next five years of his life in jail in which he can do no good at all. Or someone’s angry brother taking a shot at him with a firearm, causing him to die instantly after he’d donated only $8000 to Givewell’s top charities, as opposed to if he’d peacefully sat back and worked a high-paying job he would have donated $800,000 over the course of his life. Or someone successfully suing to get all the donated money back as a class-action suit, causing the Against Malaria Foundation to go bankrupt because it already spent it all on bed nets and couldn’t get a refund. Not that all of those are equally likely, but there are a lot of ways for his kind of plan to fail at levels of badness approaching these, and if they fail this way he definitely killed people, and I don’t find the assumption that he knew none of them would happen very persuasive.