Thanks for writing this! “EA is too focused on individual impact” is a common critique, but most versions of it fall flat for me. This is a very clear, thorough case for it, probably the best version of the argument I’ve read.
I agree most strongly with the dangers of internalizing the “heavy-tailed impact” perspective in the wrong way, e.g. thinking “the top people have the most impact → I’m not sure I’m one of the top people → I won’t have any meaningful impact → I might as well give up.” (To be clear, steps 2, 3, and 4 are all errors: if there’s a decent chance you’re one of the top, that’s still potentially worth going for. And even if not—most people aren’t—that doesn’t mean your impact is negligible, and certainly doesn’t mean doing nothing is better!)
I mostly disagree with the post though, for some of the same reasons as other commenters. The empirical case for heavy-tailed impact is persuasive to me, and while measuring impact reliably seems practically very hard / intractable in most cases, I don’t think it’s in principle impossible (e.g. counterfactual reasoning and Shapley values).
I’m also wary of arguments that have the form “even if X is true, believing / saying it has bad consequences, so we shouldn’t believe / say X.” I think there are usually ways to incorporate X while mitigating the downsides that might be associated with believing it; some of the links you included (e.g. Virtues for Real-World Utilitarians, Naive vs Prudent Utilitarianism) provide examples of this. Heavy-tailed impact is (if true) a very important fact about the world. So it’s worth putting in the effort to incorporate it into our beliefs effectively, doing our best to avoid the downsides you point out.
Thanks for your comment, very happy to hear that my post struck you as clear and thorough (I’m never sure how well I do on clarity in my philosophical writing, since I usually retain a bit of confusion and uncertainty even in my own mind).
I agree that many dangers of internalizing the “heavy-tailed impact” perspective in the wrong way are due to misguided inference, not a strictly necessary implication of the perspective itself.
Not least thanks to input from several comments below, I am back to reconsidering my stance on the claims made in the essay around empirical reality and around appropriate conceptual frameworks. I have tangentially encountered Shapley values before but not yet really tried to understand the concept, so if you think they could be useful for the contents of this post, I’ll try to find the time to read the article you linked; thanks for the input!
I share the wariness that you mention re “arguments that have the form “even if X is true, believing / saying it has bad consequences, so we shouldn’t believe / say X.”″. At the same time, I don’t think that these arguments are always completely groundless (at least the arguments around refraining from saying something; much more inclined to agree that we should never believe something just for the sake of supposed better consequences from believing it). I also tend to be more sympathetic to these arguments when X is very hard to know (“we don’t really have means to tell whether X is true, and since believing in X might well have bad side-effects, we should not claim that X and we should maybe even make an effort to debunk the certainty with which others claim that X”). But yes, agree that wariness (though maybe not unconditional rejection) around arguments of this form is generally warranted, to avoid misguided dogmatism in the flawed attempt to prevent (supposed) information hazards.
Shapley values are a great tool for divvying up attribution in a way that feels intuitively just, but I think for prioritization they are usually an unnecessary complication. In most cases you can only guess what they might be because you can’t mentally simulate the counterfactual worlds reliably, and your set of collaborators contains billions of potentially relevant actors. But as EAs we can “just” choose whatever action will bring about the world history with the greatest value regardless of any impact attribution to ourselves or anyone.
I like the Shapley value and think it would make similar recommendations, but it adds another layer of infeasability (and arbitrariness) on top of an already infeasably complex optimization problem without adding any value.
Then again many of us are strongly motivated by “number go up,” so Shapley values are probably helpful for self-motivation. :-3
(I think if EAs were more individualist, “the core” from cooperative game theory would be more popular than the Shapley value.)
Oh, and we get so caught up in the object-level here that we tend to fail to give praise for great posts: Great work writing this up! When I saw it, it reminded me of Brian Tomasik’s important article on the same topic, and sure enough, you linked it right before the intro! I’m always delighted when someone does their research so well that whatever random spontaneous associations I (as a random reader) have are already cited in the article!
Shapley values are a great tool for divvying up attribution in a way that feels intuitively just, but I think for prioritization they are usually an unnecessary complication. In most cases you can only guess what they might be because you can’t mentally simulate the counterfactual worlds reliably, and your set of collaborators contains billions of potentially relevant actors. [emphasis added]
From what I’ve learned about Shapley values so far, this seems to mirror my takeaway. I’m still giving myself another 2-3 days until I write up a more fleshed-out response to the commenters who recommended looking into Shapley values, but I might well end up just copying some version of the above; so thanks for formulating and putting it here already!
(I think if EAs were more individualist, “the core” from cooperative game theory would be more popular than the Shapley value.)
I do not understand this point but would like to (since the stance I developed in the original post went more in the direction of “EAs are too individualist”). If you find the time, could you explain or point to resources to explain what you mean by “the core from cooperative game theory” and how that links to (non-)individualist perspectives, and to impact modeling?
Oh, and we get so caught up in the object-level here that we tend to fail to give praise for great posts: Great work writing this up! When I saw it, it reminded me of Brian Tomasik’s important article on the same topic, and sure enough, you linked it right before the intro! I’m always delighted when someone does their research so well that whatever random spontaneous associations I (as a random reader) have are already cited in the article!
Very glad to read that, thank you for deciding to add that piece to your comment :)!
From what I’ve learned about Shapley values so far, this seems to mirror my takeaway.
Nice! To be sure, I want to put an emphasis on any kind of attribution being an unnecessary step in most cases rather than on the infeasibility of computing it.
There is complex cluelessness, nonlinearity from perturbations at perhaps even the molecular level, and a lot of moral uncertainty (because even though I think that evidential cooperation in large worlds can perhaps guide us toward solving ethics, that’ll take enormous research efforts to actually make progress on), so infeasibility is already the bread and butter of EA. In the end we’ll find a way to 80⁄20 it (or maybe −80/20 it, as you point out, and we’ll never know) to not end up paralyzed. I’ve many times just run through mental “simulations” of what I think would’ve happened if any subset of people on my team had not been around, so this 80/20ing is also possible for Shapley values.
If you do retroactive public goods funding, it’s important that the collaborators can, up front, trust that the rewards they’ll receive will be allocated justly, so being able to pay them out in proportion to the Shapley value would be great. But as altruists, we’re only concerned with rewards to the point where we don’t have to worry about our own finances anymore. What we really care about is the impact, and for that it’s not relevant to calculate any attribution.
I do not understand this point but would like to (since the stance I developed in the original post went more in the direction of “EAs are too individualist”).
I might be typical-minding EAs here (based on me and my friends) but my impression is that a lot of EAs are from lefty circles that are very optimistic about the ability of a whole civilization to cooperate and maximize some sort of well-being average. We’ve then just turned to neglectedness as our coordination mechanism rather than long, well-structured meetings, consensus voting, living together and other such classic coordination tools. In theory (or with flexible resources, dominant assurance contracts, and impact markets) that should work fine. Resources pour into campaigns that are deemed relatively neglected until they are not, at which point the resources can go to the new most neglected thing. Eventually nothing will be neglected anymore.
So it seems to me that the spirit is the same one of cooperativeness, community, and collective action. Just the tool we use to coordinate is a new one.
But some 99.9% (total guess) of the population are more individualist than that (well, I’ve only ever lived in WEIRD cultures, so I’m in an obvious bubble). They don’t think in terms of civilizations thriving or succumbing to infighting but in terms of the standing of their family in society or even just their own. (I’m excluding people in poverty here – almost anyone, including most altruists, well behave selfishly when they are in dire straits.)
Shapley values are useful for startups or similar enterprises that have a set goal that everyone works toward. The degree to which they work toward it is a fixed attribute of the collaborator. The core is more about trying to find an attribution split that sets just the right incentives to maximize the number of people who are interested in collaborating in the first place. (I think I’m getting this backwards, but something of this sort. It’s been too long since I research these things.)
If someone is very community- and collective-action-minded, they’ll have the tacit assumption that everyone is working towards the good of the whole community, and they’re just wondering how they can best contribute to that. That’s how I see most EAs.
If someone is very individualistic, they’ll want to travel 10 countries, have 2 kids, drive a car that can accelerate real fast, and get their brain frozen when they die. They’ll have no tacit assumptions about any kind of greater community or their civilization and never think about collective action. But if they did, their question would be what’s in it for them, and if there is something in it for them, if they can conspire with a smaller set of collaborators to get more of it. They’ll turn to cooperative game theory, crunch the numbers, and then pick out just the right co-conspiritors to form a subcoalition.
So that’s the intuition behind that overly terse remark in my last message. ^.^
Off topic: I have a badly structured, hastily written post where I argue that it’s not optimal for EAs to focus maximally on the one thing where they can contribute most (AI safety, animal rights, etc.) and neglect everything else but that it’s probably better to cooperate with all other efforts in their immediate environment that they endorse to at least the extent to which the median person in the environment cooperates with them. Or else we’re all (slightly) sabotaging each other all the time and get less change in aggregate. I feel like mainstream altruists (a small percentage of the population) do this better than some EAs, and it seems conceptually similar to individualism.
Very glad to read that, thank you for deciding to add that piece to your comment :)!
I agree with just about everything in this comment :)
(Also re: Shapley values—I don’t actually have strong takes on these and you shouldn’t take this as a strong endorsement of them. I haven’t engaged with them beyond reading the post I linked. But they’re a way to get some handle on cases where many people contribute to an outcome, which addresses one of the points in your post.)
I’m skeptical that Shapley values can practically help us much in addressing the ‘conceptual problem’ raised by the post. See critique of estimated Shapley values in another comment on this post
Thanks for the considered and considerate discussion
Thanks for writing this! “EA is too focused on individual impact” is a common critique, but most versions of it fall flat for me. This is a very clear, thorough case for it, probably the best version of the argument I’ve read.
I agree most strongly with the dangers of internalizing the “heavy-tailed impact” perspective in the wrong way, e.g. thinking “the top people have the most impact → I’m not sure I’m one of the top people → I won’t have any meaningful impact → I might as well give up.” (To be clear, steps 2, 3, and 4 are all errors: if there’s a decent chance you’re one of the top, that’s still potentially worth going for. And even if not—most people aren’t—that doesn’t mean your impact is negligible, and certainly doesn’t mean doing nothing is better!)
I mostly disagree with the post though, for some of the same reasons as other commenters. The empirical case for heavy-tailed impact is persuasive to me, and while measuring impact reliably seems practically very hard / intractable in most cases, I don’t think it’s in principle impossible (e.g. counterfactual reasoning and Shapley values).
I’m also wary of arguments that have the form “even if X is true, believing / saying it has bad consequences, so we shouldn’t believe / say X.” I think there are usually ways to incorporate X while mitigating the downsides that might be associated with believing it; some of the links you included (e.g. Virtues for Real-World Utilitarians, Naive vs Prudent Utilitarianism) provide examples of this. Heavy-tailed impact is (if true) a very important fact about the world. So it’s worth putting in the effort to incorporate it into our beliefs effectively, doing our best to avoid the downsides you point out.
Thanks for your comment, very happy to hear that my post struck you as clear and thorough (I’m never sure how well I do on clarity in my philosophical writing, since I usually retain a bit of confusion and uncertainty even in my own mind).
I agree that many dangers of internalizing the “heavy-tailed impact” perspective in the wrong way are due to misguided inference, not a strictly necessary implication of the perspective itself.
Not least thanks to input from several comments below, I am back to reconsidering my stance on the claims made in the essay around empirical reality and around appropriate conceptual frameworks. I have tangentially encountered Shapley values before but not yet really tried to understand the concept, so if you think they could be useful for the contents of this post, I’ll try to find the time to read the article you linked; thanks for the input!
I share the wariness that you mention re “arguments that have the form “even if X is true, believing / saying it has bad consequences, so we shouldn’t believe / say X.”″. At the same time, I don’t think that these arguments are always completely groundless (at least the arguments around refraining from saying something; much more inclined to agree that we should never believe something just for the sake of supposed better consequences from believing it). I also tend to be more sympathetic to these arguments when X is very hard to know (“we don’t really have means to tell whether X is true, and since believing in X might well have bad side-effects, we should not claim that X and we should maybe even make an effort to debunk the certainty with which others claim that X”). But yes, agree that wariness (though maybe not unconditional rejection) around arguments of this form is generally warranted, to avoid misguided dogmatism in the flawed attempt to prevent (supposed) information hazards.
Shapley values are a great tool for divvying up attribution in a way that feels intuitively just, but I think for prioritization they are usually an unnecessary complication. In most cases you can only guess what they might be because you can’t mentally simulate the counterfactual worlds reliably, and your set of collaborators contains billions of potentially relevant actors. But as EAs we can “just” choose whatever action will bring about the world history with the greatest value regardless of any impact attribution to ourselves or anyone.
I like the Shapley value and think it would make similar recommendations, but it adds another layer of infeasability (and arbitrariness) on top of an already infeasably complex optimization problem without adding any value.
Then again many of us are strongly motivated by “number go up,” so Shapley values are probably helpful for self-motivation. :-3
(I think if EAs were more individualist, “the core” from cooperative game theory would be more popular than the Shapley value.)
Oh, and we get so caught up in the object-level here that we tend to fail to give praise for great posts: Great work writing this up! When I saw it, it reminded me of Brian Tomasik’s important article on the same topic, and sure enough, you linked it right before the intro! I’m always delighted when someone does their research so well that whatever random spontaneous associations I (as a random reader) have are already cited in the article!
From what I’ve learned about Shapley values so far, this seems to mirror my takeaway. I’m still giving myself another 2-3 days until I write up a more fleshed-out response to the commenters who recommended looking into Shapley values, but I might well end up just copying some version of the above; so thanks for formulating and putting it here already!
I do not understand this point but would like to (since the stance I developed in the original post went more in the direction of “EAs are too individualist”). If you find the time, could you explain or point to resources to explain what you mean by “the core from cooperative game theory” and how that links to (non-)individualist perspectives, and to impact modeling?
Very glad to read that, thank you for deciding to add that piece to your comment :)!
Nice! To be sure, I want to put an emphasis on any kind of attribution being an unnecessary step in most cases rather than on the infeasibility of computing it.
There is complex cluelessness, nonlinearity from perturbations at perhaps even the molecular level, and a lot of moral uncertainty (because even though I think that evidential cooperation in large worlds can perhaps guide us toward solving ethics, that’ll take enormous research efforts to actually make progress on), so infeasibility is already the bread and butter of EA. In the end we’ll find a way to 80⁄20 it (or maybe −80/20 it, as you point out, and we’ll never know) to not end up paralyzed. I’ve many times just run through mental “simulations” of what I think would’ve happened if any subset of people on my team had not been around, so this 80/20ing is also possible for Shapley values.
If you do retroactive public goods funding, it’s important that the collaborators can, up front, trust that the rewards they’ll receive will be allocated justly, so being able to pay them out in proportion to the Shapley value would be great. But as altruists, we’re only concerned with rewards to the point where we don’t have to worry about our own finances anymore. What we really care about is the impact, and for that it’s not relevant to calculate any attribution.
I might be typical-minding EAs here (based on me and my friends) but my impression is that a lot of EAs are from lefty circles that are very optimistic about the ability of a whole civilization to cooperate and maximize some sort of well-being average. We’ve then just turned to neglectedness as our coordination mechanism rather than long, well-structured meetings, consensus voting, living together and other such classic coordination tools. In theory (or with flexible resources, dominant assurance contracts, and impact markets) that should work fine. Resources pour into campaigns that are deemed relatively neglected until they are not, at which point the resources can go to the new most neglected thing. Eventually nothing will be neglected anymore.
So it seems to me that the spirit is the same one of cooperativeness, community, and collective action. Just the tool we use to coordinate is a new one.
But some 99.9% (total guess) of the population are more individualist than that (well, I’ve only ever lived in WEIRD cultures, so I’m in an obvious bubble). They don’t think in terms of civilizations thriving or succumbing to infighting but in terms of the standing of their family in society or even just their own. (I’m excluding people in poverty here – almost anyone, including most altruists, well behave selfishly when they are in dire straits.)
Shapley values are useful for startups or similar enterprises that have a set goal that everyone works toward. The degree to which they work toward it is a fixed attribute of the collaborator. The core is more about trying to find an attribution split that sets just the right incentives to maximize the number of people who are interested in collaborating in the first place. (I think I’m getting this backwards, but something of this sort. It’s been too long since I research these things.)
If someone is very community- and collective-action-minded, they’ll have the tacit assumption that everyone is working towards the good of the whole community, and they’re just wondering how they can best contribute to that. That’s how I see most EAs.
If someone is very individualistic, they’ll want to travel 10 countries, have 2 kids, drive a car that can accelerate real fast, and get their brain frozen when they die. They’ll have no tacit assumptions about any kind of greater community or their civilization and never think about collective action. But if they did, their question would be what’s in it for them, and if there is something in it for them, if they can conspire with a smaller set of collaborators to get more of it. They’ll turn to cooperative game theory, crunch the numbers, and then pick out just the right co-conspiritors to form a subcoalition.
So that’s the intuition behind that overly terse remark in my last message. ^.^
Off topic: I have a badly structured, hastily written post where I argue that it’s not optimal for EAs to focus maximally on the one thing where they can contribute most (AI safety, animal rights, etc.) and neglect everything else but that it’s probably better to cooperate with all other efforts in their immediate environment that they endorse to at least the extent to which the median person in the environment cooperates with them. Or else we’re all (slightly) sabotaging each other all the time and get less change in aggregate. I feel like mainstream altruists (a small percentage of the population) do this better than some EAs, and it seems conceptually similar to individualism.
Awww! :-D
I agree with just about everything in this comment :)
(Also re: Shapley values—I don’t actually have strong takes on these and you shouldn’t take this as a strong endorsement of them. I haven’t engaged with them beyond reading the post I linked. But they’re a way to get some handle on cases where many people contribute to an outcome, which addresses one of the points in your post.)
I’m skeptical that Shapley values can practically help us much in addressing the ‘conceptual problem’ raised by the post. See critique of estimated Shapley values in another comment on this post
Thanks for the considered and considerate discussion