Thanks for writing this! I’ve long been suspicious of this idea but haven’t got round to investigating the claim itself, and my skepticism of it, fully, so I super appreciate you kicking off this discussion.
I also identify with ‘do I disagree with this empirically or am I just uneasy with the vibes/frame, how to tease those apart, ?’
For people who broadly agree with the idea that Sarah is critiquing: what do you think is the best defence of it, arguing from first principles and data as much as possible?
I have a couple of other queries/scepticisms about the power-law argument. I haven’t read all the other comments, so sorry if I repeat stuff said elsewhere.
1. Does it empirically hold up even assuming you can attribute stuff to individuals? You focus a lot on critiquing conceptual idea of the individual impact of one person (since most actions happen in the context of other actions and actors). I think I also have empirical disagreements with the claim even if we can tease out what impact comes from which person.
It feels to me like EAs sometimes over-generalize that finding from global health interventions — where I don’t doubt that it holds up — to other domains, where it hasn’t been established (e.g., orgs working in longtermist causes, or people compared to their peers, or actions one takes in one’s career). It’s possible that there *is* more discussion and substantiation of this idea out there, but I just haven’t seen it.
Like, even if we accept that (per your example) the President does have much more impact than the average person, or (per Jeff’s example above) a larger donor has more impact than a smaller donor to the same charity, can I generalize that to the actions available to me personally, or to questions of how impactful ‘overall’ I can be compared to my peers? What’s the empirical justification for such generalizations?
2. Is the bar low? Does this depend on how you define the space?
Benjamin Todd, in the article you linked, claims that the power-law pattern has been found in many areas of social impact. I’m sure this is true, but I want to point out that this is kind of contingent, not a law of nature. E.g., I’d guess this is due to some combination of ‘there’s not a culture of measuring outcomes and prioritization in general philanthropy’ (that’s kind of the whole point of EA) and/or ‘the world is very complicated and it’s hard to know ex ante (and sometimes even ex post) what will work/what did work’.
Like, if there were a culture shift in philanthropy across the board meaning that interventions would only be funded or carried out if they met some effectiveness bar, would we still expect interventions to be power-law distributed? Surely less so?
To frame this another way, imagine I said to you ‘the nutritional value of foods follows a power-law distribution’, and you were like ‘hmm’, but then it turned out that among ‘foods’ I was counting inedible objects like chairs and rocks and grass. So yes, only a minority of objects have most of the nutritional value, but anything we’d call food is in the heavy tail, and this is a kind of silly frame.
This point isn’t fully worked out but yeah, I wonder if ‘what counts as the distribution’ is kind of socially constructed in a way that’s not always helpful.
On 2, I like this point about the distribution being shaped by the choices of others, I think it is quite true that if more people cared about impact it would be a lot harder to counterfatually achieve very high impact actions (because there would be so much ‘competition’ with other impact seekers). Reminiscent of how financial markets are pretty efficient because so many people are seeking to make money trading—I think if a similar number of people were looking to succeed in the ‘impact market’ there wouldn’t be these super cost-effective low-hanging fruit left (lead elimination and the like).
I think this then relates to point 1, as if there was an efficient impact market, it would be quite surprising for impact to be heavy-tailed. But as long as most people are focused on things other than impact I think my default assumption is it won’t be too hard to find things that are a lot higher impact than the average. But I agree that this is not definitive and in areas like longtermist interventions where measurement is so hard we don’t have empirical evidence of this.
Thanks for writing this! I’ve long been suspicious of this idea but haven’t got round to investigating the claim itself, and my skepticism of it, fully, so I super appreciate you kicking off this discussion.
I also identify with ‘do I disagree with this empirically or am I just uneasy with the vibes/frame, how to tease those apart, ?’
For people who broadly agree with the idea that Sarah is critiquing: what do you think is the best defence of it, arguing from first principles and data as much as possible?
I have a couple of other queries/scepticisms about the power-law argument. I haven’t read all the other comments, so sorry if I repeat stuff said elsewhere.
1. Does it empirically hold up even assuming you can attribute stuff to individuals?
You focus a lot on critiquing conceptual idea of the individual impact of one person (since most actions happen in the context of other actions and actors). I think I also have empirical disagreements with the claim even if we can tease out what impact comes from which person.
It feels to me like EAs sometimes over-generalize that finding from global health interventions — where I don’t doubt that it holds up — to other domains, where it hasn’t been established (e.g., orgs working in longtermist causes, or people compared to their peers, or actions one takes in one’s career). It’s possible that there *is* more discussion and substantiation of this idea out there, but I just haven’t seen it.
Like, even if we accept that (per your example) the President does have much more impact than the average person, or (per Jeff’s example above) a larger donor has more impact than a smaller donor to the same charity, can I generalize that to the actions available to me personally, or to questions of how impactful ‘overall’ I can be compared to my peers? What’s the empirical justification for such generalizations?
2. Is the bar low? Does this depend on how you define the space?
Benjamin Todd, in the article you linked, claims that the power-law pattern has been found in many areas of social impact. I’m sure this is true, but I want to point out that this is kind of contingent, not a law of nature. E.g., I’d guess this is due to some combination of ‘there’s not a culture of measuring outcomes and prioritization in general philanthropy’ (that’s kind of the whole point of EA) and/or ‘the world is very complicated and it’s hard to know ex ante (and sometimes even ex post) what will work/what did work’.
Like, if there were a culture shift in philanthropy across the board meaning that interventions would only be funded or carried out if they met some effectiveness bar, would we still expect interventions to be power-law distributed? Surely less so?
To frame this another way, imagine I said to you ‘the nutritional value of foods follows a power-law distribution’, and you were like ‘hmm’, but then it turned out that among ‘foods’ I was counting inedible objects like chairs and rocks and grass. So yes, only a minority of objects have most of the nutritional value, but anything we’d call food is in the heavy tail, and this is a kind of silly frame.
This point isn’t fully worked out but yeah, I wonder if ‘what counts as the distribution’ is kind of socially constructed in a way that’s not always helpful.
On 2, I like this point about the distribution being shaped by the choices of others, I think it is quite true that if more people cared about impact it would be a lot harder to counterfatually achieve very high impact actions (because there would be so much ‘competition’ with other impact seekers). Reminiscent of how financial markets are pretty efficient because so many people are seeking to make money trading—I think if a similar number of people were looking to succeed in the ‘impact market’ there wouldn’t be these super cost-effective low-hanging fruit left (lead elimination and the like).
I think this then relates to point 1, as if there was an efficient impact market, it would be quite surprising for impact to be heavy-tailed. But as long as most people are focused on things other than impact I think my default assumption is it won’t be too hard to find things that are a lot higher impact than the average. But I agree that this is not definitive and in areas like longtermist interventions where measurement is so hard we don’t have empirical evidence of this.