Hey Rohin, without getting into the details, I’m pretty unsure whether correcting for impacts from multiple orgs makes 80,000 Hours look better or worse, so I’m not sure how we should act. We win out in some cases (we get bragging rights from someone who found out about EA from another source then changes their career) and lose in others (someone who finds out about GiveWell through 80k but doesn’t then attribute their donations to us).
There’s double counting yes, but the orgs are also legitimately complementary of one another—not sure if the double counting exceeds the real complementarity.
We could try to measure the benefit/cost of the movement as a whole—this gets rid of the attribution and complementarity problem, though loses the ability to tell what is best within the movement.
I’m pretty unsure whether correcting for impacts from multiple orgs makes 80,000 Hours look better or worse
I’m a little unclear on what you mean here. I see three different factors:
Various orgs are undercounting their impact because they don’t count small changes that are part of a larger effort, even though in theory from a single player perspective, they should count the impact.
In some cases, two (or more) organizations both reach out to an individual, but either one of the organizations would have been sufficient, so neither of them get any counterfactual impact (more generally, the sum of the individually recorded impacts is less than the impact of the system as a whole)
Multiple orgs have claimed the same object-level impact (eg. an additional $100,000 to AMF from a GWWC pledge) because they were all counterfactually responsible for it (more generally, the sum of the individually recorded impacts is more than the impact of the system as a whole).
Let’s suppose:
X is the impact of an org from a single player perspective
Y is the impact of an org taking a system-level view (so that the sum of Y values for all orgs is equal to the impact of the system as a whole)
Point 1 doesn’t change X or Y, but it does change the estimate we make of X and Y, and tends to increase it.
Point 2 can only tend to make Y > X.
Point 3 can only tend to make Y < X.
Is your claim that the combination of points 1 and 2 may outweigh point 3, or just that point 2 may outweigh point 3? I can believe the former, but the latter seems unlikely—it doesn’t seem very common for many separate orgs to all be capable of making the same change, it seems more likely to me that in such cases all of the orgs are necessary which would be an instance of point 3.
We could try to measure the benefit/cost of the movement as a whole
Yeah, this is the best idea I’ve come up with so far, but I don’t really like it much. (Do you include local groups? Do you include the time that EAs spend talking to their friends? If not, how do you determine how much of the impact to attribute to meta orgs vs. normal network effects?) It would be a good start though.
Another possibility is to cross-reference data between all meta orgs, and try to figure out whether for each person, the sum of the impacts recorded by all meta orgs is a reasonable number. Not sure how feasible this actually is (in particular, it’s hard to know what a “reasonable number” would be, and coordinating among so many organizations seems quite hard).
Hey Rohin, without getting into the details, I’m pretty unsure whether correcting for impacts from multiple orgs makes 80,000 Hours look better or worse, so I’m not sure how we should act. We win out in some cases (we get bragging rights from someone who found out about EA from another source then changes their career) and lose in others (someone who finds out about GiveWell through 80k but doesn’t then attribute their donations to us).
There’s double counting yes, but the orgs are also legitimately complementary of one another—not sure if the double counting exceeds the real complementarity.
We could try to measure the benefit/cost of the movement as a whole—this gets rid of the attribution and complementarity problem, though loses the ability to tell what is best within the movement.
I’m a little unclear on what you mean here. I see three different factors:
Various orgs are undercounting their impact because they don’t count small changes that are part of a larger effort, even though in theory from a single player perspective, they should count the impact.
In some cases, two (or more) organizations both reach out to an individual, but either one of the organizations would have been sufficient, so neither of them get any counterfactual impact (more generally, the sum of the individually recorded impacts is less than the impact of the system as a whole)
Multiple orgs have claimed the same object-level impact (eg. an additional $100,000 to AMF from a GWWC pledge) because they were all counterfactually responsible for it (more generally, the sum of the individually recorded impacts is more than the impact of the system as a whole).
Let’s suppose:
X is the impact of an org from a single player perspective
Y is the impact of an org taking a system-level view (so that the sum of Y values for all orgs is equal to the impact of the system as a whole)
Point 1 doesn’t change X or Y, but it does change the estimate we make of X and Y, and tends to increase it.
Point 2 can only tend to make Y > X.
Point 3 can only tend to make Y < X.
Is your claim that the combination of points 1 and 2 may outweigh point 3, or just that point 2 may outweigh point 3? I can believe the former, but the latter seems unlikely—it doesn’t seem very common for many separate orgs to all be capable of making the same change, it seems more likely to me that in such cases all of the orgs are necessary which would be an instance of point 3.
Yeah, this is the best idea I’ve come up with so far, but I don’t really like it much. (Do you include local groups? Do you include the time that EAs spend talking to their friends? If not, how do you determine how much of the impact to attribute to meta orgs vs. normal network effects?) It would be a good start though.
Another possibility is to cross-reference data between all meta orgs, and try to figure out whether for each person, the sum of the impacts recorded by all meta orgs is a reasonable number. Not sure how feasible this actually is (in particular, it’s hard to know what a “reasonable number” would be, and coordinating among so many organizations seems quite hard).