Here’s a puzzle I’ve thought about a few times recently:
The impact of an activity (I) is due to two factors, X and Y. Those factors combine multiplicatively to produce impact. Examples include:
The funding of an organization and the people working at the org
A manager of a team who acts as a lever on the work of their reports
The EA Forum acts as a lever on top of the efforts of the authors
A product manager joins a team of engineers
Let’s assume in all of these scenarios that you are only one of the players in the situation, and you can only control your own actions.
From a counterfactual analysis, if you can increase your contribution by 10%, then you increase the impact by 10%, end of story.
From a Shapley Value perspective, it’s a bit more complicated, but we can start with a prior that you split your impact evenly with the other players.
Both these perspectives have a lot going for them! The counterfactual analysis has important correspondences to reality. If you do 10% better at your job the world gets 0.1I better. Shapley Values prevent the scenario where the multiplicative impact causes the involved agents to collectively contribute too much.
I notice myself feeling relatively more philosophically comfortable running with the Shapely Value analysis in the scenario where I feel aligned with the other players in the game. And potentially the Shapley Value approach downsides go down if I actually run the math (Fake edit: I ran a really hacky guess as to how I’d calculate this using this calculator and it wasn’t that helpful).
But I don’t feel 100% bought-in to the Shapley Value approach, and think there’s a value in paying attention to the counterfactuals. My unprincipled compromise approach would be to take some weighted geometric mean and call it a day.
I’m annoyed at vague “value” questions. If you ask a specific question the puzzle dissolves. What should you do to make the world go better? Maximize world-EV, or equivalently maximize your counterfactual value (not in the maximally-naive way — take into account how “your actions” affect “others’ actions”). How should we distribute a fixed amount of credit or a prize between contributors? Something more Shapley-flavored, although this isn’t really the question that Shapley answers (and that question is almost never relevant, in my possibly controversial opinion).
Happy to talk about well-specified questions. Annoyed at questions like “should I use counterfactuals here” that don’t answer the obvious reply, “use them FOR WHAT?”
I don’t feel 100% bought-in to the Shapley Value approach, and think there’s a value in paying attention to the counterfactuals. My unprincipled compromise approach would be to take some weighted geometric mean and call it a day.
FOR WHAT?
Let’s assume in all of these scenarios that you are only one of the players in the situation, and you can only control your own actions.
If this is your specification (implicitly / further specification: you’re an altruist trying to maximize total value, deciding how to trade off between increasing X and doing good in other ways) then there is a correct answer — maximize counterfactual value (this is equivalent to maximizing total value, or argmaxing total value over your possible actions), not your personal Shapley value or anything else. (Just like in all other scenarios. Multiplicative-ness is irrelevant. Maximizing counterfactual value is always the answer to questions about what action to take.)
I think you’re right to be more uncomfortable with the counterfactual analysis in cases where you’re aligned with the other players in the game. Cribbing from a comment I’ve made on this topic before on the forum:
I think that counterfactual analysis is the right approach to take on the first point if/when you have full information about what’s going on. But in practice you essentially never have proper information on what everyone else’s counterfactuals would look like according to different actions you could take.
If everyone thinks in terms of something like “approximate shares of moral credit”, then this can help in coordinating to avoid situations where a lot of people work on a project because it seems worth it on marginal impact, but it would have been better if they’d all done something different. Doing this properly might mean impact markets (where the “market” part works as a mechanism for distributing cognition, so that each market participant is responsible for thinking through their own alternative options, and feeding that information into the system via their willingness to do work for different amounts of pay), but I think that you can get some rough approximation to the benefits of impact markets without actual markets by having people do the things they would have done with markets—and in this context, that means paying attention to the share of credit different parties would get.
Shapley values are one way to divide up that credit. They have some theoretical appeal, but it’s basically as “what would a fair division of credit be, which divides the surplus compared to outside options”. And they’re extremely complex to calculate so in practice I’d recommend against even trying. Instead just think of it as an approximate bargaining solution between the parties, and use some other approximation to bargaining solutions—I think Austin’s practice of looking to the business world for guidance is a reasonable approach here.
(If there’s nobody whom you’re plausibly coordinating with then I think trying to do a rough counterfactual analysis is reasonable, but that doesn’t feel true of any of your examples.)
So, as a self-professed mechanism geek, I feel like the Shapley Value stuff should be my cup of tea, but I must confess I’ve never wrapped my head around it. I’ve read Nuno’s post and played with the calculator, but still have little intuitive sense of how these things work even with toy examples, and definitely no idea on how they can be applied in real-world settings.
I think delineating impact assignment for shared projects is important, though I generally look to the business world for inspiration on the most battle-tested versions of impact assignment (equity, commissions, advertising fees, etc). Startup/tech company equity & compensation, for example, at least provides a clear answer to “how much does the employer value your work”. The answer is suboptimal in many ways (eg my guess is startups by default assign too much equity to the founders), but at least it provides a simple starting point; better to make up numbers and all that.
Rationally and ideally, you should just maximize the expected value of your actions, taking into account your potential influence on others and their costs, including opportunity costs. This just follows assuming expected utility theory axioms. It doesn’t matter that there are other agents; you can just capture them as part of your outcomes under consideration.
When you’re assigning credit across other actors whose impacts aren’t roughly independent, including for estimating their cost-effectiveness for funding, Shapley values (or something similar) can be useful. You want assigned credit to sum to 100% to avoid double counting or undercounting. (Credit for some actors can even be negative, though.)
But, if you were going to calculate Shapley values, which means estimating a bunch of subgroup counterfactuals that didn’t or wouldn’t happen, anyway, you may be able to just directly estimate how to best allocate resources instead. You could skip credit assignments (EDIT especially ex ante credit assignments, or when future work will be similar to past work in effects).
I think you can get closer to dissolving this problem by considering why you’re assigning credit. Often, we’re assigning some kind of finite financial rewards.
Imagine that a group of n people have all jointly created $1 of value in the world, and that if any one of them did not participate, there would only be $0 units of value. Clearly, we can’t give $1 to all of them, because then we would be paying $n to reward an event that only created $0 of value, which is inefficient. If, however, only the first guy (i=1) is an “agent” that responds to incentives, while the others (1<=i<=n) are “environment” whose behaviour is unresponsive to incentives, then it is fine to give the first guy a reward of $1.
This is how you can ground the idea that agents who cooperate should share their praise (something like a Shapley Value approach), whereas rival agents who don’t buy into your reward scheme should be left out of the shapley calculation.
I’ll give general takes in another comment, but I just wanted to call out how I think that at least for some of your examples the assumptions are unrealistic (and this can make the puzzle sound worse than it is).
Take the case of “The funding of an organization and the people working at the org”. In this case the must factors combine in a sub-multiplicative way rather than a multiplicative way. For it’s clear that if you double the funding and double the people working at the org you should approximately double the output (rather than quadruple it). I think that Cobb-Douglas production functions are often a useful modelling tool here.
In the case of managers or the Forum I suspect that it’s also not quite multiplicative—but a bit closer to it. In any case I do think that after accounting for this there’s still a puzzle about how to evaluate it.
Here’s a puzzle I’ve thought about a few times recently:
The impact of an activity (I) is due to two factors, X and Y. Those factors combine multiplicatively to produce impact. Examples include:
The funding of an organization and the people working at the org
A manager of a team who acts as a lever on the work of their reports
The EA Forum acts as a lever on top of the efforts of the authors
A product manager joins a team of engineers
Let’s assume in all of these scenarios that you are only one of the players in the situation, and you can only control your own actions.
From a counterfactual analysis, if you can increase your contribution by 10%, then you increase the impact by 10%, end of story.
From a Shapley Value perspective, it’s a bit more complicated, but we can start with a prior that you split your impact evenly with the other players.
Both these perspectives have a lot going for them! The counterfactual analysis has important correspondences to reality. If you do 10% better at your job the world gets 0.1I better. Shapley Values prevent the scenario where the multiplicative impact causes the involved agents to collectively contribute too much.
I notice myself feeling relatively more philosophically comfortable running with the Shapely Value analysis in the scenario where I feel aligned with the other players in the game. And potentially the Shapley Value approach downsides go down if I actually run the math (Fake edit: I ran a really hacky guess as to how I’d calculate this using this calculator and it wasn’t that helpful).
But I don’t feel 100% bought-in to the Shapley Value approach, and think there’s a value in paying attention to the counterfactuals. My unprincipled compromise approach would be to take some weighted geometric mean and call it a day.
Interested in comments.
I’m annoyed at vague “value” questions. If you ask a specific question the puzzle dissolves. What should you do to make the world go better? Maximize world-EV, or equivalently maximize your counterfactual value (not in the maximally-naive way — take into account how “your actions” affect “others’ actions”). How should we distribute a fixed amount of credit or a prize between contributors? Something more Shapley-flavored, although this isn’t really the question that Shapley answers (and that question is almost never relevant, in my possibly controversial opinion).
Happy to talk about well-specified questions. Annoyed at questions like “should I use counterfactuals here” that don’t answer the obvious reply, “use them FOR WHAT?”
FOR WHAT?
If this is your specification (implicitly / further specification: you’re an altruist trying to maximize total value, deciding how to trade off between increasing X and doing good in other ways) then there is a correct answer — maximize counterfactual value (this is equivalent to maximizing total value, or argmaxing total value over your possible actions), not your personal Shapley value or anything else. (Just like in all other scenarios. Multiplicative-ness is irrelevant. Maximizing counterfactual value is always the answer to questions about what action to take.)
I think you’re right to be more uncomfortable with the counterfactual analysis in cases where you’re aligned with the other players in the game. Cribbing from a comment I’ve made on this topic before on the forum:
I think that counterfactual analysis is the right approach to take on the first point if/when you have full information about what’s going on. But in practice you essentially never have proper information on what everyone else’s counterfactuals would look like according to different actions you could take.
If everyone thinks in terms of something like “approximate shares of moral credit”, then this can help in coordinating to avoid situations where a lot of people work on a project because it seems worth it on marginal impact, but it would have been better if they’d all done something different. Doing this properly might mean impact markets (where the “market” part works as a mechanism for distributing cognition, so that each market participant is responsible for thinking through their own alternative options, and feeding that information into the system via their willingness to do work for different amounts of pay), but I think that you can get some rough approximation to the benefits of impact markets without actual markets by having people do the things they would have done with markets—and in this context, that means paying attention to the share of credit different parties would get.
Shapley values are one way to divide up that credit. They have some theoretical appeal, but it’s basically as “what would a fair division of credit be, which divides the surplus compared to outside options”. And they’re extremely complex to calculate so in practice I’d recommend against even trying. Instead just think of it as an approximate bargaining solution between the parties, and use some other approximation to bargaining solutions—I think Austin’s practice of looking to the business world for guidance is a reasonable approach here.
(If there’s nobody whom you’re plausibly coordinating with then I think trying to do a rough counterfactual analysis is reasonable, but that doesn’t feel true of any of your examples.)
So, as a self-professed mechanism geek, I feel like the Shapley Value stuff should be my cup of tea, but I must confess I’ve never wrapped my head around it. I’ve read Nuno’s post and played with the calculator, but still have little intuitive sense of how these things work even with toy examples, and definitely no idea on how they can be applied in real-world settings.
I think delineating impact assignment for shared projects is important, though I generally look to the business world for inspiration on the most battle-tested versions of impact assignment (equity, commissions, advertising fees, etc). Startup/tech company equity & compensation, for example, at least provides a clear answer to “how much does the employer value your work”. The answer is suboptimal in many ways (eg my guess is startups by default assign too much equity to the founders), but at least it provides a simple starting point; better to make up numbers and all that.
Rationally and ideally, you should just maximize the expected value of your actions, taking into account your potential influence on others and their costs, including opportunity costs. This just follows assuming expected utility theory axioms. It doesn’t matter that there are other agents; you can just capture them as part of your outcomes under consideration.
When you’re assigning credit across other actors whose impacts aren’t roughly independent, including for estimating their cost-effectiveness for funding, Shapley values (or something similar) can be useful. You want assigned credit to sum to 100% to avoid double counting or undercounting. (Credit for some actors can even be negative, though.)
But, if you were going to calculate Shapley values, which means estimating a bunch of subgroup counterfactuals that didn’t or wouldn’t happen, anyway, you may be able to just directly estimate how to best allocate resources instead. You could skip credit assignments (EDIT especially ex ante credit assignments, or when future work will be similar to past work in effects).
(I endorse this.)
FYI thanks for all the helpful comments here — I promptly got covid and haven’t had a chance to respond 😅
I think you can get closer to dissolving this problem by considering why you’re assigning credit. Often, we’re assigning some kind of finite financial rewards.
Imagine that a group of n people have all jointly created $1 of value in the world, and that if any one of them did not participate, there would only be $0 units of value. Clearly, we can’t give $1 to all of them, because then we would be paying $n to reward an event that only created $0 of value, which is inefficient. If, however, only the first guy (i=1) is an “agent” that responds to incentives, while the others (1<=i<=n) are “environment” whose behaviour is unresponsive to incentives, then it is fine to give the first guy a reward of $1.
This is how you can ground the idea that agents who cooperate should share their praise (something like a Shapley Value approach), whereas rival agents who don’t buy into your reward scheme should be left out of the shapley calculation.
I’ll give general takes in another comment, but I just wanted to call out how I think that at least for some of your examples the assumptions are unrealistic (and this can make the puzzle sound worse than it is).
Take the case of “The funding of an organization and the people working at the org”. In this case the must factors combine in a sub-multiplicative way rather than a multiplicative way. For it’s clear that if you double the funding and double the people working at the org you should approximately double the output (rather than quadruple it). I think that Cobb-Douglas production functions are often a useful modelling tool here.
In the case of managers or the Forum I suspect that it’s also not quite multiplicative—but a bit closer to it. In any case I do think that after accounting for this there’s still a puzzle about how to evaluate it.