Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
Edited after more careful reading of the post
As you say in the post, I think all these things can be true:
1) The expected counterfactual value is all that matters (i.e. we can ignore Shapley values).
2) The 3 vaccine programs had zero counterfactual value in hindsight.
3) It was still the correct decision to work on each of them at the time, with the information that was available then.
At the time, none of the 3 programs knew that any of the others would succeed, so the expected value of each programme was very high. Itâs not clear to me why the â12.5%âČ figure in your probabilistic analysis is getting anything wrong.
If one vaccine program actually had known with certainty that the other two would already succeed, and if that really rendered their own counterfactual value 0 (probably not literally true in practice but granted for the sake of argument) then it seems very plausible to me that they probably should have focused on other things (despite what a Shapley value analysis might have told them).
It gets more complicated if you imagine that each of the three knew with certainty that the other two could succeed if they tried, but might not necessarily actually do it, because they will be reasoning similarly. Then it becomes a sort of game of chicken between the three of them as to which will actually do the work, and I think this is the kind of anti-cooperative nature of counterfactual value that youâre alluding to. This is a potential problem with focusing only on counterfactuals, but focusing only on Shapley values has problems too, because it gives the wrong answer in cases where the decisions of others are already set in stone.
Toby Ord left a really good comment on the linked post on Shapley values that I think itâs worth people reading, and I would echo his recommendation to read Parfitâs Five Mistakes in Moral Mathematics for a really good discussion of these problems.
The most important thing about your decision theory is that it shouldnât predictably and in expectation leave you worse off than if you had used a different approach. My claim in the post is that weâre using such an approach, and it leaves us predictably worse off in certain specific cases.
For example, I strongly disagree with the idea that itâs coherent to say that all three programs would have zero value in hindsight, and that the true value is 12.5% each, because it means that in many plausible cases, where return on investment from a single working solution is, say, only 3x the bar for funding, we should fund none of them.
And regarding Tobyâs comment, I agree with himâthe problem I pointed to in the last section is specifically and exactly relevantâweâre committing to things on an ongoing and variable basis, along with others. Itâs a game-theoretic setup, and as he suggests, âShapley [should be] applied when the other agentsâ decisions are still âliveâââwhich is the case here. When EA was small, this was less problematic. Weâre big enough to factor into other large playerâs calculus now, so we canât pretend we move last in the game. (And even when we think we know we are, in fact, moving last in committing funds after everyone else, it is an iterated game, so weâre not actually doing so.)
To arrive at the 12.5% value, you were assuming that you knew with certainty that the other two teams will try to create the vaccine without you (and that they each have a 50% chance of succeeding). And I still think that under that assumption, 12.5% is the correct figure.
If I understand your reasoning correctly for why you think this is incoherent, itâs because:
If the 3 teams independently arrive at the 12.5% figure, and each use that to decide whether to proceed, then you might end up in a situation where none of them fund it, despite it being clearly worth it overall.
But in making this argument, youâve changed the problem. The other 2 teams are now no longer funding the vaccine with certainty, they are also making decisions based on counterfactual cost-benefit. So 12.5% is no longer the right number.
To work out what the new right number is, you have to decide how likely you think it is that the other 2 teams will try to make a vaccine, and that might be tricky. Whatever arguments you think of, you might have to factor in whether the other 2 teams will be thinking similarly. But if you really do all have the same goals, and thereâs only 3 of you, thereâs a fairly easy solution here, which is to just talk to each other! As a group you can collectively figure out what set of actions distributed among the 3 of you will maximize the global good, and then just do those. Shapley values donât have to come into it.
It gets more complicated if thereâs too many actors involved to all get together and figure things out like this, or if you donât all have exactly the same goals, and maybe there is scope for concepts like Shapley values be useful in those cases. And you might well be right that EA is now often in situations like these.
Maybe we donât disagree much in that case. I just wanted to push back a bit against the way you presented Shapley values here (e.g. as the âindisputably correct way to think about counterfactual value in scenarios with cooperationâ). Shapley values are not always the right way to approach these problems. For example, the two thought experiments at the beginning of Parfitâs paper I linked above are specific cases where Shapley values would leave you predictably worse off (and all decision theories will have some cases where they leave you predictably worse off).
Letâs make the problem as simple as possible; you have a simple intervention with 3 groups pursuing it. Each has an independent 50% chance of success, per superforecasters with an excellent track record, and talking about it and coordinating doesnât help, because each one is a different approach that canât benefit from coordination.
And I agree with you that there are cases where itâs the wrong toolâbut as you said, I think âEA is now often in situations like these,â and weâre not getting the answer right!
Edit: Vasco Grilo has pointed out a mistake in the final paragraph of this comment (see thread below), as I had misunderstood how to apply Shapley values, although I think the conclusion is not affected.
If the value of success is X, and the cost of each group pursuing the intervention is Y, then ideally we would want to pick N (the number of groups that will pursue the intervention) from the possible values 0,1,2 or 3, so as to maximize:
(1-(1/â2)^N) XâN Y
i.e., to maximize expected value.
If all 3 groups have the same goals, theyâll all agree what N is. If N is not 0 or 3, then the best thing for them to do is to get together and decide which of them will pursue the intervention, and which of them wonât, in order to get the optimum N. They can base their decision of how to allocate the groups on secondary factors (or by chance if everything else really is equal). If they all have the same goals then thereâs no game theory here. Theyâll all be happy with this, and theyâll all be maximizing their own individual counterfactual expected value by taking part in this coordination.
This is what I mean by coordination. The fact that their individual approaches are different is irrelevant to them benefiting from this form of coordination.
âMaximize Shapley valueâ will perform worse than this strategy. For example, suppose X is 8, Y is 2. The optimum value of N for expected value is then 2 (2 groups pursue intervention, 1 doesnât). But using Shapley values, I think you find that whatever N is, the Shapley value of your contribution is always >2. So whatever every other group is doing, each group should decide to take part, and we then end up at N=3, which is sub-optimal.
Hi Toby,
I do not think this is correct. If we consider a game with N players where each has to pay c to have a probability p of achieving a value of V:
The actual contribution of a coalition with size n is:
va(n)=(1â(1âp)n)Vânc.
The marginal contribution to a coalition with size n is (note it tends to -c as n increases, as expected):
vm(n)=va(n+1)âva(n)=(1â(1âp)(n+1))Vâ(n+1)câ(1â(1âp)n)V+nc==p(1âp)nVâc.
Since the marginal contribution only depends on the size of the coalition, not on its specific members, the Shapley value is (note it tends to -c as n increases, as expected):
S(N)=1NâNâ1n=0vm(n)=1N(pV1â(1âp)NpâNc)=1â(1âp)NNVâc.
In Davidâs example, N = 3, and p = 0.5, so S = (7/â24) Vâc. For your values of V = 8, and c = 2, S = 1â3. This is not higher than 2.
From the formula for the Shapley value, maximising it is equivalent to maximising:
f(N)=1â(1âp)NN.
I have concluded in this Sheet the above is a strictly decreasing function of N (which tends to 0), so Shapley value is maximised for the smallest possible number of players. This makes intuitive sense, as there is less credit to be shared when N is smaller.
The smallest possible number of players is 1, in which case the Shapley value equals the counterfactual value. In reality, N is higher than 1 in expectation, because it can only be 1 or higher. So, since the Shapley value decreases with N, assuming a single player game will tend to overestimate the contribution of that single player. I think David was arguing for this.
In any case, I do not think it makes sense to frame the problem as deciding what is the best value for N. This is supposed to be the number of (âliveâ) agents in the problem we are trying to solve, not something we can select.
Thank you for this correction, I think youâre right! I had misunderstood how to apply Shapley values here, and I appreciate you taking the time to work through this in detail.
If I understand correctly now, the right way to apply Shapley values to this problem (with X=8, Y=2) is not to work with N (the number of players who end up contributing, which is unknown), but instead to work with Nâ, the number of âliveâ players who could contribute (known with certainty here, not something you can select), and then:
Nâ=3, the number of âliveâ players who are deciding whether to contribute.
With Nâ=3, the Shapley value of the coordination is 1â3 for each player (expected value of 1 split between 3 people), which is positive.
A positive Shapley value means that all players decide to contribute (if basing their decisions off Shapley values as advocated in this post), and you then end up with N=3.
Have I understood the Shapley value approach correctly? If so, I think my final conclusion still stands (even if for the wrong reasons) that a Shapley value analysis will lead to sub-optimal N (number of players deciding to participate). Since the optimal N here is 2 (or 1, which has same value).
As for whether the framing of the problem makes sense, with N as something we can select, the point I was making was that in a lot of real-world situations, N might well be something we can select. If a group of people have the same goals, they can coordinate to choose N, and then youâre not really in a game-theory situation at all. (This wasnât a central point to my original comment but was the point I was defending in the comment youâre responding to)
Even if you donât all have exactly the same goals, or if thereâs a lot of actors, it seems like youâll often be able to benefit by communicating and coordinating, and then youâll be able to improve over the approach of everyone deciding independently according to a Shapley value estimate: e.g. Givewell recommending a funding allocation split between their top charities.
Since I was calculating the Shapley value relative to doing nothing, it being positive only means taking the action is better than doing nothing. In reality, there will be other options available, so I think agents will want to maximise their Shapley cost-effectiveness. For the previous situation, it would be:
SCE(N)=1â(1âp)NNVc.
For the previous values, this would be 7â6. Apparently not very high, considering donating 1 $ to GWWC leads to 6 $ of counterfactual effective donations as a lower bound (see here). However, the Shapley cost-effectiveness of GWWC would be lower than their counterfactual cost-effectiveness⊠In general, since there are barely any impact assessments using Shapley values, it is a little hard to tell whether a given value is good or bad.
In a single person game, or one where weâre fully aligned and cooperating, we get to choose N. We should get to the point where weâre actively cooperating, but itâs not always that easy. And in a game-theoretic situation, where weâre only in control of one party, we need a different approach than either saying we can choose where to invest last, when we canât, and I agree that itâs more complex than Shapley values.
Hi Toby,
Regarding:
You are most likely aware of this, but I just wanted to clarify Shapley value is equal to counterfactual value when there is only 1 (âliveâ) agent. So Shapley value would not leave us predictably worse off if we get the number of agents right.
Point taken, although I think this is analogous to saying: Counterfactual analysis will not leave us predictably worse off if we get the probabilities of others deciding to contribute right.
Agreed. Great point, Toby!
This isnât a problem with expected utility maximization (with counterfactuals), though, right? I think the use of counterfactuals is theoretically sound, but we may be incorrectly modelling counterfactuals.
Itâs a problem with using expected utility maximization in a game theoretic setup without paying attention to other playersâ decisions and responsesâthat is, using counterfactuals which donât account for other player actions, instead of Shapley values, which are a game theoretic solution to the multi-agent dilemma.
Iâm torn with this post as while I agree with the overall spirit (that EAs can do better at cooperation and counterfactuals, be more prosocial), I think the post makes some strong claims/âassumptions which I disagree with. I find it problematic that these assumptions are stated like they are facts.
First, EA may be better at âinternalâ cooperation than other groups, but cooperation is hard and internal EA cooperation is far from perfect.
Second, the idea that correctly assessed counterfactual impact is hyperopic. Nope, hyperopic assessments are just a sign of not getting your counterfactual right.
Third, the idea that Shapley values are the solution. I like Shapley values but only within the narrow constraints for which they are well specified. That is, environments where cooperation should inherently be possible: when all agents agree on the value that is being created. In general you need an approach that can hand both cooperative and adversial environments and everything in between. Iâd call that general approach counterfactual impact. I see another commentor has noted Tobyâs old comments about this and Iâll second that.
Finally, economists may do more counterfactual reasoning than other groups but that doesnât mean they have it all figured out. Ask your average economist to quickly model a counterfactual and it could easily end up being as myopic or hyperopic too. The solution is really to get all analysts better trained on heuristics for reasoning about counterfactuals in a way that is prosocial. To me that is what you get to if you try to implement philosophies like Tobyâs global consequentialism. But we need more practical work on things like this, not repetitive claims about Shapley values.
Iâm writing quickly and hope this comes across in the right spirit. I do find the strong claims in this post frustrating to see, but I welcome that you raised the topic.
Iâm frustrated by your claim that I make strong claims and assumptions, since it seems like what you disagree with me about are conclusions youâd have from skimming, rather than engaging, and being extremely uncharitable.
First, yes, cooperation is hard, and EAs do it âpartially.â I admit that fact, and itâs certainly not the point of this post, so I donât think we disagree. Second, youâre smuggling the entire argument into âcorrectly assessed counterfactual impact,â and again, sure, I agree that if itâs correct, itâs not hyperopicâbut correct requires a game theoretic approach, which we donât generally use in practice.
Third, I donât think we should just use Shapley values, which you seem to claim I believe. I said in the conclusion, âIâm unsure if there is a simple solution to this,â and I agreed that itâs relevant only to where we have goals that are amenable to cooperation. Unfortunately, as I pointed out, in exactly those potentially cooperative scenarios, it seems that EA organizations are the ones attempting to eke out marginal attributable impact instead of cooperating to maximize total good done. Iâve responded to the comment about Tobyâs claims, and again note that those comments are assuming weâre not in a potentially cooperative scenario, or that we are pretending we get to ignore the way others respond to our decisions over time. And finally, I donât know where your attack on economists is coming from, but it seems completely unrelated to the post. Yes, we need more practical work on this, but more than that, we need to admit there is a problem, and stop using poorly reasoned counterfactuals about other groupâs behaviorâsomething you seem to agree with in your comment.
I really like this, thanks!
Another point to perhaps add (not well formed thought) is that 2 groups may be doing the exact same thing with the exact same outcome (say 2 vaccine companies), but because they have such different funding sources and/âor political influence there remains enormous counterfactual good.
For example in Covid, many countries for political reasons almost âhadâ to have their own vaccine so they could produce the vaccine themselves and garner trust in the population. I would argue that none of America, China and Russia would have freely accepted each otherâs vaccines, so they had to research and produce their own even if it didnât make economic sense. The counterfactual value was their not because the vaccine was âneededâ in a perfect world, but because it was needed in the weird geopolitcal setup that happens to exist. If those countries hadnât invented and produced their own vaccines, there would have been huge resistance in importing one from another country. Even if it was allowed and promoted how many Americans would have accepted using sinovax?
OR 2 NGOs could do the same thing (e.g. giving out bednets), but have completely different sources of funding. One could be funded by USAID and the other by DIFID. It might be theoretically inefficient to have 2 NGOs doing the same thing, but in reality they do double the good and distribute twice as many nets because their sources of income donât overlap at all.
The world is complicated
I didnât express this so well but I hope you get the jist....
Yes, this is a reason that in practice, applying Shapley values will be very trickyâyou need to account for lots of details. Thatâs true of counterfactuals as well, but Shapley values make it even harder. (But given that weâre talking about Givewell and similar orgs allocating tens of millions of dollars, the marginal gain seems obviously worth it to me.)
Thanks for the post!
One case Iâve previously thought about is that some naive forms of patient philanthropy could be like thisâtrying to take credit for spending on the âbestâ interventions.
Iâve polished a old draft and posted it as short-form with some discussion of this (in the When patient philanthropy is counterfactual section).
Thanks David, this made me think quite a bit.
This comment starts out nitpicky but hopefully gets better
âIf each team has an uncorrelated 50% chance of succeeding, and with three teams, the probability of being the sole group with a vaccine is therefore 12.5% - justifying the investment. But that means the combined value of all three was only 37.5% of the value of success, and if we invested on that basis, we would underestimate the value for eachâ
Thereâs something wrong with this calculation. I donât think it makes sense to sum marginal values in this way. Itâs like saying that if the last input has zero marginal value then the average input also has zero marginal value. But if we lower inputs based on this, the marginal value will go up and we will have a contradiction.
You want the expected value of trying to develop each vaccine, which requires an assumption about how others decide if to try or not. Then, it wonât be obvious that all three try, and in the cases they donât , the marginal value of those that do is higher.
I think youâre right that we should be careful with counterfactual thinking, but cooperative game theory wonât solve this problem. Shapley value is based on the marginal contribution of each player. So itâs also counterfactual.
Instead, I think the natural solution to the problem you outline is to think more clearly about our utility function. We sometimes talk as if itâs equal to the sum of utilities of all present and future sentient beings and sometimes as if itâs equal to the marginal contribution attributable to a specific action by EA. neither is good. The first is too broad and the latter too narrow. If we think EA is about doing good better, we should care about converting people and resources to our way of doing good and also about to be transparent and credible and rigorous about doing good. So if we cooperate with others in ways that promote these goals, great, otherwise, we should try to compete with them for resources.
Are you suggesting we maximize Shapley values instead of expected utility with counterfactuals? Thatâs going to violate standard rationality axioms, and so (in idealized scenarios with âcorrectly identifiedâ counterfactual distributions) is likely to lead to worse outcomes overall. It could recommend, both in practice and in theory, doing fully redundant work for just for the credit and no extra value. In the paramedic example, depending on how the numbers work out, couldnât it recommend perversely pushing the paramedics out of the way to do CPR yourself and injuring the individualâs spine, even knowing ahead of time you will injure them? Thereâs a coalitionâjust youâwhere your marginal/âcounterfactual impact is to save the personâs life, and giving that positive weight could make up for the injury you actually cause.
I think we should only think of Shapley values as a way to assign credit. The ideal is still to maximize expected utility (or avoid stochastic dominance or whatever). Maybe we need to model counterfactuals with other agents better, but I donât find the examples you gave very persuasive.
In scenarios where we are actually cooperating, maximizing Shapley values is the game-theoretic optimal solution to maximize surplus generated by all partiesâam I misunderstanding that? And since weâre not interested in maximizing our impact, weâre interested in improving the future, that matters more than maximizing what you get credit for.
So yes, obviously, if youâre pretending youâre the only funder, or stipulating that you get to invest last and no one will react in future years to your decisions, then yes, it âis likely to lead to worse outcomes overall.â But weâre not in that world, because weâre not playing a single player game.
Do you mean maximizing the sum of Shapley values or just your own Shapley value? I had the latter in mind. I might be mistaken about the specific perverse examples even under that interpretation, since Iâm not sure how Shapley values are meant to be used. Maximizing your own Shapley value seems to bring in a bunch of counterfactuals (i.e. your counterfactual contribution to each possible coalition) and weigh them ignoring propensities to cooperate/âcoordinate, though.
On the other hand, the sum of Shapley values is just the value (your utility?) of the âgrandâ coalition, i.e. everyone together. If youâre just maximizing this, you donât need to calculate any of the Shapley values first (and in general, you need to calculate the value of the grand coalition for each Shapley value). I think the point of Shapley values would just be for assigning credit (and anything downstream of that), not deciding on the acts for which credit will need to be distributed.
If youâre maximizing this sum, what are the options youâre maximizing over?
On one interpretation, if youâre maximizing this only over your own actions and their consequences, including on othersâ responses (and possibly acausal influence), itâs just maximizing expected utility.
On another interpretation, if youâre maximizing it over everyoneâs actions or assuming everyone else is maximizing it (and so probably that everyone is sufficiently aligned), then that would be ideal (game-theoretically optimal?), but such an assumption is often unrealistic and making it can lead to worse outcomes. For example, our contributions to a charity primarily supported (funding, volunteering, work) by non-EAs with little room for more support might displace that non-EA support towards far less cost-effective uses or even harmful uses. And in that case, it can be better to look elsewhere more neglected by non-EAs. The assumption may be fine in some scenarios (e.g. enough alignment and competence), but it can also be accommodated in 1.
So, I think the ideal target is just doing 1 carefully, including accounting for your influence over other agents and possibly acausal influence in particular.
As I said in the post, âIâm unsure if there is a simple solution to this, since Shapley values require understanding not just your own strategy, but the strategy of others, which is information we donât have. I do think that it needs more explicit consideration...â
Youâre saying that âif youâre maximizing this only over your own actions and their consequences, including on othersâ responses (and possibly acausal influence), itâs just maximizing expected utility.â
I think we agree, modulus the fact that weâre operating in conditions where much of the information we need to âjustâ maximize utility is unavailable.
It seems like youâre overselling Shapley values here, then, unless Iâve misunderstood. They wonât help to decide which interventions to fund, except for indirect reasons (e.g. assigning credit and funding ex post, judging track record).
You wrote âThen we walk away saying (hyperopically,) we saved a life for $5,000, ignoring every other part of the complex system enabling our donation to be effective. And that is not to say itâs not an effective use of money! In fact, itâs incredibly effective, even in Shapley-value terms. But weâre over-allocating credit to ourselves.â
But if $5000 per life saved is the wrong number to use to compare interventions, Shapley values wonât help (for the right reasons, anyway). The solution here is to just model counterfactuals better. If youâre maximizing the sum of Shapley values, youâre acknowledging we have to model counterfactuals better anyway, and the sum is just expected utility, so you donât need the Shapley values in the first place. Either Shapley value cost-effectiveness is the same as the usual cost-effectiveness (my interpretation 1) and redundant, or itâs a predictably suboptimal theoretical target (e.g. maximizing your own Shapley value only, as in Nunoâs proposal, or as another option, my interpretation 2, which requires unrealistic counterfactual assumptions).
The solution to the non-EA money problem is also to just model counterfactuals better. For example, Charity Entrepreneurship has used estimates of the counterfactual cost-effectiveness of non-EA money raised by their incubated charities if the incubated charity doesnât raise it.
Youâre right that Shapley values are the wrong toolâthank you for engaging with me on that, and I have gone back and edited the post to reflect that!
Iâm realizing as I research this that the problem is that act-utilitarianism fundamentally fails for cooperation, and thereâs a large literature on that fact[1] - I need to do much more research.
But âjust model counterfactuals betterâ isnât a useful response. Itâs just saying âget the correct answer,â which completely avoids the problem of how to cooperate and how to avoid the errors I was pointing at.
Kuflik, A. (1982). Utilitarianism and large-scale cooperation. Australasian Journal of Philosophy, 60(3), 224â237.
Regan, Donald H., âCo-operative Utilitarianism Introducedâ, Utilitarianism and Co-operation (Oxford, 1980)
Williams, Evan G. âIntroducing Recursive Consequentialism: A Modified Version of Cooperative Utilitarianism.â The Philosophical Quarterly 67.269 (2017): 794-812.