Curious what you think of the Giving What We Can impact evaluation? I helped write the first one, which claimed that over the 2020â2022 period it had an average multiplier of 30x. I think the more recent one claimed it had a 6x multiplier.
If an investment in an organization can yield anything over 1x in counterfactual donations, this would be worth funding.
I think thereâs an important distinction between average and marginal effectiveness â the above claim seems true in the abstract for orgs that can move >$1 on the margin but not on average. And IMO itâs much harder to estimate marginal cost-effectiveness, because itâs forward looking, whereas historical average cost-effectiveness is a bit easier.
Thanks, Iâll try to gave a look at that and comment (I might have seen it in the past).
What you say about average vs. marginal seems true in principle but
A. âwe estimate that our current grantees deliver an average adjusted return on donations of 6x across our effective giving portfolioâ
Saying âdeliverâ to me present tense implies âdeliver and will continue to deliverâ, suggesting the marginal returns should be comparable.
B. Given the nature of what these funds go for and what these organizations are doing, to me it indeed seems intuitive to expect marginal returns to be fairly similar to the previous returns.
Starting new ~regional initiatives: Okay, once the markets are saturated for âfounding new effective giving orgs in new areasâ there should be diminishing returns. But it would seem like that should already have been picked up by the data from the most recent crop.
For marketing and advertising activities, I even more expect the returns to perhaps decrease somewhat with future expenditure, but in a sort of gradual, continual way.
I might be overlooking some aspects of what the organizations and these grants are doing ⊠But generally, I tend to expect that more money brings diminishing returns, but only gradually diminishing returns. So if Estonia was seen as the ânext most promising targetâ, and founding an organization there had 5x returns, and Latvia is the next one on the priority list, you might expect that to have 4.5x returns.
Saying âdeliverâ to me present tense implies âdeliver and will continue to deliverâ, suggesting the marginal returns should be comparable.
This could still be (and Iâd guess is?) referring to the past and expected future average cost-effectiveness.
I also think that itâd be pretty reasonable to have a bar higher than 1x. (I donât know what CGâs bar actually is.) There are many contentious choices you make when coming up with a multiplier â e.g., how do you discount future donations, how do you discount donations to less cost-effective charities, do you adjust for the opportunity cost of the labor of the employees who could otherwise do impactful work or eartn to give, etc. Thereâs also just a huge amount of uncertainty in various places, especially around counterfactuality. So given all that, I think itâz reasonable to just zoom out and think: hmm, this intervention looks great overall, but I think the multiplier model isnât robust enough to justify further support of organizations that it estimates only have a 1.1x multiplier.
If itâs average future that still could justify a 1x bar, depending on what weâre averaging over.
I agree with the concerns about uncertainty, displacing less-effective charities, and counterfactuality. But Iâd rather see attempts to adjust the estimate for that rather than ~âweâre saying 6x but not really, probably lower after considering thisâ. This will help avoid temptations towards soldier/âpromotion mentality, and make it more comparable to other estimates.
(RE âopportunity cost of the labor of the employees who could otherwise do impactful work or eartn to give, etcââif EA people are putting in free labor into these efforts, that should also be factored into the cost estimates, naturally, not just the direct CG investment.)
If itâs average future that still could justify a 1x bar, depending on what weâre averaging over.
I donât think it does. Itâs conceptually coherent for an organization to have a very high average cost-effectiveness while also having a marginal cost-effectiveness below 1x. For this reason, I donât think you should have a âbarâ for average cost-effectiveness. (You might be making the point that if the average cost-effectiveness is above 1, then you are better off making the grant than burning the money, and so it clears a bar in that sense, but itâs not clear itâs worth making the grant vs making a potentially much smaller grant, and so itâs not a helpful âbarâ in the sense the term is usually used.)
I agree with the concerns about uncertainty, displacing less-effective charities, and counterfactuality. But Iâd rather see attempts to adjust the estimate for that rather than ~âweâre saying 6x but not really, probably lower after considering thisâ. This will help avoid temptations towards soldier/âpromotion mentality, and make it more comparable to other estimates.
Sure, but these are hard to account for. I agree itâs better to adjust the model when itâs possible, but youâll still be left with a model that has a tonne of uncertainty.
(RE âopportunity cost of the labor of the employees who could otherwise do impactful work or eartn to give, etcââif EA people are putting in free labor into these efforts, that should also be factored into the cost estimates, naturally, not just the direct CG investment.)
Yep! I wasnât trying to suggest you shouldnât account for that.
I want to make sure weâre talking about the same thing here. Iâd be want to know the cost-effectiveness in terms âfor each $1 we spend to promote givingâ (via starting new orgs or doing more fundraising) âhow much do we raise in truly counterfactual donations to the most effective charitiesâ and Iâd want this to be net of any donations or effective work that might be crowded out.
E.g., suppose Joe lives in the USA and earns $100k per year. Without our spending Joe, would not give anything to charity and would also not be doing socially-useful work. We spend $1 on ads and this causes Joe (a rich guy) to give $1.50 to The Humane League or The Malaria Consortium, without affecting anyone elseâs behavior. From the PoV of ~âthe EA communityâ we have earned our $1 back plus gained an additional 50 cents. Again from the global EA community perspective, wouldnât we always want to do this?
The example you gave is about marginal cost-effectiveness (we spend â$1 on adsâ). I agree that then, in this abstract/âidealized case, you should spend the $1 on ads. I think all the uncertainty you would realistically have makes it less obvious, though.
But average cost-effectiveness would be more like, we spent $1,000,000 on an organization that did a bunch of different activities, and we think that led to $1,500,000 counterfactually going to charity. This seems good on average, but thereâs a further question of whether we should give another $1 to the organization. And I think that the 6x figure of the orignal post is referring to average cost-effectiveness (âour current grantees deliver an average adjusted return on donations of 6x across our effective giving portfolioâ). This is at least conceptually coherent with the bar for the marginal $ being closer to 1x.
I think you might find the GWWC impact evals interesting, they go into an enormous amount of depth on all these issues.
Okay, thank you. I aim to take a look at these evals and hopefully learn something and maybe give some useful feedback.
And one more point which maybe is obvious but just to get it out there.
Sure, but these are hard to account for. I agree itâs better to adjust the model when itâs possible, but youâll still be left with a model that has a tonne of uncertainty.
I agree that a large amount of uncertainty will persist, but I suppose we should aim to do the modeling and adjustments is mean zero. E.g., weâd put in a large adjustment for âpotential non-counterfactualityâ for things like âmaybe the people who pledged would have pledged later on anyways and the fact that they pledged and donated now means that theyâre likely to end their pledges earlier.â
I suspect that the impact evaluations indeed consider things like these, and I am looking forward to going over them when I have a moment. Thanks for engaging.
(Flagging I work at CG, but not on this team!)
Curious what you think of the Giving What We Can impact evaluation? I helped write the first one, which claimed that over the 2020â2022 period it had an average multiplier of 30x. I think the more recent one claimed it had a 6x multiplier.
I think thereâs an important distinction between average and marginal effectiveness â the above claim seems true in the abstract for orgs that can move >$1 on the margin but not on average. And IMO itâs much harder to estimate marginal cost-effectiveness, because itâs forward looking, whereas historical average cost-effectiveness is a bit easier.
Thanks, Iâll try to gave a look at that and comment (I might have seen it in the past).
What you say about average vs. marginal seems true in principle but
A. âwe estimate that our current grantees deliver an average adjusted return on donations of 6x across our effective giving portfolioâ
Saying âdeliverâ to me present tense implies âdeliver and will continue to deliverâ, suggesting the marginal returns should be comparable.
B. Given the nature of what these funds go for and what these organizations are doing, to me it indeed seems intuitive to expect marginal returns to be fairly similar to the previous returns.
Starting new ~regional initiatives: Okay, once the markets are saturated for âfounding new effective giving orgs in new areasâ there should be diminishing returns. But it would seem like that should already have been picked up by the data from the most recent crop.
For marketing and advertising activities, I even more expect the returns to perhaps decrease somewhat with future expenditure, but in a sort of gradual, continual way.
I might be overlooking some aspects of what the organizations and these grants are doing ⊠But generally, I tend to expect that more money brings diminishing returns, but only gradually diminishing returns. So if Estonia was seen as the ânext most promising targetâ, and founding an organization there had 5x returns, and Latvia is the next one on the priority list, you might expect that to have 4.5x returns.
This could still be (and Iâd guess is?) referring to the past and expected future average cost-effectiveness.
I also think that itâd be pretty reasonable to have a bar higher than 1x. (I donât know what CGâs bar actually is.) There are many contentious choices you make when coming up with a multiplier â e.g., how do you discount future donations, how do you discount donations to less cost-effective charities, do you adjust for the opportunity cost of the labor of the employees who could otherwise do impactful work or eartn to give, etc. Thereâs also just a huge amount of uncertainty in various places, especially around counterfactuality. So given all that, I think itâz reasonable to just zoom out and think: hmm, this intervention looks great overall, but I think the multiplier model isnât robust enough to justify further support of organizations that it estimates only have a 1.1x multiplier.
If itâs average future that still could justify a 1x bar, depending on what weâre averaging over.
I agree with the concerns about uncertainty, displacing less-effective charities, and counterfactuality. But Iâd rather see attempts to adjust the estimate for that rather than ~âweâre saying 6x but not really, probably lower after considering thisâ. This will help avoid temptations towards soldier/âpromotion mentality, and make it more comparable to other estimates.
(RE âopportunity cost of the labor of the employees who could otherwise do impactful work or eartn to give, etcââif EA people are putting in free labor into these efforts, that should also be factored into the cost estimates, naturally, not just the direct CG investment.)
I donât think it does. Itâs conceptually coherent for an organization to have a very high average cost-effectiveness while also having a marginal cost-effectiveness below 1x. For this reason, I donât think you should have a âbarâ for average cost-effectiveness. (You might be making the point that if the average cost-effectiveness is above 1, then you are better off making the grant than burning the money, and so it clears a bar in that sense, but itâs not clear itâs worth making the grant vs making a potentially much smaller grant, and so itâs not a helpful âbarâ in the sense the term is usually used.)
Sure, but these are hard to account for. I agree itâs better to adjust the model when itâs possible, but youâll still be left with a model that has a tonne of uncertainty.
Yep! I wasnât trying to suggest you shouldnât account for that.
I want to make sure weâre talking about the same thing here. Iâd be want to know the cost-effectiveness in terms âfor each $1 we spend to promote givingâ (via starting new orgs or doing more fundraising) âhow much do we raise in truly counterfactual donations to the most effective charitiesâ and Iâd want this to be net of any donations or effective work that might be crowded out.
E.g., suppose Joe lives in the USA and earns $100k per year. Without our spending Joe, would not give anything to charity and would also not be doing socially-useful work. We spend $1 on ads and this causes Joe (a rich guy) to give $1.50 to The Humane League or The Malaria Consortium, without affecting anyone elseâs behavior. From the PoV of ~âthe EA communityâ we have earned our $1 back plus gained an additional 50 cents. Again from the global EA community perspective, wouldnât we always want to do this?
The example you gave is about marginal cost-effectiveness (we spend â$1 on adsâ). I agree that then, in this abstract/âidealized case, you should spend the $1 on ads. I think all the uncertainty you would realistically have makes it less obvious, though.
But average cost-effectiveness would be more like, we spent $1,000,000 on an organization that did a bunch of different activities, and we think that led to $1,500,000 counterfactually going to charity. This seems good on average, but thereâs a further question of whether we should give another $1 to the organization. And I think that the 6x figure of the orignal post is referring to average cost-effectiveness (âour current grantees deliver an average adjusted return on donations of 6x across our effective giving portfolioâ). This is at least conceptually coherent with the bar for the marginal $ being closer to 1x.
I think you might find the GWWC impact evals interesting, they go into an enormous amount of depth on all these issues.
Okay, thank you. I aim to take a look at these evals and hopefully learn something and maybe give some useful feedback.
And one more point which maybe is obvious but just to get it out there.
I agree that a large amount of uncertainty will persist, but I suppose we should aim to do the modeling and adjustments is mean zero. E.g., weâd put in a large adjustment for âpotential non-counterfactualityâ for things like âmaybe the people who pledged would have pledged later on anyways and the fact that they pledged and donated now means that theyâre likely to end their pledges earlier.â
I suspect that the impact evaluations indeed consider things like these, and I am looking forward to going over them when I have a moment. Thanks for engaging.