Hi, this is Dan from Giving Green. As you might imagine, I have a lot to say here.
First though, let me thank Alex for going about this criticism in what I would consider the right way: he brought his concerns to us, we had a discussion, and he changed some things based on the discussion. He also offered us a chance to comment on his draft to ensure he hadn’t said anything blatantly factually inaccurate. And then he aired his disagreements in a respectful post. So thanks for that Alex.
That being said, I fundamentally disagree with the majority of Alex’s points, and believe that the judgement calls we have made at Giving Green allow us to be impactful to a wider audience.
But let’s start with something else: Giving Green is a young organization, and I think we have a lot of room to improve and pivot. So criticism is welcomed, and some of Alex’s suggestions did resonate with us.
First, I think we could do a better job in promoting the donation options we think are “better” (ie policy, instead of offsets.) I think the offset research is valuable (as described below), but I agree that it’s not totally obvious to users of the website that we recommend policy over offsets, so that’s something we’d like to improve.
Second, although I do think we have some fundamental disagreements about the value of modeling uncertain situations, I do think there would be value in modeling the cost-effectiveness of offsets more explicitly. I think this is a case where the modeling assumptions are tractable, and we could provide users useful cost-effectiveness data, and may even promote certain offsets over others. This is something we’ve wanted to do for a while, but haven’t had the time to implement. (As Alex noted, we have limited funding and have relied heavily on “side of the desk” work to create Giving Green.)
Now onto the disagreements. I think to respond to every point I would have to write a book, but let me tackle the main ones.
Recommending Offsets: I made an argument defending recommending offsets (even though we believe they are less cost-effective than policy charities) on a comment previously on this post. The main idea is that there’s a tradeoff between certainty and high-risk, high-reward options, and I think there’s a market for both. I’ll paste the most fun part of the argument below.
“Finally, at the risk of going down a rabbit hole, one more point. There are a lot of parallels to this offset debate within international development/global health, an area in which EA is much more developed. Within EA communities, most people are quite comfortable with the recommendations from GiveWell, which are all direct-delivery of health services, and therefore things that can be measured with a high level of certainty. (Like offsets!) So why don’t big international development agencies (World Bank, etc) concentrate only on direct delivery of health services? It’s not because they are just stupid. It’s because they think they can have more bang for their buck investing in systemic changes that can’t be well-quantified with an RCT (like institution-building, macroeconomic stability, infrastructure, etc). Kinda like...funding charities that work on climate policy. So I would find it curious if the final consensus from EAs on global health is all about certainty, but in environment it is firmly for less-certain policy interventions. My argument would be that there is a clear place for both. “
Quantitative modeling: Alex is of the opinion that because we haven’t explicitly quantitatively modeled some of the tradeoffs we face, that the analysis isn’t to be trusted. I think we just have a fundamental difference of opinion on the value of modeling in situations of extreme uncertainty. Look, I’m a trained economist and am pro-modeling in general. But if you’re going to make a model where the outcomes are decided by key parameters that you have to make uninformed judgement calls on, what is the value of the model? Why not just make your judgement call on the outcome?
I know that modeling is in vogue in the EA community so perhaps this makes us outsiders, but I fundamentally believe that modeling in these circumstances leads only to science-y false precision, and does not actually give more clarity.
Let’s take an example, which leads into a discussion below. Let’s say we were trying to weigh the value of a donation to the Sunrise Movement Education Fund (TSM) vs Clean Air Task Force (CATF). Ok, you could model it, but at some point you’re going to have to make a judgement call on the fundamental tradeoff: CATF is more likely to cause incremental change (though some would argue that this is at the expense of entrenching fossil interests and hurting long-term progress), while TSM has a lower chance of causing more fundamental change (though at the potential expense of increasing polarization and jeopardizing incremental progress). So tell me, how are you going to get an unbiased, data-driven estimate of this key parameter that will determine the outcome of your model? I don’t think it’s possible, so don’t want to go down that rabbit hole.
Recommendation of Sunrise Movement Education Fund (TSM): Understanding how donations to organizations lead to policy change is an exercise in fundamental uncertainty, and is going to involve tough judgement calls. I understand that people could make a different judgement call on the tradeoffs with TSM and come to a different conclusion. To be honest, we’ll know a lot more over the next couple of years, as now is the time for TSM to flex its muscles and get climate on the agenda of the Biden administration (and democratic congress.) But for now, we stand by our research and think it’s a good bet. You can read our justification on the site.
A couple of specific points: it’s true that TSM’s budget has grown massively over the last few years (as has CATF’s for that matter), but I think that’s a poor proxy for neglectedness. I think that there is very little effective climate activism happening out there, and there’s huge room for effective growth.
I’m really not compelled by the “uncertainty about the sign of impact” argument, though i don’t really have a way to argue against it quantitatively since it’s theoretically possible. I would just say that this argument is lobbed at a lot of organizations, since people have different theories of political change. For instance, above I linked to an article making a similar argument about the 45Q tax credit, which is one of CATF’s big claimed accomplishments. It’s messy.
Burn Recommendation: I really think that much of the criticism is off the mark here. Berkouer and Dean (2020) focuses a lot of their analysis on credit and demand curves and other fancy economics because that’s how economists get papers published, but underpinning the paper is a strong RCT that convincingly estimates the effect of purchasing a BURN stove on fuel use. Yes, it would be nice if the sample size used for long-term follow-up was larger. And yes, this is just one study but it’s important to realize that it’s a carbon offset certification (which has a number of validation criteria) plus an RCT, which is rare and gives multiple layers of certainty. Given the difficulty of many carbon offsets, I think this is a unique level of rigor that justifies our recommendation of BURN.
The worry that purchasing offsets will not actually lead to more stoves getting distributed is more valid, as this is very hard to verify. But I’m fundamentally willing to believe that if a company like BURN gets more revenue from every stove they sell, they will sell more stoves. In other words, I think the supply curve slopes up, like it usually does.
Climeworks:
This one is a little tougher. Like Alex said, we did not take cost into account when recommending offsets, because we were just looking for any offsets that we felt offered near-certainty. And Climeworks really does offer unparalleled certainty and permanence. But yes, Climeworks is expensive (and we are up front about that on the site). In order for it to be worth it, you have to believe that direct air capture and storage of CO2 is going to be an important part of the climate solution in the future. I don’t find those Metaculus numbers Alex listed too relevant, since you are betting on the technology, not the company. But I can see how reasonable people could disagree here.
Other than the clarification in my other comment, I think the most important disagreement we have is about Sunrise, so I’m going to primarily talk about that.
Neglectedness
it’s true that TSM’s budget has grown massively over the last few years (as has CATF’s for that matter), but I think that’s a poor proxy for neglectedness. I think that there is very little effective climate activism happening out there, and there’s huge room for effective growth.
TSM’s budget growing by 1.5 orders of magnitude since 2015 isn’t sufficient to show that they aren’t neglected, but I think it is sufficient to show that donations data from 2015 should not be relied on to make the case for neglectedness, especially as arguably the most famous activist in the cause area also didn’t start campaigning until 3 years later.
The claim that “there’s very little effective climate activism happening” is very different from the claim that climate activism in general is neglected, and I think may well be true, but that claim only applies to TSM if their activism is unusually effective compared to progressive activism more broadly (which is far from neglected), and I don’t think you’ve shown that. To the extent that TSM has goals, which is somewhat limited, those goals seem to be very typical of progressive climate activism in general, which as discussed is extremely far from being neglected. Sunrise Seattle’s open opposition to cap and trade is one recent example.
Sign of Impact
I would just say that [the impact may be negative] is lobbed at a lot of organizations, since people have different theories of political change.
Isn’t the whole point of doing charity evaluation as opposed to just donating wherever you like that you can evaluate whether these sorts of claims are credible? I appreciate that you’re time pressured and am grateful for the time you’ve already given but I was really hoping for more than just “other organisations have this lobbed at them too”.
It doesn’t really feel consistent to me to take the position when comparing [donate to TSM] and [donate to CATF] that “there’s loads of uncertainty so we won’t make the call”, but then when comparing [Recommend TSM as they are +EV] to [Don’t recommend TSM as they are -EV], take the position “sure there’s loads of uncertainty but on balance the former is the best option”. What’s the difference I’m missing between the two cases?
Hi Alex, let me clarify my thoughts on the “unsure of sign” argument. Let’s say for a given charity, you are considering the sign of impact on some outcome given an increase in donations. Given inherent uncertainly, you might think of a having a probability distribution reflecting your belief on the effect of a donation on this outcome. In almost any case, you would have to believe that there is some non-zero portion of the probability mass of this distribution below zero (because we’ve seen good intentions backfire so many times.) This is my point: the sign of impact is always unknown, technically.
To make recommendation, one must use gathered evidence and judgment to determine the distribution of impacts, and whether this estimated distribution merits a recommendation. Based on our judgement, the distribution of potential impacts of Sunrise (including the mass of probability that is below zero) merits it a recommendation. You and others certainly can disagree with the estimated distribution of impacts or our judgement or whether it merits recommendation. This stuff isn’t easy. But the fact that there is some probability mass on negative impact is not disqualifying, nor should it be.
As for your first comment, it’s important to note that the local chapters of Sunrise can take policy positions at odds with the centralized movement. I agree that sometimes these are unsavory. But when you make a donation, you make it to the centralized org. Critics trying to take down Sunrise frequently pull out the most radical quote they can find from one of the local chapters and use it to disqualify the whole organization, but I don’t really think that’s valid.
It feels like we’re talking past each other a bit, so I’m going to try to clarify my position below but not add anything new. I don’t think the reply above adresses it, but that could well be due to lack of clarity on my part.
Sign of impact
I don’t think the problem with TSM is that there’s non-zero probability mass on negative outcomes. This is, as you point out, true for basically anything.
My issue with TSM is that, for the reasons laid out above, I think the probability mass on negative outcomes is extremely signficant, especially when compared to other good options, for example CATF. This would be enough to make it underperform CATF in expectation even if it had similar upside, though I don’t actually think it does.
Consistency
To make recommendations, one must use gathered evidence and judgment to determine the distribution of impacts, and whether this estimated distribution merits a recommendation...
...This stuff isn’t easy. But the fact that there is some probability mass on negative impact is not disqualifying, nor should it be.
I agree with this. In fact, I still agree with it when the following words are added:
To make recommendations about preferring one organisation over another, one must use gathered evidence and judgment to determine the distributions of impact for each organisation, and whether the estimated distribution of the difference in impact merits a recommendation of one over the other...
...This stuff isn’t easy. But the fact that there is some probability mass on negative impact if we only recommend the organisation which we think is best in expectation is not disqualifying, nor should it be.
I think both the quote from you and the one I’ve added bold text to are true.
Individual chapters of Sunrise
Critics trying to take down Sunrise frequently pull out the most radical quote they can find from one of the local chapters and use it to disqualify the whole organization, but I don’t really think that’s valid.
I pulled that quote to indicate that the decentralised nature of Sunrise means any claims about its work being in any sense atypical of progressive activism more broadly are hard to believe. This is relevant not because one bad quote should discredit an organisation, but because I showed above that climate activism in general is not neglected, and you responded that effective climate activism is not neglected. But both statements can only simultaneously be true if Sunrise’s activism is meaningfully different from progressive activism more broadly, and it doesn’t appear to be.
Thanks Dan, I’m glad to see the comment and will have a more thorough look later. I wanted to clarify one thing though.
Alex is of the opinion that because we haven’t explicitly quantitatively modeled some of the tradeoffs we face, that the analysis isn’t to be trusted. (emphasis mine)
This isn’t quite right. I don’t agree with some of your analysis, but the reason I don’t agree is not the lack of quant models, it’s the things detailed above.
Separately, I do think we disagree on whether quantitative modelling is useful even in cases of very high uncertainty (I think it is). I also think that the act of trying to quantify models tends to improve analysis, and that making explicit models makes analysis much easier for others to critique, which is a good thing if our end goal is having correct analysis.
Hi, this is Dan from Giving Green. As you might imagine, I have a lot to say here.
First though, let me thank Alex for going about this criticism in what I would consider the right way: he brought his concerns to us, we had a discussion, and he changed some things based on the discussion. He also offered us a chance to comment on his draft to ensure he hadn’t said anything blatantly factually inaccurate. And then he aired his disagreements in a respectful post. So thanks for that Alex.
That being said, I fundamentally disagree with the majority of Alex’s points, and believe that the judgement calls we have made at Giving Green allow us to be impactful to a wider audience.
But let’s start with something else: Giving Green is a young organization, and I think we have a lot of room to improve and pivot. So criticism is welcomed, and some of Alex’s suggestions did resonate with us.
First, I think we could do a better job in promoting the donation options we think are “better” (ie policy, instead of offsets.) I think the offset research is valuable (as described below), but I agree that it’s not totally obvious to users of the website that we recommend policy over offsets, so that’s something we’d like to improve.
Second, although I do think we have some fundamental disagreements about the value of modeling uncertain situations, I do think there would be value in modeling the cost-effectiveness of offsets more explicitly. I think this is a case where the modeling assumptions are tractable, and we could provide users useful cost-effectiveness data, and may even promote certain offsets over others. This is something we’ve wanted to do for a while, but haven’t had the time to implement. (As Alex noted, we have limited funding and have relied heavily on “side of the desk” work to create Giving Green.)
Now onto the disagreements. I think to respond to every point I would have to write a book, but let me tackle the main ones.
Recommending Offsets: I made an argument defending recommending offsets (even though we believe they are less cost-effective than policy charities) on a comment previously on this post. The main idea is that there’s a tradeoff between certainty and high-risk, high-reward options, and I think there’s a market for both. I’ll paste the most fun part of the argument below.
“Finally, at the risk of going down a rabbit hole, one more point. There are a lot of parallels to this offset debate within international development/global health, an area in which EA is much more developed. Within EA communities, most people are quite comfortable with the recommendations from GiveWell, which are all direct-delivery of health services, and therefore things that can be measured with a high level of certainty. (Like offsets!) So why don’t big international development agencies (World Bank, etc) concentrate only on direct delivery of health services? It’s not because they are just stupid. It’s because they think they can have more bang for their buck investing in systemic changes that can’t be well-quantified with an RCT (like institution-building, macroeconomic stability, infrastructure, etc). Kinda like...funding charities that work on climate policy. So I would find it curious if the final consensus from EAs on global health is all about certainty, but in environment it is firmly for less-certain policy interventions. My argument would be that there is a clear place for both. “
Quantitative modeling: Alex is of the opinion that because we haven’t explicitly quantitatively modeled some of the tradeoffs we face, that the analysis isn’t to be trusted. I think we just have a fundamental difference of opinion on the value of modeling in situations of extreme uncertainty. Look, I’m a trained economist and am pro-modeling in general. But if you’re going to make a model where the outcomes are decided by key parameters that you have to make uninformed judgement calls on, what is the value of the model? Why not just make your judgement call on the outcome?
I know that modeling is in vogue in the EA community so perhaps this makes us outsiders, but I fundamentally believe that modeling in these circumstances leads only to science-y false precision, and does not actually give more clarity.
Let’s take an example, which leads into a discussion below. Let’s say we were trying to weigh the value of a donation to the Sunrise Movement Education Fund (TSM) vs Clean Air Task Force (CATF). Ok, you could model it, but at some point you’re going to have to make a judgement call on the fundamental tradeoff: CATF is more likely to cause incremental change (though some would argue that this is at the expense of entrenching fossil interests and hurting long-term progress), while TSM has a lower chance of causing more fundamental change (though at the potential expense of increasing polarization and jeopardizing incremental progress). So tell me, how are you going to get an unbiased, data-driven estimate of this key parameter that will determine the outcome of your model? I don’t think it’s possible, so don’t want to go down that rabbit hole.
Recommendation of Sunrise Movement Education Fund (TSM): Understanding how donations to organizations lead to policy change is an exercise in fundamental uncertainty, and is going to involve tough judgement calls. I understand that people could make a different judgement call on the tradeoffs with TSM and come to a different conclusion. To be honest, we’ll know a lot more over the next couple of years, as now is the time for TSM to flex its muscles and get climate on the agenda of the Biden administration (and democratic congress.) But for now, we stand by our research and think it’s a good bet. You can read our justification on the site.
A couple of specific points: it’s true that TSM’s budget has grown massively over the last few years (as has CATF’s for that matter), but I think that’s a poor proxy for neglectedness. I think that there is very little effective climate activism happening out there, and there’s huge room for effective growth.
I’m really not compelled by the “uncertainty about the sign of impact” argument, though i don’t really have a way to argue against it quantitatively since it’s theoretically possible. I would just say that this argument is lobbed at a lot of organizations, since people have different theories of political change. For instance, above I linked to an article making a similar argument about the 45Q tax credit, which is one of CATF’s big claimed accomplishments. It’s messy.
Burn Recommendation: I really think that much of the criticism is off the mark here. Berkouer and Dean (2020) focuses a lot of their analysis on credit and demand curves and other fancy economics because that’s how economists get papers published, but underpinning the paper is a strong RCT that convincingly estimates the effect of purchasing a BURN stove on fuel use. Yes, it would be nice if the sample size used for long-term follow-up was larger. And yes, this is just one study but it’s important to realize that it’s a carbon offset certification (which has a number of validation criteria) plus an RCT, which is rare and gives multiple layers of certainty. Given the difficulty of many carbon offsets, I think this is a unique level of rigor that justifies our recommendation of BURN.
The worry that purchasing offsets will not actually lead to more stoves getting distributed is more valid, as this is very hard to verify. But I’m fundamentally willing to believe that if a company like BURN gets more revenue from every stove they sell, they will sell more stoves. In other words, I think the supply curve slopes up, like it usually does.
Climeworks:
This one is a little tougher. Like Alex said, we did not take cost into account when recommending offsets, because we were just looking for any offsets that we felt offered near-certainty. And Climeworks really does offer unparalleled certainty and permanence. But yes, Climeworks is expensive (and we are up front about that on the site). In order for it to be worth it, you have to believe that direct air capture and storage of CO2 is going to be an important part of the climate solution in the future. I don’t find those Metaculus numbers Alex listed too relevant, since you are betting on the technology, not the company. But I can see how reasonable people could disagree here.
Other than the clarification in my other comment, I think the most important disagreement we have is about Sunrise, so I’m going to primarily talk about that.
Neglectedness
TSM’s budget growing by 1.5 orders of magnitude since 2015 isn’t sufficient to show that they aren’t neglected, but I think it is sufficient to show that donations data from 2015 should not be relied on to make the case for neglectedness, especially as arguably the most famous activist in the cause area also didn’t start campaigning until 3 years later.
The claim that “there’s very little effective climate activism happening” is very different from the claim that climate activism in general is neglected, and I think may well be true, but that claim only applies to TSM if their activism is unusually effective compared to progressive activism more broadly (which is far from neglected), and I don’t think you’ve shown that. To the extent that TSM has goals, which is somewhat limited, those goals seem to be very typical of progressive climate activism in general, which as discussed is extremely far from being neglected. Sunrise Seattle’s open opposition to cap and trade is one recent example.
Sign of Impact
Isn’t the whole point of doing charity evaluation as opposed to just donating wherever you like that you can evaluate whether these sorts of claims are credible? I appreciate that you’re time pressured and am grateful for the time you’ve already given but I was really hoping for more than just “other organisations have this lobbed at them too”.
It doesn’t really feel consistent to me to take the position when comparing [donate to TSM] and [donate to CATF] that “there’s loads of uncertainty so we won’t make the call”, but then when comparing [Recommend TSM as they are +EV] to [Don’t recommend TSM as they are -EV], take the position “sure there’s loads of uncertainty but on balance the former is the best option”. What’s the difference I’m missing between the two cases?
Hi Alex, let me clarify my thoughts on the “unsure of sign” argument. Let’s say for a given charity, you are considering the sign of impact on some outcome given an increase in donations. Given inherent uncertainly, you might think of a having a probability distribution reflecting your belief on the effect of a donation on this outcome. In almost any case, you would have to believe that there is some non-zero portion of the probability mass of this distribution below zero (because we’ve seen good intentions backfire so many times.) This is my point: the sign of impact is always unknown, technically.
To make recommendation, one must use gathered evidence and judgment to determine the distribution of impacts, and whether this estimated distribution merits a recommendation. Based on our judgement, the distribution of potential impacts of Sunrise (including the mass of probability that is below zero) merits it a recommendation. You and others certainly can disagree with the estimated distribution of impacts or our judgement or whether it merits recommendation. This stuff isn’t easy. But the fact that there is some probability mass on negative impact is not disqualifying, nor should it be.
As for your first comment, it’s important to note that the local chapters of Sunrise can take policy positions at odds with the centralized movement. I agree that sometimes these are unsavory. But when you make a donation, you make it to the centralized org. Critics trying to take down Sunrise frequently pull out the most radical quote they can find from one of the local chapters and use it to disqualify the whole organization, but I don’t really think that’s valid.
It feels like we’re talking past each other a bit, so I’m going to try to clarify my position below but not add anything new. I don’t think the reply above adresses it, but that could well be due to lack of clarity on my part.
Sign of impact
I don’t think the problem with TSM is that there’s non-zero probability mass on negative outcomes. This is, as you point out, true for basically anything.
My issue with TSM is that, for the reasons laid out above, I think the probability mass on negative outcomes is extremely signficant, especially when compared to other good options, for example CATF. This would be enough to make it underperform CATF in expectation even if it had similar upside, though I don’t actually think it does.
Consistency
I agree with this. In fact, I still agree with it when the following words are added:
I think both the quote from you and the one I’ve added bold text to are true.
Individual chapters of Sunrise
I pulled that quote to indicate that the decentralised nature of Sunrise means any claims about its work being in any sense atypical of progressive activism more broadly are hard to believe. This is relevant not because one bad quote should discredit an organisation, but because I showed above that climate activism in general is not neglected, and you responded that effective climate activism is not neglected. But both statements can only simultaneously be true if Sunrise’s activism is meaningfully different from progressive activism more broadly, and it doesn’t appear to be.
Thanks Dan, I’m glad to see the comment and will have a more thorough look later. I wanted to clarify one thing though.
This isn’t quite right. I don’t agree with some of your analysis, but the reason I don’t agree is not the lack of quant models, it’s the things detailed above.
Separately, I do think we disagree on whether quantitative modelling is useful even in cases of very high uncertainty (I think it is). I also think that the act of trying to quantify models tends to improve analysis, and that making explicit models makes analysis much easier for others to critique, which is a good thing if our end goal is having correct analysis.