I appreciate you writing up these comments! There are some great suggestions here as well as things I disagree with. As the author of the “extremely positive” post let me share some thoughts. (I’m by no means an expert on this so feel free to tell me I’m wrong.)
1. Quantitative cost-effectiveness analysis
Summary of my view: I’m pretty torn on this one but think we may not want to require a quantitative CEA on charities working on policy change (although definitely encourage the GG team to try this exercise).
On one hand I think it’s great to at least attempt it to develop a better understanding of one’s causal model as well as sources of uncertainty, and getting a ballpark estimate if possible (though sometimes the range is too wide to be useful). On the other hand requiring quantitative cost-effectiveness estimates can restrict the type of charities one can evaluate. I took a brief look at Founders’ Pledge’s model on the Clean Air Task Force, which seems to be a combination of 1) their track record, 2) their plan, 3) subjective judgements. While the model seems reasonable (I haven’t taken a deep enough look to tell how much I agree) I do think requiring such a model would preclude evaluating orgs like the Sunrise Movement—or, if we take your concerns about them seriously (which I’ll address below) let’s just say orgs like that, of which there are many in the climate space: those with a more complex theory of change than say CATF, and any model would involve inputs that are mostly extremely subjective (compared to the CATF one) which makes it less meaningful. Perhaps you would say these orgs are precisely the ones not worth recommending—on this I agree with Giving Green that we should hedge our bets among different theories of change and hence look at different types of orgs (even though as I’ll elaborate later I agree with having a stronger recommendation for CATF, I think potentially recommending orgs more similar to TSM is valuable).
So I think it is definitely good to attempt a quantitative CEA and I highly encourage the GG team to do so, even for an org like TSM. (I would have liked to engage with Founders’ Pledges’ models more but didn’t end up doing it—that would be a nice exercise.) But I’m unsure about requiring that in a recommendation especially when you work in a space with so much uncertainty. (I was trying to look up other EA charity recommenders and saw that Animal Charity Evaluators also don’t seem to have quantitative CEA for any/all their top charities—I haven’t checked all but here’s an example without. Not saying this is a sufficient argument though.)
I have to say I’m pretty uncertain about how much to use quantitative CEA and I am happy to be convinced that I’m wrong.
I do agree Giving Green should communicate with less confidence in their recommendations as say GiveWell, which explicitly recommends charities that are amenable to be evaluated with higher quality evidence (e.g. RCTs) and hence have lower uncertainty.
2. Offsets
1) Offsets vs policy change
My read is that GG recommends offsets because they see a huge market especially among companies that want to purchase offsets, and it’s hard to convince them to instead donate the money to the maximally impactful thing. However, I agree that they should communicate this more clearly: that for more “flexible” donors they strongly recommend policy change over offsets.
2) Cost-effectiveness of offsets
I agree it would be good to come up with cost-effectiveness estimates for offsets even though they will also be pretty uncertain (probably something between the uncertainty of GiveWell current top charities and climate change orgs working on policy change). In addition to telling people to buy offsets with real additionality, it’s probably also good to put a proper price tag on things especially if they differ a lot.
3. The Sunrise Movement (TSM)
Summary of my view: I’m more positive than the author on the impact they achieved (and perhaps their impact potential), and less negative on the potential for negative impact, although I’m really unsure about it as I’m far from an expert. I do agree that GG should recommend CATF more highly than TSM.
Impact they achieved: The fact that Biden and some other Democrats adopted climate change plans similar to what’s proposed by TSM (see the “Policy consensus and promotion” section of GG’s page on TSM) is some evidence of their influence, although of course we can’t be sure. (This article argues it was valuable for groups on the left to have a more unified framework for addressing climate change, and it seems like TSM is one of the multiple groups that had an influence in the process.)
Potential for negative impact:
In terms of actual policies: I mostly trust Biden and overall the Democratic members of Congress (rather than the most “progressive” ones) to go for policies that will be less polarizing than the most radical proposals, and I’m not too worried about TSM pressuring them into doing things they don’t think are good ideas.
In terms of public opinions: Will TSM make climate change a more polarizing issue than it already was? On one hand we do see the majority of Americans being concerned about climate change; on the other hand the extreme level of polarization (even in the absence of the TSM) already shape people’s view on many things. so I’m not sure.
(I think my arguments are pretty weak here though because I don’t understand the US political system very well.)
Inside view perspective: under the Biden administration it seems like CATF has a very clear vision of what they can do (see here); for TSM it’s less clear—even if they achieved some impact before the election in getting candidates to take climate change more seriously and adopt a more unified platform, it’s less clear how they will influence policy now. If I were choosing between the two at this moment it’s definitely CATF.
(Right now they sort of do this: labeling CATF as “good bet” and TSM “shows promise”, although we probably want something more clear than those labels, and apparently the team did not mean to recommend CATF more highly.)
Thanks for engaging here. This is a thoughtful and interesting comment, and I think it’s noteworthy that we basically agree on several important conclusions, namely that Giving Green should:
Clearly indicate that, currently, CATF looks, in expectation, to be far superior to TSM, not least because even if their own research doesn’t show this, everyone else’s does.
Be more clear about the difference in expectation between Offsets and Policy change (some progress has been made on this already).
Consider cost in their offset analysis (though that doesn’t mean calculating a naive $/TCO2e and calling it a day).
Be more clear about the current quality and limitations of their original research.
Consider incorporating quantitative models, especially about their own theory of change (not because qualitative ones aren’t valid, but because it would likely improve their reasoning and make it easier to evaluate).
There are, however, a couple of misconceptions in your comment which are similar to those in Dan’s initial responses, and have been discussed elsewhere in the comments. I’m going to try to summarise those here, as this thread has got very long so it’s not surprising some things are being missed.
Quantitative research
think we may not want to require a quantitative CEA on charities working on policy change
As I mentioned in my reply to Dan when he raised a similar concern, I’m not rejecting Giving Green’s because it is not quantitative, I’m rejecting Giving Green’s analysis because of the many substantial flaws which have been extensively discussed, and I’m also saying that quantitative modelling is a useful exercise which may have prevented or helped identify many of those flaws. The way that building quantitative models can improve analysis, even if the models themselves are rough or flawed, is usefully discussed by Johannes at the start of this epic comment which is longer than the post itself so I’ll quote the relevant section.
I should also state upfront that my credence in CATF and other high-impact climate charities does not come primarily from the cost-effectiveness models, which are clearly wrong and also described as such, but by the careful reasoning that has gone into the FP climate recommendations...
...But the process of building these models and doing the research around them—for each FP recommendation there is at least 20 pages worth of additional background research examining all kinds of concerns -- combined with years of expertise working in and studying climate policy, has served the purpose of clearly delineating the theory of value creation, as well as the risks and assumptions, in a way that a completely qualitative analysis that has a somewhat loose connection between evidence, arguments, and conclusions (recommendation) has not.
The fundamental concern with Giving Green’s analysis that I, and I think (?) Alex, have is not the lack of quantitative modeling per se, but the unwillingness to make systematic arguments about relative goodness of things in a situation of uncertainty, rather treating each concern as equally weighted and taking an attitude of “when things are uncertain, everything goes and we don’t know anything”...
The Sunrise Movement
Again, I think the most important misunderstanding here has already been discussed repeatedly in the comments. The difference between “is X good” and “is X good on the margin” is a massive and fundamental part of impact evaluation. It’s easy to argue a case along the lines of “progressive activism has been broadly positive/associated with positive changes”, I wholeheartedly agree with that claim! It just has very little to do with what the potential impact will be of TSM on the current margin. It is possible for extremely good causes to be poor donation opportunities, because additional donations would not allow them to do any more good. It is similarly possible for only moderately good causes to be extremely good donation opportunities, if additional donations would be transformative for them. Neglectedness is only one aspect of judging marginal impact, but it is discussed helpfully in this comment.
There’s been a good deal of discussion in other comments here and here, as well as the substance of the original post, about the downside risks of TSM, but I think it’s worth noting that the view that “the Biden camp will probably ignore them if they suggest something too crazy” is not one which is totally compatible with thinking that donations to TSM will have high marginal impact.
There are several ways in which TSM might influence things though which don’t seem obviously like they will fail, for example (quoting from this comment):
A stronger TSM could intensify pressure on Biden to prioritize executive orders over legislative politics, because this looks more appealing than more incrementally seeming legislative politics even though legislative politics would ultimately be more impactful and/or more robust over time.
I chose this in particular because it also speaks the “who is being ambitious and transformative” discussion which seems to have popped up a few times in the comments. Ultimately, bipartisan legislation, even if it’s slower to get big wins, ensures that those wins stick around in the long run (there’s also the national vs international angle, but that’s been covered elsewhere). Quoting part of another comment from Johannes:
All of the major success stories we have seen in climate over the past 20 years – solar, wind, coal > gas in the US, electric cars and batteries – have been the result of relatively narrow and targeted policies, the kind of which CATF advances for technologies that are less popular with greens for reasons of ideology, not merit.
Thanks for your comment! I agree with Alex on his points and—apparently, a lot with you as well :) -- but adding some clarifications on questions/assumptions in your comment re FP research on this: (1) whether or not FP would research TSM or other similar interventions (absolutely!), (2) additional reasons why CATF is a robust rec and TSM is not (3) where credence in CATF comes from.
1. Would FP or similar orgs exclude TSM because of low measurability? I don’t really know where this idea originated, but the answer is clearly that we would not exclude an org like TSM because of low measurability. We would absolutely examine TSM or other similar orgs if we had reasons to believe to find something high impact in this space.
Yes, TSM is very uncertain and the path is a bit more indirect than with CATF or similar, but this is a gradual difference, not a qualitative one—there are clear quantitative ways in which one could think about TSM; indeed reading the GG work on TSM and the discussion here has already given some indications on how this would look like.
As I wrote in another reply, we constantly evaluate and recommend uncertain hit-based opportunities.
The reason for not investigating TSM more deeply at FP right now is that from GG’s analysis and this forum discussion it is pretty clear that this is not a particularly high-impact option -- (a) it’s clearly not neglected, (b) there is a lot of downside risk, and (c) there isn’t a strong marginal case—nothing that would leave us to expect that giving more money to TSM would lead to much stronger TSM, let alone a much better world.
(I) Given that it takes 120+ hours to vet a funding opportunity, (II) the goodness of existing climate recs with remaining funding gaps and (III) the vast impact differentials between excellent and average opportunities (easily 100x), at FP we believe that this time is better spent at finding things that have a plausible chance of being really high impact.
I think the most plausible case for this to be a grassroots movement would be outside the US, because a lot of the downside risk for TSM comes from features specific to its partisan nature and the structure of the American political system. If in the US, my best guess would be Republican pro-climate grassroots.
2. There are a least 4 additional reasons beyond those you outline why we should expect CATF to be very robust and TSM not to be. I discuss those in the second part of this comment:
The TL,DR of it is as follows:
1. There is a lot of expert support for the CATF recommendation and there is a lot more uncertainty regarding TSM.
2. CATF looks very good on the theory of change/frame most relevant to effective climate action—maximizing global decarbonization benefit—and the argument for TSM on that frame is not made.
3. Charity evaluation methodology is our friend and allows us to draw useful inferences even in highly uncertain situations.
4. The length and depth of engagement that led to the CATF and similar recommendations should itself be a reason for confidence, more so than the GG comment suggests.
3. Our current credence in CATF as a top-recommendation does not build primarily on the 2018 report which “discovered” CATF but in multiple re-evaluations of CATF as well as additional evaluations by other orgs. I summarize this here (emphasis new):
CATF looks very good on a theory of change focused on maximized global decarbonization impact when taking into account some of the most important stylized facts about the climate challenge (widely recognized as median views in the respective expert communities):
1. Global energy demand will grow and restricting energy demand growth is very problematic from a humanitarian perspective.
2. Effective global decarbonization requires a much larger set of technologies than those currently available. Most of those technologies are not on track and many necessary technologies are in early stages.
You can then combine this with two CATF-specific features:
5. CATF is a strong organization that translates money into effective advocacy. This is not controversial within the EA community, something GG agrees on. It was first established in the FP 2018 report and it appears that at least 4 EA orgs had multiple calls with CATF, often dozens, that reaffirmed this conclusion (FP, Legacies Now, SoGive, Giving Green).
6. CATF has very productive funding margins, projects that are currently unfunded and that make a lot of sense from the above stylized facts and the theory of change.
This is all you need to come to CATF as a likely local optimum in effective climate philanthropy.
None of this is controversial and – indeed – each of the claims above about the world in general (1-4) follow directly from median expert views on those respective topics and the CATF-specific claims (5-6) are even entirely uncontroversial across the EA community.
In contrast, motivating TSM as a top-choice requires a lot of controversial claims, such as (a) that we are sure that the impact of marginal TSM donations is not negative in expectation and (b) that additional effort can lead to significant change beyond what is already baked in despite the approach of Sunrise being partisan and thereby, quite plausibly, limited in its ultimate potential given the structure of the Senate and the Electoral College.
I appreciate you writing up these comments! There are some great suggestions here as well as things I disagree with. As the author of the “extremely positive” post let me share some thoughts. (I’m by no means an expert on this so feel free to tell me I’m wrong.)
1. Quantitative cost-effectiveness analysis
Summary of my view: I’m pretty torn on this one but think we may not want to require a quantitative CEA on charities working on policy change (although definitely encourage the GG team to try this exercise).
On one hand I think it’s great to at least attempt it to develop a better understanding of one’s causal model as well as sources of uncertainty, and getting a ballpark estimate if possible (though sometimes the range is too wide to be useful). On the other hand requiring quantitative cost-effectiveness estimates can restrict the type of charities one can evaluate. I took a brief look at Founders’ Pledge’s model on the Clean Air Task Force, which seems to be a combination of 1) their track record, 2) their plan, 3) subjective judgements. While the model seems reasonable (I haven’t taken a deep enough look to tell how much I agree) I do think requiring such a model would preclude evaluating orgs like the Sunrise Movement—or, if we take your concerns about them seriously (which I’ll address below) let’s just say orgs like that, of which there are many in the climate space: those with a more complex theory of change than say CATF, and any model would involve inputs that are mostly extremely subjective (compared to the CATF one) which makes it less meaningful. Perhaps you would say these orgs are precisely the ones not worth recommending—on this I agree with Giving Green that we should hedge our bets among different theories of change and hence look at different types of orgs (even though as I’ll elaborate later I agree with having a stronger recommendation for CATF, I think potentially recommending orgs more similar to TSM is valuable).
So I think it is definitely good to attempt a quantitative CEA and I highly encourage the GG team to do so, even for an org like TSM. (I would have liked to engage with Founders’ Pledges’ models more but didn’t end up doing it—that would be a nice exercise.) But I’m unsure about requiring that in a recommendation especially when you work in a space with so much uncertainty. (I was trying to look up other EA charity recommenders and saw that Animal Charity Evaluators also don’t seem to have quantitative CEA for any/all their top charities—I haven’t checked all but here’s an example without. Not saying this is a sufficient argument though.)
I have to say I’m pretty uncertain about how much to use quantitative CEA and I am happy to be convinced that I’m wrong.
I do agree Giving Green should communicate with less confidence in their recommendations as say GiveWell, which explicitly recommends charities that are amenable to be evaluated with higher quality evidence (e.g. RCTs) and hence have lower uncertainty.
2. Offsets
1) Offsets vs policy change
My read is that GG recommends offsets because they see a huge market especially among companies that want to purchase offsets, and it’s hard to convince them to instead donate the money to the maximally impactful thing. However, I agree that they should communicate this more clearly: that for more “flexible” donors they strongly recommend policy change over offsets.
2) Cost-effectiveness of offsets
I agree it would be good to come up with cost-effectiveness estimates for offsets even though they will also be pretty uncertain (probably something between the uncertainty of GiveWell current top charities and climate change orgs working on policy change). In addition to telling people to buy offsets with real additionality, it’s probably also good to put a proper price tag on things especially if they differ a lot.
3. The Sunrise Movement (TSM)
Summary of my view: I’m more positive than the author on the impact they achieved (and perhaps their impact potential), and less negative on the potential for negative impact, although I’m really unsure about it as I’m far from an expert. I do agree that GG should recommend CATF more highly than TSM.
Impact they achieved: The fact that Biden and some other Democrats adopted climate change plans similar to what’s proposed by TSM (see the “Policy consensus and promotion” section of GG’s page on TSM) is some evidence of their influence, although of course we can’t be sure. (This article argues it was valuable for groups on the left to have a more unified framework for addressing climate change, and it seems like TSM is one of the multiple groups that had an influence in the process.)
Potential for negative impact:
In terms of actual policies: I mostly trust Biden and overall the Democratic members of Congress (rather than the most “progressive” ones) to go for policies that will be less polarizing than the most radical proposals, and I’m not too worried about TSM pressuring them into doing things they don’t think are good ideas.
In terms of public opinions: Will TSM make climate change a more polarizing issue than it already was? On one hand we do see the majority of Americans being concerned about climate change; on the other hand the extreme level of polarization (even in the absence of the TSM) already shape people’s view on many things. so I’m not sure.
(I think my arguments are pretty weak here though because I don’t understand the US political system very well.)
Why GG should recommend CATF more highly:
Outside view perspective: even if the expected values of the two orgs look the same we should account for the fact that CATF has much more of a track record.
Inside view perspective: under the Biden administration it seems like CATF has a very clear vision of what they can do (see here); for TSM it’s less clear—even if they achieved some impact before the election in getting candidates to take climate change more seriously and adopt a more unified platform, it’s less clear how they will influence policy now. If I were choosing between the two at this moment it’s definitely CATF.
(Right now they sort of do this: labeling CATF as “good bet” and TSM “shows promise”, although we probably want something more clear than those labels, and apparently the team did not mean to recommend CATF more highly.)
Thanks for engaging here. This is a thoughtful and interesting comment, and I think it’s noteworthy that we basically agree on several important conclusions, namely that Giving Green should:
Clearly indicate that, currently, CATF looks, in expectation, to be far superior to TSM, not least because even if their own research doesn’t show this, everyone else’s does.
Be more clear about the difference in expectation between Offsets and Policy change (some progress has been made on this already).
Consider cost in their offset analysis (though that doesn’t mean calculating a naive $/TCO2e and calling it a day).
Be more clear about the current quality and limitations of their original research.
Consider incorporating quantitative models, especially about their own theory of change (not because qualitative ones aren’t valid, but because it would likely improve their reasoning and make it easier to evaluate).
There are, however, a couple of misconceptions in your comment which are similar to those in Dan’s initial responses, and have been discussed elsewhere in the comments. I’m going to try to summarise those here, as this thread has got very long so it’s not surprising some things are being missed.
Quantitative research
As I mentioned in my reply to Dan when he raised a similar concern, I’m not rejecting Giving Green’s because it is not quantitative, I’m rejecting Giving Green’s analysis because of the many substantial flaws which have been extensively discussed, and I’m also saying that quantitative modelling is a useful exercise which may have prevented or helped identify many of those flaws. The way that building quantitative models can improve analysis, even if the models themselves are rough or flawed, is usefully discussed by Johannes at the start of this epic comment which is longer than the post itself so I’ll quote the relevant section.
The Sunrise Movement
Again, I think the most important misunderstanding here has already been discussed repeatedly in the comments. The difference between “is X good” and “is X good on the margin” is a massive and fundamental part of impact evaluation. It’s easy to argue a case along the lines of “progressive activism has been broadly positive/associated with positive changes”, I wholeheartedly agree with that claim! It just has very little to do with what the potential impact will be of TSM on the current margin. It is possible for extremely good causes to be poor donation opportunities, because additional donations would not allow them to do any more good. It is similarly possible for only moderately good causes to be extremely good donation opportunities, if additional donations would be transformative for them. Neglectedness is only one aspect of judging marginal impact, but it is discussed helpfully in this comment.
There’s been a good deal of discussion in other comments here and here, as well as the substance of the original post, about the downside risks of TSM, but I think it’s worth noting that the view that “the Biden camp will probably ignore them if they suggest something too crazy” is not one which is totally compatible with thinking that donations to TSM will have high marginal impact.
There are several ways in which TSM might influence things though which don’t seem obviously like they will fail, for example (quoting from this comment):
I chose this in particular because it also speaks the “who is being ambitious and transformative” discussion which seems to have popped up a few times in the comments. Ultimately, bipartisan legislation, even if it’s slower to get big wins, ensures that those wins stick around in the long run (there’s also the national vs international angle, but that’s been covered elsewhere). Quoting part of another comment from Johannes:
Thanks for your comment! I agree with Alex on his points and—apparently, a lot with you as well :) -- but adding some clarifications on questions/assumptions in your comment re FP research on this: (1) whether or not FP would research TSM or other similar interventions (absolutely!), (2) additional reasons why CATF is a robust rec and TSM is not (3) where credence in CATF comes from.
1. Would FP or similar orgs exclude TSM because of low measurability?
I don’t really know where this idea originated, but the answer is clearly that we would not exclude an org like TSM because of low measurability. We would absolutely examine TSM or other similar orgs if we had reasons to believe to find something high impact in this space.
Yes, TSM is very uncertain and the path is a bit more indirect than with CATF or similar, but this is a gradual difference, not a qualitative one—there are clear quantitative ways in which one could think about TSM; indeed reading the GG work on TSM and the discussion here has already given some indications on how this would look like.
As I wrote in another reply, we constantly evaluate and recommend uncertain hit-based opportunities.
The reason for not investigating TSM more deeply at FP right now is that from GG’s analysis and this forum discussion it is pretty clear that this is not a particularly high-impact option -- (a) it’s clearly not neglected, (b) there is a lot of downside risk, and (c) there isn’t a strong marginal case—nothing that would leave us to expect that giving more money to TSM would lead to much stronger TSM, let alone a much better world.
(I) Given that it takes 120+ hours to vet a funding opportunity, (II) the goodness of existing climate recs with remaining funding gaps and (III) the vast impact differentials between excellent and average opportunities (easily 100x), at FP we believe that this time is better spent at finding things that have a plausible chance of being really high impact.
I think the most plausible case for this to be a grassroots movement would be outside the US, because a lot of the downside risk for TSM comes from features specific to its partisan nature and the structure of the American political system. If in the US, my best guess would be Republican pro-climate grassroots.
2. There are a least 4 additional reasons beyond those you outline why we should expect CATF to be very robust and TSM not to be. I discuss those in the second part of this comment:
3. Our current credence in CATF as a top-recommendation does not build primarily on the 2018 report which “discovered” CATF but in multiple re-evaluations of CATF as well as additional evaluations by other orgs. I summarize this here (emphasis new):