There’s a lot of good work here and I don’t have time to analyse it in detail, but I had a look at some of your estimates, and I think they depend a bit too heavily on subjective guesses about the counterfactual impact of XR to be all that useful. I can imagine that if you vary the parameter for how much XR might have brought forward net zero or the chance that it directly caused net zero pledges to be taken, then you end up with very large bounds on your ultimate effectiveness numbers. Personally, I don’t think it’s all that reasonable to suggest that, for example, making a net zero pledge one or two years earlier means a corresponding one or two year difference in the time to hit net zero (this seems highly non-linear and there could reasonably be 0 difference).
In addition, I think you discount “zeitgeist effects”—having XR gain traction at the precise time when many other climate groups and climate awareness in general were also gaining traction means that attributing specific outcomes to XR becomes very difficult, although of course XR is part of said zeitgeist. Therefore it seems possible that you could model SMOs as “riding the wave” of public sentiment—contributing to popular awareness of their cause to some extent, but acting as a manifestation of popular awareness rather than as a cause of it.
Hey Herbie—good questions/points so thank you for this response. A few points:
On “they depend a bit too heavily on subjective guesses to be useful”
I think this massively depends on what you think “useful” is! Do I think this evidence is rigorous enough to allocate tens of millions towards nonviolent protest groups? Definitely not, so it’s not useful in that regard. But do I think this is an update on the very little quantitative information we had previously? Yes, definitely useful in that case! At least for me, before I did this project, my uncertainty around the impact of XR ranged from negative overall to roughly the cost-effectiveness of CATF. So in many ways, this project definitely updated my beliefs on the possible upper bounds of cost-effectiveness and that XR was probably net good overall. Also I think it was useful because it showed that even in my pessimistic scenarios, the cost-effectiveness of XR was still not that far off CATF, which I think is the one of the strongest reasons to look into it further.
In essence, the purpose of me doing this preliminary research was to make a case that we should do further research, as this still isn’t very conclusive (as you’ve pointed out). So again, I think it was definitely useful in that regard, as it now is happening. The next steps for this research would definitely be to try reduce the reliance on subjective values for these estimates and reduce the confidence intervals, so more inline with what you’re talking about.
But overall, I think this initial research has narrowed the range of uncertainty / at least put some numbers on the potential cost-effectiveness of nonviolent protest, whereas we didn’t really have that at all before, both inside the EA community and out. The next steps would be to keep narrowing down the large bounds to make it more and more actionable for funders/people generally, so maybe that will be more useful in that regard if it succeeds!
2. On the net-zero pledges
I agree, I don’t think it’s clear that a net-zero declaration a year or two earlier will directly map to reaching net zero a year or two earlier. I don’t think the strength of this case comes from any individual argument, rather than the general direction of many pieces of weak evidence. I think I mentioned this in the document but it’s based on the many weak arguments / cluster thinking approach, similar to what CE uses here. That’s the reason why I chose to try do my CEA from 4 independent angles and compare them, rather than only doing one CEA. Given that all four gave relatively large values for the cost-effectiveness, this was a reasonably positive update overall. However, I wouldn’t place that much weight on any single one, but combined I think they’re more compelling.
To improve this, I could have weighted my average with which CEAs I found the most robust, because actually I think my net-zero one is one of the weaker ones. I’m probably the most confident in my local authority pledges CEA and possibly the one on NDCs. But really I think that these CEAs, when combined with other bits of evidence, is when I think the case becomes more compelling. For example, here are the different bits of evidence that all point in a positive direction:
4 independent CEAs showing high cost-effectiveness
Academic studies showing success of protest on public opinion
Academic studies showing impact of protest on policymakers
Historical case studies of nonviolent protest
Conversations with UK policymakers
Non-quantified impacts such as narrative adoption
A priori arguments on the INT framework (+replaceability, counterfactual impact, etc.)
I think when these are all combined, the case seems quite good, but the individual evidence for each point is not that strong (partly because it just doesn’t exist yet, so what can we do).
3. On Zeitgest effects
I talked about this a bit in my attribution section so not sure if you read that and weren’t convinced or you just didn’t get around to that section (sorry the document is very long!). I definitely do agree it’s hard to disentangle what XR or any individual organisation did to accurately get a feel for this, but I don’t think it’s impossible. To a degree, it’s what foundations do to evaluate policy organisations (as seen by this Founders Pledge document) so it would be applying those same principles to different kinds of organisations. Even applying that framework, as seen in my attribution section, I still believe the counterfactual impact XR had was reasonably large and to some degree kickstarted/boosted that wave of public sentiment.
There’s a lot of good work here and I don’t have time to analyse it in detail, but I had a look at some of your estimates, and I think they depend a bit too heavily on subjective guesses about the counterfactual impact of XR to be all that useful. I can imagine that if you vary the parameter for how much XR might have brought forward net zero or the chance that it directly caused net zero pledges to be taken, then you end up with very large bounds on your ultimate effectiveness numbers. Personally, I don’t think it’s all that reasonable to suggest that, for example, making a net zero pledge one or two years earlier means a corresponding one or two year difference in the time to hit net zero (this seems highly non-linear and there could reasonably be 0 difference).
In addition, I think you discount “zeitgeist effects”—having XR gain traction at the precise time when many other climate groups and climate awareness in general were also gaining traction means that attributing specific outcomes to XR becomes very difficult, although of course XR is part of said zeitgeist. Therefore it seems possible that you could model SMOs as “riding the wave” of public sentiment—contributing to popular awareness of their cause to some extent, but acting as a manifestation of popular awareness rather than as a cause of it.
Hey Herbie—good questions/points so thank you for this response. A few points:
On “they depend a bit too heavily on subjective guesses to be useful”
I think this massively depends on what you think “useful” is! Do I think this evidence is rigorous enough to allocate tens of millions towards nonviolent protest groups? Definitely not, so it’s not useful in that regard. But do I think this is an update on the very little quantitative information we had previously? Yes, definitely useful in that case! At least for me, before I did this project, my uncertainty around the impact of XR ranged from negative overall to roughly the cost-effectiveness of CATF. So in many ways, this project definitely updated my beliefs on the possible upper bounds of cost-effectiveness and that XR was probably net good overall. Also I think it was useful because it showed that even in my pessimistic scenarios, the cost-effectiveness of XR was still not that far off CATF, which I think is the one of the strongest reasons to look into it further.
In essence, the purpose of me doing this preliminary research was to make a case that we should do further research, as this still isn’t very conclusive (as you’ve pointed out). So again, I think it was definitely useful in that regard, as it now is happening. The next steps for this research would definitely be to try reduce the reliance on subjective values for these estimates and reduce the confidence intervals, so more inline with what you’re talking about.
But overall, I think this initial research has narrowed the range of uncertainty / at least put some numbers on the potential cost-effectiveness of nonviolent protest, whereas we didn’t really have that at all before, both inside the EA community and out. The next steps would be to keep narrowing down the large bounds to make it more and more actionable for funders/people generally, so maybe that will be more useful in that regard if it succeeds!
2. On the net-zero pledges
I agree, I don’t think it’s clear that a net-zero declaration a year or two earlier will directly map to reaching net zero a year or two earlier. I don’t think the strength of this case comes from any individual argument, rather than the general direction of many pieces of weak evidence. I think I mentioned this in the document but it’s based on the many weak arguments / cluster thinking approach, similar to what CE uses here. That’s the reason why I chose to try do my CEA from 4 independent angles and compare them, rather than only doing one CEA. Given that all four gave relatively large values for the cost-effectiveness, this was a reasonably positive update overall. However, I wouldn’t place that much weight on any single one, but combined I think they’re more compelling.
To improve this, I could have weighted my average with which CEAs I found the most robust, because actually I think my net-zero one is one of the weaker ones. I’m probably the most confident in my local authority pledges CEA and possibly the one on NDCs. But really I think that these CEAs, when combined with other bits of evidence, is when I think the case becomes more compelling. For example, here are the different bits of evidence that all point in a positive direction:
4 independent CEAs showing high cost-effectiveness
Academic studies showing success of protest on public opinion
Academic studies showing impact of protest on policymakers
Historical case studies of nonviolent protest
Conversations with UK policymakers
Non-quantified impacts such as narrative adoption
A priori arguments on the INT framework (+replaceability, counterfactual impact, etc.)
I think when these are all combined, the case seems quite good, but the individual evidence for each point is not that strong (partly because it just doesn’t exist yet, so what can we do).
3. On Zeitgest effects
I talked about this a bit in my attribution section so not sure if you read that and weren’t convinced or you just didn’t get around to that section (sorry the document is very long!). I definitely do agree it’s hard to disentangle what XR or any individual organisation did to accurately get a feel for this, but I don’t think it’s impossible. To a degree, it’s what foundations do to evaluate policy organisations (as seen by this Founders Pledge document) so it would be applying those same principles to different kinds of organisations. Even applying that framework, as seen in my attribution section, I still believe the counterfactual impact XR had was reasonably large and to some degree kickstarted/boosted that wave of public sentiment.