Hey Herbie—good questions/points so thank you for this response. A few points:
On “they depend a bit too heavily on subjective guesses to be useful”
I think this massively depends on what you think “useful” is! Do I think this evidence is rigorous enough to allocate tens of millions towards nonviolent protest groups? Definitely not, so it’s not useful in that regard. But do I think this is an update on the very little quantitative information we had previously? Yes, definitely useful in that case! At least for me, before I did this project, my uncertainty around the impact of XR ranged from negative overall to roughly the cost-effectiveness of CATF. So in many ways, this project definitely updated my beliefs on the possible upper bounds of cost-effectiveness and that XR was probably net good overall. Also I think it was useful because it showed that even in my pessimistic scenarios, the cost-effectiveness of XR was still not that far off CATF, which I think is the one of the strongest reasons to look into it further.
In essence, the purpose of me doing this preliminary research was to make a case that we should do further research, as this still isn’t very conclusive (as you’ve pointed out). So again, I think it was definitely useful in that regard, as it now is happening. The next steps for this research would definitely be to try reduce the reliance on subjective values for these estimates and reduce the confidence intervals, so more inline with what you’re talking about.
But overall, I think this initial research has narrowed the range of uncertainty / at least put some numbers on the potential cost-effectiveness of nonviolent protest, whereas we didn’t really have that at all before, both inside the EA community and out. The next steps would be to keep narrowing down the large bounds to make it more and more actionable for funders/people generally, so maybe that will be more useful in that regard if it succeeds!
2. On the net-zero pledges
I agree, I don’t think it’s clear that a net-zero declaration a year or two earlier will directly map to reaching net zero a year or two earlier. I don’t think the strength of this case comes from any individual argument, rather than the general direction of many pieces of weak evidence. I think I mentioned this in the document but it’s based on the many weak arguments / cluster thinking approach, similar to what CE uses here. That’s the reason why I chose to try do my CEA from 4 independent angles and compare them, rather than only doing one CEA. Given that all four gave relatively large values for the cost-effectiveness, this was a reasonably positive update overall. However, I wouldn’t place that much weight on any single one, but combined I think they’re more compelling.
To improve this, I could have weighted my average with which CEAs I found the most robust, because actually I think my net-zero one is one of the weaker ones. I’m probably the most confident in my local authority pledges CEA and possibly the one on NDCs. But really I think that these CEAs, when combined with other bits of evidence, is when I think the case becomes more compelling. For example, here are the different bits of evidence that all point in a positive direction:
4 independent CEAs showing high cost-effectiveness
Academic studies showing success of protest on public opinion
Academic studies showing impact of protest on policymakers
Historical case studies of nonviolent protest
Conversations with UK policymakers
Non-quantified impacts such as narrative adoption
A priori arguments on the INT framework (+replaceability, counterfactual impact, etc.)
I think when these are all combined, the case seems quite good, but the individual evidence for each point is not that strong (partly because it just doesn’t exist yet, so what can we do).
3. On Zeitgest effects
I talked about this a bit in my attribution section so not sure if you read that and weren’t convinced or you just didn’t get around to that section (sorry the document is very long!). I definitely do agree it’s hard to disentangle what XR or any individual organisation did to accurately get a feel for this, but I don’t think it’s impossible. To a degree, it’s what foundations do to evaluate policy organisations (as seen by this Founders Pledge document) so it would be applying those same principles to different kinds of organisations. Even applying that framework, as seen in my attribution section, I still believe the counterfactual impact XR had was reasonably large and to some degree kickstarted/boosted that wave of public sentiment.
Hey Herbie—good questions/points so thank you for this response. A few points:
On “they depend a bit too heavily on subjective guesses to be useful”
I think this massively depends on what you think “useful” is! Do I think this evidence is rigorous enough to allocate tens of millions towards nonviolent protest groups? Definitely not, so it’s not useful in that regard. But do I think this is an update on the very little quantitative information we had previously? Yes, definitely useful in that case! At least for me, before I did this project, my uncertainty around the impact of XR ranged from negative overall to roughly the cost-effectiveness of CATF. So in many ways, this project definitely updated my beliefs on the possible upper bounds of cost-effectiveness and that XR was probably net good overall. Also I think it was useful because it showed that even in my pessimistic scenarios, the cost-effectiveness of XR was still not that far off CATF, which I think is the one of the strongest reasons to look into it further.
In essence, the purpose of me doing this preliminary research was to make a case that we should do further research, as this still isn’t very conclusive (as you’ve pointed out). So again, I think it was definitely useful in that regard, as it now is happening. The next steps for this research would definitely be to try reduce the reliance on subjective values for these estimates and reduce the confidence intervals, so more inline with what you’re talking about.
But overall, I think this initial research has narrowed the range of uncertainty / at least put some numbers on the potential cost-effectiveness of nonviolent protest, whereas we didn’t really have that at all before, both inside the EA community and out. The next steps would be to keep narrowing down the large bounds to make it more and more actionable for funders/people generally, so maybe that will be more useful in that regard if it succeeds!
2. On the net-zero pledges
I agree, I don’t think it’s clear that a net-zero declaration a year or two earlier will directly map to reaching net zero a year or two earlier. I don’t think the strength of this case comes from any individual argument, rather than the general direction of many pieces of weak evidence. I think I mentioned this in the document but it’s based on the many weak arguments / cluster thinking approach, similar to what CE uses here. That’s the reason why I chose to try do my CEA from 4 independent angles and compare them, rather than only doing one CEA. Given that all four gave relatively large values for the cost-effectiveness, this was a reasonably positive update overall. However, I wouldn’t place that much weight on any single one, but combined I think they’re more compelling.
To improve this, I could have weighted my average with which CEAs I found the most robust, because actually I think my net-zero one is one of the weaker ones. I’m probably the most confident in my local authority pledges CEA and possibly the one on NDCs. But really I think that these CEAs, when combined with other bits of evidence, is when I think the case becomes more compelling. For example, here are the different bits of evidence that all point in a positive direction:
4 independent CEAs showing high cost-effectiveness
Academic studies showing success of protest on public opinion
Academic studies showing impact of protest on policymakers
Historical case studies of nonviolent protest
Conversations with UK policymakers
Non-quantified impacts such as narrative adoption
A priori arguments on the INT framework (+replaceability, counterfactual impact, etc.)
I think when these are all combined, the case seems quite good, but the individual evidence for each point is not that strong (partly because it just doesn’t exist yet, so what can we do).
3. On Zeitgest effects
I talked about this a bit in my attribution section so not sure if you read that and weren’t convinced or you just didn’t get around to that section (sorry the document is very long!). I definitely do agree it’s hard to disentangle what XR or any individual organisation did to accurately get a feel for this, but I don’t think it’s impossible. To a degree, it’s what foundations do to evaluate policy organisations (as seen by this Founders Pledge document) so it would be applying those same principles to different kinds of organisations. Even applying that framework, as seen in my attribution section, I still believe the counterfactual impact XR had was reasonably large and to some degree kickstarted/boosted that wave of public sentiment.