Thanks very much for posting this reply. And thanks a lot for all the work ACE does in general.
Some clarifications were useful to have, e.g. “The Relationship Between our Intervention Research and our Charity Reviews”—I had felt confused about this when I first looked through the reviews in depth.
Here are some specific comments:
Reviews of existing literature
I agree that the new intervention reports are much better on this front. I’m especially keen on the clear tables summarising existing literature in the protest report. I suspect that there’s still room for more depth here, especially since the articles summarized are probably just the most relevant parts of much wider debates within the social movement studies literature. For example, I notice a couple of items by S.A. Soule; although I haven’t read the book and analysis you (or whoever wrote the protest report) cite, I have read another article of her’s which was partially directed at considering the importance of the “political mediation” and “political opportunity structure” theories for assessing the impact of social movement organizations, and suspect that some of the works you cite might consider similar issues. I think the protest report goes into an appropriate amount of depth, given limited time and resources etc, but I’ve recently gained the impression that a literature review of social movement impact theory in a broad sense, or more systematic reviews of some of the more specific sub-areas, is a high priority in EAA research. I’d be keen to hear views about how useful this would be, and I’m happy to share more specific thoughts if that would help.
Unclear sources of figures
With some older intervention reports I agree with John Halstead that there are some confusing, unexplained numbers, although I think he exaggerates the extent of this (perhaps unintentionally), since some of the figures are explained. I don’t think this needs further comment since, as noted, the new intervention report style is much clearer.
My impression was that the Guesstimate models from more recent charity evaluations also had some slightly unexplained figures on there. E.g. THL guesstimate model – “Rough estimate of number of farmed animals spared per dollar THL spent on campaigns” is −52 to 340. Tracking this back through the model takes you to a box which notes “THL did not provide estimates for the number of animals affected by cage-free campaigns they were involved with. We have roughly based this estimate on estimates from other groups active in promoting cage-free policies and have attempted to take into account the greater amount of resources THL dedicates towards this program area.” I feel like some explanation of this (perhaps a link to an external Google sheet) might have been helpful? I don’t think this is a big issue though. There’s also a chance I’ve just missed something / don’t fully understand Guesstimate yet.
General comment on use of CEEs
ACE does make very clear that it only sees CEEs as one part of a charity evaluation. I’d just suggest that, in spite of these warnings, individuals looking at the reports will naturally gravitate towards the CEEs as one of the more tangible/concrete/easily quotable areas of the report. E.g. when I’ve organised events and created resources for Effective Animal Altruism London, I’ve quoted some of the CEEs for charities (and pretty much nothing else from the report) to make broad points about the rough ballpark for cost effectiveness of different groups. Given this, it still makes sense to treat the CEEs as more important than some other parts of the report, and to try and be especially rigorous in these sections.
So doing things like using a single disputed paper by De Mol et al (2016) (although this example is from the old corporate campaigns intervention report) as a key part of a cost effectiveness analysis seems inadvisable, if it is avoidable.
Thanks for those thoughts. I agree that there’s room for more depth in the literature review portion of our intervention reports. We’ve prioritized breadth over depth in our intervention research so far. That’s because there’s usually no existing survey of the literature on a given intervention, and beginning with a survey helps us identify the areas that we’d like to explore more in depth. (We usually identify “questions for further research” at the end of our reports.) I agree that a review of the literature on social movement impact theory would likely be very useful for the movement. I’m not sure whether ACE is the best-positioned group to do that kind of research, but we can certainly consider it!
Regarding the sources of the figures in our CEEs, I agree that this is an area where we can improve. I do think Guesstimate can be a little hard to read, and that might be part of it, but there are also some places where our 2017 CEEs did not include enough information. We are being more careful about this in 2018, and are publishing a separate “CEE metric library” that will explain the figures that crop up in every CEE.
Yes, we’ve definitely noticed that people naturally gravitate towards our CEEs : ) That corporate outreach report will be archived, and we are focusing on improving our research every year.
Thanks for the reply. Just wanted to note that I agree with ACE’s breadth over depth strategy, and that ACE might not be best-placed for a fuller review of social movement impact literature. It’s something I’m considering prioritizing doing personally in my work for Sentience Institute.
Thanks very much for posting this reply. And thanks a lot for all the work ACE does in general. Some clarifications were useful to have, e.g. “The Relationship Between our Intervention Research and our Charity Reviews”—I had felt confused about this when I first looked through the reviews in depth.
Here are some specific comments:
Reviews of existing literature
I agree that the new intervention reports are much better on this front. I’m especially keen on the clear tables summarising existing literature in the protest report. I suspect that there’s still room for more depth here, especially since the articles summarized are probably just the most relevant parts of much wider debates within the social movement studies literature. For example, I notice a couple of items by S.A. Soule; although I haven’t read the book and analysis you (or whoever wrote the protest report) cite, I have read another article of her’s which was partially directed at considering the importance of the “political mediation” and “political opportunity structure” theories for assessing the impact of social movement organizations, and suspect that some of the works you cite might consider similar issues. I think the protest report goes into an appropriate amount of depth, given limited time and resources etc, but I’ve recently gained the impression that a literature review of social movement impact theory in a broad sense, or more systematic reviews of some of the more specific sub-areas, is a high priority in EAA research. I’d be keen to hear views about how useful this would be, and I’m happy to share more specific thoughts if that would help.
Unclear sources of figures
With some older intervention reports I agree with John Halstead that there are some confusing, unexplained numbers, although I think he exaggerates the extent of this (perhaps unintentionally), since some of the figures are explained. I don’t think this needs further comment since, as noted, the new intervention report style is much clearer. My impression was that the Guesstimate models from more recent charity evaluations also had some slightly unexplained figures on there. E.g. THL guesstimate model – “Rough estimate of number of farmed animals spared per dollar THL spent on campaigns” is −52 to 340. Tracking this back through the model takes you to a box which notes “THL did not provide estimates for the number of animals affected by cage-free campaigns they were involved with. We have roughly based this estimate on estimates from other groups active in promoting cage-free policies and have attempted to take into account the greater amount of resources THL dedicates towards this program area.” I feel like some explanation of this (perhaps a link to an external Google sheet) might have been helpful? I don’t think this is a big issue though. There’s also a chance I’ve just missed something / don’t fully understand Guesstimate yet.
General comment on use of CEEs
ACE does make very clear that it only sees CEEs as one part of a charity evaluation. I’d just suggest that, in spite of these warnings, individuals looking at the reports will naturally gravitate towards the CEEs as one of the more tangible/concrete/easily quotable areas of the report. E.g. when I’ve organised events and created resources for Effective Animal Altruism London, I’ve quoted some of the CEEs for charities (and pretty much nothing else from the report) to make broad points about the rough ballpark for cost effectiveness of different groups. Given this, it still makes sense to treat the CEEs as more important than some other parts of the report, and to try and be especially rigorous in these sections. So doing things like using a single disputed paper by De Mol et al (2016) (although this example is from the old corporate campaigns intervention report) as a key part of a cost effectiveness analysis seems inadvisable, if it is avoidable.
Hi Jamie,
Thanks for those thoughts. I agree that there’s room for more depth in the literature review portion of our intervention reports. We’ve prioritized breadth over depth in our intervention research so far. That’s because there’s usually no existing survey of the literature on a given intervention, and beginning with a survey helps us identify the areas that we’d like to explore more in depth. (We usually identify “questions for further research” at the end of our reports.) I agree that a review of the literature on social movement impact theory would likely be very useful for the movement. I’m not sure whether ACE is the best-positioned group to do that kind of research, but we can certainly consider it!
Regarding the sources of the figures in our CEEs, I agree that this is an area where we can improve. I do think Guesstimate can be a little hard to read, and that might be part of it, but there are also some places where our 2017 CEEs did not include enough information. We are being more careful about this in 2018, and are publishing a separate “CEE metric library” that will explain the figures that crop up in every CEE.
Yes, we’ve definitely noticed that people naturally gravitate towards our CEEs : ) That corporate outreach report will be archived, and we are focusing on improving our research every year.
Best, Toni
Thanks for the reply. Just wanted to note that I agree with ACE’s breadth over depth strategy, and that ACE might not be best-placed for a fuller review of social movement impact literature. It’s something I’m considering prioritizing doing personally in my work for Sentience Institute.