b. I think it’s worth reminding that animal welfare interventions are less cost-effective than they were when Simcikas conducted his analysis.
Admittedly I haven’t been following work on animal welfare cost-effectiveness analysis closely, but this is news to me; can you point me to further readings on this?
c. I generally feel much more comfortable standing behind Givewell’s estimates but Givewell doesn’t analyse cost-effectiveness of advocacy work. My biggest misgivings about cost-effectiveness estimates are due to the difficulty of assessing advocacy work. I think we should make a lot more progress on this.
I agree with the need for the latter; I’m thinking in particular of Animal Ask’s systematic review finding “insufficient evidence to break down overall policy success into the baseline rate of success and the counterfactual impact of lobbying”. I default to the evaluative framework in Founders Pledge’s guide to evaluating policy advocacy organisations but would be keen to learn how to improve upon it.
re: the former, here are some GiveWell policy advocacy-related CEAs:
For both BOTECs GiveWell explicitly mentioned that they rely”on a number of difficult best-guess assumptions and judgment calls about modeling structure. It therefore contains less information value than cost-effectiveness estimates for our top charities, which limits its comparability”, so I’m not sure you’d feel as comfortable standing behind these estimates as with the top charity CEAs. And none of the models address the counterfactual estimation issue Animal Ask identified, again at a quick skim—correct me if I’m wrong on this.
(None of this changes my general sense that funding top animal welfare interventions are more cost-effective on the margin than GHW.)
This seems to be a representative publicly available estimate from 4 years ago by Lewis Bollard:
“This is a major question for us, and one we continue to research. Our current very rough estimate is that our average $ spent on corporate campaigns and all supporting work (which is ~40% of our total animal grant-making) achieves the equivalent of ~7 animals spared a year of complete suffering. We use this a rough benchmark for BOTECs on new grants, and my best guess is this reflects roughly the range we should hope for the last pro-animal dollar. ”
I think several more up to date estimates will be available soon.
For advocacy evaluation, a concrete area for improvement is the following. Saulius’s analysis has a really nice section titled “Ways this estimate could be misleading”. Other advocates cite concerns similar to those when they argue against corporate welfare campaigns. They usually don’t have empirical evidence, but I don’t have super strong evidence to show them wrong either. I’m not very happy about that.
What did you think of the GiveWell policy advocacy CEAs & BOTECs I linked? I shared them in response to your ”...but Givewell doesn’t analyse cost-effectiveness of advocacy work” so I wondered if you had a different take.
I appreciate the correction. When I said “I generally feel much more comfortable standing behind Givewell’s estimates” that was for their main page recommendations. I currently won’t prioritise reviewing these BOTECS in detail in the short term but as a future exercise I will look into the linked analyses and compare them to animal welfare ones.
Admittedly I haven’t been following work on animal welfare cost-effectiveness analysis closely, but this is news to me; can you point me to further readings on this?
I agree with the need for the latter; I’m thinking in particular of Animal Ask’s systematic review finding “insufficient evidence to break down overall policy success into the baseline rate of success and the counterfactual impact of lobbying”. I default to the evaluative framework in Founders Pledge’s guide to evaluating policy advocacy organisations but would be keen to learn how to improve upon it.
re: the former, here are some GiveWell policy advocacy-related CEAs:
2017 CEA of the Centre for Pesticide Suicide Prevention (grant writeup, 2018 blog post where they explained their reasoning in considering policy advocacy orgs in general), and 2021 skeleton BOTEC of the same org (grant writeup)
2021 BOTEC of Vital Strategies (grant writeup) to reduce harms of excessive alcohol consumption in LMICs.
For both BOTECs GiveWell explicitly mentioned that they rely”on a number of difficult best-guess assumptions and judgment calls about modeling structure. It therefore contains less information value than cost-effectiveness estimates for our top charities, which limits its comparability”, so I’m not sure you’d feel as comfortable standing behind these estimates as with the top charity CEAs. And none of the models address the counterfactual estimation issue Animal Ask identified, again at a quick skim—correct me if I’m wrong on this.
(None of this changes my general sense that funding top animal welfare interventions are more cost-effective on the margin than GHW.)
This seems to be a representative publicly available estimate from 4 years ago by Lewis Bollard:
“This is a major question for us, and one we continue to research. Our current very rough estimate is that our average $ spent on corporate campaigns and all supporting work (which is ~40% of our total animal grant-making) achieves the equivalent of ~7 animals spared a year of complete suffering. We use this a rough benchmark for BOTECs on new grants, and my best guess is this reflects roughly the range we should hope for the last pro-animal dollar. ”
I think several more up to date estimates will be available soon.
For advocacy evaluation, a concrete area for improvement is the following. Saulius’s analysis has a really nice section titled “Ways this estimate could be misleading”. Other advocates cite concerns similar to those when they argue against corporate welfare campaigns. They usually don’t have empirical evidence, but I don’t have super strong evidence to show them wrong either. I’m not very happy about that.
Thanks for the pointers, much appreciated.
What did you think of the GiveWell policy advocacy CEAs & BOTECs I linked? I shared them in response to your ”...but Givewell doesn’t analyse cost-effectiveness of advocacy work” so I wondered if you had a different take.
I appreciate the correction. When I said “I generally feel much more comfortable standing behind Givewell’s estimates” that was for their main page recommendations. I currently won’t prioritise reviewing these BOTECS in detail in the short term but as a future exercise I will look into the linked analyses and compare them to animal welfare ones.