Thanks for asking this question. I support and follow the approach of asking relevant people in the space for input to a research agenda. I am happy to see that other organizations are also doing it.
Meta-thoughts on how to approach this selection task would also be handy.
Your question inspired me to write a short post on a methodology of systematically integrating stakeholders’ and decision-makers’ input into the research agenda. You might find this meta-methodology helpful. Out of the areas you mention, I’d be very interested in the following:Animal product alternatives 6⁄10
Pain relief in developing countries 6⁄10Improving science 9⁄10
Ideas not included on your list:GiveWell recently published its list of areas they are planning to explore. I think some of them might be of interest to donors focused on improving the welfare of the current generation of humans and high-income countries’ problems.
Tobacco, alcohol, and sugar control
Air pollution regulation
Micronutrient fortification and biofortification
Improving government program selection
Improving government implementation
Mosquito gene drives advocacy and research
Mental health (interventions comparison)
Sleep quality improvement
As you know, GW’s research is very diligent. Consequently, it takes a long time to finalize. I would be interested in having preliminary research conducted by other organizations. Regarding donors focused on animal welfare:
Producers’ outreach, for example,. providing subsidization for farmers interested in higher-welfare farming
CRISPR-based gene drives to address wild animals’ suffering
WAS intervention comparison
Affecting law and law enforcement focused on welfare improvements for chicken and fish in Asia
Insects’ welfare, intervention comparison, for example, reduction of the production of silk, painkillers for insects used in research, etc.
I am currently working on CE’s agenda for the next year in the area of global poverty/health, animal advocacy, and mental health. I will be able to list more areas and research questions worth investigating that CE cannot cover this year at the end of September. I am narrowing down a list of research ideas from 400 ideas (in three cases). Let me know if you are interested in hearing more about it.
Thanks Saulius for pointing this out. We spent a considerable amount of time trying to find a more relevant number, but that was the closes proxy we’ve found. If you have any other source we could use, that would increase the accuracy of the estimate.
When analyzing 35 Independent Pieces of Evidence for Why New Corporate Campaigns Might (or Might Not) Work, I’ve found some commonalities:
Follow-through rate is lower in the case of companies that affect the highest number of hens.
Financial situation: When companies broke or delayed the commitment, they blamed the recession or claimed that there were insufficient consumer demand and lack of funds for making the necessary changes on farms.
ACE claimed that the success of the campaigns depends on the public perception of targeted issues. I have not fack-check that though.
Characteristics that don’t correlate with follow-through:
No correlation between the number of pledges and the % of cage-free eggs in a given country before the campaign. Meaning that campaigns do not seem to be simply riding already existing trends.
I wouldn’t be surprised if those factors were somewhat predictive:
If the pledge was made voluntary form the company or was forced upon by strong negative campaign.
If a given company gave itself some wiggle room in the phrasing of their commitments, which they could later be used as a justification for breaking their commitments.
What brand and public image they try to create
Thanks for the suggestions! As we were discussing above, combining this estimate with a prior estimate using Bayes’ rule might be a solution here. Taking the uncertainty of the model into account, we indeed score this approach quite poorly when it comes to the evidence-base aspect of it. We have a different research template for approaches than the one you linked. I expect to publish the whole report on corporate outreach pretty soon.
When it comes to the step between research questions and the probability distribution, full research, answering each question, can be seen in the full report. In the report, we also address some of the concerns you have with the judgement calls on each of the “qualitative” parameters.
Each update incorporates the weight we put on this factor, the directionality and strength. Those factors, again, rely on other information. With the example, you cited ”what ACE thinks makes an effective campaign” vs “probability that all companies defect in a Prisoner’s Dilemma scenario”. For example, ACE’s opinion on the importance of public support when launching corporate campaigns is formed based on the intervention report they have researched in November 2014, and as they currently claim “is not up to our current standards.”. The landscape has changed since then. As of recent, we can observe that there is a strong track record of successful corporate campaigns in countries where the society didn’t have sympathetic views toward animals (e.g. Lithuania or Japan). I think we can rely more and more on rigorous and generalizable conclusions from research on real-life examples and on the application of game theory to predict the behaviour of the companies.
I agree I wish we had enough time to flesh out the reasoning for each of the factors. Sadly, due to limited time we are constantly having to make trade-offs about whether we should put time into explaining the reasoning more deeply to the broader community vs discussing with the CE candidates vs researching more to get a deeper internal understanding. We generally plan on going deeply into these factors with the specific entrepreneurs looking to start this project or others, who are going to work/are working in the field in the near term, but not publish much more on the topic publicly after our full report.
Thank you for the suggestion. I agree that we can’t extrapolate the conclusion about predicted follow-through rate based on what percentage of companies have followed through on the commitment so far. I looked at it again, and I think that if analyzed correctly it still provides valuable information, so I will leave it in the model, but I’ll move it to the section on updates based on qualitative data and change the values based on the information below. I think that a good proxy for the information is whether the top 20 biggest companies are making progress and will eventually switch is if they responded to EggTrack. We can break it down to:* Companies that passed their deadline: Whole Foods (2004) − 100% follow-throughCostco (2018) - as of July 2017, 78% and 100% converted to liquid*Those that didn’t respond to EggTrack are Walmart, Albertsons, McDonald’s, Target, Sysco, ALDI, Burger King, Tim Hortons, Southeastern, and Wendy’s. * Those that did respond and reported progress: Kroger (21% as of 2017), Publix (50%), SUPERVALU (as of 2016, 12%)* Those in 20 that I do not have information on US Foods, IGA, Inc., Associated Grocers of Florida.I will estimate the value in this cell based on this information unless you have info that could fill in the gaps. I generally think that with all very uncertain estimates, whatever the result, it should be only treated as a cautious update and be combined with prior estimates of value. As a meta comment, I think I’m less concerned about an error in one of the parameters than you seem to be because of the different goals of the research. My goal is to reach broadly good conclusions about which intervention should be executed from a given list of options given a limited amount of time, rather than get the right answer to a specific question, even if it takes me an extremely large amount of time. I think that using cluster approach is superior in such cases. If you are using cluster approach, the more perspectives you take into account the lower the odds of your decision being wrong, and so I trade other aspects (eg. number of interventions compared in a given time frame and how accurate a single estimate need to be) differently. One contradicting factor also cannot overpower the whole decision, etc. A completely different method should be used when we are trying to have as accurate beliefs about the world as possible vs getting to a good decision.
When evaluating cost-effectiveness of interventions or charities, GiveWell only looks at how the action affects the most important metric that the charity is trying to adress. For example, when analysing the cost-effectiveness of SMS vaccine reminders, they only take into account the effect on the vaccination rate, but not on breastfeeding rates, which is also promoted by the intervention. We look at the effect on multiple disperse metrics, including health effects, reduced birth rate, woman empowerment, effect on animal welfare and the environment, etc. Additionally, we have not yet determined that condom distribution is going to be the intervention the charity is going to pursue. We are also considering SMS for reproductive health, community reproductive education, advance provision of EC etc.
I agree that there is no obvious way to model it and the method would even depend on the goal of the model, and it might not necessarily cross-apply to seemingly similar cases.
The estimate reflects a probability distribution of the percentage of corporations that have pledged a welfare improvement that will follow through on those pledges. Note here that it doesn’t inform about what percentage of companies in a country that the organization operate will implement the improvement, but rather the percentage of companies out of companies that have already pledged. Here the 39% − 50% is the most plausible outcome, but the model also includes, for example, the small probability of just 5% of companies following-through. We are also trading the accuracy of the result for the value of the information it provides. Of course, I feel fully confident that the true outcome will be somewhere between 0% and 100%, but this result is not that informative when we need to make a call.
I was modelling in mostly having in mind CE’s asks recommendations: food fortification and management of DO levels. That enabled us to narrow it down and make it more generalizable. I agree it won’t be generalizable for other asks, like the one that you used or even for the broiler asks for the same reasons.
Given your aims, you can use my estimates but just give any prior estimate, given that presumably, your priors aren’t flat or 1.
An alternative to that method might be estimating number of animals affected rather than percentage of corporations since presumably animals aren’t distributed evenly across corporations and so it seems possible that you might hit >x% of animals with x% of corporations. That would require modelling it for a very specific case if you want to get a “usable” result.
Thanks for the suggestions. For our poverty year we mainly focused on GiveWell priority programs although considered some interventions in the hits based area. Next year we plan on writing up some views comparing hits based giving to evidence-based giving and how we think they compare in expected value.
Yes, we do have reports on both of those ideas. I will link it here as soon as they are published.
Thank you for the recommendation. Yes, we took some ideas or their variations from that book.
Thanks, I will definitely post the results. I also encourage others to test it as well, so we have more generalizable data.
Thank you for the suggestion. I’m always open for ideas on productivity improvements, especially if they directly affect charity entrepreneurs ;)
We generated a list of 100 ideas and prioritized them based on things like expected effect on the general population, on me and Joey, ease of testing, etc. As far as I remember, rotating positions from sitting on an office chair to standing to sit on a ball or laying on a couch are more strongly recommended than any single one of those. I think testing all of the tools you can use to be physically active would be an interesting separate experiment in itself. Have you ever tried a mini-stepper? How did you find the effects of a treadmill compared to a mini-stepper?