I’m a little puzzled about how to interpret the results though, and it’s related with a maths problem that I’ve been confused about for a while. However, I have to warn that this is confusing and it might be counterproductive to think about it because of that.
Do you mean that if you start a new campaign for a new ask, then you expect 39% − 50% of companies that make commitments to follow through? If that is the case, the confidence interval seems to be very narrow. My 90% Subjective Confidence Interval (SCI) for that would be 0% − 100%. For example, there can be commitments like stopping chick culling which depend on the creation of new technologies. Scientists might fail to create such technologies in which case it’s 0%. Or they might make them very cheap and then everyone fulfils their commitments (100%).
Another way to interpret the result is that it is your subjective probability that a given company will follow through. But then I’m not sure it makes sense to have a 90% SCI of what your own subjective probability is. What would that even mean? How would you check that the true value is in the confidence interval?
But even if in your cost-effectiveness estimate you would use a point estimate (44%) instead of a SCI (39% − 50%) for the probability, I wouldn’t be sure about how to interpret the results. The result would still be a SCI because you will probably use SCIs in other parts of your calculations. But then that wouldn’t be a 90% SCI of the number of animal affected. It would be a 90% SCI of the expected value of the number of animals affected. But then again, I don’t know how to interpret a 90% SCI of an expected value.
I think that one way to model cost-effectiveness in a way that makes mathematical sense is to have a probability distribution of the percentage of companies that will follow through. The distribution would have some weight on 0%, some weight on 100% and some weight in between. Another way would be to use point estimates everywhere and say that it is an expected value. Of course, no one will die if you mix these two things, but the result might be difficult to interpret.
If anyone thinks that my reasoning here is wrong, I’d be very curious to hear because I encounter this problem quite often nowadays. And currently I am making a cost-effectiveness model of corporate campaigns myself, and I don’t quite know what to do with the uncertainty about following through...
I agree that there is no obvious way to model it and the method would even depend on the goal of the model, and it might not necessarily cross-apply to seemingly similar cases.
The estimate reflects a probability distribution of the percentage of corporations that have pledged a welfare improvement that will follow through on those pledges. Note here that it doesn’t inform about what percentage of companies in a country that the organization operate will implement the improvement, but rather the percentage of companies out of companies that have already pledged. Here the 39% − 50% is the most plausible outcome, but the model also includes, for example, the small probability of just 5% of companies following-through. We are also trading the accuracy of the result for the value of the information it provides. Of course, I feel fully confident that the true outcome will be somewhere between 0% and 100%, but this result is not that informative when we need to make a call.
I was modelling in mostly having in mind CE’s asks recommendations: food fortification and management of DO levels. That enabled us to narrow it down and make it more generalizable. I agree it won’t be generalizable for other asks, like the one that you used or even for the broiler asks for the same reasons.
Given your aims, you can use my estimates but just give any prior estimate, given that presumably, your priors aren’t flat or 1.
An alternative to that method might be estimating number of animals affected rather than percentage of corporations since presumably animals aren’t distributed evenly across corporations and so it seems possible that you might hit >x% of animals with x% of corporations. That would require modelling it for a very specific case if you want to get a “usable” result.
Of course, I feel fully confident that the true outcome will be somewhere between 0% and 100%, but this result is not that informative when we need to make a call.
If your 90% CI is between 0% and 100%, it can be a little bit informative to put that in the model (preferably with a custom probability distribution), because it would help to distinguish between interventions that help 0-2 animals per dollar spent, and interventions that help 1 animal per dollar spent. You should of course prefer the latter to avoid the optimizer’s curse. If you end up not having actual 90% subjective confidence intervals because you want to make things simpler, I guess you should keep that in mind when filling the column for the strength of evidence in your Priority Asks table.
Thanks for the suggestions! As we were discussing above, combining this estimate with a prior estimate using Bayes’ rule might be a solution here. Taking the uncertainty of the model into account, we indeed score this approach quite poorly when it comes to the evidence-base aspect of it. We have a different research template for approaches than the one you linked. I expect to publish the whole report on corporate outreach pretty soon.
Another way to interpret the result is that it is your subjective probability that a given company will follow through. But then I’m not sure it makes sense to have a 90% SCI of what your own subjective probability is. What would that even mean? How would you check that the true value is in the confidence interval?
Karolina has already answered, my response here is very late and you’ve already worked on corporate campaign research, so I’d guess you’re already satisfied, but hopefully this will be useful for anyone else who comes by or might still be useful to you.
Polling, e.g. for elections, is a useful comparison, if the follow-through rate in cage-free campaigns is interpreted as as the proportion of pledged companies that will actually go cage-free, rather than a probability that a given (or random) company will go cage-free.
Here, for each pledged company (ignoring those that haven’t pledged), we have a binary random variable (Bernoulli trial) whose outcome is whether or not it follows through on its commitment. In polling, the binary variable for each individual from the population from which respondents are sampled is whether or not they would vote for candidate X if an election were held the day of polling (and then we might extrapolate the estimate to the actual election held in the future, but this would not really be justified under frequentism).
Checking that the true value is in the confidence interval would just mean checking for each pledged company (pledged by a given date, or pledged by the time we check), whether or not it has followed through, and calculating the proportion that have followed through.
In polling, the population of interest is the votes of actual voters on the day of the actual election, but not all poll respondents will vote, and surely not all actual voters were polled. Furthermore, even respondents may change their mind between the day of polling and the actual election, so even if we polled all and only the actual future voters, because they can change their mind, the statistical populations, in the technical sense, between poll responses and actual future votes could still differ, since it’s the responses/votes that make up the statistical populations, not the people.
For cage-free campaigns, the population of interest is the set of companies that have made pledges by some date (e.g. before today, so only companies that have already made pledges). If we base our estimate only on companies that have already made pledges, we don’t “poll anyone who won’t vote”, so our sample population is a subset of (possibly equal to) the population of interest. I suppose this might be complicated if we consider companies that have actually followed through by the time we’ve finished collecting data for our estimate and then start using cages again; this would be like poll respondents changing their minds.
This is very interesting and useful, thank you!
I’m a little puzzled about how to interpret the results though, and it’s related with a maths problem that I’ve been confused about for a while. However, I have to warn that this is confusing and it might be counterproductive to think about it because of that.
Do you mean that if you start a new campaign for a new ask, then you expect 39% − 50% of companies that make commitments to follow through? If that is the case, the confidence interval seems to be very narrow. My 90% Subjective Confidence Interval (SCI) for that would be 0% − 100%. For example, there can be commitments like stopping chick culling which depend on the creation of new technologies. Scientists might fail to create such technologies in which case it’s 0%. Or they might make them very cheap and then everyone fulfils their commitments (100%).
Another way to interpret the result is that it is your subjective probability that a given company will follow through. But then I’m not sure it makes sense to have a 90% SCI of what your own subjective probability is. What would that even mean? How would you check that the true value is in the confidence interval?
But even if in your cost-effectiveness estimate you would use a point estimate (44%) instead of a SCI (39% − 50%) for the probability, I wouldn’t be sure about how to interpret the results. The result would still be a SCI because you will probably use SCIs in other parts of your calculations. But then that wouldn’t be a 90% SCI of the number of animal affected. It would be a 90% SCI of the expected value of the number of animals affected. But then again, I don’t know how to interpret a 90% SCI of an expected value.
I think that one way to model cost-effectiveness in a way that makes mathematical sense is to have a probability distribution of the percentage of companies that will follow through. The distribution would have some weight on 0%, some weight on 100% and some weight in between. Another way would be to use point estimates everywhere and say that it is an expected value. Of course, no one will die if you mix these two things, but the result might be difficult to interpret.
If anyone thinks that my reasoning here is wrong, I’d be very curious to hear because I encounter this problem quite often nowadays. And currently I am making a cost-effectiveness model of corporate campaigns myself, and I don’t quite know what to do with the uncertainty about following through...
I agree that there is no obvious way to model it and the method would even depend on the goal of the model, and it might not necessarily cross-apply to seemingly similar cases.
The estimate reflects a probability distribution of the percentage of corporations that have pledged a welfare improvement that will follow through on those pledges. Note here that it doesn’t inform about what percentage of companies in a country that the organization operate will implement the improvement, but rather the percentage of companies out of companies that have already pledged. Here the 39% − 50% is the most plausible outcome, but the model also includes, for example, the small probability of just 5% of companies following-through. We are also trading the accuracy of the result for the value of the information it provides. Of course, I feel fully confident that the true outcome will be somewhere between 0% and 100%, but this result is not that informative when we need to make a call.
I was modelling in mostly having in mind CE’s asks recommendations: food fortification and management of DO levels. That enabled us to narrow it down and make it more generalizable. I agree it won’t be generalizable for other asks, like the one that you used or even for the broiler asks for the same reasons.
Given your aims, you can use my estimates but just give any prior estimate, given that presumably, your priors aren’t flat or 1.
An alternative to that method might be estimating number of animals affected rather than percentage of corporations since presumably animals aren’t distributed evenly across corporations and so it seems possible that you might hit >x% of animals with x% of corporations. That would require modelling it for a very specific case if you want to get a “usable” result.
If your 90% CI is between 0% and 100%, it can be a little bit informative to put that in the model (preferably with a custom probability distribution), because it would help to distinguish between interventions that help 0-2 animals per dollar spent, and interventions that help 1 animal per dollar spent. You should of course prefer the latter to avoid the optimizer’s curse. If you end up not having actual 90% subjective confidence intervals because you want to make things simpler, I guess you should keep that in mind when filling the column for the strength of evidence in your Priority Asks table.
Thanks for the suggestions! As we were discussing above, combining this estimate with a prior estimate using Bayes’ rule might be a solution here. Taking the uncertainty of the model into account, we indeed score this approach quite poorly when it comes to the evidence-base aspect of it. We have a different research template for approaches than the one you linked. I expect to publish the whole report on corporate outreach pretty soon.
Karolina has already answered, my response here is very late and you’ve already worked on corporate campaign research, so I’d guess you’re already satisfied, but hopefully this will be useful for anyone else who comes by or might still be useful to you.
See, for the frequentist approach, https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval .
Polling, e.g. for elections, is a useful comparison, if the follow-through rate in cage-free campaigns is interpreted as as the proportion of pledged companies that will actually go cage-free, rather than a probability that a given (or random) company will go cage-free.
Here, for each pledged company (ignoring those that haven’t pledged), we have a binary random variable (Bernoulli trial) whose outcome is whether or not it follows through on its commitment. In polling, the binary variable for each individual from the population from which respondents are sampled is whether or not they would vote for candidate X if an election were held the day of polling (and then we might extrapolate the estimate to the actual election held in the future, but this would not really be justified under frequentism).
Checking that the true value is in the confidence interval would just mean checking for each pledged company (pledged by a given date, or pledged by the time we check), whether or not it has followed through, and calculating the proportion that have followed through.
In polling, the population of interest is the votes of actual voters on the day of the actual election, but not all poll respondents will vote, and surely not all actual voters were polled. Furthermore, even respondents may change their mind between the day of polling and the actual election, so even if we polled all and only the actual future voters, because they can change their mind, the statistical populations, in the technical sense, between poll responses and actual future votes could still differ, since it’s the responses/votes that make up the statistical populations, not the people.
For cage-free campaigns, the population of interest is the set of companies that have made pledges by some date (e.g. before today, so only companies that have already made pledges). If we base our estimate only on companies that have already made pledges, we don’t “poll anyone who won’t vote”, so our sample population is a subset of (possibly equal to) the population of interest. I suppose this might be complicated if we consider companies that have actually followed through by the time we’ve finished collecting data for our estimate and then start using cages again; this would be like poll respondents changing their minds.
(Or, as done in https://forum.effectivealtruism.org/posts/WvgrGLDBko6rZ5qax/did-corporate-campaigns-in-the-us-have-any-counterfactual or as you’ve done in https://forum.effectivealtruism.org/posts/L5EZjjXKdNgcm253H/corporate-campaigns-affect-9-to-120-years-of-chicken-life , instead of binary random variables for following through, we could estimate progress by a given date.)