I agree that there is no obvious way to model it and the method would even depend on the goal of the model, and it might not necessarily cross-apply to seemingly similar cases.
The estimate reflects a probability distribution of the percentage of corporations that have pledged a welfare improvement that will follow through on those pledges. Note here that it doesnāt inform about what percentage of companies in a country that the organization operate will implement the improvement, but rather the percentage of companies out of companies that have already pledged. Here the 39% ā 50% is the most plausible outcome, but the model also includes, for example, the small probability of just 5% of companies following-through. We are also trading the accuracy of the result for the value of the information it provides. Of course, I feel fully confident that the true outcome will be somewhere between 0% and 100%, but this result is not that informative when we need to make a call.
I was modelling in mostly having in mind CEās asks recommendations: food fortification and management of DO levels. That enabled us to narrow it down and make it more generalizable. I agree it wonāt be generalizable for other asks, like the one that you used or even for the broiler asks for the same reasons.
Given your aims, you can use my estimates but just give any prior estimate, given that presumably, your priors arenāt flat or 1.
An alternative to that method might be estimating number of animals affected rather than percentage of corporations since presumably animals arenāt distributed evenly across corporations and so it seems possible that you might hit >x% of animals with x% of corporations. That would require modelling it for a very specific case if you want to get a āusableā result.
Of course, I feel fully confident that the true outcome will be somewhere between 0% and 100%, but this result is not that informative when we need to make a call.
If your 90% CI is between 0% and 100%, it can be a little bit informative to put that in the model (preferably with a custom probability distribution), because it would help to distinguish between interventions that help 0-2 animals per dollar spent, and interventions that help 1 animal per dollar spent. You should of course prefer the latter to avoid the optimizerās curse. If you end up not having actual 90% subjective confidence intervals because you want to make things simpler, I guess you should keep that in mind when filling the column for the strength of evidence in your Priority Asks table.
Thanks for the suggestions! As we were discussing above, combining this estimate with a prior estimate using Bayesā rule might be a solution here. Taking the uncertainty of the model into account, we indeed score this approach quite poorly when it comes to the evidence-base aspect of it. We have a different research template for approaches than the one you linked. I expect to publish the whole report on corporate outreach pretty soon.
I agree that there is no obvious way to model it and the method would even depend on the goal of the model, and it might not necessarily cross-apply to seemingly similar cases.
The estimate reflects a probability distribution of the percentage of corporations that have pledged a welfare improvement that will follow through on those pledges. Note here that it doesnāt inform about what percentage of companies in a country that the organization operate will implement the improvement, but rather the percentage of companies out of companies that have already pledged. Here the 39% ā 50% is the most plausible outcome, but the model also includes, for example, the small probability of just 5% of companies following-through. We are also trading the accuracy of the result for the value of the information it provides. Of course, I feel fully confident that the true outcome will be somewhere between 0% and 100%, but this result is not that informative when we need to make a call.
I was modelling in mostly having in mind CEās asks recommendations: food fortification and management of DO levels. That enabled us to narrow it down and make it more generalizable. I agree it wonāt be generalizable for other asks, like the one that you used or even for the broiler asks for the same reasons.
Given your aims, you can use my estimates but just give any prior estimate, given that presumably, your priors arenāt flat or 1.
An alternative to that method might be estimating number of animals affected rather than percentage of corporations since presumably animals arenāt distributed evenly across corporations and so it seems possible that you might hit >x% of animals with x% of corporations. That would require modelling it for a very specific case if you want to get a āusableā result.
If your 90% CI is between 0% and 100%, it can be a little bit informative to put that in the model (preferably with a custom probability distribution), because it would help to distinguish between interventions that help 0-2 animals per dollar spent, and interventions that help 1 animal per dollar spent. You should of course prefer the latter to avoid the optimizerās curse. If you end up not having actual 90% subjective confidence intervals because you want to make things simpler, I guess you should keep that in mind when filling the column for the strength of evidence in your Priority Asks table.
Thanks for the suggestions! As we were discussing above, combining this estimate with a prior estimate using Bayesā rule might be a solution here. Taking the uncertainty of the model into account, we indeed score this approach quite poorly when it comes to the evidence-base aspect of it. We have a different research template for approaches than the one you linked. I expect to publish the whole report on corporate outreach pretty soon.