One of my the biggest take aways from looking at your model was the importance of the Mean Years of Impact parameter. Looking at guesstimate’s sensitivity analysis the r^2 value is about 0.75 [1], meaning approximately ~75% of the variance in the bottom line result is due to the variance estimating Mean Years of Impact.
Your choice of SCI is also significantly more optimistic than the figures that ACE or Lewis Bollard use. ACE seems to use a log-normal distribution with SCI 1.6 to 13 [2]. Using this in your model gives a bottom-line estimate of 16 (4.2 to 43) years of life affected per dollar. Simply using Bollard’s estimate of 5 years (with no uncertainty) gives 14 (7.5 to 24) years of life affected.
You note above that you are estimating different things from both ACE and Bollard, as your counterfactual is for a world without undercover investigations. This does explain some of the discrepancy, but even so the estimates seem quite far apart. This suggests it’s worth doing more research into the parameter and both your model and ACE’s model should be more uncertain about it.
Additionally, I’d like to see reports like this contain a short display of which input parameters have the largest effect with the result. I think it can be the important information for seeing how robust the result is!
[1] The exact r^2 value changes on refresh and if I’m reading from the small graph or the large graph after hovering over, but it’s consistently around 0.7-0.8. More simulation samples in would likely be necessary to compute the exact figure.
[2] The exact parameter ACE uses points to a link I don’t have view permission for, but this distribution replicates the graph.
TL;DR: Nobody seems to know what the value of mean years of impact should be, and I don’t see how this uncertainty could be reduced. I think that indirect effects are more important and it would be better to research them.
Good points and I’m happy you brought it up.
Firstly, I know you know this, but Lewis wrote that in his view, “the assumption that these campaigns only accelerated pledges by five years is very conservative.” It makes sense to use a conservative value when doing a point estimate like he did. And I did use a similar value (4 years) in my conservative estimate in Appendix 1. ACE did not really describe their choice for the value so I didn’t pay much attention to it. There’s also Capriati (2018) which assumed that THL’s cage-free and broiler campaigns moved policies forward by only one year (I just added the description. But this assumes that other organizations would have still done everything they did. And even then, I don’t think it is reasonable.
To be honest, I think that nobody has a clue about what value to use here. Hence, everybody uses random conservative values in order for the end result to be more believable, because the estimated cost-effectiveness of campaigns is unbelievably high even with a conservative values. I did that to a degree as well. I asked some people who work on corporate campaigns what value would they use for mean years of impact. They thought that my range was reasonable, but I think they would have said that about many different ranges because it’s difficult to think about. Only one person was able to say what range for mean years of impact would they use without looking at my range, and they said 40 to 100 years. If I weren’t anchored by other estimates and didn’t want to be a bit conservative to be more convincing to skeptics, I think I would have chosen a higher value as well, especially for the upper bound. In general, my impression is that all these “estimates” (including mine) are basically guesses that look more legitimate than they should because they are surrounded by neatly formatted text. We just can’t predict the future.
I also don’t see how to do more research on this topic. Perhaps seeing how long corporate commitments in other domains lasted could give a little bit information on that but only a little bit because situations are so different. If you have any ideas, I’d be interested to hear them.
But I’m uncertain if it’s worthwhile to put more effort into cost-effectiveness estimates of corporate campaigns. I don’t know what decisions depend on whether it’s 7.5-24 years or 9-120 years. I think it would be more decision-relevant to research indirect effects, or in which regions corporate campaigns are more cost-effective, or how corporate campaigns compare with legislative campaigns for welfare reforms.
your model and ACE’s model should be more uncertain about it.
I use a subjective confidence interval of 4 to 36 years with a log-normal distribution, so it’s already very uncertain. Perhaps I should have used a flatter custom distribution, but after experimenting, it seems that it wouldn’t change the answer much.
Additionally, I’d like to see reports like this contain a short display of which input parameters have the largest effect with the result. I think it can be the important information for seeing how robust the result is!
Interesting, I will consider it. Do you know of any report that I could use as an example of how to write such a thing?
First of all, great model and write-up.
One of my the biggest take aways from looking at your model was the importance of the Mean Years of Impact parameter. Looking at guesstimate’s sensitivity analysis the r^2 value is about 0.75 [1], meaning approximately ~75% of the variance in the bottom line result is due to the variance estimating Mean Years of Impact.
Your choice of SCI is also significantly more optimistic than the figures that ACE or Lewis Bollard use. ACE seems to use a log-normal distribution with SCI 1.6 to 13 [2]. Using this in your model gives a bottom-line estimate of 16 (4.2 to 43) years of life affected per dollar. Simply using Bollard’s estimate of 5 years (with no uncertainty) gives 14 (7.5 to 24) years of life affected.
You note above that you are estimating different things from both ACE and Bollard, as your counterfactual is for a world without undercover investigations. This does explain some of the discrepancy, but even so the estimates seem quite far apart. This suggests it’s worth doing more research into the parameter and both your model and ACE’s model should be more uncertain about it.
Additionally, I’d like to see reports like this contain a short display of which input parameters have the largest effect with the result. I think it can be the important information for seeing how robust the result is!
[1] The exact r^2 value changes on refresh and if I’m reading from the small graph or the large graph after hovering over, but it’s consistently around 0.7-0.8. More simulation samples in would likely be necessary to compute the exact figure.
[2] The exact parameter ACE uses points to a link I don’t have view permission for, but this distribution replicates the graph.
TL;DR: Nobody seems to know what the value of mean years of impact should be, and I don’t see how this uncertainty could be reduced. I think that indirect effects are more important and it would be better to research them.
Good points and I’m happy you brought it up.
Firstly, I know you know this, but Lewis wrote that in his view, “the assumption that these campaigns only accelerated pledges by five years is very conservative.” It makes sense to use a conservative value when doing a point estimate like he did. And I did use a similar value (4 years) in my conservative estimate in Appendix 1. ACE did not really describe their choice for the value so I didn’t pay much attention to it. There’s also Capriati (2018) which assumed that THL’s cage-free and broiler campaigns moved policies forward by only one year (I just added the description. But this assumes that other organizations would have still done everything they did. And even then, I don’t think it is reasonable.
To be honest, I think that nobody has a clue about what value to use here. Hence, everybody uses random conservative values in order for the end result to be more believable, because the estimated cost-effectiveness of campaigns is unbelievably high even with a conservative values. I did that to a degree as well. I asked some people who work on corporate campaigns what value would they use for mean years of impact. They thought that my range was reasonable, but I think they would have said that about many different ranges because it’s difficult to think about. Only one person was able to say what range for mean years of impact would they use without looking at my range, and they said 40 to 100 years. If I weren’t anchored by other estimates and didn’t want to be a bit conservative to be more convincing to skeptics, I think I would have chosen a higher value as well, especially for the upper bound. In general, my impression is that all these “estimates” (including mine) are basically guesses that look more legitimate than they should because they are surrounded by neatly formatted text. We just can’t predict the future.
I also don’t see how to do more research on this topic. Perhaps seeing how long corporate commitments in other domains lasted could give a little bit information on that but only a little bit because situations are so different. If you have any ideas, I’d be interested to hear them.
But I’m uncertain if it’s worthwhile to put more effort into cost-effectiveness estimates of corporate campaigns. I don’t know what decisions depend on whether it’s 7.5-24 years or 9-120 years. I think it would be more decision-relevant to research indirect effects, or in which regions corporate campaigns are more cost-effective, or how corporate campaigns compare with legislative campaigns for welfare reforms.
I use a subjective confidence interval of 4 to 36 years with a log-normal distribution, so it’s already very uncertain. Perhaps I should have used a flatter custom distribution, but after experimenting, it seems that it wouldn’t change the answer much.
Interesting, I will consider it. Do you know of any report that I could use as an example of how to write such a thing?