Thanks for this detailed response. I don’t have much to add over what I said in my post. I have three comments.
Clarificatory question—I calculated the percentage of impact of the different interventions by multiplying the spending on the intervention by the c-e of the intervention. Could you clarify how I should have calculated the impact? Also, I’m not sure I understand how the percentages you give can be right. You say “By our calculations, corporate outreach accounts for about 63% of THL’s 2017 CEE and about 36% of Animal Equality’s 2017 CEE”. But you model the effect of only 3 interventions carried out by Animal Equality—grassroots outreach, investigations, and corporate outreach. But as far as I can see from the CEE, your mean estimate of the effect of grassroots outreach and investigations is negative i.e. they do harm. So, I don’t see how they could comprise 64% of the impact of Animal Equality. I would assume that similar things apply to the CEE of THL. Though maybe I have misunderstood
Clarifying my own position—I don’t argue that the evidence on the impact of THL and Animal Equality comes from the CEE. My argument is that the arguments and evidence provided for the CEEs, and the views in the intervention reports and the charity reviews is not of the standard we should expect.
Corporate campaigns—it’s fine to rely on the open phil report on cage-free welfare, but ACE’s research doesn’t tell the reader this or provide any acceptable justification for the view that corporate campaigns are beneficial.
1- Sure, happy to discuss this further. In the example we gave in footnote 3, we only used the proportional expenditure (PE) to calculate the weighting of each program’s “animal years averted” (AYA) estimate (i.e., weighting for AYA_1 = PE_1/Sum(PE_modelled)). So this gives a weighting that we apply to each AYA estimate, and is independent from the AYA estimate itself. Stopping here is not ideal, however it is not as straightforward to use a similar method for the AYA estimates, due to their distributions.
Including the mean values of the AYA estimates without the rest of the distributions introduces some inconsistencies that make this approach of questionable use. If you consider example 1 in this model, we have two calculations for total AYA. They would be identical if it weren’t for for the distribution of the third AYA. The impacts of the third AYAs would have the same result using your method of calculation, however they clearly impact the model differently (with 3a having a much larger impact on the overall result). In example 2, we have the issue of the mean being very small for one AYA. While the two distributions are of even size and have the same expenditure weighting, an estimate using the mean would attribute 99% of the impact to program 2.
A different way of considering the impact of each part of the model is not to consider the proportional magnitude of each program but to use a sensitivity analysis (Guesstimate has one built in). This tests which parts of the model would have the biggest impact on the final result, should they be adjusted. Running this for both models indicates that the THL model is most sensitive to corporate outreach, while the Animal Equality model fluctuates between corporate and grassroots outreach, depending on how guesstimate populates the model.
2- That’s fair! I agree that we did not sufficiently explain all of the evidence we used in our CEEs, and I agree that our old intervention reports were not of our current standard. You did not state explicitly that the evidence for supporting THL and Animal Equality comes only from their CEEs. However, you seemed to conclude that our reviews provide only weak evidence for supporting each charity simply because our CEEs are weak evidence. My point is just that we provide a lot of other evidence, as well.
3- Agreed—we should have mentioned this! We are trying to do better this year, and we appreciate your insights as our Criterion 3 consultant : )
Hi Toni,
Thanks for this detailed response. I don’t have much to add over what I said in my post. I have three comments.
Clarificatory question—I calculated the percentage of impact of the different interventions by multiplying the spending on the intervention by the c-e of the intervention. Could you clarify how I should have calculated the impact? Also, I’m not sure I understand how the percentages you give can be right. You say “By our calculations, corporate outreach accounts for about 63% of THL’s 2017 CEE and about 36% of Animal Equality’s 2017 CEE”. But you model the effect of only 3 interventions carried out by Animal Equality—grassroots outreach, investigations, and corporate outreach. But as far as I can see from the CEE, your mean estimate of the effect of grassroots outreach and investigations is negative i.e. they do harm. So, I don’t see how they could comprise 64% of the impact of Animal Equality. I would assume that similar things apply to the CEE of THL. Though maybe I have misunderstood
Clarifying my own position—I don’t argue that the evidence on the impact of THL and Animal Equality comes from the CEE. My argument is that the arguments and evidence provided for the CEEs, and the views in the intervention reports and the charity reviews is not of the standard we should expect.
Corporate campaigns—it’s fine to rely on the open phil report on cage-free welfare, but ACE’s research doesn’t tell the reader this or provide any acceptable justification for the view that corporate campaigns are beneficial.
Hi John!
1- Sure, happy to discuss this further. In the example we gave in footnote 3, we only used the proportional expenditure (PE) to calculate the weighting of each program’s “animal years averted” (AYA) estimate (i.e., weighting for AYA_1 = PE_1/Sum(PE_modelled)). So this gives a weighting that we apply to each AYA estimate, and is independent from the AYA estimate itself. Stopping here is not ideal, however it is not as straightforward to use a similar method for the AYA estimates, due to their distributions.
Including the mean values of the AYA estimates without the rest of the distributions introduces some inconsistencies that make this approach of questionable use. If you consider example 1 in this model, we have two calculations for total AYA. They would be identical if it weren’t for for the distribution of the third AYA. The impacts of the third AYAs would have the same result using your method of calculation, however they clearly impact the model differently (with 3a having a much larger impact on the overall result). In example 2, we have the issue of the mean being very small for one AYA. While the two distributions are of even size and have the same expenditure weighting, an estimate using the mean would attribute 99% of the impact to program 2.
A different way of considering the impact of each part of the model is not to consider the proportional magnitude of each program but to use a sensitivity analysis (Guesstimate has one built in). This tests which parts of the model would have the biggest impact on the final result, should they be adjusted. Running this for both models indicates that the THL model is most sensitive to corporate outreach, while the Animal Equality model fluctuates between corporate and grassroots outreach, depending on how guesstimate populates the model.
2- That’s fair! I agree that we did not sufficiently explain all of the evidence we used in our CEEs, and I agree that our old intervention reports were not of our current standard. You did not state explicitly that the evidence for supporting THL and Animal Equality comes only from their CEEs. However, you seemed to conclude that our reviews provide only weak evidence for supporting each charity simply because our CEEs are weak evidence. My point is just that we provide a lot of other evidence, as well.
3- Agreed—we should have mentioned this! We are trying to do better this year, and we appreciate your insights as our Criterion 3 consultant : )
Best, Toni