1- Sure, happy to discuss this further. In the example we gave in footnote 3, we only used the proportional expenditure (PE) to calculate the weighting of each program’s “animal years averted” (AYA) estimate (i.e., weighting for AYA_1 = PE_1/Sum(PE_modelled)). So this gives a weighting that we apply to each AYA estimate, and is independent from the AYA estimate itself. Stopping here is not ideal, however it is not as straightforward to use a similar method for the AYA estimates, due to their distributions.
Including the mean values of the AYA estimates without the rest of the distributions introduces some inconsistencies that make this approach of questionable use. If you consider example 1 in this model, we have two calculations for total AYA. They would be identical if it weren’t for for the distribution of the third AYA. The impacts of the third AYAs would have the same result using your method of calculation, however they clearly impact the model differently (with 3a having a much larger impact on the overall result). In example 2, we have the issue of the mean being very small for one AYA. While the two distributions are of even size and have the same expenditure weighting, an estimate using the mean would attribute 99% of the impact to program 2.
A different way of considering the impact of each part of the model is not to consider the proportional magnitude of each program but to use a sensitivity analysis (Guesstimate has one built in). This tests which parts of the model would have the biggest impact on the final result, should they be adjusted. Running this for both models indicates that the THL model is most sensitive to corporate outreach, while the Animal Equality model fluctuates between corporate and grassroots outreach, depending on how guesstimate populates the model.
2- That’s fair! I agree that we did not sufficiently explain all of the evidence we used in our CEEs, and I agree that our old intervention reports were not of our current standard. You did not state explicitly that the evidence for supporting THL and Animal Equality comes only from their CEEs. However, you seemed to conclude that our reviews provide only weak evidence for supporting each charity simply because our CEEs are weak evidence. My point is just that we provide a lot of other evidence, as well.
3- Agreed—we should have mentioned this! We are trying to do better this year, and we appreciate your insights as our Criterion 3 consultant : )
Hi John!
1- Sure, happy to discuss this further. In the example we gave in footnote 3, we only used the proportional expenditure (PE) to calculate the weighting of each program’s “animal years averted” (AYA) estimate (i.e., weighting for AYA_1 = PE_1/Sum(PE_modelled)). So this gives a weighting that we apply to each AYA estimate, and is independent from the AYA estimate itself. Stopping here is not ideal, however it is not as straightforward to use a similar method for the AYA estimates, due to their distributions.
Including the mean values of the AYA estimates without the rest of the distributions introduces some inconsistencies that make this approach of questionable use. If you consider example 1 in this model, we have two calculations for total AYA. They would be identical if it weren’t for for the distribution of the third AYA. The impacts of the third AYAs would have the same result using your method of calculation, however they clearly impact the model differently (with 3a having a much larger impact on the overall result). In example 2, we have the issue of the mean being very small for one AYA. While the two distributions are of even size and have the same expenditure weighting, an estimate using the mean would attribute 99% of the impact to program 2.
A different way of considering the impact of each part of the model is not to consider the proportional magnitude of each program but to use a sensitivity analysis (Guesstimate has one built in). This tests which parts of the model would have the biggest impact on the final result, should they be adjusted. Running this for both models indicates that the THL model is most sensitive to corporate outreach, while the Animal Equality model fluctuates between corporate and grassroots outreach, depending on how guesstimate populates the model.
2- That’s fair! I agree that we did not sufficiently explain all of the evidence we used in our CEEs, and I agree that our old intervention reports were not of our current standard. You did not state explicitly that the evidence for supporting THL and Animal Equality comes only from their CEEs. However, you seemed to conclude that our reviews provide only weak evidence for supporting each charity simply because our CEEs are weak evidence. My point is just that we provide a lot of other evidence, as well.
3- Agreed—we should have mentioned this! We are trying to do better this year, and we appreciate your insights as our Criterion 3 consultant : )
Best, Toni