I’m writing this comment as both a fundraiser and someone who has been involved in leading an organization’s participation in an evaluation process. I don’t have the knowledge base to engage on whether ACE or Sinergia’s claims or evaluations are accurate, but reading your posts makes me worry that you lack some essential understanding of how charities work, which would be greatly improved if you discussed your reviews with charities prior to publishing.
I think you’re underestimating how difficult cost-effectiveness estimates are in animal advocacy. The work is highly complex and interdependent, and reliable impact data is not always available. My organization does not currently publish our own cost-effectiveness estimate because we have found it too complex and time consuming to meet the standard of accuracy we anticipate some supporters or potential supporters might hold—essentially out of fear of reviews like yours—and rely on external evaluators like ACE and Rethink Priorities, though their estimates are not really suitable for a general donor audience as they are not widely accessible.
As we have been developing our measurement & evaluation capacity, we are working on creating a cost-effectiveness estimate for our work. Seeing your reviews makes me very hesitant to go through with that process, even though I think overall publishing such an estimate would increase our funding and enable us to spare more animals from suffering. We know that our estimate would be imperfect (different reviewers use different methodologies—there is simply disagreement about what matters and the best methodology) and we would plan to publish our assumptions and work, in line with your expectation that “charities to provide sufficient and publicly stated evidence to justify their publicly stated important claims.”
In fact, I think reviewers reaching out to us makes us more likely to publish evidence for our claims, contrary to your Other Reason #1, as we understand the expectation for this and what kind of evidence we might need to collect and publish. On the other hand, anonymous, scathing reviews make us less likely to publish anything at all. In addition, I disagree with the level of seriousness you assign these types of mistakes and the idea that there needs to be some sort of severe public accountability. There is enormous pressure to produce cost-effectiveness estimates because they are such effective fundraising tools, and too make them broadly legible, contrary to what you or an EA reviewer would expect of them. I would conservatively estimate that I’ve been asked by a potential supporter, meta-charity, or other interested party for a cost-effectiveness estimate at least every month since I started fundraising for an animal advocacy non-profit over 8 years ago. In that time, I’ve seen plenty of cost-effectiveness estimates be revised, or fall in and out of favor. I’m not defending deliberate misinformation, but without talking to charities I think you have no way of knowing whether those mistakes are deliberate, simply mistakes, or valid philosophical disagreements.
I’m writing this comment as both a fundraiser and someone who has been involved in leading an organization’s participation in an evaluation process. I don’t have the knowledge base to engage on whether ACE or Sinergia’s claims or evaluations are accurate, but reading your posts makes me worry that you lack some essential understanding of how charities work, which would be greatly improved if you discussed your reviews with charities prior to publishing.
I think you’re underestimating how difficult cost-effectiveness estimates are in animal advocacy. The work is highly complex and interdependent, and reliable impact data is not always available. My organization does not currently publish our own cost-effectiveness estimate because we have found it too complex and time consuming to meet the standard of accuracy we anticipate some supporters or potential supporters might hold—essentially out of fear of reviews like yours—and rely on external evaluators like ACE and Rethink Priorities, though their estimates are not really suitable for a general donor audience as they are not widely accessible.
As we have been developing our measurement & evaluation capacity, we are working on creating a cost-effectiveness estimate for our work. Seeing your reviews makes me very hesitant to go through with that process, even though I think overall publishing such an estimate would increase our funding and enable us to spare more animals from suffering. We know that our estimate would be imperfect (different reviewers use different methodologies—there is simply disagreement about what matters and the best methodology) and we would plan to publish our assumptions and work, in line with your expectation that “charities to provide sufficient and publicly stated evidence to justify their publicly stated important claims.”
In fact, I think reviewers reaching out to us makes us more likely to publish evidence for our claims, contrary to your Other Reason #1, as we understand the expectation for this and what kind of evidence we might need to collect and publish. On the other hand, anonymous, scathing reviews make us less likely to publish anything at all. In addition, I disagree with the level of seriousness you assign these types of mistakes and the idea that there needs to be some sort of severe public accountability. There is enormous pressure to produce cost-effectiveness estimates because they are such effective fundraising tools, and too make them broadly legible, contrary to what you or an EA reviewer would expect of them. I would conservatively estimate that I’ve been asked by a potential supporter, meta-charity, or other interested party for a cost-effectiveness estimate at least every month since I started fundraising for an animal advocacy non-profit over 8 years ago. In that time, I’ve seen plenty of cost-effectiveness estimates be revised, or fall in and out of favor. I’m not defending deliberate misinformation, but without talking to charities I think you have no way of knowing whether those mistakes are deliberate, simply mistakes, or valid philosophical disagreements.