I had arrived at similar conclusions. Lately we were busy preparing material to put up on our freshly minted website at EA France. When it comes to recommendations, we naturally turn to charities such as GiveWell or ACE and refer to their work. But before putting these recommendations onto our website, I wanted to double-check that the evaluations are sound, because I want that we be ready to back them up publicly with confidence. So I tried to locate contradicting voices. For GiveWell, I found out that they disagree with other organizations such as the Campbell institute or Cochrane on the effectiveness of deworming programs. So I spent quite a bit of time reading the arguments of each party, and after that I came out extremely impressed by the depth and seriousness of the analysis of GiveWell. Of course I did not check all of what they do, but this experience gave me very high confidence that they are doing an outstanding work.
Then I moved to ACE. I took Nathan’s article as a starting point for the disagreeing voice. Of course I was appalled by some of the points raised there, in particular in relation with leafletting. Also, a friend at AEF dig up a Facebook thread that happened around the time of publication of this article, and my recollection is that half of the people discussing this where just busy explaining that Nathan was really a very very mean person that we could not possibly imagine talking to.
I understand that this is old news, but I want to pause for a moment and reflect on the bigger picture. On human poverty, GiveWell is one among several very serious actors. It engages very thoroughly in discussions and explanations when diverging views emerge. We can argue about whether Nathan was diplomatic enough, etc, certainly he did not take all the precautions that Halstead has taken when writing this piece. But we have to realize that when it comes to animal suffering, as far as I know ACE is the only game in town. In my opinion, this is a precarious state of affairs, and we should do our best to protect criticism of ACE, even when it does not come with the highest level of politeness. Of course, I do not mean that people at ACE have bad intentions, but checks and balances are important, we are all human beings, and right now there seems to be precious little of these.
And as it turns out, as pointed out here by Halstead, at least some of the criticism of Nathan was actually correct, and is now acknowledged on the website of ACE (e.g. on leafletting).
And then I must stay that I fell off of my chair when I looked up the evaluation of corporate outreach and found out that the single argument in support of cage-free reform was this paper by De Mol et al (2016). To give an element of context, I have no previous exposure to animal wellfare, and yet it jumped at me that this was very bogus. How can this possibly happen?? I know that now ACE changed their position about this, but how they could come up with that in the first place, and how on Earth can this argument be still available online completely eludes me. All this while, as far as I can tell, corporate outreach is one of the flagship interventions advocated by ACE.
But again, I want to pause and think about the bigger picture for a while. The fact is that at the time of writing this argument, the organisation Direct Action Everywhere (DxE) had put up a rather comprehensive report explaining that they had come up with the opposite conclusion! (That cage-free reform is actually detrimental to animal wellfare.) I will refrain from discussing it at length here because this comment is already long, but this report of DxE was, in my opinion, dismissed with precious little good argument.
So again I see a small dissenting voice in the otherwise rather monopolistic position of ACE which is being dismissed without due consideration. And of course I get more worried. (To be perfectly clear, my point has nothing to do with whether DxE’s conclusions were right or not; only with the fact that they were dismissed without proper consideration.)
I know that ACE no longer considers the article of De Mol et al (2016) as relevant, and things are clearly moving in a positive direction. Yet my confidence in ACE’s work is at an all-time low. For full disclosure, I was planning to propose to my colleagues at AEF that we spend some time doing a “case study” in relation with the recommendations of ACE (similarly to studying the controversy about deworming for GiveWell). The “case study” I have in mind is the comparative evaluations of the Good Food Institute vs. New Harvest (As a side remark, I am interested in any previous discussion about this point.)
To sum up, I want to stress that I write this with the best intentions, and I appreciate that ACE has been improving a lot on all the points I have raised so far. Also, I understand that we cannot “go meta” and ask for evaluators of evaluators, evaluators of evaluators of evaluators, etc. Yet it is my impression that when it comes to human poverty, the situation is much, much healthier. In this area, there are several influential actors that have their own decision processes, can then compare their conclusions, and engage in serious debate about them. For animal suffering, I think it would do us a lot of good to make sure that dissenting voices are heard and protected, and that ACE engages with their arguments with much greater consideration than has been the case so far.
It sounds like AEF is doing a fantastic job of ensuring rigour in its messaging!
But we have to realize that when it comes to animal suffering, as far as I know ACE is the only game in town. In my opinion, this is a precarious state of affairs, and we should do our best to protect criticism of ACE, even when it does not come with the highest level of politeness.
I think in cases where there is little primary research, it’s all the more important to ensure that discourse remain not merely polite, but friendly and kind. Research isn’t easy at the best of times, and the animal space has a number of factors making it harder than others like global poverty (eg historic neglect and the difficulty of understanding experiences unlike our own). In cases like this where people are pushing ahead despite difficulty, it is all the more important to make sure that the work is actively appreciated, and at baseline that people do not end up feeling attacked simply for doing it. Criticisms that are framed badly can easily be worse than nothing, in leading those working in this area to think that their work isn’t useful and they should leave the area, and by dissuading others from joining the area in the first place.
This makes me all the more grateful to John for being so thoughtful in his feedback—suggesting improvements directly to ACE in the first instance, running a public piece by them before publishing, and for highlighting reasons for being optimistic as well as potential problems.
The problem with HN’s article wasn’t just that it was “impolite” but that it mixed in a number of unfounded ad hominem attacks along with the more serious criticisms, arguments that implied people at ACE and other organisations were deliberately acting in bad faith. It seems to me that the proper way to respond to a mix of good arguments and bad arguments is to take the good arguments into account while dismissing the bad ones, and that seems to be what happened.
And just as a broader point, if someone regularly mixes ad hominem attacks with more serious points, it’s not a good idea to signal boost the mixed arguments. If I say, “so-and-so is a liar and a thief who only cares about self-promotion, and also so-and-so incorrectly reported a particular study,” the best response is to take the valid point about the study into consideration without promoting the argument as a whole.
Moreover, the methodological arguments put forward weren’t entirely new; HL Lab director Harish Sethu’s presentation at the EAA conference in Princeton, for example, had extremely detailed methodological criticisms that far surpassed the other criticisms I had seen to that point.
In other words, there’s not a binary decision between ignoring the points altogether and praising the deeply flawed HN article. The good methodological points are being taken into consideration, as seen in some of the changes made and in this essay.
On human poverty, GiveWell is one among several very serious actors. It engages very thoroughly in discussions and explanations when diverging views emerge.
The diverging views in the case of the GiveWell example you gave are from respected research organizations Campbell and Cochrane, with all parties arguing in good faith. This was very different from the case of Nathan’s criticisms of ACE.
So again I see a small dissenting voice in the otherwise rather monopolistic position of ACE which is being dismissed without due consideration.
But ACE did reply to Nathan Harrison’s criticisms:
But again, I want to pause and think about the bigger picture for a while. The fact is that at the time of writing this argument, the organisation Direct Action Everywhere (DxE) had put up a rather comprehensive report explaining that they had come up with the opposite conclusion! (That cage-free reform is actually detrimental to animal wellfare.) I will refrain from discussing it at length here because this comment is already long, but this report of DxE was, in my opinion, dismissed with precious little good argument.
My understanding is that ACE did in fact take DxE’s arguments into consideration, and that their relatively pessimistic estimate that cage-free represents a ~5% improvement is informed by many different views, including DxE’s arguments that cage-free is harmful. (This is from conversations I’ve had and I’m not sure if ACE has published this. I agree it would be helpful if ACE published their reasons for this estimate.)
But we have to realize that when it comes to animal suffering, as far as I know ACE is the only game in town.
I’m not sure how you define “the only game in town.” There are currently a number of other organizations who do research on effective animal advocacy, including Open Philanthropy, Sentience Institute, Rethink Charity, Faunalytics, Humane Society of the United States, Humane League Labs, Animal Welfare Action Lab, Wild Animal Suffering Research, etc.
The fact of the matter is that ACE exists since 2013, that corporate outreach is central to its strategy, and that Nathan, DxE, and now Halstead have serious reservations about how the impact of this intervention is assessed. Moreover, it seems to be agreed upon that past mistakes have been substantial and yet have stayed online way too long. So I’m not sure if I found the best way to express it, and I surely wasted a lot of time on material that is regarded as outdated, but I think the point remains that ACE’s research has not been sufficiently “tested”. I am very happy to see all the progress that has been made and is to be coming at ACE. But I also hope with Halstead that these observations “will encourage external scrutiny of ACE research going forward”.
Wouldn’t referring to other groups likely confirm that it is the only game in town? If they were working on similar issues then there would be cross referencing and a greater degree of accountability. But it seems that hasn’t happened at least in some cases and it may or may not be the case there are further issues to be examined elsewhere. In my view there are around moral theory (particularly managing more polarising issues), whilst i would disagree with Jc that meta evaluation isn’t useful. Likely it would provide some useful information to consider in one sweep if other organisations aren’t doing that work or people too time constrained or just willing to trust in the process. I think it would at least be worthwhile seeing whether it has value in this context and it could also give people more confidence in the process.
I had arrived at similar conclusions. Lately we were busy preparing material to put up on our freshly minted website at EA France. When it comes to recommendations, we naturally turn to charities such as GiveWell or ACE and refer to their work. But before putting these recommendations onto our website, I wanted to double-check that the evaluations are sound, because I want that we be ready to back them up publicly with confidence. So I tried to locate contradicting voices. For GiveWell, I found out that they disagree with other organizations such as the Campbell institute or Cochrane on the effectiveness of deworming programs. So I spent quite a bit of time reading the arguments of each party, and after that I came out extremely impressed by the depth and seriousness of the analysis of GiveWell. Of course I did not check all of what they do, but this experience gave me very high confidence that they are doing an outstanding work.
Then I moved to ACE. I took Nathan’s article as a starting point for the disagreeing voice. Of course I was appalled by some of the points raised there, in particular in relation with leafletting. Also, a friend at AEF dig up a Facebook thread that happened around the time of publication of this article, and my recollection is that half of the people discussing this where just busy explaining that Nathan was really a very very mean person that we could not possibly imagine talking to.
I understand that this is old news, but I want to pause for a moment and reflect on the bigger picture. On human poverty, GiveWell is one among several very serious actors. It engages very thoroughly in discussions and explanations when diverging views emerge. We can argue about whether Nathan was diplomatic enough, etc, certainly he did not take all the precautions that Halstead has taken when writing this piece. But we have to realize that when it comes to animal suffering, as far as I know ACE is the only game in town. In my opinion, this is a precarious state of affairs, and we should do our best to protect criticism of ACE, even when it does not come with the highest level of politeness. Of course, I do not mean that people at ACE have bad intentions, but checks and balances are important, we are all human beings, and right now there seems to be precious little of these.
And as it turns out, as pointed out here by Halstead, at least some of the criticism of Nathan was actually correct, and is now acknowledged on the website of ACE (e.g. on leafletting).
And then I must stay that I fell off of my chair when I looked up the evaluation of corporate outreach and found out that the single argument in support of cage-free reform was this paper by De Mol et al (2016). To give an element of context, I have no previous exposure to animal wellfare, and yet it jumped at me that this was very bogus. How can this possibly happen?? I know that now ACE changed their position about this, but how they could come up with that in the first place, and how on Earth can this argument be still available online completely eludes me. All this while, as far as I can tell, corporate outreach is one of the flagship interventions advocated by ACE.
But again, I want to pause and think about the bigger picture for a while. The fact is that at the time of writing this argument, the organisation Direct Action Everywhere (DxE) had put up a rather comprehensive report explaining that they had come up with the opposite conclusion! (That cage-free reform is actually detrimental to animal wellfare.) I will refrain from discussing it at length here because this comment is already long, but this report of DxE was, in my opinion, dismissed with precious little good argument.
So again I see a small dissenting voice in the otherwise rather monopolistic position of ACE which is being dismissed without due consideration. And of course I get more worried. (To be perfectly clear, my point has nothing to do with whether DxE’s conclusions were right or not; only with the fact that they were dismissed without proper consideration.)
I know that ACE no longer considers the article of De Mol et al (2016) as relevant, and things are clearly moving in a positive direction. Yet my confidence in ACE’s work is at an all-time low. For full disclosure, I was planning to propose to my colleagues at AEF that we spend some time doing a “case study” in relation with the recommendations of ACE (similarly to studying the controversy about deworming for GiveWell). The “case study” I have in mind is the comparative evaluations of the Good Food Institute vs. New Harvest (As a side remark, I am interested in any previous discussion about this point.)
To sum up, I want to stress that I write this with the best intentions, and I appreciate that ACE has been improving a lot on all the points I have raised so far. Also, I understand that we cannot “go meta” and ask for evaluators of evaluators, evaluators of evaluators of evaluators, etc. Yet it is my impression that when it comes to human poverty, the situation is much, much healthier. In this area, there are several influential actors that have their own decision processes, can then compare their conclusions, and engage in serious debate about them. For animal suffering, I think it would do us a lot of good to make sure that dissenting voices are heard and protected, and that ACE engages with their arguments with much greater consideration than has been the case so far.
It sounds like AEF is doing a fantastic job of ensuring rigour in its messaging!
I think in cases where there is little primary research, it’s all the more important to ensure that discourse remain not merely polite, but friendly and kind. Research isn’t easy at the best of times, and the animal space has a number of factors making it harder than others like global poverty (eg historic neglect and the difficulty of understanding experiences unlike our own). In cases like this where people are pushing ahead despite difficulty, it is all the more important to make sure that the work is actively appreciated, and at baseline that people do not end up feeling attacked simply for doing it. Criticisms that are framed badly can easily be worse than nothing, in leading those working in this area to think that their work isn’t useful and they should leave the area, and by dissuading others from joining the area in the first place.
This makes me all the more grateful to John for being so thoughtful in his feedback—suggesting improvements directly to ACE in the first instance, running a public piece by them before publishing, and for highlighting reasons for being optimistic as well as potential problems.
The problem with HN’s article wasn’t just that it was “impolite” but that it mixed in a number of unfounded ad hominem attacks along with the more serious criticisms, arguments that implied people at ACE and other organisations were deliberately acting in bad faith. It seems to me that the proper way to respond to a mix of good arguments and bad arguments is to take the good arguments into account while dismissing the bad ones, and that seems to be what happened.
And just as a broader point, if someone regularly mixes ad hominem attacks with more serious points, it’s not a good idea to signal boost the mixed arguments. If I say, “so-and-so is a liar and a thief who only cares about self-promotion, and also so-and-so incorrectly reported a particular study,” the best response is to take the valid point about the study into consideration without promoting the argument as a whole.
Moreover, the methodological arguments put forward weren’t entirely new; HL Lab director Harish Sethu’s presentation at the EAA conference in Princeton, for example, had extremely detailed methodological criticisms that far surpassed the other criticisms I had seen to that point.
In other words, there’s not a binary decision between ignoring the points altogether and praising the deeply flawed HN article. The good methodological points are being taken into consideration, as seen in some of the changes made and in this essay.
The diverging views in the case of the GiveWell example you gave are from respected research organizations Campbell and Cochrane, with all parties arguing in good faith. This was very different from the case of Nathan’s criticisms of ACE.
But ACE did reply to Nathan Harrison’s criticisms:
https://animalcharityevaluators.org/blog/responses-to-common-critiques/ https://animalcharityevaluators.org/blog/response-to-a-recent-critique-of-our-research/
My understanding is that ACE did in fact take DxE’s arguments into consideration, and that their relatively pessimistic estimate that cage-free represents a ~5% improvement is informed by many different views, including DxE’s arguments that cage-free is harmful. (This is from conversations I’ve had and I’m not sure if ACE has published this. I agree it would be helpful if ACE published their reasons for this estimate.)
I’m not sure how you define “the only game in town.” There are currently a number of other organizations who do research on effective animal advocacy, including Open Philanthropy, Sentience Institute, Rethink Charity, Faunalytics, Humane Society of the United States, Humane League Labs, Animal Welfare Action Lab, Wild Animal Suffering Research, etc.
The fact of the matter is that ACE exists since 2013, that corporate outreach is central to its strategy, and that Nathan, DxE, and now Halstead have serious reservations about how the impact of this intervention is assessed. Moreover, it seems to be agreed upon that past mistakes have been substantial and yet have stayed online way too long. So I’m not sure if I found the best way to express it, and I surely wasted a lot of time on material that is regarded as outdated, but I think the point remains that ACE’s research has not been sufficiently “tested”. I am very happy to see all the progress that has been made and is to be coming at ACE. But I also hope with Halstead that these observations “will encourage external scrutiny of ACE research going forward”.
Wouldn’t referring to other groups likely confirm that it is the only game in town? If they were working on similar issues then there would be cross referencing and a greater degree of accountability. But it seems that hasn’t happened at least in some cases and it may or may not be the case there are further issues to be examined elsewhere. In my view there are around moral theory (particularly managing more polarising issues), whilst i would disagree with Jc that meta evaluation isn’t useful. Likely it would provide some useful information to consider in one sweep if other organisations aren’t doing that work or people too time constrained or just willing to trust in the process. I think it would at least be worthwhile seeing whether it has value in this context and it could also give people more confidence in the process.