apparently the main evaluator for the animal rights wing of the EA movement has already decided to join it and throw out actually having discourse on effectiveness in favour of plundering their reputation for more donations
This seems like an exaggerated and unhelpful thing to say.
Perhaps. It’s certainly what the people suggesting that deliberate dishonesty would be okay are suggesting, and it is what a large amount of online advocacy does, and it is in effect what they did, but they probably didn’t consciously decide to do it. I’m not sure how much credit not having consciously decided is worth, though, because that seems to just reward people for not thinking very hard about what they’re doing, and they did it from a position of authority and (thus) responsibility.
I stand by the use of the word ‘plundering’- it’s surprising how some people are willing to hum and har about maybe it being worth it, when doing it deliberately would be a very short-sighted, destroy-the-future-for-money-now act. It calls for such a strong term. And I stand by the position that it would throw out actually having discourse on effectiveness if people played those sorts of games, withheld information that would be bad for causes they think are good, etc, rather than being scrupulously honest. But again to say they ‘decided’ to do those things is perhaps not entirely right.
I think in an evaluator, which is in a sense a watchdog for other peoples’ claims, these kind of things really are pretty serious- it would be scandalous if e.g. GiveWell were found to have been overexcited about something and ignored issues with it on this level. Their job is to curb enthusiasm, not just be another advocate. So I think taking it seriously is pretty called for. As I mentioned in a comment below, though, maybe part of the problem is that EA people tried to take ACE as a more robust evaluator than it was actually intending to be, and the consequence should be that they shift to regarding it as a source for pointers whose own statements are to be taken with a large grain of salt, the way individual charity statements are.
ACE’s primary output is its charity recommendations, and I would guess that it’s “top charities” page is viewed ~100x more than the leafleting page Sarah links to.
ACE does not give the “top charity” designation to any organization which focuses primarily on leafleting, and e.g. the page for Vegan Outreach explicitly states that VO is not considered a top charity because of its focus on leafleting and the lack of robust research on that:
We have some concerns that Vegan Outreach has relied too heavily on poor sources of evidence to determine the effectiveness of leafleting as compared to other interventions…
Why didn’t Vegan Outreach receive our top recommendation?
Although we are impressed with Vegan Outreach’s recent openness to change and their attempts to measure their effectiveness, we still have reservations about their heavy focus on leafleting programs
You are proposing that ACE says negative things on its most prominent pages about leafleting, but left some text buried in a back page that said good things about leafleting as part of a dastardly plot to increase donations to organizations they don’t even recommend.
This seems unlikely to me, to put it mildly, but more importantly: it’s incredibly important that we assume others are acting in good faith. I disagree with you about this, but I don’t think that you are trying to “throw out actually having discourse on effectiveness”. This, more than any empirical fact about the likelihood of your hypothesis, is why I think your comment is unhelpful.
This definitely isn’t the kind of deliberate where there’s an overarching plot, but it’s not distinguishable from the kind of deliberate where a person sees a thing they should do or a reason to not write what they’re writing and knowingly ignores it, though I’d agree in that I think it’s more likely they flinched away unconsciously.
I don’t think it is good to laud positive evidence but refer to negative evidence only via saying “there is a lack of evidence”, which is what the disclaimers do- in particular there’s no mention of the evidence against there being any effect at all. Nor is it good to refer to studies which are clearly entirely invalid as merely “poor” while still relying on their data. It shouldn’t be “there is good evidence” when there’s evidence for, and “the evidence is still under debate” when there’s evidence against, and there shouldn’t be a “gushing praise upfront, provisos later” approach unless you feel the praise is still justified after the provisos. And “have reservations” is pretty weak. These are not good acts from a supposedly neutral evaluator.
Until the revision in November 2016, the VO page opened with: “Vegan Outreach (VO) engages almost exclusively in a single intervention, leafleting on behalf of farmed animals, which we consider to be among the most effective ways to help animals.”, as an example of this. Even now I don’t think it represents the state of affairs well.
If in trying to resolve the matter of whether it has high expected impact or not, you went to the main review on leafleting (https://animalcharityevaluators.org/research/interventions/leafleting/), you’d find it began with “The existing evidence on the impact of leafleting is among the strongest bodies of evidence bearing on animal advocacy methods.”.
This is a very central Not Technically a Lie (http://lesswrong.com/lw/11y/not_technically_lying/); the example of a not-technically-a-lie in that post being using the phrase “The strongest painkiller I have.” to refer to something with no painkilling properties when you have no painkillers. I feel this isn’t something that should be taken lightly:
“NTL, by contrast, may be too cheap. If I lie about something, I realize that I’m lying and I feel bad that I have to. I may change my behaviour in the future to avoid that. I may realize that it reflects poorly on me as a person. But if I don’t technically lie, well, hey! I’m still an honest, upright person and I can thus justify visciously misleading people because at least I’m not technically dishonest.”
The disclaimer added now helps things, but good judgement should have resulted in an update and correction being transparently issued well before now.
The part which strikes me as most egregious was in the deprioritising of updating a review on what was described in a bunch of places as the most cost effective (and therefore most effective) intervention. I can’t see any reason for that, other than that the update would have been negative.
There may not have been conscious intent behind this- I could assume that this was as a result of poor judgement rather than design- but it did mislead the discourse on effectiveness, that already happened, and not as a result of people doing the best thing given information available to them but as a result of poor decisions given this information. Whether it got more donations or not is unclear- it might have tempted more people into offsetting, but on the other hand each person who did offsetting would have paid less because they wouldn’t have actually offset themselves.
However something like this is handled is also how a bad actor would be handled, because a bad actor would be indistinguishable from this; if we let this by without criticism and reform, then bad actors would also be let by without criticism and reform.
I think when it comes to responding to some pretty severe stuff of this sort, even if you assume the people made them in good faith and just made some rationality failings, more needs to be said than “mistakes were made, we’ll assume you’re doing the best you can to not make them again”. I don’t have a grand theory of how people should react here, but it needs to be more than that.
My inclination is to at the least frankly express how severe I think it is- even if it’s not the nicest thing I could say.
Thanks for the response, it helps me understand where you’re coming from.
I agree that the sentence you cite could be better written (and in general ACE could improve, as could we all). I disagree with this though:
However something like this is handled is also how a bad actor would be handled, because a bad actor would be indistinguishable from this; if we let this by without criticism and reform, then bad actors would also be let by without criticism and reform.
At the object level: ACE is distinguishable from a bad actor, for example due to the fact that their most prominent pages do not recommend charities which focus on leafleting.
At the metalevel: I don’t think we should have a conversational norm of “everyone should be treated as a bad actor until they can prove otherwise”. It would be really awful to be a member of a community with that norm.
All this being said, it seems that ACE is responding in this post now, and it may be better to let them address concerns since they are both more knowledgeable and more articulate than me.
in particular there’s no mention of the evidence against there being any effect at all.
To be clear, it’s inaccurate to describe the studies as showing evidence of no effect. All of the studies are consistent with a range of possible outcomes that include no effect (and even negative effect!) but they’re also consistent with positive effect.
That isn’t to say that there is a positive effect.
But it isn’t to say there’s a negative effect either.
I think it is best to describe this as a “lack of evidence” one way or another.
-
I don’t think it is good to laud positive evidence but refer to negative evidence only via saying “there is a lack of evidence”, which is what the disclaimers do
I don’t think there’s good evidence that anything works in animal rights and if ACE suggests anything anywhere to the contrary I’d like to push against it.
This seems like an exaggerated and unhelpful thing to say.
Perhaps. It’s certainly what the people suggesting that deliberate dishonesty would be okay are suggesting, and it is what a large amount of online advocacy does, and it is in effect what they did, but they probably didn’t consciously decide to do it. I’m not sure how much credit not having consciously decided is worth, though, because that seems to just reward people for not thinking very hard about what they’re doing, and they did it from a position of authority and (thus) responsibility.
I stand by the use of the word ‘plundering’- it’s surprising how some people are willing to hum and har about maybe it being worth it, when doing it deliberately would be a very short-sighted, destroy-the-future-for-money-now act. It calls for such a strong term. And I stand by the position that it would throw out actually having discourse on effectiveness if people played those sorts of games, withheld information that would be bad for causes they think are good, etc, rather than being scrupulously honest. But again to say they ‘decided’ to do those things is perhaps not entirely right.
I think in an evaluator, which is in a sense a watchdog for other peoples’ claims, these kind of things really are pretty serious- it would be scandalous if e.g. GiveWell were found to have been overexcited about something and ignored issues with it on this level. Their job is to curb enthusiasm, not just be another advocate. So I think taking it seriously is pretty called for. As I mentioned in a comment below, though, maybe part of the problem is that EA people tried to take ACE as a more robust evaluator than it was actually intending to be, and the consequence should be that they shift to regarding it as a source for pointers whose own statements are to be taken with a large grain of salt, the way individual charity statements are.
ACE’s primary output is its charity recommendations, and I would guess that it’s “top charities” page is viewed ~100x more than the leafleting page Sarah links to.
ACE does not give the “top charity” designation to any organization which focuses primarily on leafleting, and e.g. the page for Vegan Outreach explicitly states that VO is not considered a top charity because of its focus on leafleting and the lack of robust research on that:
You are proposing that ACE says negative things on its most prominent pages about leafleting, but left some text buried in a back page that said good things about leafleting as part of a dastardly plot to increase donations to organizations they don’t even recommend.
This seems unlikely to me, to put it mildly, but more importantly: it’s incredibly important that we assume others are acting in good faith. I disagree with you about this, but I don’t think that you are trying to “throw out actually having discourse on effectiveness”. This, more than any empirical fact about the likelihood of your hypothesis, is why I think your comment is unhelpful.
This definitely isn’t the kind of deliberate where there’s an overarching plot, but it’s not distinguishable from the kind of deliberate where a person sees a thing they should do or a reason to not write what they’re writing and knowingly ignores it, though I’d agree in that I think it’s more likely they flinched away unconsciously.
It’s worth noting that while Vegan Outreach is not listed as a top charity it is listed as a standout charity, with their page here: https://animalcharityevaluators.org/research/charity-review/vegan-outreach/
I don’t think it is good to laud positive evidence but refer to negative evidence only via saying “there is a lack of evidence”, which is what the disclaimers do- in particular there’s no mention of the evidence against there being any effect at all. Nor is it good to refer to studies which are clearly entirely invalid as merely “poor” while still relying on their data. It shouldn’t be “there is good evidence” when there’s evidence for, and “the evidence is still under debate” when there’s evidence against, and there shouldn’t be a “gushing praise upfront, provisos later” approach unless you feel the praise is still justified after the provisos. And “have reservations” is pretty weak. These are not good acts from a supposedly neutral evaluator.
Until the revision in November 2016, the VO page opened with: “Vegan Outreach (VO) engages almost exclusively in a single intervention, leafleting on behalf of farmed animals, which we consider to be among the most effective ways to help animals.”, as an example of this. Even now I don’t think it represents the state of affairs well.
If in trying to resolve the matter of whether it has high expected impact or not, you went to the main review on leafleting (https://animalcharityevaluators.org/research/interventions/leafleting/), you’d find it began with “The existing evidence on the impact of leafleting is among the strongest bodies of evidence bearing on animal advocacy methods.”.
This is a very central Not Technically a Lie (http://lesswrong.com/lw/11y/not_technically_lying/); the example of a not-technically-a-lie in that post being using the phrase “The strongest painkiller I have.” to refer to something with no painkilling properties when you have no painkillers. I feel this isn’t something that should be taken lightly:
“NTL, by contrast, may be too cheap. If I lie about something, I realize that I’m lying and I feel bad that I have to. I may change my behaviour in the future to avoid that. I may realize that it reflects poorly on me as a person. But if I don’t technically lie, well, hey! I’m still an honest, upright person and I can thus justify visciously misleading people because at least I’m not technically dishonest.”
The disclaimer added now helps things, but good judgement should have resulted in an update and correction being transparently issued well before now.
The part which strikes me as most egregious was in the deprioritising of updating a review on what was described in a bunch of places as the most cost effective (and therefore most effective) intervention. I can’t see any reason for that, other than that the update would have been negative.
There may not have been conscious intent behind this- I could assume that this was as a result of poor judgement rather than design- but it did mislead the discourse on effectiveness, that already happened, and not as a result of people doing the best thing given information available to them but as a result of poor decisions given this information. Whether it got more donations or not is unclear- it might have tempted more people into offsetting, but on the other hand each person who did offsetting would have paid less because they wouldn’t have actually offset themselves.
However something like this is handled is also how a bad actor would be handled, because a bad actor would be indistinguishable from this; if we let this by without criticism and reform, then bad actors would also be let by without criticism and reform.
I think when it comes to responding to some pretty severe stuff of this sort, even if you assume the people made them in good faith and just made some rationality failings, more needs to be said than “mistakes were made, we’ll assume you’re doing the best you can to not make them again”. I don’t have a grand theory of how people should react here, but it needs to be more than that.
My inclination is to at the least frankly express how severe I think it is- even if it’s not the nicest thing I could say.
Thanks for the response, it helps me understand where you’re coming from.
I agree that the sentence you cite could be better written (and in general ACE could improve, as could we all). I disagree with this though:
At the object level: ACE is distinguishable from a bad actor, for example due to the fact that their most prominent pages do not recommend charities which focus on leafleting.
At the metalevel: I don’t think we should have a conversational norm of “everyone should be treated as a bad actor until they can prove otherwise”. It would be really awful to be a member of a community with that norm.
All this being said, it seems that ACE is responding in this post now, and it may be better to let them address concerns since they are both more knowledgeable and more articulate than me.
To be clear, it’s inaccurate to describe the studies as showing evidence of no effect. All of the studies are consistent with a range of possible outcomes that include no effect (and even negative effect!) but they’re also consistent with positive effect.
That isn’t to say that there is a positive effect.
But it isn’t to say there’s a negative effect either.
I think it is best to describe this as a “lack of evidence” one way or another.
-
I don’t think there’s good evidence that anything works in animal rights and if ACE suggests anything anywhere to the contrary I’d like to push against it.