Criticism on the EA Forum
I’m writing this on behalf of the mod team. They’ve reviewed and commented on this post, but mistakes are mine.
We want and value criticism on the EA Forum. EA organisations often make their decisions transparent to the Forum audience in a way which makes good criticism possible. But transparency has its challenges, one of which is that not all criticism is good.
By ‘good’ criticism, I mean criticism which is valuable to spend time engaging with. Criticism which can improve your project, or improve the case you make for your project. I also mean criticism which is good for the world but not for the criticised person or organisation. For example, perhaps an approach to solving a problem, or a the work of a particular organisation, is getting too much funding or attention relevant to its merits. Then, the recipient of the criticism may protest — but the criticism is good.
Bad criticism, by contrast, is criticism which is time-consuming to respond to and complicatedly wrong. Criticism which might still tarnish a brand, but which doesn’t give readers a more accurate picture of the world, or help the target of the criticism to improve.
Part of running an epistemically healthy discussion space is accepting a certain amount of bad criticism along with the good. The ideal wouldn’t be a space with no bad criticism — interventions that ensure no bad criticism would doubtless also reduce the amount of good criticism.
This is partially why the moderation team does not hide criticism that we think is bad. However, we encourage all Forum users to share their opinions on which criticisms are valuable by voting and commenting. As users, moderators can also do these things.
This doesn’t mean that criticism is never moderated. Some of the accounts, comments or posts that are hidden or lead to banning are also critical. However, they are hidden or banned because they break strong forum norms, generally involving hostility, rudeness, or off-topic content, rather than because they are critical.
Practices we’d like to encourage
This doesn’t mean the moderation team takes no responsibility for bad criticism on the Forum. We can promote and assist the development of practices around criticism, without resorting to drastic action like banning. Some practices we’d like to promote are:
Reach out to people before posting criticism of their work (in almost all cases).
Lizka previously wrote a post about why, how and when to share a critique with the subject of your criticism. I highly recommend reading that post — she also includes a helpful guide with template emails for critics.
The most compelling reason to do this is to ensure that you, as a critic, are interpreting the subject of your criticism correctly. It isn’t in anyone’s interest to make complicated, false criticism—it’s bad for the person or organisation being criticised and bad for the critic.
The best reason not to involve the criticised person or org is if doing so would in practice stop you from posting your criticism. You can always ask the mod team for support if this is the case. Simply email forum-moderation@effectivealtruism.org with a link to your post, and we’ll coordinate with the subject of the criticism to get a response.
NB- This practice isn’t usually necessary for critical comments. Reaching out for a reply to a comment before posting would mean it is posted far too late to be seen by readers of the post. In this post, I’m thinking about critical posts.
Where possible, give an organisation or individual sufficient notice that they can respond along with your post, i.e. something like a journalistic “right of reply”.
There is a practice (and in Brazil, a constitutional right) in journalism known as the ‘right of reply’. It’s the idea that the person being critiqued should have the opportunity to respond at the same time the critique is published. This can be particularly important because, as rational as we might aspire to be, seeing criticism can update our opinion without causing us to look out for later rebuttals.
This is important on the Forum, where you can easily read a post critiquing something, and then never see a comment under that post rebutting the criticism (unless you have subscribed to the post or commented on a thread yourself).
In a Forum post, this could look like:
Sending the post to the criticised person or organisation to be commented on, and then sharing those comments as footnotes.
Letting the criticised person or org pre-write their comment, and know when you plan to post, so that they can post their comment along with the post.
I recognise that these practices may be time-consuming for the critic to actualise, and I don’t want the result to be substantially less good criticism.
We aim to alleviate the burden of this process for both critics and those being critiqued, wherever possible. If we can assist in coordinating this, please feel free to reach out to forum-moderation@effectivealtruism.org. We could help by (for example):
Coordinating with the subject of the criticism on your behalf.
Collecting comments from the organisation on a google doc of your post, and then publishing the critique, with comments as a dialogue (via your account).
To emphasise, we will not take strong moderation action (banning users or moving posts to drafts, for example) if a user doesn’t break strong forum norms, and this post establishes no new strong norms. This post is a statement of our hopes for good criticism on the Forum and a statement of intention to be more involved and proactive in stewarding it.
Note that I changed some uses of the word “norm” to “practice” to make clearer that in the “practices we’d like to encourage” section of the post, I’m not setting up new Forum norms.
Let me know how you feel about this in the comments below, or on Jason’s polls here.
I am somewhat against a norm of reaching out to people before criticizing their work. Dynomight’s Arguing Without Warning has IMO the strongest arguments on this topic.
First: Criticism is difficult; requiring more effort on the part of critics makes criticism less likely to happen. OP did acknowledge this with
But realistically, there is a social penalty to saying “I would not have posted this criticism if I’d been required to reach out to orgs first.” It makes you look lazy. A norm of “reach out to people, unless it would stop you from posting your criticism” is not a viable norm because critics who would’ve been stopped from posting criticism are unable to defend themselves. In some cases, I expects this norm to stop criticisms from getting written. So I think a better norm is “you don’t have to reach out to people”.
Second: A big reason to reach out to people is to resolve misunderstandings. But it’s even better to resolve misunderstandings in public, after publishing the criticism. Readers may have the same misunderstandings, and writing a public back-and-forth is better for readers.
(Dynomight’s post also gives a couple other arguments that I don’t think are quite as important.)
My preferred norms are:
Critics don’t have to do anything.
If they want to be nice, critics can reach out to the subject of criticism to inform them of the criticism after it has already been published.
That’s what I did for my recent critical review of one of Social Change Lab’s reports.
There are some circumstances where it makes sense to reach out to people before publishing, but I don’t think that should be the norm, and I don’t think we should have any expectation that critics do it.
For some reason I find this title delightful. I kind of wish I could have an “argues without warning” flair or something.
I agree with arguments you present above and your conclusion about preferred norms. That said, I think people might have in mind certain types of cases that might justify the need for reaching out beyond the case of general “criticism”. For example, imagine something like this:
Now, my view is, even if this is what happens, this is still a positive outcome, because, like you say:
Transparency has costs, but I think they are usually internal costs to the org, while transparency also has external benefits, and thus would be expected to be systematically under-supplied by orgs.
At the same time, I think most cases of criticism are realistically more mixed, with the critic making reasonable points but also some mistakes, and the org having some obvious corrections to the criticism but also some places where the back-and-forth is very enlightening. Requiring people to reach out I think risks losing a lot of the value that comes from such “debatable” cases for the reasons you mention.
Another set of cases that is worth separating out are allegations of intentional misconduct. I think there are particular reasons why it might make sense to have a higher expectation for critics to reach out to an org if they are accusing that org of intentional misconduct. I think this may also vary by whether the critic personally observed misconduct, in which case I think issues like a risk of retaliation or extreme difficulty for the critic may weigh in favor of not expecting the critic to reach out.
One of the challenges here is defining what “criticism” is for purposes of the proposed expectation. Although the definition can be somewhat murky at the margin, I think the intent here is to address posts that are more fairly characterized as critical of people or organizations, not those that merely disagree with intellectual work product like an academic article or report.
For what it’s worth, I think your review was solidly on the “not a criticism of a person or organization” side of the ledger.
That’s consistent with reaching out, I think. My recollection is that people who advocate for the practice have generally affirmed that advance notification is sufficient; the critic need not agree to engage in any pre-publication discourse.
I think this is much messier than suggested here. Consider a situation where charity evaluator org A performs and publicly publishes a cost-effective analysis of org B. A critic publishes a re-analysis that suggests the cost-effectiveness is much lower than the original analysis, perhaps far below the expected funding bar. Org A may feel the criticism goes to its competence as a charity evaluator, and org B may consider this an existential threat that could result in loss of funding, yet I think a public cost effectiveness analysis simply has to be considered “intellectual work product”.
I don’t think this is hypothetical. A while ago there was a case where a critic posted some criticism of a cost-effectiveness analysis of a mental health charity, I believe the charity being evaluated was called “StrongMinds”. I think that case is similar to what I describe above.
This presents a similar problem to the laziness allegations mentioned above except worse, since the critic may face allegations that failure to change their criticism pre-publication demonstrates how the critic is insufficiently “truthseeking”.
In fact, to the extent that their is no community accepted safe-harbor for what is expected, I think there is likely to be a death-by-a-thousand-cuts problem. Critics can predictably expect that they will need to litigate their conduct regarding these meta-issues when they publish (even if they actually do a lot of the what is suggested!), likely in a way that moves discussion away from the content of their object-level criticism.
Again, not hypothetical. The Nonlinear situation goes to this, I think.
FWIW I do not expect people to run cost-effectiveness analyses by orgs before publishing them.
“The best reason not to involve the criticised person or org is if doing so would in practice stop you from posting your criticism. You can always ask the mod team for support if this is the case. Simply email forum...”
One of the biggest reasons I’ve considered, is just time delay. Posts on the forum disappear from the front page quickly, and memory fades quckly as well. For example right now I’m about to post a response to RP’s great analysis on Health Systems Strengthening interventions. Only part of is is criticism and not really very heavy. I probably won’t send it to them because I think I need to give 3 days really for a fair response, and its already over a week after the initial post so I’m already late.
If most of it was criticism, I would send it to them first
This is a tricky one with no clear solutionI don’t think
Yep, I think that’s fair, and that’s part of why these are general practices that we’ll be trying to promote/ helping happen rather than strong norms we would moderate. Getting a response isn’t always a reasonable ask, but if I can, I’d like to lower friction for Forum authors.
I’m not familiar with the context, but my comment might address this sort of situation.
I’m guessing this is probably a response to the post that unfairly accused a charity of fraud? (The post I’m thinking of currently has −60 karma, 0 agree votes, 6 disagree votes, and 4 top-level comments that are all critical.)
Some criticism might be friendly and constructive enough that giving the organization a chance to write a reply before publishing is not that important. Or if the organization is large, powerful, and has lots of money, like Open Philanthropy, and especially if your criticisms are of a more general or a more philosophical kind, it might not be important to send them a copy before you publish. This depends partly on how influential you are in EA and on how harsh your criticisms are.
Definitely accusing a small charity of fraud is something you should run by the charity beforehand. In that case, though, the charity was already so frustrated with the critic’s poor-quality criticism that they had publicly stated (before the fraud accusation) they didn’t want to engage with it anymore.
Hi Yarrow, thank you for your comment.
We posted an article about Sinergia on February 20th indicating that their 354 piglets per dollar claim was wrong.
On March 21st, Sinergia commented acknowledging that their 354 piglets per dollar claim was wrong. Their comment also included the follow advertisement from their main donation page.
(image from Sinergia’s March 21st comment)
On April 9th (8 days before publication), we sent Sinergia a follow-up article expressing our concern that they were misleading donors with their advertisements.
On April 21st, Sinergia still had not stopped advertising the 354 piglets per dollar claim on their main donation page.
We then made a post stating Sinergia was committing fraud. Less than 24 hours after we made this post, Sinergia took down the 354 piglets per dollar claim.
Our priority is protecting donors, not upvotes.
I think you behaved inappropriately, as I and others explained in the comments on that post about the dubious “fraud” accusation. I completely understand why Sinergia said they don’t want to engage with your criticism anymore.
Upvotes/downvotes are not a meaningless number in this context, but a sign of EA Forum users’ opinion on whether you behaved appropriately and whether your claim that Sinergia committed fraud was true or misleading. You can see this in the comments on that post as well. It seems like there is, so far, unanimous agreement that you behaved inappropriately and that your claim was misleading or false.
I’m not sure if Vetted Causes is a salvageable project at this point. Its reputation is badly damaged. It might be best to put the project to an end and move on to something else.
Speaking for myself, I will never trust any evaluation that Vetted Causes ever publishes about any charity, and I would feel an obligation to warn people in this community that Vetted Causes is an untrustworthy and unreliable source for charity evaluations.
Thanks for your opinion.
I’m already on record in this comment thread that I don’t agree with the norms laid out above regarding reaching out to orgs and a right to reply. At the same time, I’m extremely worried you all at VettedCauses will take the focus on those issues and assume that you have some amazing criticism that the community is ignoring because they can’t look past the fact you didn’t color within the lines on those issues. Although I haven’t followed whatever is going on with you all, I very seriously doubt that is the case. It seems extremely likely to me that people have given valid push-back to your criticism, and you yourselves are now ignoring that push-back. You can’t just criticize others but ignore criticism directed at yourselves!
Yarrow points out that this could be an existential issue for your organization, I think you all need to really seriously take that type of criticism onboard and think long and hard about it. I hope the fact that I disagree about the things like right to reply but still share these concerns helps get it through to you all that the criticism you are receiving is not solely because you failed on those counts. I don’t give a single fuck about whether you gave someone a right to reply or whatever, but I still think what you all have been doing has a very high chance of being inappropriate and your criticisms probably have many serious issues on their merits.
Editing to add: Although I agree with what I said above, I also thought about this whole situation a bit, and I’m sure this situation feels terrible for you all, and I’m sorry for that. I think you all probably made some mistakes, but so have all of us. You all aren’t bad people or bad EAs or anything like that. Giving criticism is hard, and its unfortunately an area where mistakes that wouldn’t even get noticed in other places can earn you a lot of negative attention. Although I think there is important stuff for you all to learn from here, I hope you don’t take this experience as an indictment of you all as people.
It seems to be just the one overconfident young guy
I strongly disagree with the idea that there is a general obligation to reach out to someone before you publicly criticize them, and I’ve been considering writing a post explaining my case. I’d like to ask some questions to better understand the positions that people on the forum/EA community hold on this topic.
You talk about practices you’d like to “encourage” but later speak of “these norms”, which I take to mean the obligation to reach out and to offer a “right of reply”. There are some things that it is good to do, but where one does not violate a norm when failing to do that thing. If someone makes a post that criticizes someone on the forum but does not reach out to the target of their criticism first, would you consider that to be violating a norm of the forum, even if that violation won’t result in any enforcement?
Some posts that express similar views focus on criticism directed at organizations (e.g. “run posts by orgs”). Does the entity at which criticism is directed impact what a critic is expected to do? For example, it would surprise me if I was expected to reach out to OpenAI, the DOJ, or Amazon prior to making a post criticizing one of those entities on the forum. Similarly, people sometimes make posts that respond to criticism of EA or EA institutions that is published in other venues. Those responses are sometimes critical of the authors of the original criticism. I would also be surprised if the expectation was that such posts offer a right of reply to the original critics.
Appendix 3 of this post mentions this:
What is in scope for “criticism” in this context? People may reasonably disagree on whether a particular piece of critical writing is more about public arguments/evidence (and thus is like disagreement with someone’s arguments) or not. This also seems to suggest that if an org does something and publishes some reasons for doing it the critic might not need to reach out to them (but its unclear to me what the standard is), while if they simply state they are doing something and don’t state any reasons a critic would have to reach out.
The other appendices mention cases when the target of criticism is not expected to act in good faith, and the “run posts by orgs” post mentions a similar exception to the expectation when the person/org being criticized may behave badly when a critic reaches out. I think its not uncommon that critics and their targets have major disagreements about whether these types of beliefs are reasonable. When can one invoke this type of reasoning for not reaching out?
This post is about criticism of EA organizations, so it doesn’t apply to OpenAI or the U.S. government.
I interpreted this post as mostly being about charities with a small number of employees and relatively small budgets that either actively associate themselves with EA or that fall into a cause area EA generally supports, such as animal welfare or global poverty.
For example, if you wanted to criticize 80,000 Hours, New Harvest, or one of these charities focusing on mental health in poor countries, then I’d say you should send them a copy of your criticism before publishing and give them a chance to prepare a reply before you post. These organizations are fairly small in terms of their staff, have relatively little funding, and aren’t very well-known. So, I think it’s fair to give them more of an opportunity to defend their work.
If you wanted to criticize Good Ventures, Open Philanthropy, GiveWell, GiveDirectly, or the Against Malaria Foundation, then I think you could send them a courtesy email if you wanted, but they have so much funding and — in the case of Open Philanthropy at least — a large staff. They’re also already the subject of media attention and public discourse. With one of the smaller charities, you could plausibly hurt them with your post, so I think more caution is warranted. With these larger charities with more resources that are already getting debated and criticized a lot, an EA Forum post has a much lower chance of doing accidental harm.
I take this to be the case as well, but I think it would be worth making this explicit.
I think this is a fairly reasonable heuristic, I myself personally think the concept of punching up vs punching down is helpful in terms of calibrating criticism, but I don’t think this means there should be a norm that one must reach out to orgs before criticizing or that a right of reply is required. I think we can judge criticisms on their reasonableness, and the individual critic should be responsible for navigating these factors and others and deciding when these things (reaching out, allowing a reply) would be appropriate and make sense.
Most commentary I have read on the EA forum about this includes what is essentially a bad faith exception. That if you are worried about the org your are criticizing acting in bad faith, retaliating in some way, etc. that you don’t need to do these things. I think that probably applies to small orgs just as much as large orgs. This seems to suggest there is no general requirement to do these things for small orgs, just maybe you should have a lower bar in your reasonableness calculation for small orgs vs large ones.
Part of my interest here is in understanding what the actual norm is that people intend to apply. If the norm is that large orgs aren’t included, I think it would be worth having that stated explicitly. I’m somewhat doubtful that is what is intended in the OP, but if so it would be good to know.
I think that’s half complete. No one is having their posts deleted for not reaching out, so the choice is ultimately up to the critic. But the community also has a role to play here. If community members believe the critic failed to provide appropriate advance notice, and has not demonstrated sufficient cause for doing so, they can elect to:
Downvote the criticism, and/or
Decline to engage with the criticism, at least until the organization has had a reasonable amount of time to reply (even though they may not remember to come back to it later).
I agree people should downvote criticism based on whether the person reached out based on their own judgements. I might have a different assessment of any given case compared to the typical EA forum voter, but people should be allowed to vote based on their own views.
I also agree that an organization has no obligation to respond to any given criticism, even if the critic did reach out in advance.
I would distinguish between a few things:
What the moderation team considers norm-violating
What the community considers norm-violating
What the community considers ideal vs sub-optimal (but norm-compliant) criticism
I think you can downvote something that is sub-optimal but not norm-violating, although I think its debatable exactly what the balance should be, so I can see an argument that 2 and 3 kind of bleed together.
On the other hand, I think its pretty fair to want to distinguish 1 from 2⁄3, and that it is reasonable to expect a reasonable degree of clarity on 1. I think its reasonable to want to understand what the moderators consider a norm even if they won’t remove posts for violating that norm. I understand moderators can’t give 100% exact standards because then people would abuse that by tip-toeing up to the line, but I believe my questions above go to pretty fundamental aspects of the issue, they aren’t just random nitpicks.
I would also like to understand to what degree the norm in question respects some version of viewpoint neutrality. The OP to me seems to portray the ask as essentially viewpoint neutral (with-in the category of “criticism” anyway). I’m not so sure this would be the case if we really ran down the answers to my questions above. I have no problem with people up and downvoting based on non-viewpoint neutral considerations (it would be kind of crazy to do otherwise). I think moderation being highly dependent on viewpoint could be more of an issue.
Thanks for this comment! I think you’ve pointed out a few places where this post clearly isn’t comprehensive. I’m not sure how frequently asked these questions will be, but in case they are, some quickfire answers:
No—I mistakenly used the word norms in an ambiguous sentence in the second section. I’ve changed the word to practices. Reaching out to a critiqued organisation or person, or giving right of reply are ‘practices we’d like to encourage’ rather than new norms. In practice this means that we (the mods) will advise people to follow these practices in many cases, and in many cases, will help reduce friction (by doing the reaching out on the critics behalf for example).
This is a good question. I could cop out with a “I know it when I see it” which is partially true. But broadly I think the type of criticism we are more concerned about/ would more strongly encourage to follow these practices is criticism which could damage the reputation of an organisation or individual if was read without a response. General disagreement/ critical engagement with the ideas of an organisation could technically fall into this category, but is generally read as more collaborative than as an accusation of wrongdoing. Tone probably matters a bit here. Others on the mod team may have different views on this question.
When it’s reasonable to do so. I think the Forum is naturally quite sceptical and won’t let bad faith arguments stand for long, so in many cases, I don’t think it will matter if a bad faith response is published alongside a critique. But it’s a little hard to form a principle here (hence practices, not norms).
NOTE: I will abbreviate (“reaching out” + “right to reply” as R+R)
Appreciate the clarification. Do you have any advice for people like myself who have a very different perspective on the value of what you recommend (i.e. R+R)? The way you have described it, I would normal consider the decision of what to do to be within my discretion as a poster. As an analogy, I try to write well reasoned arguments, but I understand that not too infrequently I will probably fail to do so. I might write something and think that I could refine the arguments if I took more time but decide what I have is good enough. But R+R seems much more binary than “make well reasoned arguments”. Its hard for me to shake the feeling that it would be perceived as doing something distinctly “wrong” to fail to do so in certain cases.
This seems like it could get awfully messy. I think strong disagreements tend to coincide with different views on the nature of the criticism and how accusatory it is, what appropriate tone is etc. It seems like the exact cases where some guidance is most needed are when people will heavily contest these types of issues.
Related to that, one of my concerns is focusing too much on R+R may predictably lead to what I consider unproductive discussions. I think back-and-forth among people who disagree has great value. I worry focusing on R+R has a “going meta” problem. People will argue about whether the degree of R+R done by the critic was justified instead of focusing on the object level merits of the criticism. The R+R debate can become a proxy war for people who’s main crux is really the merits.
I also worry that expectations around R+R won’t necessarily be applied consistently. I worry that R+R is in a sense a “regressive tax” on criticism, that R+R may in practice advantage orgs with more influence over orgs/people with less influence. I also worry that there may a “right targets” dynamic, where people criticizing “the right targets” may not be subject to the same expectations as people targeting well-liked EA orgs. This is why some of my questions above relate to “who” R+R applies to.
I agree with this, but the logic to me suggests that R+R might not really be needed? The OP raises the concern that orgs will have to scramble to answer criticism, but if they think people on the Forum will find the criticism might be warranted, doesn’t that indicate that it would in fact be valuable for the org to respond? I personally think this overall would produce a better result without R+R, because orgs could in some (perhaps many) cases rely on commenters to do the work, and only feel the need to respond when a response provides information that the community doesn’t have but would find useful. The fact that they feel the need to respond is a signal that there is information missing that they may uniquely have access to. Are you saying you think the Forum can identify bad faith but can’t identify criticism quality as accurately?
I agree, but I think similar reasoning applies to the initial criticism. Its obviously not good to have bad criticism, but its not the end of the world, and I think its often reasonably likely that the Forum will respond appropriately. I think to the extend possible it would be good to have symmetry where there aren’t specific things required because a post is “criticism”.
Will respond to other points in a sec, but on the “encourage” and “norms” distinction—I’ll go and edit the post to make sure I’m consistent here. The aim was to separate strong forum norms from these practices that I’d like to promote and make easier around criticism. I don’t think it would be reasonable to make right of reply a norm in the way not being rude on the Forum is a norm.
This would benefit from one of those polls, I think. Unfortunately, I don’t think they are available in comments. E.g.,
Giving advance notice of critical Forum posts to EA organizations should be:
seen as optional in almost all cases
done in almost all cases
(with at least one footnote to define “critical”)
Based on prior discussions, my guess is that the median voter would vote about 70% toward done in almost all cases . . . so this would be evidence for a community-supported norm, albeit one that is more flexible than Toby advocates for here.
Good idea! If you make one I’d link it in the post.
Done! The wording was trickier than I expected, but I decided it was better to post than not.
Fantastic! I’ve linked it at the bottom of the post.