New EA Forum LLM-use policy
This policy does not apply to anything posted before this post’s time of publication.
New policy:
You are welcome to use AI to help you write posts, but we ask that you disclose it when you do. Not disclosing that your post is AI-assisted could mean a rate-limit or a ban.[1] We won’t enforce this policy for comments and quick takes, though we’d appreciate a norm of disclosure there as well.
We are (and have been) moving more low quality or off-topic writing off of the Frontpage. Fully AI written text is (at time of writing) overwhelmingly likely to be one of these.
More detail:
Disclosure: If it is likely that there is substantial portions of AI-generated text[2] in your post at time of publishing, you must note this at the top of your post. You are not required to note if you used AI for research/ ideation. If you read a post which seems AI-generated, and you don’t see a disclosure — please report it.
To help get you started, we added a button to our post editor that provides some example disclosure statements. Like so:
If you’re unsure if your case applies, feel free to ask the Forum team before publishing.
Removing content from the Frontpage.
When a new user writes their first comment or post, it is reviewed by the mods. At this point we decide whether to allow the post on the Frontpage (the majority of non-spam posts go here), to put it on personal blog (a minority go here), or to ban the user (this is mostly used for clear spam).
Over the past year, we’ve also been moving low-quality and off-topic posts to personal blog more often, so that Forum users don’t have to spend time reading them. LLM-generated posts are more likely than human-written ones to be moved to personal blog, due to their lower quality.
Reasoning:
This policy represents a decision not to go for an authoritarian or a laissez-faire option to LLM-generated text.
The authoritarian option would be to ban LLM-generated text. To find a good text detector and to remove from the Frontpage all posts which contain generated text.
A laissez-faire option would be to allow LLM-generated text and hope that the forces of karma, upvotes, and downvotes would take care of the low-quality content that would likely result.
Both options are flawed. Taking the authoritarian route, and banning all LLM text is flawed because some writers find LLMs very helpful for getting their ideas across, and some readers don’t mind reading LLM-generated text[3].
The laissez-faire option is flawed because LLM-generated writing is increasingly difficult to detect. There are posts (I’ve seen a lot of these) which have the form of a good quality post which is worth reading, but on closer analysis turn out not to contain any ideas, or just to contain a couple of bullet points’ worth of ideas, surrounded by a lot of fluff and repetition. This leads to quite a large waste of time for the reader.
We’re opting for what I’d call the ‘liberal’ option. We’ll discourage LLM generated content by lowering the visibility of low quality posts and enforcing disclosure of LLM use, but we’ll ultimately be leaving the decision of whether to read LLM-generated content down to the Forum audience[4].
LessWrong is doing something reasonably similar, but with some caveats. Under their new policy, AI-written text must be published within labelled sections in a post. Our policy is therefore a chunk less onerous. I see our policies a little like this[5]:
We’re trying to have the best of both worlds, and I hope that we can. However, if it turns out that increasing amounts of content on the Forum is low-effort AI slop, or if valued authors find the Forum increasingly less valuable because of AI generated content, we are prepared to change our policy.
Good and bad uses of LLMs
Note that this section goes slightly beyond our policy, and into what we’d like to promote/ discourage. Treat these as strong recommendations, rather than laws.
Examples of recommended use of LLMs
A user uses an LLM to track down statistics on laws about gestation crates by country. They check the sources provided by the LLM, conclude that the statistics are accurate, and reference them in their post.
A user sends a draft of a forum post to an LLM asking it for feedback. They make edits to their post in response to its feedback.
A user who speaks English non-natively sends a draft of a forum post to an LLM asking it to correct any grammatical issues, and corrects the grammatical issues it raised.
[requires disclosure] Alternatively, they allow the LLM to redraft their post, and include a note at the top of their post explaining that they have done so.
A user creates a post discussing evaluation awareness in LLMs, in which they include several quotes from LLMs that appear to indicate evaluation awareness.
[requires disclosure] A user has an idea for a forum post, then co-writes it with an LLM, turning a verbal mind-dump into bullet points, into an essay, into bullets again, etc… It turns out good.
Examples of discouraged use of LLMs
A user wants to grow their reputation on the forum, so they feed popular forum posts into an LLM and ask it to write a thoughtful reply. This would lead to a ban under our existing rules against spam.
[requires disclosure] A user sends a list of bullet points to an LLM, asks the LLM to write a post based on their outline, and posts the content to the EA forum without making any edits. The post is bad. This post would be moved to personal blog.
A user uses an LLM to track down statistics on laws about gestation crates by country, but makes no effort to verify whether the information provided by the LLM is accurate.
As always, if you have any questions about our forum norms, feel free to contact the moderation team. Also, you are very welcome to share feedback on this policy below. I’m open to changing the policy if you change my mind.
PS—Thanks to the entire moderation and facilitation team for multiple rounds of feedback and discussion about this policy, and especially to @Francis for writing the first draft.
- ^
If we suspect that your post is AI-assisted and you did not include a disclosure at the top, we may hide it from readers, for example by moving it back to your drafts. You’re welcome to re-publish if you add a disclosure, or contact us if you think we made a mistake.
- ^
Specifically, if more than 10% of your post is the output of a chatbot.
- ^
I was personally surprised by how many people had this view on Nick Laing’s poll. Though a fellow mod points out that the poll was interpreted by some as a hypothetical ‘if AI could write as well as humans…’ and others were thinking of current models.
- ^
We are considering adding an option so that users can filter out AI-assisted content if they prefer. Currently we are testing pangram for accuracy.
- ^
NB- I edited this section after a message from Habryka. He pointed out that our policies were a little closer than I’d thought.
Disclosure is a reasonable idea, but mandating it at the top is awful, because the first line of a essay generally should be a hook, or convey the most information about the essay (after the title, anyways; especially because EA Forum doesn’t have a subtitle the way eg Substack does).
I would recommend allowing the author to put the disclosure anywhere in their essay. After the intro section might be a more natural place, or at the bottom similar to acknowledgements.
[requires disclosure] A user has an idea for a forum post, then co-writes it with an LLM, turning a verbal mind-dump into bullet points, into an essay, into bullets again, etc… It turns out good.
I think this category, back and forth idea and drafting iteration between human and AI(s), has an enormous amount of value. The chatbots are very good at both generalizing from pithy insights and organizing material pretty effectively. Discussion and feedback over several rounds, at least for me, can produce content that well conveys my ideas much more quickly than if I were to just do it myself.
I think it is unfortunate that most of the discussion seems to be about demonizing the use of all AI for writing-generation, rather than distinguishing the good for the bad, and encouraging its positive use to enable contributions that otherwise just would not have happened.
I’m yet to see a brilliant article which I think is written in this manner. (I might have missed it), and I think we should be aiming high here on the forum. I also think that stylistically, if everyone starts co-writing with AI, posts will become boring and start sounding samey. I agree this method could enable contributions that otherwise wouldn’t have happened, but I’m happy to sacrifice OK/Good contributions to maintain human voice, and what I perceive at least to be higher overall quality of writing.
That’s actually a fantastic challenge: What post that gets above e.g. 200 karma has had the most AI usage? I mean if AI-heavy workflows can generate excellent content, that’s a win, no? More along the line of AI for epistemics things some EAs are working on, and might even help if AIs can help with sharing information on AI safety—at some point AI safety might need to become mostly automated to keep up with AI progress. In addition to the limit, I would be super excited to have some competitions on using AI heavily to write the best content, do the best research, and share lessons on how to use it to do even more good.
I think that’s a good starting point and I’d encourage people to share if there are good examples. It’s not necessarily a win though even if they can generate high quality argument. I don’t like the sameyness of LLM writing, I much prefer huam thought from the brainsto the page.
I’ve got no problem with AI safety work becoming automated if need be, it’s just the writing for human consumption that I don’t like right now. I suspect I’ll change my mind at some point when LLM writing becomes genuinely indistinguishable from good human writing, because then what can actually be done about it even if there is an objection?
I think something like the EA forum has a decent chance of LLM writing which is OK with people because here substance rules over style, although style is still valued.
Hmm, I’ve used LLMs to varying degrees in writing articles. Usually not to the point of writing significant amounts of text, but a case where I think it clearly helped to improve the output is this story: https://strangecities.substack.com/p/some-days-soon
Functionally, I wrote a complete draft, then got Claude to redraft, then I went through and stitched the best bits of the two drafts together (or wrote new versions where that seemed best). (If you thought the original draft was better I’d be interested to hear that: https://docs.google.com/document/d/1icY2wpcgvKszfzHFButKcOwV8B9xMypTAk48kjnOGz0/edit?usp=drivesdk )
(I notice that I’m more likely to find LLMs helpful in drafting things when writing fiction. I think it’s least likely to help when it’s important to convey my precise epistemic status towards the things I’m saying.)
“Usually not to the point of writing significant amounts of text”
This is the key point for me. Using AI for research, brainstorming and even to some degree structure makes perfect sense to me. It’s the actual final writing itself that’s the topic of this post and conversation.
‘higher overall qualirt of writing’ lol Nick was that on purpose.
nope lol. That was just bad spelling to show lack of LLMs.
I genuinely think comments should have the lowest bar of writing imaginable lol. Just get your thoughts out there and move the discussion somewhere!
I agree a lot of the time but there are also some absolute banger comments on the Forum that I’m glad people sweated over. And there is a difference between thinking seriously and being loose about form and just being loose all around.
100% comments can be any level—the best ones of course are far better than most posts. I’ve sweated over a handul myself ;).
Hang on, the category/example you cite is listed in the ‘Recommended use of LLMs’ section. So, I’m not sure what you’re disagreeing with?
Indeed, almost half the post is about distinguishing good from bad uses of LLMs, thus I’m struggling to make sense of your last paragraph. Are you referring to discussion (which demonizes all AI use for writing) that has happened elsewhere?
Requiring disclosures to be at the top of the post (rather than e.g. allowing them to be at the bottom) does feel like it’s sending some implicit “this is kind of bad so people need to be warned about it” message, even if it’s in a “recommended uses” section.
Like I think people might reasonably worry about others pre-judging posts with this disclaimer, and hence (perhaps, sometimes) prefer workflows where they don’t need to include the disclaimer, even if this makes their posts worse.
I don’t think there’s an easy answer here—like, presumably the point of the policy is to allow this kind of pre-judging and let people make differently-informed choices about what they engage with. But I think the post kind of papers over this tension.
I can clarify that in writing this policy that was definitely part of my reasoning (i.e. to make it slightly costlier to use AI for final drafting).
I do think that “even if this makes their posts worse” is going to be fairly rare.
Though, as AI gets better at writing, we might all come to look on disclaimers differently. At some point readers may even prefer to know an AI has already checked over a post before they bother to read it.
I think there are very few justifications for consumers not knowing what they are buying. We should know as much as possible. When we eat food all the ingredients should be there on the packet. If some people think AI written posts are likely to be better, than people might even be more likely to read them? We should have the right to read or not read heavily AI written posts.
Labelling from my perspective is not about it being “good” or “bad” persay, but helping people make informed decisions.
Ok so I can kind of tune into what you’re saying here, but I also feel kind of uneasy about it. I guess I’d be curious what you make of the following potential arguments:
Ingredients are important because we can’t directly discern what’s in food. But with writing we can see exactly what’s there and judge that directly without needing to judge the process. (This perspective would endorse reviews being posted warning people not to read low-quality stuff.)
Requiring disclosure is an inappropriate form of thought policing—people should have the right to use whatever cognitive processes and augmentation methods they like, and take responsibility for the words they then share. If this produces LLM garbage it’s not on them to label that up front, but this should have the natural consequence that people stop listening to them.
I’m not disagreeing with this post (or, in any event, not in the comment to which you replied). I am noting that most of the discussion that I have seen has been pretty against AI-generated writing writ large, conflating the good use of it with the bad use. I am noting my opinion that there is a lot of value in this usage. When I am saying “most of the discussion”, I am not talking about this post specifically, but the broader discussion there has been about the use of AI to generate writings.
LLM disclosure in general is just a good idea to do. The internet is absolutely flooded with LLM-written spam at the moment, so if people detect LLM writing with no context it’s natural to assume your post is spam as well. This is a shame when someone who is a non-native speaker has just used it for translation or whatnot.
Personally I’d recommend against using LLM-written text if you can help it, as in the age of spam the value of cultivating your own stylistic voice is increasing.
I would add that whilst I understand non-native speakers wanting to use LLMs to write more idiomatic English, I would rather read their own thoughts with a few grammar errors and unusual word choices than their own thoughts mixed up with vaguely similar ideas that aren’t theirs and pithy summaries that are actually expressing something different...
Seems like a good compromise. The examples at the end are also helpful.
About this, however:
While this is true, and indeed happens a lot everywhere nowadays, let’s not forget about the option for actual malice—manipulation by posts that look good or convincing but are actually written to persuade you to serve someone’s interests. Which can be done by anyone ranging from individuals, to companies, to industry lobbies to state governments.
Allowing LLM-generated content not only leaves the door open to heaps of slop, but also allows all of this. So some sort of defence is definitely warranted.
I assume this is going forward and there’s no requirement to backlabel?
Yep, should have made that clearer. I’ll edit.
Thanks this is helpful. I weakly disagree with this approach and would prefer something more like the LessWrong policy, but your reasoning makes sense to me. We don’t seem to have a big problem with AI slop at the moment, and that might be because of the good job that the mods are doing in cleaning it up before it sullies our brains. The Karma system is protective as well.
”However, if it turns out that increasing amounts of content on the Forum is low-effort AI slop, or if valued authors find the Forum increasingly less valuable because of AI generated content, we are prepared to change our policy.”
That’s great to hear! Linkedin was a decent platform 2 years ago and now its almost not worth being on due to AI generated trash. Posts sound mechanical, samey and you almost feel the lack of humanity there. There’s nothing written terrible, and nothing brilliant there any more. Among African authors it’s the worst, most very capable African writers now resort to AI for writing which makes me super sad.
I’m concerned that substack might go the same way, but it seems to be mostly free of that rubbish for now at least! Apart from in the comments sections…
Thanks Nick, I appreciate this take. I’m personally hoping that mandatory disclosure will serve to discourage AI-writing where it isn’t necessary, but we will see. Let me know if you’re seeing signs of the policy being insufficient.
PS- I agree with substack, but that’s partially just a good algorithm. If you look at the best-sellers … it’s not so good.
Has anyone done any thinking about posting the actual conversations for LLM ideation and other things? I’d be kind of interested. Most authors don’t include the revision history of their papers when they publish them, but I feel like this could be helpful. Especially thinking about the case with the non-native English speaker using an LLM to translate their thoughts. I think that actually for us English primary-language speakers, funnily enough, it might be helpful to be able to see the full history.