Effective Altruism: The First Decade (Forum Review)

Aaron Gertler1 Dec 2021 22:18 UTC
93 points
45 comments6 min readEA link

Following in the footsteps of LessWrong, we are running our first-ever EA Forum Review.

The purpose of the Review is to figure out which writings from the EA community have produced actual value, so that we can build on their ideas and share them with many more readers.

The Review uses the following process:

  1. Users cast preliminary votes to nominate posts they’ve found valuable for the final voting stage. (December 1 - December 14)

  2. Users review posts — explaining how the posts have been useful (or not), which of their ideas seem best (or worst), and so on. (December 15 - January 15)

  3. Users cast final votes on posts[1], based on their own experience and others’ reviews. The posts with the most votes are, hopefully, those that have provided the most collective value to the community. (January 16 - February 1) To see what this looks like in practice, check out LessWrong’s 2019 Review and its results.[2]

Why hold a review?

Effective altruism is focused on impact. But it’s hard to tell which intellectual work has actually made an impact.

The karma system shows us how many people liked a post right away. But that’s different from knowing what a post has achieved over the full span of its existence. Did it change the movement forever? Did people forget about it completely? Is it still useful?

The Review is our way of trying to answer questions like:

  • “What are the most valuable things we’ve written? How might they be improved, or built upon?”

  • “Which ideas are most important for someone to understand if they want to do as much good as possible? And where can they find the best writing on those ideas?”

  • “What should people do if they want their own writing to be impactful? What can they learn by seeing trends and patterns in our best writing so far?”

Other benefits:

  • This is a chance for people to share reflective commentary on important posts, with the benefit of hindsight. What still seems true, a year or five years later? Which ideas didn’t stand the test of time?

  • It’s also a chance for authors to look back at their own posts and fix errors or make changes. The voting system lets them see which posts might be most impactful to update.

    • To quote LessWrong: “There needs to be an incentive to clean up ideas that turned out to be important but badly presented.”

  • Asking people to think about content they’ve valued might also lead them to crosspost it, making the Forum a more complete repository of EA’s intellectual contributions.

How does it work?

The Review will be visible to anyone who had a Forum account before December 1st, 2021.[3]

Our first review covers all posts published before 2021.[4] To quote LessWrong:

A year is probably enough time to no longer be swept away in the news or excitement of the day, but recent enough that we can still remember and write down how an idea or explanation has affected us.

(But don’t be shy about taking your best guess at how something much older affected you.)

Stage 1: Nominations /​ Preliminary Votes

From today through January 15th, eligible users will see this feature on eligible posts:

You can vote for as many posts as you like; a post will need at least one positive vote to be nominated to the second round. Negative votes don’t affect nomination status — only how posts are ranked.

The nomination scoring system

Here’s how we interpret each voting score[5]:

0 //​ Neutral: You have no strong feelings about the post.[6]

1 //​ Good: You found the post valuable. You’ve referred back to it before, or would recommend it to others.

4 //​ Important: Like “Good”, but more so. Perhaps the post includes one or more key insights that have changed the way you think, or led you to take useful actions.

9 //​ Crucial: The post is timeless — one of the most valuable on the Forum. You’d recommend it to almost everyone, and you think the community should be discussing and building on it for years to come.

-1 //​ Bad: The post is misleading, slightly harmful, or exceedingly unimportant. You wouldn’t personally recommend it to anyone.

-4 //​ Very bad: Like “Bad”, but more so. Perhaps the post includes one or more exceptionally bad ideas that have damaged the community, or led you to take actions you regret.

-9 //​ Awful: The post has had a worse effect on EA’s intellectual landscape than almost any other piece of writing.

How voting works

  • You can vote for any eligible post, unless you were its author/​co-author.

  • You can cast an unlimited number of votes.

  • However, the more votes you cast, and the higher your total “score” (where a “9” vote counts for 9x the score of a “1” vote), the less influential each of your votes will be. We normalize voting strength so that all users who are past a certain “score” threshold exert roughly the same amount of total influence.[7]

On the back end, we use a modified quadratic voting system, which allocates a fixed number of “points” across your votes based on how strong they are.

You can vote at the top of a post, or anywhere the post appears in a list (like the All Posts page).

You can also use the voting dashboard to see and vote on many older posts at once:

There’s also a new section on the front page:

The buttons on the bottom, from left to right, show you:

  • Posts you’ve upvoted from before 2021 (a good way to find things to vote on for the review).

  • A list of all eligible posts that haven’t yet been nominated.

  • Posts that have already been nominated, so you can see what other people have been voting on.

Stage 2: Reviews

On December 14th, the list of nominated posts will close; no new posts can be nominated after that.

However, you can write reviews of posts. Here are some things you might discuss:

  • How has this post been useful, to you or others you’ve known?

  • How does it connect to the broader intellectual landscape of EA?

  • Is this post epistemically sound? Does it say true things, and use language responsibly?

  • Does the post still stand up to scrutiny today?

  • How could the post be improved?

  • How would you like to see someone build on this post with other work?

See these great examples from LessWrong’s 2018 review.

We hope that authors and other readers will find these comments valuable! To quote LessWrong:

Now is the time to give your opinions much more detail, distinguish between a post being an interesting hypothesis versus a robust argument, and generally help others understand what you think, so that we can discover exciting new disagreements and build much more robustly on past and future work.

Reviews can be voted on like other comments, but will have a special flair.

Only posts which have been reviewed with make it to the Final Voting phase.

What if some really good writing isn’t on the Forum?

  1. Crosspost it. If it was valuable to you, let’s make it easier to find!

  2. Add the review crosspost tag. This lets other people see all the new material that has gone up for review. If it was valuable to you, it may have been valuable to others!

  3. Contact us. Send an Intercom message or email forum@effectivealtruism.org, and we’ll backdate the post to its original publication date, making it eligible for nomination.

Stage 3: Final Votes

In the Review’s final phase, we hope that people will focus primarily on reading reviews, updating their posts, and finalizing their votes. (However, you’ll still be able to write new reviews.)

Voting will remain open through February 1st. You can only cast one vote for each post, but you’ll be able to change your votes until the review period ends.

Throughout the final voting period, the voting dashboard will continue to order posts by how many total “points” they’ve received.

What happens after the Review?

When the review ends, we’ll end up with a public list (on the voting dashboard) of which posts the Forum’s users found most valuable.

We’ll definitely create a sequence of these posts, and add it to our list of core readings. We may also follow LessWrong in creating a book, but that’s not guaranteed.

Prizes for great reviews and posts

What is guaranteed: Prizes!

We’re allocating up to $15,000 in total prize funding — $5,000 for the top posts, up to $5,000 for authors who update their posts during the Review, and up to $5,000[8] for the best reviews (as judged by a team of moderators and experienced authors).[9]

What should you do next?

Start finding posts to nominate and review!

Some ideas for how to start:

  1. Use our lists of posts you’ve upvoted before, and posts others have nominated.

  2. Look at posts that won the EA Forum Prize

  3. Find tags for topics you care about and look at the posts that use them.

  4. Check your history in your email, Messenger, or other apps to see which Forum posts you’ve shared (searching for “forum.effectivealtruism.org” should turn them up)

Questions and bug reports

Leave a comment if you have any questions about the review. (It’s a very detailed process, and I could easily have left out some helpful details in this post.)

We’re still working out a few bugs in the system, and changing some of our UI text to be more accurate. Leave a comment if you see anything that appears to need fixing!

Thank you, LessWrong!

The LessWrong team developed the concept for this review, did lots of coding to make it all work (including many changes this year), and wrote some excellent materials that made it much easier for me to write this post. We couldn’t have done this without them.

  1. ↩︎

    Or change their initial votes — you can only vote on each post once through the entire process.

  2. ↩︎

    Note that we’ve combined the nomination and review periods, so our process looks a bit different than theirs.

  3. ↩︎

    If you’ve been reading Forum content since before 2021, don’t have an account, and want to participate in the review, you can! Just make an account and email forum@effectivealtruism.org; we’ll make your account eligible.

  4. ↩︎

    We say “2011-2020” in some places because this period covers all but five published Forum posts, the term “effective altruism” was coined in 2011, and “The Decade Review” has a nice ring to it. However, posts from before 2011 are also eligible.

  5. ↩︎

    We wanted to include suggestions, but you are welcome to use your own definitions.

  6. ↩︎

    We expect that you will feel this way about almost all of the Forum’s 4,000+ posts.

  7. ↩︎

    We’ll share more specific figures on this later.

  8. ↩︎

    We’re saying “up to” in case very few reviews are written or very few posts are updated. However, we think it’s likely that we’ll allocate the full $10,000 across these categories.

  9. ↩︎

    The exact distribution of prize funding will depend on the specifics of how people allocate points, and how our judges feel about various reviews.

Aaron Gertler1 Dec 2021 22:18 UTC
93 points
45 comments6 min readEA link