Read The Sequences

There is heavy overlap among the effective altruism and rationality communities but they are not the same thing. Within the effective altruism community, especially among those who are newer to the movement and were introduced to it through a university group, I’ve noticed some tension between the two. I often sense the vibe that sometimes people into effective altruism who haven’t read much of the canonical LessWrong content write off the rationalist stuff as weird or unimportant.

I think this is a pretty big mistake.

Lots of people doing very valuable work within effective altruism got interested in it via first interacting with rationalist content, in particular The Sequences and Harry Potter and the Methods of Rationality. I think that is for good reason. If you haven’t come across those writings before, here’s a nudge to give The Sequences a read.

The Sequences are a (really long) collection of blog posts written by Eliezer Yudkowsky on the science and philosophy of human rationality. They are divided into sequences—a list of posts on a similar topic. Most of the posts would have been pretty useful to me on their own but I also got more value from reading posts in a particular sequence to better internalise the concepts.

There are slightly fewer posts in The Sequences than there are days in the year so reading the whole thing is a very doable thing to do in the coming year! You can also read Highlights from the Sequences which cover 50 of the best essays.

Below, I’ll list some of the parts that I have found especially helpful and that I often try to point to when talking to people into effective altruism (things I wish they had read too).

Fake Beliefs is an excellent sequence if you already know a bit about biases in human thinking. The key insight there is about making beliefs pay rent (“don’t ask what to believe—ask what to anticipate”) and that sometimes your expectations can come apart from your professed beliefs (fake beliefs). The ideas were helpful for me noticing when that happens, for example when I believe I believe something but actually do not. It happens a bunch when I start talking about abstract, wordy things but forget to ask myself what I would actually expect to see in the world if the things I am saying were true.

Noticing Confusion is a cool sequence that talks about things like:

  • What is evidence? (“For an event to be evidence about a target of inquiry, it has to happen differently in a way that’s entangled with the different possible states of the target”)

  • Your strength as a rationalist is your ability to be more confused by fiction than by reality—noticing confusion when something doesn’t check out and going EITHER MY MODEL IS FALSE OR THIS STORY IS WRONG

  • Absence of evidence is evidence of absence, and conservation of expected evidence (“If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction”)

I am often surrounded by people who are very smart and say convincing-sounding things all the time. The ideas mentioned above have helped me better recognise when I’m confused and when a smooth-sounding argument doesn’t match up with how I think the world actually works.

Against rationalisation has things that are useful to remember:

  • Knowing about biases can hurt people. Exposing subjects to an apparently balanced set of pro and con arguments will exaggerate their initial polarisation. Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to some biases.

  • Not to avoid your belief’s real weak points. “Ask yourself what smart people who disagree would say to your first reply, and your second reply. Whenever you catch yourself flinching away from an objection you fleetingly thought of, drag it out into the forefront of your mind”

  • Motivated stopping and motivated continuation. You should suspect motivated stopping when you close off search, after coming to a comfortable conclusion, and yet there’s a lot of fast cheap evidence you haven’t gathered yet. You should suspect motivated continuation when some evidence is leaning in a way you don’t like, but you decide that more evidence is needed—expensive evidence that you know you can’t gather anytime soon

  • Is that your true rejection? “Is that simple straightforward-sounding reason your true rejection [for a position you disagree with], or does it come from intuition-X or professional-zeitgeist-Y?”

  • Entangled truths, contagious lies

I once facilitated an effective altruism intro fellowship. Sometimes, the participants in this fellowship would have criticisms or questions that I hadn’t thought of. Even so, my mind would quickly come up with a convincing-sounding response and I would feel very rational. That’s rationalisation. This also happens when I’m alone, in the privacy of my own mind: the urge to find a convincing argument against something I don’t want to believe and quickly move on, and the urge to put a lot of effort into gathering evidence for something I want to believe. Scary! But I notice it more now.

Cached Thoughts was useful for recognising when I am simply accepting and repeating ideas without actually evaluating or understanding them. Rohin Shah, an AI safety researcher, has previously mentioned that he estimates there are ~50 people in the world who can make a case for working on AI alignment that he wouldn’t consider clearly flawed. Lots of people would disagree with Rohin about what counts as a not-clearly-flawed argument but I think the general pattern of “there are way more people who think they know the arguments and can parrot them compared to people who can actually generate them” is true in lots of areas. This is one type of thing that the ideas in this post can help with:

  • What patterns are being completed, inside your mind, that you never chose to be there?

  • If this idea had suddenly occurred to you personally, as an entirely new thought, how would you examine it critically?

  • Try to keep your mind from completing the pattern in the standard, unsurprising, already-known way. It may be that there is no better answer than the standard one, but you can’t think about the answer until you can stop your brain from filling in the answer automatically.

  • But is it true? Don’t let your mind complete the pattern! Think!

Every Cause Wants to Be a Cult points out something that happens because of human nature regardless of how worthy your cause is. It points out the need to actively push back against sliding into cultishness. It doesn’t just happen as a result of malevolence or stupidity but whenever you have a group of people with an unusual goal who aren’t always making a constant effort to resist the cult attractor. It doesn’t have suggestions on how to do this (there are other posts that cover that) but just points out that cultishness is the default unless you are actively making an effort to prevent it. For me, this is helpful to remember as I am often a part of groups with unusual goals.

Letting Go is a sequence on, well, how to let go of untrue beliefs when you change your mind instead of holding on. It has posts on The Importance of Saying “Oops”, on using the power of innocent curiosity, on leaving a line of retreat so that you can more easily evaluate the evidence for beliefs that make you uncomfortable, and how to stage a Crisis of Faith when there is a belief you have had for a while that is surrounded by a cloud of known arguments and refutations, that you have invested a lot in, and that has emotional consequences.

I first read these posts when I had doubts about my religious beliefs but they were still a huge part of my identity. The tools presented in the sequence made it easier for me to say “oops” and move on instead of just living with a cloud of doubts. I have found it useful to come back to these ideas when I start noticing uncomfortable doubts about a major belief where changing my mind on it would have emotional and social consequences.

Fake Preferences has some blog posts that I found valuable, especially Not For the Sake of Happiness Alone (helped me notice that my values aren’t reducible to just happiness), Fake Selfishness (people usually aren’t genuinely selfish—I do actually care about things outside myself), and Fake Morality (“The fear of losing a moral compass is itself a moral compass”).

The Quantified Humanism sequence has some bangers that have always been relevant for effective altruists and are especially relevant today. Ends Don’t Justify Means (Among Humans) and Ethical Injunctions caution against doing unethical things for the greater good because we run on corrupted hardware and having rules to not do certain things even if it feels like the right thing to do protects us from our own cleverness. “For example, you shouldn’t rob banks even if you plan to give the money to a good cause.”

The Challenging the Difficult sequence is about solving very difficult problems, to make an extraordinary effort to do the impossible. It’s not just inspiring but also helpful for looking at how I am approaching my goals, if I am actually doing what needs to be done and aiming to win or just acting out my role and trying to try.

In summary, I think the Sequences have lots of valuable bits for people aiming to have a positive impact. I have found them valuable for my thinking. If you haven’t encountered them before, I recommend giving them a try.