EA is about maximization, and maximization is perilous

This is not a contest submission; I don’t think it’d be appropriate for me to enter this contest given my position as a CEA funder. This also wasn’t really inspired by the contest—I’ve been thinking about writing something like this for a little while—but I thought the contest provided a nice time to put it out.

This piece generally errs on the concise side, gesturing at intuitions rather than trying to nail down my case thoroughly. As a result, there’s probably some nuance I’m failing to capture, and hence more ways than usual in which I would revise my statements upon further discussion.

For most of the past few years, I’ve had the following view on EA criticism:

  • Most EA criticism is—and should be—about the community as it exists today, rather than about the “core ideas.”

  • The core ideas are just solid. Do the most good possible—should we really be arguing about that?

Recently, though, I’ve been thinking more about this and realized I’ve changed my mind. I think “do the most good possible” is an intriguing idea, a powerful idea, and an important idea—but it’s also a perilous idea if taken too far. My basic case for this is that:

  • If you’re maximizing X, you’re asking for trouble by default. You risk breaking/​downplaying/​shortchanging lots of things that aren’t X, which may be important in ways you’re not seeing. Maximizing X conceptually means putting everything else aside for X—a terrible idea unless you’re really sure you have the right X. (This idea vaguely echoes some concerns about AI alignment, e.g., powerfully maximizing not-exactly-the-right-thing is something of a worst-case event.)

  • EA is about maximizing how much good we do. What does that mean? None of us really knows. EA is about maximizing a property of the world that we’re conceptually confused about, can’t reliably define or measure, and have massive disagreements about even within EA. By default, that seems like a recipe for trouble.

The upshot is that I think the core ideas of EA present constant temptations to create problems. Fortunately, I think EA mostly resists these temptations—but that’s due to the good judgment and general anti-radicalism of the human beings involved, not because the ideas/​themes/​memes themselves offer enough guidance on how to avoid the pitfalls. As EA grows, this could be a fragile situation.

I think it’s a bad idea to embrace the core ideas of EA without limits or reservations; we as EAs need to constantly inject pluralism and moderation. That’s a deep challenge for a community to have—a constant current that we need to swim against.

How things would go if we were maximally “hard-core”

The general conceptual points behind my critique—“maximization is perilous unless you’re sure you have the right maximand” and “EA is centrally about maximizing something that we can’t define or measure and have massive disagreements about”—are hopefully reasonably clear and sufficiently explained above.

To make this more concrete, I’ll list just some examples of things I think would be major problems if being “EA” meant embracing the core ideas of EA without limits or reservations.

We’d have a bitterly divided community, with clusters having diametrically opposed goals.

For example:

  • Many EAs think that “do the most good possible” ends up roughly meaning “Focus on the implications of your actions for the long-run future.”

  • Within this set, some EAs essentially endorse: “The more persons there are in the long-run future, the better it is” while others endorse something close to the opposite: “The more persons there are in the long-run future, the worse it is.”1

In practice, it seems that people in these two camps try hard to find common ground and cooperate. But it’s easy to envision a version of EA so splintered by this sort of debate that learning someone is an EA most-often tells you that they are dedicated to maximizing something other than what you’re dedicated to maximizing, and that you should take a fundamentally adversarial and low-trust stance toward each other.

We’d have a community full of low-integrity people, and “bad people” as most people define it.

A lot of EAs’ best guess at the right maximand is along the lines of utilitarianism.

Does utilitarianism recommend that we communicate honestly, even when this would make our arguments less persuasive and cause fewer people to take action based on them? Or does utilitarianism recommend that we “say whatever it takes” to e.g. get people to donate to the charities we estimate to be best?

Does utilitarianism recommend that we stick to promises we made? Or does utilitarianism recommend that we go ahead and break them when this would free us up to pursue our current best-guess actions?

It seems that the answers to these questions are, at best, unclear, and different people have very different takes on them. In general, it seems to be extremely uncertain and debatable what utilitarianism says about a given decision, especially from a longtermist point of view.

Even if, say, 80% of utilitarian EAs thought that utilitarianism supported honesty and integrity, while 20% thought it did not, I think the result would be a noticeably and unusually low-integrity community, full of deception and “bad actors” by the standards of its reference class. I also think that high-integrity behavior works better in a setting where it’s common; 20% of EAs behaving in noticeably low-integrity ways might change the calculus for the other 80%, making things worse still.

My view is that—for the most part—people who identify as EAs tend to have unusually high integrity. But my guess is that this is more despite utilitarianism than because of it. (One hypothesis I’ve heard is that people who care a lot about holding themselves to a high ethical standard, and achieving consistency between their views, statements and actions, are drawn to both utilitarianism and high-integrity behavior for this reason.)

We’d probably have other issues that should just generally give us pause.

I’d expect someone who is determined to derive all their actions from utilitarianism—or from any other explicit maximization of some best-guess maximand—to be at high risk of things like being a bad friend (e.g., refusing to do inconvenient or difficult things when a friend is in need), bad partner (same), narrow thinker (not taking an interest in topics that don’t have clear relevance to the maximand), etc. This is because I doubt there is a maximand and calculation method available that reliably replicates all of the many heuristics people use to be “virtuous” on a variety of dimensions.

Can we avoid these pitfalls by “just maximizing correctly?”

You could argue that for nearly any maximand, it’s a good idea to be the sort of person other people trust and like; to keep lots of life options open; and generally to avoid the sorts of behaviors I worry about above, unless you’re quite confident that you have a calculation pointing that way.

You could make a case for this either from an instrumental point of view (“Doing these things will ultimately make you better at maximizing your maximand”) or using various “nonstandard decision theories” that some EAs are fond of (including myself to some degree).

But I doubt you can make a case that’s robustly compelling and is widely agreed upon, enough to prevent the dynamics I worry about above. Especially if you accept my claim that a significant minority of people behaving badly can be extremely bad. To the extent that the EA community is avoiding these pitfalls, I don’t think this is enough to explain it.

(I do in fact think—though not with overwhelming confidence—that the “pitfalls” I describe would be bad for most plausible EA maximands. How can I simultaneously think this, while also fearing that non-tempered acceptance of EA would result in these “pitfalls?” I address this in a brief appendix.)

Avoiding the pitfalls

I think the EA community does mostly avoid these pitfalls—not in the sense that the dynamics I worry about are absent, but in the sense that they don’t seem more common among EAs than in other analogous communities.

I think a major reason for this is simply that most EAs are reasonable, non-fanatical human beings, with a broad and mixed set of values like other human beings, who apply a broad sense of pluralism and moderation to much of what they do.

My sense is that many EAs’ writings and statements are much more one-dimensional and “maximizy” than their actions. Most EAs seem to take action by following a formula something like: “Take a job at an organization with an unusually high-impact mission, which it pursues using broadly accepted-by-society means even if its goals are unusual; donate an unusual amount to carefully chosen charities; maybe have some other relatively benign EA-connected habits like veganism or reducetarianism; otherwise, behave largely as I would if I weren’t an EA.”

I’m glad things are this way, and with things as they stand, I am happy to identify as an EA. But I don’t want to lose sight of the fact that EA likely works best with a strong dose of moderation. The core ideas on their own seem perilous, and that’s an ongoing challenge.

And I’m nervous about what I perceive as dynamics in some circles where people seem to “show off” how little moderation they accept—how self-sacrificing, “weird,” extreme, etc. they’re willing to be in the pursuit of EA goals. I think this dynamic is positive at times and fine in moderation, but I do think it risks spiraling into a problem.

Brief appendix: spreading the goal of maximizing X can be bad for the goal of maximizing X

There’s a potentially confusing interplay of arguments here. To some degree, I’m calling certain potential dynamics “pitfalls” because I think they would (in fact) be bad for most plausible EA maximands. You might think something like: “Either these dynamics would be bad for the right maximand, in which case you can’t complain that a maximizing mindset is the problem (since proper maximizing would avoid the pitfalls) … or they wouldn’t be bad for the right maximand, and maybe that means they’re just good.” I have a couple of responses to this:

  • First, I think the “pitfalls” above are just broadly bad and should give us pause. The fact that a low-trust, bitterly divided EA community would probably be less effective is part of why I think it would be a bad thing, but only part of it. I think honesty is good partly because it seems usually instrumentally valuable, but I also think it’s just good, and would have some trouble being totally comfortable with any anti-honesty conclusion even if the reasoning seemed good.

  • Second, I think you can simultaneously believe “X would be bad for the maximand we care about” and “Broadly promoting and accepting a goal of maximizing that maximand would cause X.” EA isn’t just a theoretical principle, it’s a set of ideas and messages that are intended to be broadly spread and shared. It’s not contradictory to believe that spreading a goal could be bad for the goal, and it seems like a live risk here.


  1. I’m thinking of classical utilitarianism for the former, suffering-focused ethics for the latter. Some additional assumptions are needed to reach the positions I list. In particular, some assumption (which I find very plausible) along the lines of: “If we condition on the long-run future having lots of persons in it, most of those lives are probably at least reasonably good [the persons would prefer to exist vs. not exist], but there’s a significant risk that at least some are very bad.”