Error

I appreciate you taking the effort to write this. However, like other commentators I feel that if these proposals were implemented, EA would just become the same as many other left wing social movements, and, as far as I can tell, would basically become the same as standard forms of left wing environmentalism which are already a live option for people with this type of outlook, and get far more resources than EA ever has. I also think many of the proposals here have been rejected for good reason, and that some of the key arguments are weak.

1. You begin by citing the Cowen quote that “EAs couldn’t see the existential risk to FTX even though they focus on existential risk”. I think this is one of the more daft points made by a serious person on the FTX crash. Although the words ‘existential risk’ are the same here, they have completely different meanings, one being about the extinction of all humanity or things roughly as bad as that, and the other being about risks to a particular organisation. The problem with FTX is that there wasn’t enough attention to existential risks to FTX and the implications this would have for EA. In contrast, EAs have put umpteen person hours into assessing existential risks to humanity and the epistemic standards used to do that are completely different to those used to assess FTX.

2. You cite research purporting to show that diversity of some form is good for collective epistemics and general performance. I haven’t read the book that you cite, but I have looked into some of this literature, and as one might expect for a topic that is so politically charged, a lot of the literature is not good, and some of the literature actually points in the opposite direction, even though it is career suicide to criticise diversity, and there are likely personal costs even for me discussing counter-arguments here. For example, this paper suggests that group performance is mainly determined by the individual intelligence of the group members not by things like gender diversity. This paper lists various costs of team diversity that are bad for collective dynamics. You say that diversity “essentially along all dimensions” is good for epistemics. This is the sort of claim that sounds good, but also seems to be clearly false. I seldom see people who make this argument suggest that we need more Trump supporters, religious fundamentalists, homophobes or people without formal education in order to improve our performance as a community. These are all huge chunks of the national/​global community but also massively underrepresented in EA. There are lots of communities that are much more diverse than EA but which also seem to have far worse epistemics than EA. Examples include Catholicism, Trumpism, environmentalism, support of Bolsonaro/​Modi etc.

3. Relatedly, I think value alignment is very important. I have worked in organisations with a mix of EA and non EA people and it definitely made things much harder than if everyone were aligned, holding other things equal. On one level, it is not surprising that a movement trying to achieve something would agree not just at a very abstract level, but also about many concrete things about the world. If I think that stopping AI progress is good and you think it is bad, it is going to be much harder (though not impossible, per moral trade) for us to achieve things in the world. Same for speeding up progress in virus synthesis. The 80,000 Hours articles on goal directed groups are very good on this.

4. I don’t agree that EA is hostile to criticism. In fact it seems unusually open to criticism, and rational discussion of ideas rather than dismissing them on the basis of vibe/​mood affiliation/​political amenability. Aside from the controversial Cremer and Kemp case (who didn’t publish pseudonymously) what are the major critiques that have been presented pseudonymously or have caused serious personal consequences for the critics? By your definition, I think my critique of GiveWell counts as deep, but I have been rewarded for this because people thought the arguments were good. To stress, mine and Hauke’s claim was that most of the money EA has spent has been highly suboptimal.

5. You say “For instance, (intellectual) ability is implicitly assumed within much of EA to be a single variable[32], which is simply higher or lower for different people.” This isn’t just an assumption of EA, but a central finding of psychological science that things that are usually classed as intellectual abilities are strongly correlated—the g factor. eg maths ability is correlated with social science ability, and english literature ability etc.

6. I just don’t think it is true that we align well with the interests of tech billionaires. We’ve managed to persuade two billionaires of EA and one believed in EA before he became a billionaire. The vast majority of billionaires evidently don’t buy it and go off and do their own thing, mainly donating to things that sound good in their country, to climate change, or not donating at all. Longtermist EAs would like lots more money to spent on AI alignment, slowing down AI progress, on slowing down progress in virology or increasing spending on counter-measures, and on preventing major wars. I don’t see how any of these things promise to benefit tech founders as a particular constituency in any meaningful way. That being said, I agree that there is a problem with rich people becoming spokespeople for the community or overly determining what gets done and we need far better systems to protect against that in future. eg FTX suddenly deciding to do all this political stuff was a big break from previous wisdom and wasn’t questioned enough.

7. On a personal note, I get that I am a non-expert in climate, and so am wide open to criticism as an interloper (though I have published a paper on climate change). But then it is also true that getting climate people to think in EA terms is very difficult. Also, the view I recently outlined is basically in line with all climate economics. In that sense the view I hold and I think is widely held in longtermist EA is in line with one expert consensus. Indeed, it is striking that this is the one group that actually tries to quantify the aggregate costs of climate change. I also don’t think there are any areas where I disagree with the line taken by the IPCC which is supposed to express the expert consensus on climate. The view that 4ºC is going to kill everyone is one held by some activists and a small number of scientists. Either way we need to explain why we are ignoring all the climate economists and listening to Rockstrom/​Lenton instead. On planetary boundaries, as far as I know, I am the only EA to have criticised planetary boundaries, and I don’t dismiss it in passing, but in fact at considerable length. The reviewer I had for that section is a prof and strongly agreed with me.

8. Differential tech progress has been subject to peer review. The Bostrom articles on it are peer reviewed.

9. The implications of democratising EA are mindboggling. Suppose that Open Phil’s spending decisions are made democratically by EAs. This would put EAs in charge of ~$10bn. We’d then need to decide who counts as an EA. Because so much money would be on the table, lots of people who we wouldn’t class as EAs would want a say, and it would be undemocratic to exclude them (I assume). So, the ‘EA franchise’ would expand to anyone who wants a say (?) I don’t know where the money would end up after all this, but it’s fair to say that money spent on reducing engineered pandemics, AI and farm animal welfare would fall from the current pitiful sum to close to zero.

10. You say that worker self-management has been proven to be better for mission-oriented work than top-down rule. This is clearly false. There is a tiny pocket of worker cooperatives (eg in the Basque region) who have been fairly successful. But almost all companies are run oligarchially in a top-down fashion by boards or leadership groups.

Overall, we need to learn hard lessons from the FTX debacle. But thus far, the collapse has mainly been used to argue for things that are completely unrelated to FTX, and mainly to an advance an agenda that has been disfavoured in EA so far, and with good reason. For Cowen, this was neoliberal progress, here it is left wing environmentalism.

Overall this post seems like a grab-bag of not very closely connected suggestions. Many of them directly contradict each other. For example, you suggest that EA organizations should prefer to hire domain experts over EA-aligned individuals. And you also suggest that EA orgs should be run democratically. But if you hire a load of non-EAs and then you let them control the org… you don’t have an EA org any more. Similarly, you bemoan that people feel the need to use pseudonyms to express their opinions and a lack of diversity of political beliefs … and then criticize named individuals for being ‘worryingly close to racist, misogynistic, and even fascist ideas’ in essentially a classic example of the cancel culture that causes people to choose pseudonyms and causes the movement to be monolithically left wing.

I think this is in fact a common feature of many of the proposals: they generally seek to reduce what is differentiated about EA. If we adopted all these proposals, I am not sure there would be anything very distinctive remaining. We would simply be a tiny and interchangeable part of the amorphous blob of left wing organizations.

It is true this does not apply to all of the proposals. I agree that, for example, EAs should re-invent the wheel less and utilize domain expertise more. But I can’t say this post really caused me to update in their favour vs just randomly including some proposals I already agreed with. I think you would have been better off focusing on a smaller number of proposals and developing the arguments for them in more depth—and in particular considering counterarguments.

In a post this long, most people are probably going to find at least one thing they don’t like about it. I’m trying to approach this post as constructively as I can, i.e. “what I do find helpful here” rather than “how I can most effectively poke holes in this?” I think there’s enough merit in this post that the constructive approach will likely yield something positive for most people as well.

First off, thank you to everyone who worked on this post. Although I don’t agree with everything in it, I really admire the passion and dedication that went into this work—and I regret that the authors feel the need to remain anonymous for fear of adverse consequences.

For background: I consider myself a moderate EA reformer—I actually have a draft post I’ve been working on that argues that the community should democratically hire people to write moderately concrete reform proposals. I don’t have a ton of the “Sam” characteristics, and the only thing of value I’ve accepted from EA is one free book (so I feel free to say whatever I think). I am not a longtermist and know very little about AI alignment (there, I’ve made sure I’d never get hired if I wanted to leave my non-EA law career?).

Even though I agree with some of the suggested reforms here, my main reaction to this post is to affirm that my views are toward incremental/​moderate—and not more rapid/​extensive—reform. I’m firmly in the Global Health camp myself, and that probably colors my reaction to a proposal that may have been designed more with longtermism in mind. There is too much in the post for anyone to fully react to without several days of thinking and writing time, so I’ll hit on a few high points.

1. I think it’s critical to look at EA activity in the broader context of other actors working in the same cause area.^[1] I suggest that much of EA’s value in spaces where it is a low-percentage funder is in taking a distinctive approach that zeroes in on the blind spots of bigger fish. ^[2] In other words, an approach that maximizes diversity of perspective within EA may not maximize the effective diversity of perspective in the cause area as a whole. Also, the important question is not necessarily how good EA’s epistemic tools would work in a vacuum with no one else in a given space.

In spaces where EA is a niche player, I am concerned that movements in the direction of looking like other actors may well be counterproductive. In addition to GH&D, I believe that EA remains a relatively small player in animal advocacy and even in fields like pandemic prevention compared to the total amount of resources in those areas.

2. I feel that the proposal is holding EA up to a much higher standard at certain points than comparable movements. World Vision doesn’t (to my knowledge) host a message board where everyone goes to discuss various decisions its leadership has made. I doubt the Gates Foundation devotes a percentage of its spend on hiring opposition researchers. And most major charitable funders don’t crowdsource major funding decisions. Other than certain meta spending (which raises some conflict-of-interest flags for me), I don’t see anything that justifies making demands of EA unless one is simultaneously making demands of similar outfits.

Given that a large portion of the authors’ critique is about learning from others outside EA, I think that the lack of many of their proposed reforms in many similarly-sized, mature charitable movements is a significant data point. Although I believe in more process, consultation, and “bureaucracy” than I think the median EA does, I think there has to be a recognition that these things incur significant costs as well.

3. Portions of this reform package sound to my ears like the dismantling of EA and its replacement with a new movement, Democratic Altruism (“DA”). It seems unlikely that much of classic EA would be left after at least radical democratization—there are likely to be a flood of incoming people, many with prior commitments, attracted by the ability to vote on how to spend $500MM of Uncle (Open) Phil’s money every year. Whoever controls the voter registration function would ultimately control the money.

Now, I think DA is a very interesting idea, and if I had a magic wand to slice off a tiny slice of each Western charitable spend and route it to a DA movement, I think that would more likely than not be net positive. I’m just not clear on why EA should feel obliged to be the ashes from which DA arises—or why EA’s funders should feel obliged to fund DA while all the other big-money interests get to keep their handpicked people making their funding-allocation decisions.

4. As I noted in a comment elsewhere on this thread, I don’t think the community has much leverage over its funders. Unfortunately, it is much easier to come up with interesting ideas than people who want to and can fund them. Especially Grade-A funders—the proposal suggests a rejection, or at least minimization, of various classes of less-desirable donors.

As a recent post here reminds us, “[o]nly the young and the saints are uncompromised.” There’s rarely a realistic, easy way to get large sums of money for one’s cause without becoming compromised to some extent in the process. There’s the traditional way of cultivating an army of small/​mid-size donors, but that takes many years, and you end up spending lots of resources and energy on care and feeding of the donors instead of on getting stuff done. I suspect most movements will spend a lot of time waiting to launch, and seeking funding, if they will only take Grade-A donor money. That’s a massive tradeoff—I really value my bednets! -- and it’s not one I am personally desirous of making.

One final, more broadly conciliatory point: EA can be too focused on what happens under the EA brand name and can seem relatively less interested in empowering people to do good effectively outside the brand. It doesn’t have a monopoly on either effectiveness or altruism, and I’ve questioned (without getting much in the way of upvotes) whether it makes sense to have a unified EA movement at this point.

I like the idea of providing different options for people where they can do good as effectively as possible in light of their unique skills, passions, and interests. For some people, that’s going to be classic GiveWell-style EA (my own likely best fit), for others it is going to be something like the current meta, for yet others it’s going to be something like what is in this proposal, and there are doubtless many other potential flavors I haven’t thought about. Some people in the community are happy with the status quo; some people are not. The ideal might be to have spaces where everyone would be locally happy and effective, rather than try to preserve or reform the entire ecosystem into something one personally likes (but isn’t conducive to others).

1. ^

   For example, in Global Health & Development, you have a number of NGOs [e.g., World Vision at $1.2B is several times EA’s entire spend on GH&D; see generally here for a list of big US charities] plus the truly big fish like the Gates Foundation [$6.7B, although not all GH&D] and various governments. So the vast majority of this money is being moved through democratic processes, Gates-type subject-matter experts, and traditional charities—not through EA.

2. ^

   It’s scandalous to me that some of the opportunities GiveWell has found were not quickly swallowed up by the big fish.

[For context, I’m definitely in the social cluster of powerful EAs, though don’t have much actual power myself except inside my org (and my ability to try to persuade other EAs by writing posts etc); I had more power when I was actively grantmaking on the EAIF but I no longer do this. In this comment I’m not speaking for anyone but myself.]

This post contains many suggestions for ways EA could be different. The fundamental reason that these things haven’t happened, and probably won’t happen, is that no-one who would be able to make them happen has decided to make them happen. I personally think this is because these proposals aren’t very good. And so:

• people in EA roles where they could adopt these suggestions choose not to

• and people who are capable/​motivated enough that they could start new projects to execute on these ideas (including e.g. making competitors to core EA orgs) end up deciding not to.

And so I wish that posts like this were clearer about their theory of change (henceforth abbreviated ToC). You’ve laid out a long list of ways that you wish EA orgs behaved differently. You’ve also made the (IMO broadly correct) point that a lot of EA organizations are led and influenced by a pretty tightly knit group of people who consider themselves allies; I’ll refer to this group as “core org EAs” for brevity. But I don’t understand how you hope to cause EA orgs to change in these ways.

• Maybe your ToC is that core org EAs read your post (and similar posts) and are intellectually persuaded by your suggestions, and adopt them.

  • If that’s your goal, I think you should try harder to understand why core org EAs currently don’t agree with your suggestions, and try to address their cruxes. For this ToC, “upvotes on the EA Forum” is a useless metric—all you should care about is persuading a few people who have already thought about this all a lot. I don’t think that your post here is very well optimized for this ToC.

    • (Note that this doesn’t mean that they think your suggestions aren’t net positive, but these people are extremely busy and have to choose to pursue only a tiny fraction of the good-seeming things (which are the best-seeming-to-them things) so demonstrating that something is net positive isn’t nearly enough.)

  • Also, if this is the ToC, I think your tone should be one of politely suggesting ways that someone might be able to do better work by their own lights. IMO, if some EA funder wants to fund an EA org to do things a particular way, you have no particular right to demand that they do things differently, you just have the ability to try to persuade them (and it’s their prerogative whether to listen).

  • For what it’s worth, I am very skeptical that this ToC will work. I personally think that this post is very unpersuasive, and I’d be very surprised if I changed my mind to agree with it in the next year, because I think the arguments it makes are weak (and I’ve been thinking about these arguments for years, so it would be a bit surprising if there was a big update from thinking about them more.)

• Maybe your ToC is that other EAs read your post and are persuaded by your suggestions, and then pressure the core org EAs to adopt some of your suggestions even though they disagree with them.

  • If so, you need to think about which ways EAs can actually apply pressure to the core org EAs. For example, as someone who prioritizes short-AI-timelines longtermist work over global health and development, I am not very incentivized to care about whether random GWWC members will stop associating with EA if EA orgs don’t change in some particular way. In contrast, if you convinced all the longtermist EAs that they should be very skeptical of working on longtermism until there was a redteaming process like the one you described, I’d feel seriously incentivized to work on that redteaming process. Right now, the people I want to hire mostly don’t agree with you that the redteaming process you named would be very informative; I encourage you to try to persuade them otherwise.

  • Also, I think you should just generally be scared that this strategy won’t work? You want core org EAs to change a bunch of things in a bunch of really specific ways, and I don’t think that you’re actually going to be able to apply pressure very accurately (for similar reasons that it’s hard for leaders of the environmentalist movement to cause very specific regulations to get passed).

  • (Note that I don’t think you should engage in uncooperative behavior (e.g. trying to set things up specifically so that EA orgs will experience damage unless they do a particular thing). I think it’s totally fair game to try to persuade people of things that are true because you think that that will cause those people to do better things by their own lights; I think it’s not fair game to try to persuade people of things because you want to force someone’s hand by damaging them. Happy to try to explain more about what I mean here if necessary; for what it’s worth I don’t think that this post advocates being uncooperative.)

• Perhaps you think that the core org EAs think of themselves as having a duty to defer to self-identified EAs, and so if you can just persuade a majority of self-identified EAs, the core org EAs will dutifully adopt all the suggestions those self-identified EAs want.

  • I don’t think this is realistic–I don’t think that core EA orgs mostly think of themselves as executing on the community’s wishes, I think they (as they IMO should) think of themselves as trying to do as much good as possible (subject to the constraint of being honest and reliable etc).

  • I am somewhat sympathetic to the perspective that EA orgs have implied that they do think of themselves as trying to represent the will of the community, rather than just viewing the community as a vehicle via which they might accomplish some of their altruistic goals. Inasmuch as this is true, I think it’s bad behavior from these orgs. I personally try to be clear about this when I’m talking to people.

• Maybe your ToC is that you’re going to start up a new set of EA orgs/​projects yourself, and compete with current EA orgs on the marketplace of ideas for funding, talent, etc? (Or perhaps you hope that some reader of this post will be inspired to do this?)

  • I think it would be great if you did this and succeeded. I think you will fail, but inasmuch as I’m wrong it would be great if you proved me wrong, and I’d respect you for actively trying much more than I respect you for complaining that other people disagree with you.

  • If you wrote a post trying to persuade EA donors that they should, instead of other options, donate to an org that you started that will do many of the research projects you suggested here, I would think that it was cool and admirable that you’d done that.

  • For many of these suggestions, you wouldn’t even need to start orgs. E.g. you could organize/​fundraise for research into “the circumstances under which individual subjective Bayesian reasoning actually outperforms other modes of thought, by what criteria, and how this varies by subject/​domain”.

  • I’ll generically note that if you want to make a project happen, coming up with the idea for the project is usually a tiny fraction of the effort required. Also, very few projects have been made to happen by someone having the idea for them and writing it up, and then some other person stumbling across that writeup and deciding to do the project.

• Maybe you don’t have any hope that anything will change, but you heuristically believe that it’s good anyway to write up lists of ways that you think other people are behaving suboptimally. For example, I have some sympathy for people who write op-eds complaining about ways that their government is making poor choices, even if they don’t have a detailed theory of change.

  • I think this is a fine thing to do, when you don’t have more productive ways to channel your energy. In the case of this post in particular, I feel like there are many more promising theories of change available, and I think I want to urge people who agree with it to pursue those.

Overall my main complaint about this post is that it feels like it’s fundamentally taking an unproductive stance–I feel like it’s sort of acting as if its goal is to persuade core EAs, but actually it’s just using that as an excuse to ineffectually complain or socially pressure; if it were trying to persuade, more attention would be paid to tradeoffs and cruxes. People sympathetic to the perspective in this post should either seriously attempt to persuade, or they should resort to doing things themselves instead of complaining when others don’t do those things.

(Another caveat on this comment: there are probably some suggestions made in this post that I would overall agree should be prioritized if I spent more time thinking about them.)

(In general, I love competition. For example, when I was on the EAIF I explicitly told some grantees that I thought that their goal should be to outcompete CEA, and I’ve told at least one person that I’d love it if they started an org that directly competes with my org.)

I agree the we ignore experts over people who are more value aligned. Seems like a mistake.

Nice. Thanks. Really well written, very clear language, and I think this is pointed in a pretty good direction. Overall I learned a lot.

I do have the sense it maybe proves too much—i.e. if these critiques are all correct then I think it’s surprising that EA is as successful as it is, and that raises alarm bells for me about the overall writeup.

I don’t see you doing much acknowledging what might be good about the stuff that you critique—for example, you critique the focus on individual rationality over e.g. deferring to external consensus. But it seems possible to me that the movement’s early focus on individual rationality was the cause of attracting great people into the movement, and that without that focus EA might not be anything at all! If I’m right about that then are we ready to give up on whatever power we gained from making that choice early on?

Or, as a metaphor, you might be saying something like “EA needs to ‘grow up’ now” but I am wondering if EA’s childlike nature is part of its success and ‘growing up’ would actually have a chance to kill the movement.

Where I agree:

• Experimentation with decentralised funding is good. I feel it’s a real shame that EA may not end up learning very much from the FTX regrant program because all the staff at the foundation quit (for extremely good reasons!) before many of the grants were evaluated.

• More engagement with experts. Obviously, this trades off against other things and it’s easier to engage with experts when you have money to pay them for consultations, but I’m sure there are opportunities to engage with them more. I suspect that a lot of the time the limiting factor may simply be people not knowing who to reach out to, so perhaps one way to make progress on this would be to make a list of experts who are willing for people at EA orgs to reach out to them, subject to availability?

• I would love to see more engagement from Disaster Risk Reduction, Future Studies, Science and Technology Studies, ect. I would encourage anyone with such experience to consider posting on the EA forum. You may want to consider extracting out this section in a separate forum post for greater visibility.

• I would be keen to see experiments where people vote on funding decisions (although I would be surprised if this were the right funding mechanism for the vast majority of funds rather than a supplement).

Where I disagree:

• I suspect it would be a mistake for EA to shift too much towards always just adopting the expert consensus. As EAs we need to back ourselves, but without becoming overconfident. If EA’s had just deferred to the consensus of development studies experts, EA wouldn’t have gotten off the ground. If EA’s had just deferred to the most experienced animal advocates, that would have biased us towards the wrong interventions. If EA’s had just deferred to ML researchers, we would have skipped over AI Safety as a cause area.

• I don’t think EA is too focused on AI safety. In fact, I suspect that in a few years, we’ll probably feel that we underinvested in it given how fast it’s developing.

• I see value-alignment as incredibly important for a movement that actually wants to get things done, rather than being pulled in several different directions. I agree that it comes with significant risks, such as those you’ve identified, however, I think that we just have to trust in our ability to navigate those risks.

• I agree that we need to seek critiques beyond what the existing red-teaming competition and cause exploration prizes have produced, although I’m less of a fan of your specific proposals. My ideal proposal would be to get a few teams of smart, young EA’s who already have a strong understanding of why things are the way that they are in EA and give them a grant to spend time thinking about how they would construct the norms and institutions of EA if they were building them from the ground up. Movements tend to advance by having the youth break with tradition, so I would favour accelerating this natural process over the suggestions presented.

• While I would love to see EA institutions being able to achieve a broader base of funding, this feels more like something that would be nice to have, rather than something that you should risk disrupting your operations over.

• Voting isn’t a panacea. Countries have a natural answer to who gets to vote—every citizen. I can’t see open internet polls as a good idea due to how easily they can be manipulated, so we’d then require a definition of a member. This would require either membership fees or recording attendance at EA events, so there would be a lot of complexity in making this work.

If 100% of these suggestions were implemented I would expect in 5 years’ time EA to look significantly worse (less effective, helping less people/​animals and possibly having more FTX type scandals).

If the best 10% were implemented I could imagine that being an improvement.

I appreciate the large effort put into this post! But I wanted to throw out one small part that made me distrust it as a whole. I’m a US PhD in cognitive science, and I think it’d be hard to find a top cognitive scientist in the country (e.g., say, who regularly gets large science grants from governmental science funding, gives keynote talks at top conferences, publishes in top journals, etc.) who takes Iain McGilchrist seriously as a scientist, at least in the “The Master & His Emissary” book. So citing him as an example of an expert whose findings are not being taken seriously makes me worry that you handpicked a person you like, without evaluating the science behind his claims (or without checking “expert consensus”). Which I think reflects the problems that arise when you start trying to be like “we need to weigh together different perspectives”. There’s no easy heuristics for differentiating good science/​reasoning from pseudoscience, without incisive, personal inquiry—which is, as far as I’ve seen, what EA culture earnestly tries to do. (Like, do we give weight to the perspective of ESP people? If not, how do we differentiate them from the types of “domain experts” we should take seriously?)

I know this was only one small part of the post, and doesn’t necessarily reflect the other parts—but to avoid a kind of Gell-Mann Amnesia, I wanted to comment on the one part I could contribute to.

Thanks a lot for writing this detailed and thoughtful post, I really appreciate the time you spent on putting this information and thinking together.

So, let’s assume I am a ‘leader’ in the EA community being involved in some of the centralised decision-making you are talking about (which might or might not be true). I’m very busy but came across this post, it seemed relevant enough and I spent maybe a bit less than an hour skim-ish reading it. I agree with the vast majority of the object-level points you make. I didn’t really have time to think about any of the concrete proposals you are making and there are a lot of them, so it seems unlikely I will be able to find the time. However, since—as I said—I broadly agree with lots of what you’re saying I might be interested in supporting your ideas. What, concretely do you want me to do tomorrow? Next week?

I did a close read of “Epistemic health is a community issue.” The part I think is most important that you’re underemphasizing is that, according to the source you cite, “The diversity referred to here is diversity in knowledge and cognitive models,” not, as you have written, diversity “across essentially all dimensions.” In other words, for collective intelligence, we need to pick people with diverse knowledge and cognitive models relevant to the task at hand, such as having relevant but distinct professional backgrounds. For example, if you’re designing a better malaria net, you might want both a materials scientist and an epidemiologist, not two materials scientists.

Age and cultural background might be relevant in some cases, but that really depends on what you’re working on any why these demographic categories seem especially pertinent to the task at hand. If I was designing a nursing home in a team comprised of young entrepreneurs, I would want old people either to be on the team, or to be consulted with routinely as the project evolved, because adding that component of diversity would be relevant ot the project. If I was developing a team to deploy bed nets in Africa, I might want to work with people from the specific villages where they will be distributed.

Worryingly, EA institutions seem to select against diversity. Hiring and funding practices often select for highly value-aligned yet inexperienced individuals over outgroup experts, university recruitment drives are deliberately targeted at the Sam Demographic (at least by proxy) and EA organisations are advised to maintain a high level of internal value-alignment to maximise operational efficiency.

As your own source says:

For a group of diverse agents to work together, an important aspect is cognitive alignment, such as commitment to group goals and shared beliefs.

EA institutions are struggling to find people who are both value-aligned and have relevant diverse cognitive models and knowledge, both of which are prerequisites for collective intelligence. This is a natural problem for EA institutions to have, so you need to explain why EA is making this tradeoff of value alignment vs. cognitive/​knowledge diversity suboptimally for your critique to bite.

The 80,000 Hours website seems purpose-written for Sam, and is noticeably uninterested in people with humanities or social sciences backgrounds,^[14] or those without university education. Unconscious bias is also likely to play a role here – it does everywhere else.

Where your post and the collective intelligence research you’re basing it off seem to diverge is that while you want EA to select for diversity across all dimensions, perhaps for its own sake, the CI research you cited argues that you need to select for forms of cognitive models and knowledge relevant to the task at hand. 80,000 Hours might be wrong in ignoring humanities and social science backgrounds, or those without a university education, but I think your argument would be much stronger if you articulated something specific about what those disciplines would bring to the table.

Whole disciplines exist that are overwhelmingly not value-aligned with EA. Making an effort to include them seems to me like it would add a skillset of questionable task-relevant value while creating fundamental and destructive internal conflict. Because EA does have a specific set of theses on what constitutes “effectiveness” and “altruism,” which we can try to define in the abstract but which can perhaps be better articulated by the concrete types of interventions we tend to support, such as global health and X risk issues. Not everybody supports those kinds of projects, or at least not prioritizing them as highly as we do, or using the kinds of underlying models we use to prioritize them (i.e. the ITN framework), and if they are that far removed from the mission of our movement, then we should probably not try to include them in it.

As other comments have noted, a lot of the proposals seem to be bottlenecked by funding and/​or people leading them.

I would recommend people interested in these things to strongly consider Earning to Give or fundraising, or even just actually donating much more of their existing income or wealth.
If the 10 authors of this post can find other 10 people sympathetic to their causes, and each donate or fundraise on average 50k/​year, they would have $1M/​year of funding for causes that they think are even better than the ones currently funded by EA! If they get better results than existing EA funds people and resources would flock to them!

If you think the current funding allocation is bad, the value of extra funding that you would be able to allocate better becomes much higher.

Especially if you want to work on climate change, I suspect fundraising would be easier than for any other global cause area. Instead of asking Moskovitz/​Openphil for funding, it might be even higher EV to ask Gates, Bezos, or other billionaires. Anecdotally, when I talk to high net-worth people about EA, the first comment is almost always “but what about climate change, which clearly is the most important thing to fund?”

I disagree with a lot of of the content and the general vagueness/​EA Should of this post, but I appreciate the huge effort that went into it.

After reading most of the post, as a person that is in EA to give money/​support, it’s not clear to me how can I help you.

This piece is pretty long and I didn’t think I’d like it, but I put that aside. I think it’s pretty good with many suggestions I agree with.

Thanks for writing it. I guess it wasn’t easy, I know it’s hard to wrestle with communities you both love and disagree with. Thanks for taking the time and energy to write this.

On democratic control:

Any kind of democratic control that tries to have “EAs at large” make decisions will need to decide on who will get to vote. None of the ways I can think of for deciding seem very good to me (donating a certain amount? having engaged a certain amount in a visible way?). I think they’re both bad as methods to choose a group of decisionmakers and more broadly harmful. “You have done X so now you are A Real EA” is the message that will be sent to some and “Sorry, you haven’t done X, so you’re not A Real EA” to others, regardless of the method used for voter selection. I expect that it will become a distraction or discouragement from the actual real work of altruism.

I also worry that this discussion is importing too much of our intuitions about political control of countries. Like most people who live in democracies, I have a lot of intuitions about why democracy is good for me. I’d put them into two categories:

1. Democracy is good for me because I am a better decisionmaker about myself than other people are about me

   1. Most of this is a feeling that I know best about myself: I have the local knowledge that I need to make decisions about how I am ruled

   2. But other parts of it are procedural: I think that when other people decide on my behalf, they’ll arrange things in their own favor

2. Democracy is good for me because it’s deontologically wrong for other people to rule me

I don’t think either of those categories really apply here. EA is not about me, is not for me, does not rule me, and should not take my own desires as a Sam into account any more than it does, probably.

I wasn’t planning on commenting, but since you addressed me by name, I felt compelled to respond.

Strongly agree with the idea that we should stop saying “EA loves criticism”.

I think everyone should have a very strong prior that they are bad at accepting criticism, and everyone should have a very strong prior that they overestimate how good they are at accepting criticism.

In fact, we have good reasons to believe that democratic decisions outperform other kinds, in large part due to the collective intelligence properties we mentioned in previous sections. If the question of the Twitter purchase had been put to the membership or a representatively-sampled assembly of members, what would the outcome have been?

Uh do we? My sense is that democracies are often slow and that EA expert consensus has lead rather than followed the democractic consensus over time. I might instead say “democractic decisions avoid awful outcomes”. I’m not a big reader of papers, but my sense is that democracies avoid wars and famines but also waste a lot of time debating tax policy. I might suggest that EA should feel obliged to explain things to members but that members shouldn’t vote.

Consensus building tools gather the views of many people, identify cruxes, and help build consensus. Pol.is, for instance, has seen significant success when implemented in Taiwan, even on deeply polarised issues. EA could easily employ tools such as these to discover what the membership really believes about certain issues, create better-informed consensus on key issues, and rigorously update our views. Indeed, certain community members have already started doing this.

This on the other hand I do agree with—we could easily have a better sense of what the community thinks, what research could be done etc etc. I guess I’ve run more polis polls than anyone else, but it’s clunks and doesn’t present a clear “what next”.

I talked to the builders of pol.is over the weekend about this, because we were at a conference on collective decisionmaking and Vitalik Buterin funded my dev agency to try and build pol.is for manifold markets and forum magnun (if the forum team accept). So early days but we could see easier use here.

Putting numbers on things is good.

We already do it internally, and surfacing that allows us to see what we already think. Though I agree that we can treat made up numbers too seriously. A number isn’t more accurate than “high likely” it’s more precise. They can both just as easily be mistaken.

I have time for people who say that quantifying feels exhausting or arrogant but I think those are the costs, to be weighed against the precision of using numbers.

You should probably take out the claim that FLI offered 100k to a neo nazi group as it doesn’t seem to be true

Thank you so much for writing this! I don’t have much of substance to add, but this is a great post and I agree with pretty much everything.

We align suspiciously well with the interests of tech billionaires (and ourselves)

So I think this is true of Sam, and ourselves, but I’m really convinced that Dustin defers to OpenPhil more than the other way around (see below). I guess I like Dustin so feel loyalty to him that biases me.

I guess I am wary that I work on things I think are cool and interesting to me. Seems convenient.

I guess my main disagreement with this piece is that I think core EAs do a pretty good job:

• Decisions generally seem pretty good to me (though I guess I am pretty close to the white-male etc artchitype, but still). I think that that community decisionmaking wouldn’t have fixed FTX, the FLI issue or the bostrom email. In fact the criticism of the CEA statement is more from the white-male EA, I guess

• You want people empowered to take quick decisions who can be trusted based on their track record.

• I wish there was more understanding of community opinion and discussing with the community. I have long argued for a kind of virtual copy of many EA orgs that the community can discuss and criticise.

It will take a while to break all of this down, but in the meantime, thank you so much for posting this. This level of introspection is much appreciated.

I strongly agree with this particular statement from the post, but have refrained stating it publicly before out of concern that it would reduce my access to EA funding and spaces.

EAs should consciously separate:

• An individual’s suitability for a particular project, job, or role

• Their expertise and skill in the relevant area(s)

• The degree to which they are perceived to be “highly intelligent”

• Their perceived level of value-alignment with EA orthodoxy

• Their seniority within the EA community

• Their personal wealth and/​or power

I’ve been surprised how many researchers, grant-makers, and community organizers around me do seem to interchange these things. For example, I recently was surprised to hear someone who controls relevant funding and community space access remark to a group “I rank [Researcher X] as an A-Tier researcher. I don’t actually know what they work on, but they just seem really smart.” I found this very epistemically concerning, but other people didn’t seem to.

I’d like to understand this reasoning better. Is there anyone who disagrees with the statement (aka, disagrees that these factors should be consciously separated) who could help me to understand their position?

Thanks for posting this. I have lot’s of thoughts about lots of things, that will take longer to think about. So I start with one of the easier questions.

Regarding pear review, you suggest

• EAs should place a greater value on scientific rigour

  • We should use blogposts, Google Docs, and similar works as accessible ways of opening discussions and providing preliminary thoughts, but rely on peer-reviewed research when making important decisions, creating educational materials, and communicating to the public

  • When citing a blogpost, we should be clear about its scope, be careful to not overstate its claims, and not cite it as if it is comparable to a piece of peer-reviewed research

Have you had any interaction with the academic pear review system? Have you seen some of the stuff that passes though pear review? I’m in favour of scientific rigour, but I don’t think pear review solves that. In reality, my impression is that academia relies as much as name recognition and informal consensus mechanisms, as we (the blogpost community) does. The only reason academia has higher standards (in some fields) is that these fields are older and have developed a consensus around what methods are good enough vs not good enough.

I think pear review have the potential to be good. My impression is that it does really work in math, and that this has a lot to do with that reviewers receive recognition. But in many other fields it mainly serves to slow down research publication, and to gatekeep papers that are not written in the right format, or are not sufficiently interesting.

I did a PhD in theoretical physics and I was not impressed by the pear review responses I got on my paper. It was almost always very shallow comments. Which is not suppressing given that at least in that corner of physics pear review was unpaid and unrecognised work.

• EA institutions should commission peer-reviewed research far more often, and be very cautious of basing decisions on shallow-dives by non-experts

  • For important questions, commission a person/​team with relevant expertise to do a study and subject it to peer review

  • For the most important/​central questions, commission a structured expert elicitation

Can one just do that? Isn’t it very hard to find a person who has the right expertise and who you can verify have the right expertise?

I liked this post, but I’m interested in hearing from someone who disagrees with the authors: Do you think it would be a bad idea to try these ideas, period? Or do you just object to overhauling existing EA institutions?

There were a few points at which the authors said “EA should...” and I’m wondering if it would be productive to replace that with “We, the authors of this post, are planning to… (and we want your feedback on our plans)”

I suppose the place to start would be with some sort of giving circle that operates according to the decisionmaking processes the authors advocate. I think this could generate a friendly and productive rivalry with EA Funds.

Thanks for the time you’ve put into trying to improve EA, and it’s unfortunate that you feel the need to do so anonymously!

Below are some reactions, focused on points that you highlighted to me over email as sections you’d particularly appreciate my thoughts on.

On anonymity—as a funder, we need to make judgments about potential grantees, but want to do so in a way that doesn’t create perverse incentives. This section of an old Forum post summarizes how I try to reconcile these goals, and how I encourage others to. When evaluating potential grantees, we try to focus on what they’ve accomplished and what they’re proposing, without penalizing them for holding beliefs we don’t agree with.

• I understand that it’s hard to trust someone to operate this way and not hold your beliefs against you; generally, if one wants to do work that’s only a fit for one source of funds (even if those funds run through a variety of mechanisms!), I’m (regretfully) sympathetic to feeling like the situation is quite fragile and calls for a lot of carefulness.

• That said, for whatever it’s worth, I believe this sort of thing shouldn’t be a major concern w/​r/​t Open Philanthropy funding; “lack of output or proposals that fit our goals” seems like a much more likely reason not to be funded than “expressed opinions we disagree with.”

On conflicts of interest: with a relatively small number of people interested in EA overall, it doesn’t feel particularly surprising to me that there are a relatively small number of particularly prominent folks who are or have been involved in multiple of the top organizations. More specifically:

• Since Open Philanthropy funds a large % of the orgs focused on our priority issues, it doesn’t seem surprising or concerning that many of the people who’ve spent some time working for Open Philanthropy have also spent some time working for Open Philanthropy grantees. I think it is generally common for funders to hire people who previously worked at their grantees, and in turn for ex-employees of funders to leave for jobs at grantees.

• It doesn’t seem surprising or concerning that people who have written prominent books on EA-connected ideas have also helped build community infrastructure organizations such as the Centre for Effective Altruism.

To be clear, I think it’s important for conflicts of interest to be disclosed and handled appropriately, and there are some conflicts of interest that concern me for sure—I don’t at all mean to minimize the importance of conflicts of interest or potential concerns around them. I still thought it was worth sharing those reactions to the specific takes given in that section of the post.

On our concentration on a couple of existential risks: here I think we disagree. OP works on a wide variety of causes, but I don’t think we should be diversifying more than we are *within* existential risk given our picture of the size and neglectedness of the different risks.

On being in line with the interests of billionaires: I understand the misgivings that people have about EA being so reliant on a small number of funders, and address that point below. And I understand skepticism that funders who have made their wealth in the technology industry have only global impact in mind when they focus their philanthropy on technology issues. For what it’s worth, in the case of the particular billionaires I know best, Cari and Dustin were pretty emotionally reluctant to work on x-risks (and I was as well) - this felt at least to me like a case of them reluctantly concluding that these are important issues rather than coming in with pet causes.

On centralization of funding: I’m having trouble operationalizing the calls for less centralization of funding decision-making, which seems to be the main driver of much of your concerns. I agree that heavy concentration of funding for a given area brings some concerns that would be reduced if the same amount of funding were more spread out among funders; but I haven’t seen an alternative funding mechanism proposed that seems terribly promising.

I was broadly in sync with Dustin’s thoughts here, though not saying I’d endorse every word. I don’t see a good way to define the “members” of EA without keeping a lot of judgment/​discretion over what counts (and thus keeping the concentration of power around), or eroding the line between EA and the broader world with its very different priorities. To me, it looks like EA is fundamentally a self-applied label for a bunch of individuals making decisions using an intellectual framework that’s both unusual and highly judgment-laden; I think there are good and bad things about this, but I haven’t seen a way to translate it into more systematic or democratic formal structures without losing those qualities.

I’m not confident here and don’t pretend to have thought it fully through. I remain interested in suggestions for approaches to spending Cari’s and Dustin’s capital that could improve how it’s spent—the more specific and mechanistic, the better.

This post has convinced me to stay in the EA community. If I could give all the votes I have given to my own writings to this post, I would. Many of the things in this post I’ve been saying for a long time (and have been downvoted for) so I’m happy to see that this post has at least a somewhat positive reaction.

To add to what this post outlines. While the social sciences are often ignored in the EA community one notable exception to that is (orthodox) economics. I find it ironic that one of the few fields where EA’s are willing to look outside their own insular culture, is in itself extremely insular. Other social studies like philosophy, political science, history, sociology, gender studies all make a lot of attempts to integrate themselves with all the other social sciences. This makes it so that learning about one discipline also teaches you about the other disciplines. Economists meanwhile have a tendency to see their discipline as better than the others starting papers with things like:

Economics is not only a social science, it is a genuine science. Like the physical sciences, economics uses a methodology that produces refutable implications and tests these implications using solid statistical techniques.

In the paper “The Superiority of Economists” by Fourcade et al, economists were found to be the only group that thought interdisciplinary research was worse than research from a singular field. Furthermore they looked at top papers from political science, economics and sociology. They found that political science and sociology cited economics papers many times more than the other way around:

This lack of citing other social sciences was later confirmed by Angrist et al:

Take for example supply and demand. The standard supply and demand model will tell you that having/​increasing the minimum wage will automatically increase unemployment. But if we look at actual empirical evidence it shows us that it doesn’t. Learning the basics of economics might mislead people about which policies will actually help people and a more holistic look at the social sciences as a whole may counter that.

By focussing our recruitment on only a couple disciplines we inherit the problems of those disciplines. It is no wonder then that just like the EA community, economics is also very homogeneous. Bayer and Rouse show us that women are given less degrees in economics than other disciplines:

And get only 13.7% of the authorship of economics papers. Women are also less likely to get tenure in their first academic job compared to men and face a lot of discrimination in general. An AEA survey found that half of women say they were treated unfairly because of their sex and almost half say they’ve avoided conferences/​seminars because of fear of harassment.
One of the only other social studies that is popular in EA is philosophy, which has similar issues with underrepresentation and discrimination.

For the longest time the EA topics page had a segment with ‘key figures’. This is obviously bad if you want to combat ‘hero worship’. My memory might be off but I seem to remember that the only social studies that were present were philosophy and economics, and practically everyone on that page was a man. [EDIT: Found an image, my memory was pretty close and in fact that page still exists! AND SBF WAS STILL ON IT!]

Even now we have a tag for philosophy and economics, but not for sociology, anthropology, political science, gender studies or many other social studies.

These are also the two social studies that the rationalists are the most interested in. The accusation of ‘bad epistemics’ seems to coincide a lot with ‘non-rationalist epistemic’. I’ve wanted to push back on the dominant rationalists framework for quite a while now, and one post in particular I wanted to write is an attack on scientific-realism and a defense of social constructivism. I never did it because I feared it would get downvoted, nominally because it’s not directly ‘effective altruism’. But the hidden premises in how we research, how we categorize the world and what gets to count as ‘genuine science’ have a huge effect on what gets to count as ‘effective altruism’.

Personally I think this problem boils down to how effective altruism represents itself, and how it is actually governed.

For instance in my own case, I became interested in effective altruism and started getting more involved in the space with the idea that it was a loose collection of aligned, intelligent people who want to take on the world’s most pressing problems. Over time, I’ve realize that like the post mentions, effective altruism is in fact quite hierarchical and not disposed to giving people a voice based solely on the amount of time and effort they put in the movement.

Admittedly, this is a pretty naive view to take when going into any social movement.

While I am sympathetic to the arguments that a small tightly knit group can get things done more quickly and more efficiently, there is a bit of a motte and bailey going on between the leadership of effective altruism and the recruiting efforts. From my perspective a lot of new folks that join the movement are implicitly sold on a dream, commit time and energy, then are not given the voice they feel they’ve earned.

Whether or not a more democratic system would be more effective, I still think many of the internal problems that have been surfacing recently would be fixed with better communication within effective altruism about how we make decisions and who has influence.

EA seems to have a bit of a “not invented here” problem, of not taking onboard tried and tested mechanisms from other areas. E.g. with the boring standard conflict of interest and transparency mechanisms that are used by charitable organisations in developed countries.

Part of this seems to come from only accepting ideas framed in certain ways, and fitting cultural norms of existing members. (To frame it flippantly, if you proposed a decentralised blockchain based system for judging the competence of EA leaders you’d get lots of interest, but not if you suggested appointing non-EA external people to audit.)

There might be some value to posts taking existing good practices in other domains and presenting them in ways that are more palatable to the EA audience, though ideally you wouldn’t need to.

Background
First, I want to say that I really like seeing criticism that’s well organized and presented like this. It’s often not fun to be criticized, but the much scarier thing is for no one to care in the first place.

This post was clearly a great deal of work, and I’m happy to see so many points organized and cited.

I obviously feel pretty bad about this situation where, several people all felt like they had to do this in secret in order to feel safe. I think tensions around these issues feel much more heated than I’d like them to. Most of the specific points and proposals seem like things that in a slightly different world, all sides could feel much more chill discussing.

I’m personally in a weird position, where I don’t feel like one of the main EAs who make decisions (outside of maybe RP), but I’ve been around for a while and know some of them. I did some grantmaking, and now am working on an org that tries to help figure out how to improve community epistemics (QURI).

Some Quick Impressions
I think one big division I see in discussions like this, is that between:

1. What’s in the best interest of EA leadership/​funding, conditional on them not dramatically changing their beliefs about key things (this might be very unlikely).

2. What’s an ~irreconcilable different opinion (with a reasonable time of debate/​investigation, say, a few days of solid reading).

Bucket 1 is more about convincing and informing each other. The way to make progress there is by deeply understanding those with power, and explaining how it helps their goals.

Bucket 2 is more about relative power. No two people are perfectly aligned, even after years of deliberation. Frustratingly, the main ways to make progress here are to either move it from some players to others, or doing things like just making power moves (taking actions that help your interests, in comparison to other stakeholders).

Right now, in EA, the vast majority of funding (and thus control) ultimately comes from one source. This is a really uncomfortable position, in many ways.

However, other members of the community clearly have some power. They could do some nice things like write friendly posts, or some not so nice things (think of strikes) like leaking information or complaining/​ranting to antagonistic journalists.

I imagine that eventually we could find better ways to do group bargaining, like some sort of voting system (similar to what you recommend).

Back to this post, I think that some of the way this post is written reminds me of “lists of demands” that I’m used to seeing in fairly antagonistic negotiations, in the style of Bucket 2.

My guess is that this wasn’t your intention. Given that it’s so long (and must have involved a lot of coordination to write), I could definitely sympathize with “let’s just get it out there” instead of making sure it’s style is optimized for Bucket 1 (if that was your intention). That said, If I were a grantmaker now, I could easily see myself putting this in my “some PR fire to deal with” bucket rather than “some useful information for me to eventually spend time with”.

I think I’m probably sympathetic to your claims in “EA is open to some kinds of critique, but not to others”, but I think it would be helpful for there to be some discussion around Scott Alexander’s post on EA criticism. In it, he argued that “EA is open to some kinds of critique, but not to others” was an inevitable “narrative beat”, and that “shallow” criticisms which actually focus on the more actionable implications hit closer to home and are more valuable.

I was primed to dismiss your claims on the basis of Scott Alexander’s arguments, but on closer consideration I suspect that might be too quick.

I feel it would be easier for me to judge this if someone (not necessarily the authors of this post) provided some examples of the sorts of deep critiques (e.g. by pointing to examples of deep critiques made of things other than EA). The examples of deep critiques given in the post did help with this, but it’s easier to triangulate what’s really meant when there are more examples.

I think one of the reasons I loved this post is that my experience of reading it echoed in an odd way my own personal journey within EA. I remember thinking even at the start of EA there was a lack of diversity and a struggle at accept “deep critiques”. Mostly this did not affect me – until I moved into an EA longtermist role a few years ago. Finding existing longtermist research to be lacking for the kind of work I was doing I turned to the existent disciplines on risk (risk management, deep uncertainty, futures tool, etc). Next thing I know a disproportionately large amount of my time seemed to be being sunk into trying and failing to get EA thinkers and funders take seriously governance issues and those aforementioned risk disciplines. Ultimately I gave up and ended up partly switching away from that kind of work. Yet despite all this I still find the EA community to be the best place for helping me mend the world.

I loved your post but I want to push back on one thing – these problems are not only in the longermist side of EA. Yes neartermist EA is epistemically healthier (or at minimum currently having less scandals), but that there are still problems and we should still be self reflective and looking to learn from posts like this and to consider if there are issues around: diversity of views, limited funding to high impact areas due to over centralisation, rejection of deep critiques, bad actors, and so on. As one example consider the (extremely laudable) criticism contents from GiveWell which was focused heavily on looking at how their quantitative analyses are 10% inaccurate, but not finding ways to highlight where their approach might be fundamentally failing to make good decisions. [section edited]

PS. One extra idea for the idea list: run CEA (or other EA orgs) on a cooperative model where every donor/​member gets a vote on key issues or leadership decisions.

Thanks for this thoughtful and excellently written post. I agree with the large majority of what you had to say, especially regarding collective vs. individual epistemics (and more generally on the importance of good institutions vs. individual behavior), as well as concerns about insularity, conflicts of interest, and underrating expertise and overrating “value alignment”. I have similarly been concerned about these issues for a long time, but especially concerned over the past year.

I am personally fairly disappointed by the extent to which many commenters seem to be dismissing the claims or disagreeing with them in broad strokes, as they generally seem true and important to me. I would value the opportunity to convince anyone in a position of authority in EA that these critiques are both correct and critical to address. I don’t read this forum often (was linked to this thread by a friend), but feel free to e-mail me (jacob.steinhardt@gmail.com) if you’re in this position and want to chat.

Also, to the anonymous authors, if there is some way I can support you please feel free to reach out (also via e-mail). I promise to preserve your anonymity.

We should probably read more widely

Summary: EA reading lists are typically narrow, homogenous, and biased, and EA has unusual social norms against reading more than a handful of specific books. Reading lists often heavily rely on EA Forum posts and shallow dives over peer-reviewed literature. EA thus remains intellectually insular, and the resulting overconfidence makes many attempts by external experts and newcomers to engage with the community exhausting and futile. This gives the false impression that orthodox positions are well-supported and/​or difficult to critique.

Another option I like is to have the wiki include pages on “feminism”, “psychology” etc with summaries written in EA language of things people have found most valuable. I would read those.

You argue that funding is centralised much more than it appears. I find myself learning that this is the case more and more over time.

I suspect it probably is good to decentralise to some degree, however there is a very real downside to this:

• some projects are dangerous and probably shouldn’t happen

• the most dangerous of those are ones run by a charismatic leader and appear very good

• if we have multiple funders who are not “informally centralised” (i.e. talking to each other) then there’s a risk that dangerous projects will have multiple bites at the cherry, and with enough different funders, someone will fund them

I appreciate that there are counters to this, and I’m not saying this is a slam-dunk argument against decentralisation.

I appreciate the extent of thoughtful consideration that has been put into this post. I looked at the list of proposed reforms to consider which ones I should implement in my (much smaller) organisation.

I currently find it very difficult to weigh the benefits and costs of this entire list. I understand that a post shouldn’t be expected to do everything at once. But I would really appreciate it if someone explained which of these specific policies are standard practices in other contexts like universities/​political parties/​other NGOs.

Small point:
> Finally, we ask that people upvote or downvote this post on the basis of whether they believe it to have made a useful contribution to the conversation, rather than whether they agree with all of our critiques.

I think this makes a false dilemma, and recommends what seems like an unusual standard that other posts probably don’t have.

“believe it to have made a useful contribution to the conversation” → This seems like arguably a really low bar to me. I think that many posts, even bad ones, did something useful to the conversation.

“whether they agree with all of our critiques.” → I never agree with all of basically any post.

I think that more fair standards of voting would be things more like:
”Do I generally agree with these arguments?”
″Do I think that this post, as a whole, is something I want community members to pay attention to, relative to other posts?”

Sadly we don’t yet have separate “vote vs. agreement” markers for posts, but I think those would be really useful here.

More specifically, EA shows a pattern of prioritising non-peer-reviewed publications – often shallow-dive blogposts^[36] – by prominent EAs with little to no relevant expertise.

This is my first time seeing the “climate change and longtermism” report at that last link. Before having read it, I imagined the point of having a non-expert “value-aligned” longtermist applying their framework to climate change would be things like

• a focus on the long-run effects of climate change

• a focus on catastrophic scenarios that may be very unlikely but difficult to model or quantify

Instead, the report spends a lot of time on

• recapitulation of consensus modeling (to be clear, this is a good thing that’s surprisingly hard to come by), which mainly goes out to 2100

• plausible reasons models may be biased towards negative outcomes, particularly in the most likely scenarios

The two are interwoven, which weakens the report even as a critical literature review. When it comes to particular avenues for catastrophe, the analysis is often perfunctory and dismissive. It comes off less as a longtermist perspective on climate change than as having an insider evaluate the literature because only “we” can be trusted to reason well.

I don’t know how canonical that report has become. The reception in the thread where it was posted looks pretty critical, and I don’t mean to pile on. I’m commenting because this post links the report in a way that looks like a backhanded swipe, so once I read it myself I felt it was worth sketching out my reaction a bit further.

Context: I’ve worked in various roles at 80,000 Hours since 2014, and continue to support the team in a fairly minimal advisory role.

Views my own.

I agree that the heavy use of a poorly defined concept of “value alignment” has some major costs.

I’ve been moderately on the receiving end of this one. I think it’s due to some combination of:

1. I take Nietzsche seriously (as Derek Parfit did).

2. I have a strong intellectual immune system. This means it took me several years to get enthusiastically on board with utilitarianism, longtermism and AI safety as focus areas. There’s quite some variance on the speed with which key figures decide to take an argument at face value and deeply integrate it into their decision-making. I think variance on this dimension is good—as in any complex ecosystem, pace layers are important.

3. I insisted on working mostly remotely.

4. I’ve made a big effort to maintain an “FU money” relationship to EA community, including a mostly non-EA friendship group.

5. I am more interested in “deep” criticism of EA than some of my peers. E.g. I tweet about Peter Thiel on death with dignity, Nietzsche on EA, and I think Derek Parfit made valuable contributions but was not one of the Greats.

6. Some of my object-level views have been quite different to those of my peers over the years. E.g. I’ve had reservations about the maximisation meme ever since I got involved, along the lines of those recently expressed by Holden.

7. I mostly quit 80,000 Hours in autumn 2015, mainly due to concerns about strategy, messaging and fit with my colleagues. I took on a 90% time role again in 2017, for roughly 4 years.

8. I have some beliefs and traits that some people find suggestive of a lack of moral seriousness (e.g. I’m into two-thirds utilitarianism; I don’t lose sleep over EA/​LT concerns; I’m fairly normie by EA standards).

There are some advantages to the status quo, and I don’t have a positive proposal for improving this.

If someone at 80K or CEA wants to pay me to think about it for a day, I’d be up for that. Maybe I’ll do it anyway. I hesitate because I’m not sure how tractable this is, and I would not be surprised if, on further reflection, I came to think the status quo is roughly optimal at current margins.

Reminder for many people in this thread:

“Having a small clique of young white STEM grads creates tons of obvious blindspots and groupthink in EA, which is bad.”

is not the same belief as

“The STEM/​techie/​quantitative/​utilitarian/​Pareto’s-rule/​Bayesian/​”cold” cluster-of-approaches to EA, is bad.”

You can believe both. You can believe neither. You can believe just the first one. You can believe the second one. They’re not the same belief.

I think the first one is probably true, but the second one is probably false.

Thinking the first belief is true, is nowhere near strong enough evidence to think the second one is also true.

(I responded to… a couple similar ideas here.)

This post is much too long and we’re all going to have trouble following the comments.

It would be much better to split this up and post as a series. Maybe do that, and replace this post with links to the series?

Thanks for all the care and effort which went into writing this!

At the same time, while reading, my reactions were most of the time “this seems a bit confused”, “this likely won’t help” or “this seems to miss the fact that there is someone somewhere close to the core EA orgs who understands the topic pretty well, and has a different opinion”.

Unfortunately, to illustrate this in detail for the whole post would be a project for …multiple weeks.

At the same time I thought it could be useful to discuss at least one small part in detail, to illustrate how the actual in-the-detail disagreement could look like.

I’ve decided to write a detailed response for a few paragraphs about rationality and Bayesianism. This is from my perspective not cherry-picked part of the of the original text which is particularly wrong, but a part which seems representatively wrong/​confused. I picked it for convenience, because I can argue and reference this particularly easily.

Individual Bayesian Thinking (IBT) is a technique inherited by EA from the Rationalist subculture, where one attempts to use Bayes’ theorem on an everyday basis. You assign each of your beliefs a numerical probability of being true and attempt to mentally apply Bayes’ theorem, increasing or decreasing the probability in question in response to new evidence. This is sometimes called “Bayesian epistemology” in EA, but to avoid confusing it with the broader approach to formal epistemology with the same name we will stick with IBT.

This seems pretty strange characterization. Even though I participated at multiple CFAR events, do teach various ‘rationality techniques’ and know decent amount of stuff about Bayesian inference, I think this is misleading/​confused.

What’s called in rationalist circles “Bayesian epistemology” is basically what’s the common understanding of the term:
- you don’t hold beliefs to be true or false, but have credences in them
- normatively, you should update the credences based on evidence. normatively, the proper rule for that is the Bayes rule; this is intractable in practice, so you do various types of approximations
- you should strive for coherent beliefs if you don’t want to be Dutch-booked

It’s important to understand that in this frame, it is a normative theory. Bayes theorem in this perspective is not some sort of “minor aid for doing some sort of likelihood calculation” but a formal foundation for large part of epistemology.

The view that you believe different things to different degrees, and these credences basically are Bayesian probabilities and are normatively governed by the same theory isn’t an ‘EA’ or ‘rationalist’ thing but a standard Bayesian take. (cf Probability Theory: The Logic of Science, E. T. Jaynes)

Part of what Eliezer’s approach to ‘applied rationality’ aimed for is taking Bayesian epistemology seriously and applying this frame to improve everyday reasoning.

But this is almost never done by converting your implicit probability distributions to numerical credences , doing the explicit numerical math, and blindly trusting the result!

What’s done instead is
-noticing that your brain already internally used credences and probabilities all the time. you can easily access your “internal” (/​S1/​...) probabilities in an intuitive way by asking yourself questions like “how surprised you would be to see [a pink car today| Donald Trump reelection | SpaceX rocket landing in your backyard]” (the idea that brains do this is pretty mainstream eg Confidence as Bayesian Probability: From Neural Origins to Behavior, Meyniel et.al.)
- noticing you brain often clearly does something a bit similar to what Bayesians suggests as the normative idea - e.g. if two SpaceX rockets already landed in your backyard today, you would be way less surprised by the third one
- noticing there is often a disconnect between this intuitive /​ internal /​ informal calculation, and the explicit, verbal reasoning (cf alief/​belief)
- …and using all of that to improve both the “implicit” and “explicit” reasoning!

The actual ‘techniques’ derived from this are often implicit. For example, one actual technique is: Imagine you are an alien who landed in one world of two. They differ in the respect that in one a proposition is true, in the other, the opposite is true. You ask yourself how the world would look like in the different worlds, and then look at the actual world.

For example, consider this proposition: “democratic vote is the optimal way how to make decision making in organizations”: How would the world where this is true look like? There are parts of the world with intense competition between organizations, e.g. companies in highly competitive industries, optimizing hard and measurable things. In the world where the proposition is true, I’d expect a lot of voting in these companies. We don’t see that, which decreases my credence in the proposition.

It is relatively easy to see how this both connected to Bayes and not asking people to do any explicit odd multiplications.

There is nothing wrong with quantitative thinking, and much of the power of EA grows from its dedication to the numerical. However, this is often taken to the extreme, where people try to think almost exclusively along numerical lines, causing them to neglect important qualitative factors or else attempt to replace them with doubtful or even meaningless numbers because “something is better than nothing”. These numbers are often subjective “best guesses” with little empirical basis.^[27]

While some people make the error of trying to replace complex implicit calculations by the over-simplified spreadsheets with explicit numbers, this paragraph seems to conflate multiple things together as “numerical” or “quantitative”.

Assuming fairly standard standard cognitive science and neuroscience, at some level all thinking is “numerical”, including thinking which feels intuitive or qualitative. People usually express such thinking in words like “I strongly feel” or “I’m pretty confident”

What’s a classical rationalist move in such cases is to try to make the implicit explicit. E.g., if you are fairly confident, at what odds would you be willing to bet on it?

When done correctly, the ability and willingness to do that mostly exposes what’s already there. People already act based on the implicit credences and likelihoods, even if they don’t try to express them as probability distributions, and you don’t have access to them.

E.g., when some famous experts recommended ‘herd immunity’ strategy to deal with covid, using strong and confident words, such recommendation actually were subjective “best guess” with little empirical basis. Same is actually true for many expert opinions on policy topics!

Rationalist habit of reporting credences and predictions using numbers … basically exposes many things to possibility of being proved wrong, and exposes many personal best guesses for what they are.

Yes, for someone who isn’t used to this at all this may create fake aura of ‘certainty’, because use of numbers often signals ‘this is more clear’ and use of words signals ‘this is more slippery’ in common communication. But this is just a communication protocol.

Yes, as I wrote before, some people may make the mistake of trying to convert some basic things to numbers and replace their brains with spreadsheets with Bayes formulas in the next step, but it does not seem common at least in my social neighborhood.

For instance, Bayesian estimates are heavily influenced by one’s initial figure (one’s “prior”), which, especially when dealing with complex, poorly-defined, and highly uncertain and speculative phenomena, can become subjective (based on unspecified values, worldviews, and assumptions) to the point of arbitrary.^[28] This is particularly true in existential risk studies where one may not have good evidence to update on.

I would be curious how the authors imagine the non-Bayesian thinking not depending on any priors internally works.

We assume that, with enough updating in response to evidence, our estimates will eventually converge on an accurate figure. However, this is dependent on several conditions, notably well-formulated questions, representative sampling of (accurate) evidence, and a rigorous and consistent method of translating real-world observations into conditional likelihoods.^[29] This process is very difficult even when performed as part of careful and rigorous scientific study; attempting to do it all in your head, using rough-guess or even purely intuitional priors and likelihoods, is likely to lead to more confidence than accuracy.

This seems confused (the “common response” mentioned below applies here exactly). How do you imagine, for example, a group of people looking at a tree, manages to agree on seeing a tree? The process of converting raw sensory data to the tree-hypothesis is way more complicated than a typical careful and rigorous scientific study, and also way more reliable than a typical published scientific study.

Again: correctly understood, the applied rationalist idea is not to replace our mind’s natural ways of recognizing a tree by a process where you would assign numbers to statements like “green in upper left part of visual field” and do explicit Bayesian calculation in S2 way, but just to be …less wrong.

This is further complicated by the fact that probabilities are typically distributions rather than point values – often very messy distributions that we don’t have nice neat formulae for. Thus, “updating” properly would involve manipulating big and/​or ugly matrices in your head. Perhaps this is possible for some people.

A common response to these arguments is that Bayesianism is “how the mind really works”, and that the brain already assigns probabilities to hypotheses and updates them similarly or identically to Bayes’ rule. There are good reasons to believe that this may be true. However, the fact that we may intuitively and subconsciously work along Bayesian lines does not mean that our attempts to consciously “do the maths” will work.

I think the “common response” is partially misunderstood here? The common response does not imply you can consciously explicitly multiply the large matrices or do the exact Bayesian inferences, any more than someone a catching a ball would be consciously and explicitly solving the equations of motion.

The correct ideas here are:
- you can often make some parts or results of the implicit subconscious calculations explicit and numeric (cf forecasting, betting, …)
- the implicit reasoning is often biased and influenced by wishes and wants
- explicitly stating things or betting on things sometimes exposes the problems
- explicit reasoning can be good for that
- explicit reasoning is also good for understanding what the normatively good move is in simple or idealized cases
- on the other hand, explicit reasoning alone is computationally underpowered for almost anything beyond very simple models. (compare how many FLOPs is your brain using, vs. how fast you can explicitly multiply numbers)
- what you usually need to do is use both, and watch for flaws

In addition, there seems to have been little empirical study of whether Individual Bayesian Updating actually outperforms other modes of thought, never mind how this varies by domain. It seems risky to put so much confidence in a relatively unproven technique.

Personally I don’t know anyone who would propose people should do the “Individual Bayesian Thinking” mode of thoughts in the way you describe, and I don’t see much reason to make a study on this. Also while a lot people in EA orgs subscribe to basically Bayesian epistemology, I don’t know anyone who would try to live by the “IBT”, so you should probably be less worried about the risks from the use of it.

The process of Individual Bayesian Updating can thus be critiqued on scientific grounds,

So, to me, this is characteristic—and, frankly, annoying—about the whole text. I don’t think you have properly engaged with Bayesian epistemology, state of art applied rationality practice, or relevant cognitive science. “critiqued on scientific grounds” sounds serious and authoritative … but where is the science?

but there is also another issue with it and hyper-quantitative thinking more generally: motivated reasoning. With no hard qualitative boundaries and little constraining empirical data, the combination of expected value calculations and Individual Bayesian Thinking in EA allows one to justify and/​or rationalise essentially anything by generating suitable numbers.

This is both sad and funny. One of the good things about rationalist habits and techniques is, stating explicit numbers often allows one to spot and correct motivated reasoning. In relation to existential risk and similar domains, often the hope is that by practicing this in domains with good feedback and bets which are possible to empirically evaluate, you get better at thinking clearly… and this will at least partially generalize to epistemically more challenging domains.

Yes, you can overdo it, or do stupid or straw versions of this. Yes, it is not perfect.

But what’s the alternative? Honestly, in my view, in many areas of expertise the alternative is to state views, claims and predictions in sufficiently slippery and non-quantitative way that it is very difficult to clearly disprove them.

Take for example your text and claims about diversity. I think given the way you are using it, it seems anyone trying to refute the advise on empirical grounds would have really hard time, and you would be always able to write some story why some dimension of diversity is not important, or why some other piece of research states something else. (It seems a common occurrence in humanities that some confused ideas basically never die, unless they lose support on the level of ‘sociology of science’.)

Bottom line:
- these 8 paragraphs did not convince me about any mistake people at e.g. FHI may be making
- suggestion “Bayes’ theorem should be applied where it works” is pretty funny; I guess Bayesians wholeheartedly agree with this!
- suggestions like “studies of circumstances under which individual subjective Bayesian reasoning actually outperforms other modes of thought” seems irrelevant given the lack of understanding of actual ‘rationality techniques’
- we have real world evidence that getting better at some of the traditional “rationalist” skills makes you better at least at some measurable things, e.g. forecasting

I suspect … that even what I see as a wrong model of ‘erros of EA’ maybe points to some interesting evidence. For example maybe some EA community builders are actually teaching “individual bayesian thinking” as a technique you should do in the way described?

I agree that we shouldn’t pretend to be particularly good at self-criticim. I don’t think we are. We are good at updating numbers, but I have giving criticism to orgs that’s not been acted on for years before someone saying it was a great idea. Honestly I’d have preferred if they just told me they didn’t want criticism than saying they did then ignoring it.

I find it very concerning/​dissapointing that this post has so many downvotes. Like the authors said, don’t use upvotes/​downvotes to indicate agreement/​disagreement!

I strongly upvoted this post because I think it’s very valuable and I highly appreciate the months of effort put into it. Thanks for writing it! I don’t know whether I agree or not on most proposals, but I think they’re interesting nonetheless.

Personally I’m most skeptical of some of the democratization proposals. Like how would you decide who can vote? And I think it would drastically slow down grant making etc., making us less effective. Others have already better worded these concerns elsewhere.

In general I would love to see most of these ideas being tried out. Either incrementally so we can easily revert back, or as small experiments. Even with ideas I’m only say 30% sure they will work. If the ideas fail, then at least we’ll have that information.

I very much agree that we should rely more on other experts and try to reinvent the wheel less.
I had indeed never heard of the fields you mentioned before, which is sad.

I didn’t know your votes on the forum had more power the more karma you had! I’m really surprised. I was already wondering whether my strong votes had increased recently. I’m not sure I agree strong votes should be removed, I do think they’re really useful in expressing how much I appreciate a certain post. But the fact that your strong votes keep becoming more powerful the more karma you have is weird. I do think it makes sense to give more longtime/​experienced EA’s more voting power to make sure the forum doesn’t become low-quality. But perhaps it can simply be that new users’ strong vote equals two votes, and the moment you get say 50-100 karma it equals 4-5 votes. But after that it doesn’t keep increasing.

I’m wondering how true the ‘heretics/​critics don’t get funding’ is. Personally I aim to be honest/​vocal about concerns I have, but since I don’t have highly valuable skills (according to the economic market) perhaps I should be more careful? I don’t want to and don’t feel like I should be more careful! And I hope I can trust EA leaders when they say they want more diversity of thought.

This reads more like a list of suggestions than an argument. You’re making twenty or thirty points, but as far as I’ve read (I admit I didn’t get thru the whole thing) not giving any one of them the level of argumentation that would unpack the issue for a skeptic. There’s a lot that I could dispute, but I feel like arguing things here is a waste of time; it’s just not a good format to discuss so many issues in a single comment section.

I will mention one thing:

The EA community is notoriously homogenous, and the “average EA” is extremely easy to imagine: he is a white male^[9] in his twenties or thirties from an upper-middle class family in North America or Western Europe. He is ethically utilitarian and politically centrist; an atheist, but culturally protestant. He studied analytic philosophy, mathematics, computer science, or economics at an elite university in the US or UK. He is neurodivergent. He thinks space is really cool. He highly values intelligence, and believes that his own is significantly above average. He hung around LessWrong for a while as a teenager, and now wears EA-branded shirts and hoodies, drinks Huel, and consumes a narrow range of blogs, podcasts, and vegan ready-meals. He moves in particular ways, talks in particular ways, and thinks in particular ways. Let us name him “Sam”, if only because there’s a solid chance he already is.^[10]

It’s one thing to argue that organizations function more efficiently when they are more diverse, or to say that it’s disproportionately important that we recruit different types of people, but rattling off a list of attributes to make a negative stereotype with a dumb joke at their expense makes it extremely difficult to take you seriously. When you write something like this, it is not obvious that you are doing so with inoffensive intentions, and even if you don’t have offensive intentions the propagation of negative stereotypes may cause harm all the same. If you actually believe in tolerance and equality then you would do well to take five minutes out of your day to clarify that there is nothing wrong with the things on your list, in individual or in aggregate, and that you value such people equally within the EA movement.

Also, given the current and irreversible trend towards capitalization of Black, it is prudent to capitalize White as well.

I would appreciate a TL;DR of this article, and I am sure many others would too! It helps me to decide if it’s worth spending more time digging into the content.

It was even too long for chatGPT to summarize 🫠

Thank you so much for writing this, it feels like a breath of fresh air. There are a huge number of points in here that I strongly agree with. A lot of them I was thinking of writing up into full length posts eventually (I still might for a few), but I was unsure if anyone would even listen. This must have taken an immense amount of effort and even emotional stress, and I think you should be proud.

I think if EA ever does break out of its current mess, and fulfills it’s full potential as a broad, diverse movement, then posts like this are going to be the reason why.

(It feels important to disclaim before commenting that I’m not an EA, but am very interested in EA’s goal of doing good well.)

Thank you!! This post is a breath of fresh air and makes me feel hopeful about the movement (and I’m not just saying that because of the Chumbawamba break in the middle). In particular I appreciated the confirmation that I’m not the only person who has read some of the official EA writings regarding climate change and thought (paraphrasing here) ”?!!!!!!!!!”

I know this cannot have been trivial or relaxing to write. A huge thank you to the authors. I really hope that your suggestions are engaged with by the community with the respect that they deserve.

We should use blogposts, Google Docs, and similar works as accessible ways of opening discussions and providing preliminary thoughts, but rely on peer-reviewed research when making important decisions, creating educational materials, and communicating to the public

Bold type mine. No I think peer review is cumbersome and that to use it would slow work down a lot. Are there never mistakes in peer reviewed science? No, there are. I think we should aim to build better systems of peer review.

Epistemic status: Hot take

networking retreats taking place in the Bahamas

To me, FTX hosting EA events and fellowships in the Bahamas reeked of neocolonialism (which is a word I don’t like to bandy about willy-nilly). 90.6% of The Bahamas’ population are Black,^[1] whereas the majority of EAs are White.^[2] Relative to many countries in the Americas, The Bahamas has a superficially strong economy with a GDP (PPP) per capita of $40,274, but a lot of this economic activity is due to tourism and offshore companies using it as a tax haven,^[1] and it’s unclear to me how much of this prosperity actually trickles down to the majority of Bahamians. (According to UN data, The Bahamas has a Gini coefficient of 0.57, the highest in the Caribbean.^[3]) Also, I’ve never heard anyone talk about recruiting Bahamians into the EA movement or EA orgs.

1. ^

   The Bahamas—Wikipedia

2. ^

   Every EA community survey ever, e.g. EA Survey 2020

3. ^

   Inequality in the Bahamas—Etonomics

I have lots of things I want to say, but I will not say it publicly, because I’m currently working for a project that is dependent on EA Funding. I can’t risk that. Even though I think this conversation is important, I think it is even more important that I can continue the work that I’m doing with that org, and similar orgs is similar situations. And I can’t post anonymously, because I can’t explain what I want to say without referring to specific anecdotes that will identify me.

My sense is (from the votes on this post) is that most of these reforms are not broadly popular. Which while it doesn’t undermine them in my opinion does create a contradiction for the authors.

As it stands, EA neglects the importance of collective epistemics and overemphasises individual rationality, and as a result cultivates a community that is homogenous, hierarchical, and intellectually insular. EA is overwhelmingly white, male, upper-middle-class, and of a narrow range of (typically quantitative) academic backgrounds. EA reading lists and curricula over-emphasise a very narrow range of authors, which are assumed to be highly intelligent and thus to be deferred to. This narrows the scope of acceptable thought, and generates path dependencies and intellectual blind-spots.

Yeah seems pretty accurate. I guess I agree with this like 70%.

Quick thoughts on Effective Altruism and a response to Dr. Sarah Taber’s good thread calling EA a cult (which it functionally is).

I have believed EA thinking for about 8 years, and continue to do so despite the scandals. HOWEVER, I agree fully with this part of “Doing EA Better” [2], which I’ll call “the critique” from now on. : “Many beliefs accepted in EA are surprisingly poorly supported, and we ignore entire disciplines with extremely relevant and valuable insights”.

As a leftist who wants to see the end of capitalism, cops, climate change, wealth inequality, and the dooming of entire nations to death and despair to uphold a white supremacist order, I am not particularly attached to EA or its specific GiveWell top charities. I think EA works best when it’s only an attempt to be “less wrong”, and always iterating through a SCIENTIFIC process.

Without having spent much time in the weeds, I think there’s a strong moral case that a GiveWell cause (say, mosquito nets) is superior to St. Jude’s or the Red Cross. Keep in mind that EA is a small minority of all charitable giving, and all charitable mindshare (you’ve never seen a bell ringer for EA). The charities you see in public and are asked to contribute to at checkout are almost objectively less efficient at turning dollars into lives saved or vastly improved than charities that quietly work in the poorest parts of the world. EA is having a moral crisis in a way that the Catholic Church has never meaningfully had to face.

What’s odd is that EA ends up supporting billionaire philanthropy as a de facto good, rather than the best option we have under the current system which we should also be trying to dismantle. For about a year now I’ve had a post kicking around in my head that there’s no EA interest in putting numerical bounds on the value of, for example, a strong tenant movement, the end of mass incarceration, a strong labor movement, the end of the drug war, the end of war in general. EA is tainted with centrist stink that looks at problems which are “controversial” in mass media and therefore *must* be actually controversial (ex. Palestine-Israel). Getting more people involved in “doing good” seems like an obvious answer yet most EA solutions seem to be power-flattering answers that want you to make a lot of money and donate it to EA, while making 0 effort to convince your friends or society.

As someone with a statistics degree, EA falls for the same “data wonk” trap a lot of wonks fall into—difficulty to quantify leads you to focusing on quantifiable areas, which are usually correlated with power and the status quo. EA is a very young movement yet they lack the historical knowledge to realize people were once very sure the best thing to do with mentally unwell people was lock them in an asylum for their lives. (The critique calls these last two paragraphs “EA orthodoxy”.)

I’m going to skim through the critique and add stuff where I think it matters.

I’ve avoided participating in EA because I’m not at a point where I’m doing significant philanthropy, and the heavy vibe of all the posts in the community is you need a bundle of keywords to participate. Words like epistemology, prior, Bayesian (I know this one, but seemingly not in the way people use it), utilitarian, objectively. I have a 4 year degree and can’t participate, so we know that around 90% of the world population cannot participate in EA forums. I feared getting torn to shreds for making the point which should be obvious- how do we know what 3 generations from now will want, let alone 3 million? I am especially unconvinced that intelligence in the universe is an automatic good; nor do I believe that 10^1000 intelligent minds is inherently better than 10^10.

Anyone who hasn’t seen bitcoin specifically and 99.9% of all crypto as a full stop ponzi scheme does not deserve the descriptor “rational”. SBF should’ve been viewed as a useful con artist’s mark; EA should’ve been happy to take his money but assumed it was going to collapse. There are dozens of very rational takedowns of crypto out there.

100% this: “At the very least, critics have learned to watch their tone at all costs, and provide a constant stream of unnecessary caveats and reassurances in order to not be labelled “emotional” or “overconfident”.” Centrists hate emotion, and if you’re emotional you have already lost the argument. I have a strong emotional response to people being locked in cages for stealing baby formula, and because of that I am a loser in the eyes of EA. (Does emotion not guide deworming initiatives? Or are EAs just happy to make a number go up? I can’t tell)

“When Stuart Russell argues that AI could pose an existential threat to humanity, he is held up as someone worth listening to –”He wrote the book on AI, you know!” However, if someone of comparable standing in Climatology or Earth-Systems Science, e.g. Tim Lenton or Johan Rockström, says the same for their field, they are ignored, or even pilloried.[39] Moderate statements from the IPCC are used to argue that climate change is “not an existential risk”, but given significant expert disagreement among experts on e.g. deep learning capabilities, it seems very unlikely that a consensus-based “Intergovernmental Panel on Artificial Intelligence” would take a stance anything like as extreme as that of most prominent EAs. This seems like a straightforward example of confirmation bias to us. ”

EAs care a lot about pandemic risk. They were proven right with COVID. What they are still unable to reckon with is the inability of the science to affect mass change. 5% of the world(?) has had COVID, we have no idea if that will lower those people’s life expediencies or QALYs but doctors say it probably will, it has conservatively caused $10 trillion in damage and killed ~7 million people, and ruined most people’s entire 2020 if not continues to ruin their lives on a day to day basis. And yet most of the western world has done very little to contain the disease, regardless of political affiliation or economic status. People called COVID a “baby pandemic” or “baby catastrophe”, a trial run for how humanity will deal with a much harder threat. We completely failed.

Let’s say you had a billion dollars to address “pandemic risk” in the world. Could you actually meaningfully reduce pandemic risk? I’m not sure you can—the reasons for this are heavily aligned with leftist thought, so of course the centrists in EA have no vocabulary to deal with it. (Things to consider are capitalism, white supremacy, the rich at Davos have rigorous covid mitigation in place and therefor do not care that the poor at Walmart do not, the rich control the mass media and their wealth depends on people continuing to shop...)

On twitter I follow the likes of health reporter Helen Branswell, and she retweets mainstream scientists from ex. Stanford, MIT, Harvard; and most health reporters, experts, and scientists cannot fathom the voluntary preventable mass death and disabling event that COVID continues to be. They were told, like many EA practitioners, that “politics” is bad, and we just need “data and science”, and fail to realize that even showing up to work is political, not saying “the government is a death cult” is political, not saying “the hospital I work at is forcing nurses to work 12 hour shifts with crappy COVID mitigation, to save the lives of patients who don’t believe they have COVID yet are infecting the same nurses, and these nurses deserve fair hazard pay for these horrible conditions” is political. If you worked remote for UPMC (largest hospital network in PA, US), you were offered a COVID vaccine before the contractor nurses and janitors who actually walked into COVID hospitals every day. This is a class issue, like it or not, and dumping a billion dollars into it won’t solve class.

One thing that’s been nipping at me that I haven’t seen anyone in EA say—fraud is not inherently bad. I would happily defraud a billionaire who made their fortune grinding up human babies. It seems like EA distanced themselves so thoroughly from SBF because it made the movement look bad and fraudulent, not that the actions were themselves bad. In SBF’s case, I don’t agree with defrauding regular people. But if some moronic 10 millionaire lost 5 of those millions by putting money into something that reeked of a ponzi scheme with no actual value, I’m not losing sleep over that. You could even argue it’s a net good if say, 50% of that ended up donated to EA, and that millionaire wouldn’t have donated it otherwise. I’m not sure if I will get banned for saying this, which is asinine in a movement which will readily praise you letting literal living humans today starve so that we can maybe grow more brains in a jar in 10 million years. If you ban me you’re literally admitting that stealing money is worse than letting someone die.

Most centrists believe whatever elites have defined as “crime” is bad, which makes them fit right into the status quo. Centrists don’t question why nobody is prosecuted for clean water act violations or wealthy tax fraud. Every shoplifting prosecution should give you pause: “objectively” the damage of elite crimes is far worse. https://​​www.yalelawjournal.org/​​forum/​​the-punishment-bureaucracy

I like the callout of “theories of change” and “Funding bodies should within 6 months publish lists of sources they will not accept money from, regardless of legality”. Poisonous funders are, IMO: any recreational drug (drugs are fine but drug companies are not), fast food and hard-to-define junk food, advertising, policing/​prosecution/​jailing/​surveillence/​support thereof (Palantir and ICE), 99.9% of crypto (when a good crypto comes out, we’ll know), fossil fuels, cars, probably a supermajority of animal products companies (and I’m not vegetarian).

[1] https://​​twitter.com/​​sarahtaber_bww/​​status/​​1617194799261487108

[2] https://​​forum.effectivealtruism.org/​​posts/​​54vAiSFkYszTWWWv4/​​doing-ea-better-1

I think the best way to discuss these criticisms is on a case by case basis, so I’ve put a number of them in a list here:

https://​​forum.effectivealtruism.org/​​posts/​​fjaLYSNTggztom29c/​​possible-changes-to-ea-a-big-upvoted-list

“EA should cut down its overall level of tone/​language policing”.

Strongly agree.

EAs should be more attentive to how motivated reasoning might affect tone /​ language policing.

You‘re probably more likely to tone /​ language police criticism of EA rather than praise, and you’re probably less likely to seriously engage with the ideas in the criticism if you are tone /​ language policing.

I think the point regarding epidemics, and how EA excessively focuses on the individual aspects of good epistemics rather than the group aspect, is a really good point which I have surprisingly never heard before.

In thinking about the democratization—what role would real people impacted by the organization’s decisions play? This is probably mostly related to global health & development—but it seems very strange to democratize to “effective altruists” but not to mention involving the people actually impacted by an EA organization’s work. I don’t think “democratizing” per se will probably be the right way to involve them—but finding some way of gathering the insights and perspectives of people impacted by each organizaiton’s work would help with the expertise, epistemics, and power problems mentioned.

The most important solution is simple: one person, one vote.

I disagree with this: I may have missed a section where you seriously engaged with the arguments in favor of the current karma-weighted vote system, but I think there are pretty strong benefits of a system that puts value on reputation. For example, it seems fairly reasonable that the views of someone who has >1000 karma are considered with more weight than someone who just created an account yesterday or who is a known troll with −300 karma.

I think there are some valid downsides to this approach, and perhaps it would be good to put a tighter limit on reputation weighting (e.g., no more than 4x weight), but “one person one vote” is a drastic rejection of the principle of reputation, and I’m disappointed with how little consideration was apparently given to the potential negatives of this reform /​ positives of the current system.

Great fun post!

I read the whole post. Thanks for your work. It is extensive. I will revisit it. More than once. You cite a comment of mine, a listing of my cringy ideas. That’s fine, but my last name is spelled “Scales” not “Scale”. :)

About scout mindset and group epistemics in EA

No. Scout mindset is not an EA problem. Scout and soldier mindset partition mindset and prioritize truth-seeking differently. To reject scout mindset is to accept soldier mindset.

Scout mindset is intellectual honesty. Soldier mindset is not. Intellectual honesty aids epistemic rationality. Individual epistemic rationality remains valuable. Whether in service of group epistemics or not. Scout mindset is a keeper. EA suffers soldier mindset, as you repeatedly identified but not by name. Soldier mindset hinders group epistemics.

We are lucky. Julia Galef has a “grab them by the lapel and shake them” interest in intellectual honesty. EA needs scout mindset.

Focus on scout mindset supports individual epistemics. Yes.

• scout mindset

• critical thinking skills

• information access

• research training

• domain expertise

• epistemic challenges

All those remain desirable.

Epistemic status

EA’s support epistemic status announcements to serve group epistemics. Any thoughts on epistemic status? Did I miss that in your post?

Moral uncertainty

Moral uncertainty is not an everyday problem. Or remove selfish rationalizations. Then it won’t be. Or revisit the revised uncertainty, I suppose.

Integrity

Integrity combines:

• intellectual honesty

• introspective efficacy

• interpersonal honesty

• behavioral self-correction

• assess->plan->act looping efficacy

Personal abilities bound those behaviors. So do situations. For example, constantly changing preconditions of actions bound integrity. Another bound is your interest in interpersonal honesty. It’s quite a lever to move yourself through life, but it can cost you.

Common-sense morality is deceptively simple

Common-sense morality? Not much eventually qualifies. Situations complicate action options. Beliefs complicate altruistic goals. Ignorance complicates option selection. Internal moral conflicts reveal selfish and altruistic values. Selfishness vs altruism is common-sense moral uncertainty.

Forum karma changes

Yes. Lets see that work.

• Allow alternate karma scoring. One person one vote. As a default setting.

• Allow karma-ignoring display. On homepage. Of Posts. And latest comments. As a setting.

• Allow hide all karma. As a setting.

Leave current settings as an alternate.

Diversifying funding sources and broader considerations

Tech could face lost profits in the near future. “Subprime Attention Crisis” by Tim Hwang suggests why. An unregulated ad bubble will gut Silicon Valley. KTLO will cost more, percentage-wise. Money will flow to productivity growth without employment growth.′

Explore income, savings, credit, bankruptcy and unemployment trends. Understand the implications. Consumer information will be increasingly worthless. The consumer class is shrinking. Covid’s UBI bumped up Tech and US consumer earnings temporarily. US poverty worsened. Economic figures now mute reality. Nevertheless, the US economic future trends negatively for the majority.

“Opportunity zones” will be a predictive indicator despite distorted economic data, if they ever become reality. There are earlier indicators. Discover some.

Financial bubbles will pop, plausibly simultaneously. Many projects will evaporate. Tech’s ad bubble will cost the industry a lot.

Conclusion

Thanks again for the post. I will explore the external links you gave.

I offered one suggestion (among others) in a red team last year: to prefer beliefs to credences. Bayesianism has a context alongside other inference methods. IBT seems unhelpful, however. It is what I advocate against, but I didn’t have a name for it.

Would improved appetite regulation, drug aversion, and kinesthetic homeostasis please our plausible ASI overlords? I wonder. How do you all feel about being averse to alcohol, disliking of pot, and indifferent to chocolate? The book “Sodium Hunger: The Search for a Salty Taste” reminds me that cravings can have a benefit, in some contexts. However, drugs like alcohol, pot, and chocolate would plausibly get no ASI sympathy. Would the threat of intelligent, benevolent ASI that take away interest in popular drugs (e.g ,through bodily control of us) be enough to halt AI development? Such a genuine threat might defeat the billionaire-aligned incentives behind AI development.

By the way, would EA’s enjoy installing sewage and drinking water systems in small US towns 20-30 years from now? I am reminded of “The End Of Work” by Jeremy Rifkin. Effective altruism will be needed from NGO’s working in the US, I suspect.

I noticed this sentence in a footnote:

A full-length post exploring EA’s historical links to reactionary thought will be published soon

I don’t think this post would be very useful—see genetic fallacy.

“EAs should assume that power corrupts”—strongly agree.

I perceive this as a very good and thoughtful collection of criticism and good ideas for reform. It’s also very long and dense and I’m not sure how to best interact with it.

As a general note, when evaluating the goodness of a pro-democratic reform in a non-governmental context, it’s important to have a good appreciation of why one has positive feelings/​priors towards democracy. One really important aspect of democracy’s appeal in governmental contexts is that for most people, government is not really a thing you consent to, so it’s important that the governmental structure be fair and representative.

The EA community, in contrast, is something you have much more agency to choose to be a part of. This is not to say “if you don’t like the way things are, leave,”—I am definitely pro-criticism/​feedback—it’s just important to avoid importing wholesale one’s feelings towards democracy in governmental settings vs. settings like EA where people have more agency/​freedom to participate, especially since democratic decision-making does have many disadvantages.

We should not be afraid of consulting outside experts, both to improve content/​framing and to discover blind-spots

If anything, we should be afraid of any tendency to stigmatize consulting outside experts. If anything, it’d be preferable all effective altruists are afraid of discouraging consultation from outside experts.

If you’re also reading the “diversify funding sources” and thinking BUT HOW? In a post where I make some similar suggestions, I suggest via encouraging entrepreneurship-to-give:

https://​​forum.effectivealtruism.org/​​posts/​​SBSC8ZiTNwTM8Azue/​​a-libertarian-socialist-s-view-on-how-ea-can-improve

Diversity is always a very interesting word and it’s interesting that the call for more comes after two of the three scandals mentioned in the opening posts are about EA being diverse along an axis that many EAs disagree with.

Similarly, it’s very strange that a post that talks a lot about the problems of EAs caring too much about other people being value aligned and afterward talk in the recommendations about how there should be more scrutiny to checking whether funders are aligned with certain ethical values.

This gives me the impression that the main thesis of the post is that EA values differ from woke values and should be changed to be more aligned with woke values.

The post doesn’t seem to have any self-awareness about pushing in different axis. If your goal is to convince people to think differently about diversity or about the importance of value alignment, it would make sense to make arguments that are more self-aware.

When Stuart Russell argues that AI could pose an existential threat to humanity, he is held up as someone worth listening to –”He wrote the book on AI, you know!” However, if someone of comparable standing in Climatology or Earth-Systems Science, e.g. Tim Lenton or Johan Rockström, says the same for their field, they are ignored, or even pilloried.

To me, this looks like it mistakes why people hold the views that they do and strawman people.

Saying “Stuart Russell” is worth listening to because of his book boils down to “If you actually want to understand why AI is an existential threat to humanity, read his book it’s likely to convince you”. On the other hand, Tim Lenton or Johan Rockström have not written books that make arguments for the importance of climate change that seem convincing to many EAs.

Quantification

When it comes to the topic of quantification, it seems that this post criticizes at the same time that EAs quantify everything and that they don’t quantify the value of paying community organizers relatively high salaries.

EAs seem to me, very willing to do a lot of thing, especially in the field of long-termism without quantification being central. Generally, EA thought leaders don’t tend to hold positions naive positions on topics like diversity or quantification but complex ones. If you want to change views you need to be more clear about cruxes and how you think about the underlying tradeofffs.

EA reading lists are typically narrow, homogenous, and biased, and EA has unusual social norms against reading more than a handful of specific books. Reading lists often heavily rely on EA Forum posts and shallow dives over peer-reviewed literature.

Apparently, the problem of reading too narrowly also applies to many scientific research fields. Derek Thompson writes:

Chu and the University of Chicago scholar James Evans found that progress has slowed in many fields because scientists are so overwhelmed by the glut of information in their domain that they’re reading and riffing on the same limited canon of famous papers. It’s more or less the same principle as a weekend couch potato overwhelmed by streaming options who opts to just watch the top-ranked TV show on Netflix. In both science and streaming, a surplus of options might be entrenching a small number of massive hits.

Visavis peer review:

Here’s a simple question: does peer review actually do the thing it’s supposed to do? Does it catch bad research and prevent it from being published?

It doesn’t. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this. In this study reviewers caught 30% of the major flaws, in this study they caught 25%, and in this study they caught 29%. These were critical issues, like “the paper claims to be a randomized controlled trial but it isn’t” and “when you look at the graphs, it’s pretty clear there’s no effect” and “the authors draw conclusions that are totally unsupported by the data.” Reviewers mostly didn’t notice.

In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time. If reviewers were doing their job, we’d hear lots of stories like “Professor Cornelius von Fraud was fired today after trying to submit a fake paper to a scientific journal.” But we never hear stories like that. Instead, pretty much every story about fraud begins with the paper passing review and being published. Only later does some good Samaritan—often someone in the author’s own lab!—notice something weird and decide to investigate. That’s what happened with this this paper about dishonesty that clearly has fake data (ironic), these guys who have published dozens or even hundreds of fraudulent papers, and this debacle:

https://​​experimentalhistory.substack.com/​​p/​​the-rise-and-fall-of-peer-review

I appreciated “Some ideas we should probably pay more attention to”. I’d be pretty happy to see some more discussion about the specific disciplines mentioned in that section, and also suggestions of other disciplines which might have something to add.

Speaking as someone with an actuarial background, I’m very aware of the Solvency 2 regime, which makes insurers think about extreme/​tail events which have a probability of 1-in-200 of occurring within the next year. Solvency 2 probably isn’t the most valuable item to add to that list; I’m sure there are many others.

Nathan commenting atomistically is frustrating and I wish he’d put them all in one comment.

I was inspired by your post, and I wrote a post about one way I think grant-making could be less centralized and draw more on expertise. One commenter told me grant-making already makes use of more expert peer reviewers than I thought, but it sounds like there is much more room to move in that direction if grant-makers decide it is helpful.

https://​​forum.effectivealtruism.org/​​posts/​​fNuuzCLGr6BdiWH25/​​doing-ea-better-grant-makers-should-consider-grant-app-peer

I mostly agree with Buck’s comment and I think we should probably dedicate more time to this at EAGs than we have in the past (and probably some other events). I’m not sure what is the best format, but I think having conversations about it would allow us to feel it much more in our bones rather than just discussing it on the EA forum for a few days and mostly forget about it.

I believe that most of these in-person discussions seem to have mostly happened at the EA Leaders Forum, so we should probably change that.

That said, I’m concerned that a lot of people will be scared to give their opinion on some things (for a variety of reasons). There‘s some benefits to doing this in person, but there’s probably some benefits to making it anonymous too.

The right way to handle the suggested reforms section is to put them all as comments.

I will not be taking questions at this time. Sarcasm.

I’ll add that EAs seem particularly bad at steelmanning criticisms - (eg—if a criticism doesn’t explicitly frame ideas on a spectrum and discuss trade offs, the comments tend to view the ideas as black and white and reject the criticisms because they don’t like the other extreme of the spectrum)

I find writing pretty hard and I imagine it was quite a task to compile all of these thoughts, thanks for doing that.

I only read the very first section (on epistemic health) but I found it pretty confusing. I did try and find explanations in the rest of the epistemics section.

EA’s focus on epistemics is almost exclusively directed towards individualistic issues like minimising the impact of cognitive biases and cultivating a Scout Mindset. The movement strongly emphasises intelligence, both in general and especially that of particular “thought-leaders” . An epistemically healthy community seems to be created by acquiring maximally-rational, intelligent, and knowledgeable individuals, with social considerations given second place. Unfortunately, the science does not bear this out. The quality of an epistemic community does not boil down to the de-biasing and training of individuals;^[3] more important factors appear to be the community’s composition, its socio-economic structure, and its cultural norms.^[4]

The footnotes and sources that you linked to don’t give me much evidence to update towards your position and saying “the science says x” (at least to me) implies that there is some kind of consensus view within the literature which I think you should be able to point to. This reads more like your hot takes rather than something you have thought about deeply. Superforecasting (footnote 4) does talk a bit about prediction markets but much more of the book is focused on how a few people with certain traits can beat most people at forecasting which I think runs counter to the point you are making so it seems misleading to me to link to it as if it supports your view.

I think it can be fine to give hot takes but I feel like the general vibe of the post was trying to persuade me rather than explain your view. Things that might have helped are focusing on a smaller set of points and trying to make a more rigorous case for them or communicating that you are not very confident in many of the key points if that is the case. I also felt like you were trying to make a claim that the ‘science’ supports your view—which based on the sources you linked to is really hard to verify.

I don’t think everything you wrote was clearly incorrect but in my view you made strong claims without demonstrating appropriate epistemic rigour.