AI governance/grantmaking. Formerly at the Center for AI Safety and Yale EA organizer.
ThomasW
This post is mostly making claims about what a very, very small group of people in a very, very small community in Berkeley think. When throwing around words like “influential leaders” or saying that the claims “often guide EA decision-making” it is easy to forget that.
The term “background claims” might imply that these are simply facts. But many are not: they are facts about opinions, specifically the opinions of “influential leaders”
Do not take these opinions as fact. Take none for granted. Interrogate them all.
“Influential leaders” are just people. Like you and I, they are biased. Like you and I, they are wrong (in correlated ways!). If we take these ideas as background, and any are wrong, we are destined to all be wrong in the same way.
If you can, don’t take ideas on background. Ask that they be on the record, with reasoning and attribution given, and evaluate them for yourself.
In the past two years, the technical alignment organisations which have received substantial funding include
Your post does not actually say this, but when I read it I thought you were saying that these are all the organizations that have received major funding in technical alignment. I think it would have been clearer if you had said “include the following organizations based in the San Francisco Bay Area:” to make it clearer you’re discussing a subset.
Anyway, here are the public numbers, for those curious, of $1 million+ grants in technical AI safety in 2021 and 2022 (ordered by total size) made by Open Philanthropy:
Redwood Research: $9.4 million, and then another grant for $10.7 million
Many professors at a lot of universities: $14.4 million
CHAI: $11.3 million
Aleksander Madry at MIT: $1.4 million
Hofvarpnir Studios: $1.4 million
Berkeley Existential Risk Initiative—CHAI collaboration: $1.1 million
Berkeley Existential Risk Initiative—SERI MATS Program: $1 million
The Alignment Research Center received much less: $265,000.
There isn’t actually any public grant saying that Open Phil funded Anthropic. However, that isn’t to say that they couldn’t have made a non-public grant. It was public that FTX funded Anthropic.
having strong or intimate connections with employees of Open Philanthropy greatly enhances the chances of having funding, and it seems almost necessary
Based on spending some time in Berkeley, I think a more accurate way to describe this is as follows:
People who care about AI safety and are involved in EA tend to move to Berkeley because that is where everyone else is. It really can increase your productivity if you can easily interact with others working in your field and know what is going on, or so the established wisdom goes. The people who have been around the longest are often leading research organizations or are grantmakers at Open Phil. They go to the same parties, have the same friends, work in the same offices, and often spend nearly all of their time working with little time to socialize with anyone outside their community. Unless they make a special effort to avoid dating anyone in their social community, they may end up dating a grantmaker.
If we want these conflicts of interest to go away, we could try simply saying it should be a norm for Open Phil not to grant to organizations with possible conflicts of interest. But knowing the Berkeley social scene, this means that many Open Phil grantmakers wouldn’t be able to date anyone in their social circles, since basically everyone in their social circles is receiving money from Open Phil.
The real question is as you say one of structure: whether so many of the EA-aligned AI safety organizations should be headquartered in close proximity and whether EAs should live together and be friends with basically only other EAs. That’s the dynamic that created the conflicts. I don’t think the answer to this is extremely obvious, but I don’t really feel like trying to argue both sides of it right now.
It’s possibly true that regrantors would reduce this effect in grantmaking, because you could designate regrantors in other places or who have different friends. But my suspicion would be that regrantors would by default be the same people who are already receiving grants.
Thank you for writing this. For a while, I have been thinking of writing a post with many similar themes and maybe I still will at some point. But this post fills a large hole.
As is obligatory for me, I must mention Derek Parfit, who tends to have already well-described many ideas that resurface later.
In Reasons and Persons, Part 1 (especially Chapter 17), Derek Parfit argues that good utilitiarians should self-efface their utilitarianism. This is because people tend to have motivated reasoning, and tend to be wrong. Under utilitarianism, it is possible to justify nearly anything, provided your epistemics are reasonably bad (your epistemics would have to be very bad to justify murder under deontological theories that prohibit murder; you would have to claim that something was not in fact murder at all). Parfit suggests adopting whatever moral system seems to be most likely to produce the highest utility for that person in the long run (perhaps some theory somewhat like virtue ethics). This wasn’t an original idea, and Mill said similar things.
One way to self-efface your utilitiarianism would be to say “yeah, I know, it makes sense under utilitarianism for me to keep my promises” (or whatever it may be). Parfit suggests that may not be enough, because deep down you still believe in utilitarianism; it will come creeping through (if not in you, in some proportion of people who self-efface this way). He says that you may instead need to forget that you ever believed in utilitarianism, even if you think it’s correct. You need to believe a lie, and perhaps even convince everyone else of this lie.
He also draws an interesting caveat: what if the generally agreed upon virtues or rules are no longer those with the highest expected utility? If nobody believed in utilitarianism, why would they ever be changed? He responds:
This suggests that the most that could be true is that C [consequentialism] is partly self-effacing. It might be better if most people caused themselves to believe some other theory, by some process of self-deception that, to succeed, must also be forgotten. But, as a precaution, a few people should continue to believe C, and should keep convincing evidence about this self-deception. These people need not live in Government House, or have any other special status. If things went well, the few would do nothing. But if the moral theory believed by most did become disastrous, the few could then produce their evidence. When most people learnt that their moral beliefs were the result of self-deception, this would undermine these beliefs, and prevent the disaster.
This wasn’t an original idea either; Parfit here is making a reference to Sidgwick’s “Government House utilitarianism,” which seemed to suggest only people in power should believe utilitarianism but not spread it. Parfit passingly suggests the utilitarians don’t need to be the most powerful ones (and indeed Sidgwick’s assertion may have been motivated by his own high position).
Sometimes I think that this is the purpose of EA. To attempt to be the “few people” to believe consequentialism in a world where commonsense morality really does need to change due to a rapidly changing world. But we should help shift commonsense morality in a better direction, not spread utilitarianism.
Maybe utilitarianism is an info hazard not worth spreading. If something is worth spreading, I suspect it’s virtues.
Which virtues? Some have suggestions.
People have been having similar thoughts to yours for many years, including myself. Navigating through EA epistemic currents is treacherous. To be sure, so is navigating epistemic currents in lots of other environments, including the “default” environment for most people. But EA is sometimes presented as being “neutral” in certain ways, so it feels jarring to see that it is clearly not.
Nearly everyone I know who has been around EA long enough to do things like run a university group eventually confronts the fact that their beliefs have been shaped socially by the community in ways that are hard to understand, including by people paid to shape your beliefs. It’s challenging to know what to do in light of that. Some people reject EA. Others, like you, take breaks to figure things out more for themselves. And others press on, while trying to course correct some. Many try to create more emotional distance, regardless of what they do. There’s not really an obvious answer, and I don’t feel I’ve figured it fully out myself. All this is to just say: you’re not alone. If you or anyone else reading this wants to talk, I’m here.
Finally, I really like this related post, as well as this comment on it. When I ran the Yale EA in depth fellowship, I assigned it as a reading.
Sorry not to weigh in on the object-level parts about university groups and what you think they should do differently, but as I’ve graduated I’m no longer a community builder so I’m somewhat less interested in weighing in on that.
is apparently not shared by the (center-left) former Swedish government, which not only certified the Foundation as charitable but granted $30,000 in government funding and support to Nya Dagbladet in 2021
Disclaimer: I previously knew nothing about the Swedish press; I still know almost nothing. I just thought this seemed weird and spent about 20 minutes looking into it.
Some context which I think would be useful to evaluate this claim.
It appears that in Sweden the government subsidizes newspapers in the form of “press support.” From reading the Wikipedia page on press support, which is mostly actually about Norway not Sweden, it seems like support does not really constitute a government endorsement, but rather is provided to a lot of different newspapers and is mostly to ensure a healthy press. It’s possible this differs between Norway and Sweden though.
The $30,000 figure comes from the expo.se piece, which says:
Nya Dagbladet applied to the Swedish Press and Broadcasting Authority for public funding in 2020, but was turned down. The platform reacted angrily to the decision, and published a series of articles where specific officials at the agency were named and pictured. The publications caused distress among employees at the agency who felt menaced and pressured, as Dagens Nyheter reported at the time. The following year, Nya Dagbladet made another application for public funding; this time they were successful and received about $30,000 in various grants.
The article linked (archive here) Google translates the article as referring to the Norwegian press. I thought that was pretty weird, but from googling the Swedish (Myndigheten för press, radio och tv), I think the Google translate is wrong and it is indeed about the Swedish press (here is the website of the Swedish press agency). The expo.se piece might seem to imply that the government officials may have been intimidated into making the later grant, but I think that’s a bit less clear if press support is supposed to be widely distributed to newspapers in any case.
Regardless, to me it does not seem like the reception of this grant really indicates that the organization is not pro-Nazi, and certainly it doesn’t seem to imply endorsement of that claim from the Swedish government, at least as far as I can tell. A good understanding would require a better understanding of the Swedish press support system, which I neither I or presumably the vast majority of readers of this comment have.
- 20 Jan 2023 17:18 UTC; 26 points) 's comment on FLI FAQ on the rejected grant proposal controversy by (
- 20 Jan 2023 23:10 UTC; 3 points) 's comment on FLI FAQ on the rejected grant proposal controversy by (
This post lines up with my outsider perspective on FHI, and it seems to be quite measured. I encourage anyone who thinks that Bostrom is really the best leader for FHI to defend that view here (anonymously, if necessary).
I’ve never felt comfortable in EA broadly construed, not since I encountered it about three years ago. And yet I continue to be involved to a certain extent. Why? Because I think that doing so is useful for doing good, and many of the issues that EA focuses on are sadly still far too neglected elsewhere. Many of the people who come closest to sharing my values are in EA, so even if I didn’t want to be “in EA,” it would be pretty difficult to remove myself entirely.
I also love my university EA group, which is (intentionally, in part by my design, in part by the design of others) different from many other groups I’ve encountered.
I work in AI safety, and so the benefit of staying plugged into EA for me is probably higher than it would be for somebody who wants to work in global health and development. But I could still be making a (potentially massive) miscalculation.
If you think that EA is not serving your aims of doing good (the whole point of EA), then remember to look out the window. And even if you run an “EA” group, you don’t need to feel tied to the brand. Do what you think will actually be good for the world. Best of luck.
This is great!
In the intro article, I don’t think I really like the comparison between pandemic prevention and counterterrorism.
A couple reasons:
First, counterterrorism might be construed to include counter bio terrorism. In which case, it’s not obvious to me that pandemic prevention and counterterrorism are even exclusive.
Second, both pandemics and counterterrorism are heavy-tailed and dominated by tail events. Tail events don’t happen...until they do. To give an example, here is the same graph but for 2009-2019:
Essentially no deaths from COVID-19! Looks like it’s unimportant!
Knowing almost nothing about terrorism, I would expect that a terrorism tail event, such as the detonation of a nuclear dirty bomb, could be similar: we wouldn’t see it in the statistics until it was too late.
When we broaden the scope, we can see that many more people died in global pandemics (other than COVID, since COVID barely existed) in that time period than terrorism:
However, this is extremely influenced by another tail event: HIV/AIDS. In a world without HIV/AIDS, it would look like this:
This would imply that in some counterfactual world where nothing was different except that AIDS did not exist, I should have thought in 2019 that global pandemics were about equal to terrorism in scale. This is not a conclusion that should be drawn from the data, because for tail-dominated phenomena, you can’t just consider historical average data (certainly not from a ten year period), you have to consider black swan events: events unlike any that have ever happened.
Comparing the most recent pandemic tail event to the average statistics on terrorism doesn’t make sense: it’s comparing apples to oranges. Either compare the average statistics over a long time period or the probability and severity of possible tail events. For newly emerging threats like engineered pandemics, average statistics doesn’t even make sense at all, since we’ve never had an engineered pandemic.
I thought this comment was valuable and it’s also a concern I have.
It makes me wonder if some of the “original EA norms”, like donating a substantial proportion of income or becoming vegan, might still be quite important to build trust, even as they seem less important in the grand scheme of things (mostly, the increase in the proportion of people believing in longtermism). This post makes a case for signalling.
It also seems to increase the importance of vetting people in somewhat creative ways. For instance, did they demonstrate altruistic things before they knew there was lots of money in EA? I know EAs who spent a lot of their childhoods volunteering, told their families to stop giving them birthday presents and instead donate to charities, became vegan at a young age at their own initiative, were interested in utilitarianism very young, adopted certain prosocial beliefs their communities didn’t have, etc. When somebody did such things long before it was “cool” or they knew there was anything in it for them, this demonstrates something, even if they didn’t become involved with EA until it might help their self-interest. At least until we have Silicon Valley parents making sure their children do all the maximally effective things starting at age 8.
It’s kind of useful to consider an example, and the only example I can really give on the EA forum is myself. I went to one of my first EA events partially because I wanted a job, but I didn’t know that there was so much money in EA until I was somewhat involved (also this was Fall 2019, so there was somewhat less money). I did some of the things I mentioned above when I was a kid (or at least, so I claim on the EA forum)! Would I trust me immediately if I met me? Eh, a bit but not a lot, partially because I’m one of the hundreds of undergrads somewhere near AI safety technical research and not (e.g.) an animal welfare person. It would be significantly easier if I’d gotten involved in 2015 and harder if I’d gotten involved in 2021.
Part of what this means is that we can’t rely on trust so much anymore. We have to rely on cold, hard, accomplishments. It’s harder, it’s more work, it feels less warm and fuzzy, but it seems necessary in this second phase. This means we have to be better about evaluating accomplishments in ways that don’t rely on social proof. I think this is easier in some fields (e.g. earning to give, distributing bednets) than others (e.g. policy), but we should try in all fields.
For me, a big change happened when I had been around in EA long enough, done enough things, and spoken to enough people to be able to say, “if I say something disagreeable to somebody and it turns out they are one of those people who will judge me personally for disagreeing with the dominant paradigm on x thing, it’s their loss, not mine.” I also feel I can say something disagreeable to people and they will tend to hear me out rather than ignore me as a newbie who doesn’t know anything (in fairness, when I was just starting, I actually didn’t know much at all!).
For the newest people (I’ve only been in EA for 2 years, so I am still quite new) with few legible achievements and almost no connections, this is more difficult. If you constantly feel you have to think about how to get into x office or get in the good graces of y person or receive z grant, you feel far more pressure to fight against your own doubt for fear that people will judge you for your disagreements. This is unhealthy.
Obviously there is a continuum: it’s not that you can either disagree all the time or never disagree, but there are varying amounts of disagreement that people feel comfortable having.
Sometimes people talk about “f**k you money” to mean money that you can use to ride out unemployment if you decide you don’t like your job anymore and want to quit. In EA circles there is an analogous concept, something like “I respectfully disagree with your worldview social capital” or “I respectfully disagree with your worldview concrete achievements that you cannot ignore”. The more of that you have, especially the latter, the better. Luckily, the latter is also relatively correlated with how much you’ve been able to achieve, which is (hopefully) correlated with impact.
- 8 Feb 2023 16:52 UTC; 27 points) 's comment on Why People Use Burner Accounts: A Commentary on Blacklists, the EA “Inner Circle”, and Funding by (
One thing you didn’t mention is grant evaluation. I personally do not mind grants being given out somewhat quickly and freely in the beginning of a project. But before somebody asks for money again, they should need to have their last grant evaluated to see whether it accomplished anything. My sense is that this is not common (or thorough) enough, even for bigger grants. I think as the movement gets bigger, this seems pretty likely to lead to unaccountability.
Maybe more happens behind the scenes than I realize though, and there actually is a lot more evaluation than I think.
Students have by far the most flexibility in their careers. It’s not uncommon for university students (in the US—maybe some other age in other countries) to do things like switch their major from biology to economics; except in very rare circumstances, 40 year old biologists do not become economists. If you suppose that certain high-impact career paths require very special skills not common in the population, then you might need people to develop those skills early rather than try to find people who already have them. There are some areas of EA that probably do have this property, though the popular perception of it is maybe overblown.
I do think it would be good if there could be more experienced older people in EA, since I think there are probably many people out there with highly relevant and useful experience who haven’t heard of EA but would be receptive.
I like this post, thanks Thomas!
I want to make a comment for maybe newer people especially with some of the uses of the word “EA” here. I’ll take an example to illustrate: “People who are not totally dedicated to EA will...”
I actually think this means (or if it doesn’t, it should mean), “people who are not totally dedicated to impartially maximizing impact as defined under a plausible moral theory [not the point of this to debate which are plausible] will...” or something like that. In other words, “people who are not totally dedicated to the basic principles of EA”.
It doesn’t (or shouldn’t) mean “people who are not totally dedicated to the EA community” or something else that might imply only working at an EA-branded org, only having EA friends, or only working on a cause area that some proportion of EAs think is worthwhile. The EA community is probably a good way to find multipliers and a useful signal for what is valuable, but it is not the final goal at all and doesn’t have all the answers.
I could imagine some case in which it makes sense to do something “less EA” (in the sense that fewer people in EA think it’s valuable) because it’s actually “more EA” (in the sense that it’s actually more valuable for maximizing impact). The point of this example isn’t to establish how likely this is, just to point out that the final goal is maximizing impact, not EA the community, and that “more EA/less EA” is a bit ambiguous.
This might be totally obvious to most readers of this comment, but I wanted to write it anyway just in case there are people who don’t find it obvious (or it isn’t at all obvious, or not what Thomas meant).
I think this is putting too much on 80k. They have hundreds of jobs in many different areas listed on their website, and it’s a very daunting task to evaluate each one of them when the evaluators are generalists who often have to rely on (conflicting) opinions from people in the community. On top of that, what is career capital for one person could plausibly be direct impact for some other person, so it doesn’t really seem one size fits all.
If somebody can’t evaluate jobs on the job board for themselves, I’m not that confident that they’ll take a good path regardless. People have a tendency to try to offload thinking to places like 80k, and I actually think it could be bad if 80k made it easier to do that on extremely fine grained topics like individual jobs.
I do like the idea of having comments on particular jobs. And it also would be good for 80k to be more clear they don’t expect all these jobs to necessarily have direct impact.
I don’t know whether it’s the case that many people on the internet are looking at the job board and deciding which jobs to apply to when they don’t have a strong engagement with EA ideas, and that these sorts of people are the types who would actually get the jobs. If that’s the case (80k would know better than me), then I think it maybe does make sense to restrict to jobs that aren’t going to be bad if such a person gets them. That seems like an empirical question.
(I’m sorry if this comment is not accessible to many people. I also might have missed something in reading the paper, please do let me know if that’s the case!)
I do think many EAs who haven’t studied philosophy of mind probably (implicitly) believe functionalism a bit too credulously. It doesn’t matter very much right now, but maybe it will later.
But I’m not really convinced by this paper. I skimmed it and here are some initial thoughts:
A TV screen flickers between frames. If you watch an old or broken TV, you’ll notice this, but once the frame rate is high enough, you experience it continuously. Of course, it’s not actually continuous. A similar phenomenon occurs with pixels: they are discrete, but we experience them continuously.
You say that light is continuous, not discrete, but light (and every other elementary particle) behaves as discrete packets (photons) as well as waves. This makes me wonder if there is a real difference between analog and digital at extremely high levels of fidelity.
You give the example of mechanical watches, but I’m pretty sure the ones that seem continuous (i.e. have “sweeping hands” rather than ticking) are actually just higher frequency, and still move discretely rather than continuously. See here. Again, we experience them continously.
You mention hue, saturation, and brightness. We represent these in a computer just fine with only 24 (8 for each) bits most of the time, and we still get most of the same experiences. There are higher-fidelity color schemes but we barely notice this.
You argue in the paper that neurons are analog rather than digital. I agree that neurons are not simply on/off. But again, does this show anything about whether you can properly represent a neuron if you just allocate more than one bit to it? Something I find particularly problematic is that the evidence that neurons are analog presumably came from measuring action potentials and analyzing them. But aren’t the action potential measurements not represented digitally in a computer? How could that show evidence of analog behavior?
In the paper you have this caveat (and further explanation), which seems to dismantle many of my objections above:
On the Lewis-Maley view we adopt, to be analog does not require continuity, but only monotonic covariation in magnitude between the representation and what is represented. That increase or decrease can happen in steps, or it can happen smoothly (i.e., discretely or continuously). For example, consider the fuel gauges found in cars. Older cars often have a physical dial that more-or-less continuously moves from ‘F’ to ‘E’ as fuel is consumed; this dial is an analog representation of the amount of fuel in the car, because as fuel decreases, so does the literal angle of the dial. Newer cars often have a different way of displaying fuel. Instead of a physical dial, fuel is displayed as a bar graph on an LED or LCD display. Importantly, that bar graph is composed of discrete segments. Nevertheless, this is still an analog representation of the amount of fuel: as fuel decreases, the number of segments decreases. In contrast, we can imagine a fuel gauge that simply displays the number of gallons (or liters) of fuel in the tank (e.g., ‘6.5’ for six and a half gallons). Here, as fuel decreases, the digits displayed do not increase or decrease (the way they would if we changed the font in a paper): they simply change.
The suggestion here is that if we encode something in unary (number of on bits = magnitude) it is qualitatively different from being encoded in binary or decimal. This is not a straightforward claim. In your paper, it relies on panpsychism with microexperience: roughly speaking, the idea that consciousness arises from microconsciousness of fundamental particles. I think you’ve done a pretty decent job of arguing the conclusion given that premise, but I find the premise pretty unconvincing myself, and anyone else who does is similarly unlikely to be convinced by your argument.
TLDR: Yes, the magnitude of discrete representations of bits is completely contingent and arbitrary. But to say that two functionally identical organisms that are also isomorphic (usually more than we could ever ask for) are different in terms of the consciousness they produce seems to require some kind of microphenomenal laws. If you don’t believe in such laws, you shouldn’t believe this argument.
I’m quite happy that you are thinking critically about what you are reading! I don’t think you wrote a perfect criticism (see below), but the act of taking the time to write a criticism and posting it to a public venue is not an easy step. EA always needs people who are willing and eager to probe its ethical foundations. Below I’m going to address some of your specific points, mostly in a critical way. I do this not because I think your criticism is bad (though I do disagree with a lot of it), but because I think it can be quite useful to engage with newer people who take the time to write reasonably good reactions to something they’ve read. Hopefully, what I say below is somewhat useful for understanding the reasons for longtermism and what I see as some flaws in your argument. I would love for you to reply with any critiques of my response.
This has been quoted several times, even though it’s an absurd argument on its face. Imagine the world where Cleopatra skipped dessert. How does this cure cancer?
It doesn’t, and that’s not Parfit’s point. Parfit’s point is that if one were to employ a discount rate, Cleopatra’s dessert would matter more than nearly anything today. Since (he claims) this is clearly wrong, there is something clearly wrong with a discount rate.
Most of the 80000 hours article attempts to persuade the reader that longtermism is morally good, by explaining the reasons that we should consider future people. But the part about how we are able to benefit future people is very short.
Well yes, but that’s because it’s in the other pages linked there. Mostly, this has to do with thinking about whether existential risks exist soon, and whether there is anything we can do about them. That isn’t really in the scope of that article but I agree the article doesn’t show it.
The world is a complex system, and trying to affect the far future state of a complex system is a fool’s errand.
That isn’t entirely true. There are some things that routinely affect the far future of complex systems. For instance, complex systems can collapse, and if you can get them to collapse, you can pretty easily affect its far future. If it’s about to collapse due to an extremely rare event, then preventing that collapse can affect its far future state.
Let’s look at a few major breakpoints and see whether longtermism was a significant factor.
Obviously, it wasn’t. But of course it wasn’t! There wasn’t even longtermism at all, so it wasn’t a significant factor in anyone’s decisions. Maybe you are trying to say “people can make long term changes without being motivated by longtermism.” But that doesn’t say anything about whether longtermism might make them better at creating long term changes than they otherwise would be.
We can achieve longtermism without longtermism
I generally agree with this and so do many others. For instance see here and here. However, I think it’s possible that this may not be true at some time in the future. I personally would like to have longtermism around, in case there is really something where it matters, mostly because I think it is roughly correct as a theory of value. Some people may even think this is already the case. I don’t want to speak for anyone, but my sense is that people who work on suffering risk are generally considering longtermism but don’t care as much about existential risk.
The main point is that intervening for long term reasons is not productive, because we cannot assume that interventions are positive. Historically, interventions based on “let’s think long term”, instead of solving an immediate problem, have tended to be negative or negligible in effect.
First, I agree that interventions may be negative, and I think most longtermists would also strongly agree with this. In terms of whether historical “long term” interventions have been negative, you’ve asserted it but you haven’t really shown it. I would be very interested in research on this; I’m not aware of any. If this were true, I do think that would be a knock against longtermism as a theory of action (though not decisive, and not against longtermism as a theory of value). Though it maybe could still be argued that we live at “the hinge of history” where longtermism is especially useful.
I made some distinguishment between theory of value and theory of action. A theory of value (or axiology) is a theory about what states of the world are most good. For instance, it might say that a world with more happiness, or more justice, is better than a world with less. A theory of action is a theory about what you should do; for instance, that we should take whichever action produces the maximum expected happiness. Greaves and MacAskill make the case for longtermism as both. But it’s possible you could imagine longtermism as a theory of value but not a theory of action.
For instance, you write:
Some blood may be shed and lives may be lost, but the expected value is strongly positive.
Various philosophers, such as Parfit himself, have suggested that for this reason, many utilitarians should actually “self-efface” their morality. In other words, they should perhaps start to believe that killing large numbers of people is bad, even if it increases utility, because they might simply be wrong about the utility calculation, or might delude themselves into thinking what they already wanted to do produces a lot of utility. I gave some more resources/quotes here.
Thanks for writing!
The problem goes beyond guardrails. Any attempts to reduce these conflicts of interest would have to contend with the extremely insular social scene in Berkeley. Since grantmakers frequently do not interact with many people outside of EA, and everyone in EA might end up applying for a grant from Open Phil, guardrails would significantly disrupt the social lives of grantmakers.
Let’s not forget that you can not just improperly favor romantic partners, but also just friends. The idea of stopping Open Phil from making grants to organizations where employees are close friends with (other) grantmakers is almost laughable because of how insular the social scene is—but that’s not at all normal for a grantmaking organization.
Even if Open Phil grantmakers separated themselves from the rest of the community, anyone who ever wanted to potentially become a grantmaker would have to do so as well because the community is so small. What if you become a grantmaker and your friend or romantic partner ends up applying for a grant?
In addition, many grants are socially evaluated at least partially, in my experience. Grantmakers have sometimes asked me what I think of people applying for grants, for example. This process will obviously favor friends of friends.
As such, the only way to fully remove conflicts of interest is likely to entail significant disruptions to the entire EA social scene (the one that involves everyone living/partying/working with the same very small group of people). I think that would be warranted, but that’s another post and I recognize I haven’t justified it fully here.
These dynamics are one reason (certainly not the only one) why I turned down an offer to be a part time grantmaker, choose not to live in Berkeley, and generally avoid dating within EA. Even if I cannot unilaterally remove these problems, I can avoid being part of them.
- 8 Feb 2023 15:58 UTC; -6 points) 's comment on Why People Use Burner Accounts: A Commentary on Blacklists, the EA “Inner Circle”, and Funding by (
I have not (yet) known myself to ever be negatively affected for speaking my mind in EA. However, I know others who have. Some possible reasons for the difference:
My fundamental ethical beliefs are pretty similar to the most senior people.
On the EA Forum, I make almost extreme effort to make tight claims and avoid overclaiming (though I don’t always succeed). If I have vibes-based criticisms (I have plenty) I tend to keep them to people I trust.
I “know my audience:” I am good at determining how to say things such that they won’t be received poorly. This doesn’t mean “rhetoric,” it means being aware of the most common ways my audience might misinterpret my words or the intent behind them, and making a conscious effort to clearly avoid those misinterpretations.
Related to the above, I tend to “listen before I speak” in new environments. I avoid making sweeping claims before I know my audience and understand their perspective inside and out.
I’m a techy white man working in AI safety and I’m not a leftist, so I’m less likely to be typed by people as an “outsider.” I suspect this is mostly subconscious, except for the leftist part, where I think there are some community members who will consciously think you are harmful to the epistemic environment if they think you’re a leftist and don’t know much else about you. Sometimes this is in a fair way, and sometimes it’s not.
I’m very junior, but in comparison to even more junior people I have more “f*** you social capital” and “f*** you concrete achievements you cannot ignore”.
I still believe that there were significant problems with a section of the original statement from Max Tegmark, and they have been reinforced, not undermined, by this FAQ. To be clear, I am not referring to problems like “they left out detail x”; I am referring to the fact that a particular section was actively misleading. I understand FLI was under a lot of pressure to churn out a statement fast, so I’m not totally surprised the original statement wasn’t good quality. Still, I think FLI has a responsibility not to make misleading statements that they know, or should know, are misleading.
In this FAQ, FLI states the following as a main reason they rejected the grant:
we found the term “ethnopluralism” endorsed in Nya Dagbladet.
However, in their initial statement, they wrote:
We also point out that the claim by Expo.se that NDF is “pro-Nazi” [the lede in the article] is apparently not shared by the (center-left) former Swedish government, which not only certified the Foundation as charitable but granted $30,000 in government funding and support to Nya Dagbladet in 2021. This is exactly $30,000 more than the zero dollars FLI granted to them.
The invocation of Swedish government funding was never appropriate in the first place, as I wrote at the time in my only previous comment on this situation (and also there are some good replies). This is perhaps an understandable mistake for somebody might not know about the Swedish press support system or how it supports papers with essentially all political leanings. However, given the fact that FLI now states that they already knew that the Swedish government was supporting a newspaper that favors ethnopluralism (a view that wikipedia says has been linked to neo-fascist groups), surely they could not have thought that this funding was any kind of endorsement from a “center-left” government. As a result, this part of the statement appears even more misleading to me than it did when I originally pointed it out.
This is an interesting post! I agree with most of what you write. But when I saw the graph, I was suspicious. The graph is nice, but the world is not.
I tried to create a similar graph to yours:
In this case, fun work is pretty close to impactful toll. In fact, the impact value for it is only about 30% less than the impact value of impactful toll. This is definitely sizable, and creates some of the considerations above. But mostly, everywhere on the pareto frontier seems like a pretty reasonable place to be.
But there’s a problem: why is the graph so nice? To be more specific: why are the x and y axes so similarly scaled?
Why doesn’t it look like this?
Here I just replaced x in the ellipse equation with log(x). It seems pretty intuitive that our impact would be power law distributed, with a small number of possible careers making up the vast majority of our possible impact. A lot of the time when people are trying to maximize something it ends up power law distributed (money donated, citations for researchers, lives saved, etc.). Multiplicative processes, as Thomas Kwa alluded to, will also make something power law distributed. This doesn’t really look power law distributed quite yet though. Maybe I’ll take the log again:
Now, fun work is unfortunately 100x less impactful than impactful toll. That would be unfortunate. Maybe the entire pareto frontier doesn’t look so good anymore.
I think this is an inherently fatal flaw with attempts to talk about trading off impact and other personal factors in making choices. If your other personal factors are your ability to have fun, have good friendships, etc., you now have to make the claim that those things are also power-law distributed, and that your best life with respect to those other values is hundreds of times better than your impact maximizing life. If you don’t make that claim, then either you have to give your other values an extremely high weight compared with impact, or you have to let impact guide every decision.
In my view, the numbers for most people are probably pretty clear that impact should be the overriding factor. But I think there can be problems with thinking that way about everything. Some of those problems are instrumental: if you think impact is all that matters, you might try to do the minimum of self-care, but that’s dangerous.
I think people should think in the frame of the original graph most of the time, because the graph is nice, and a reminder that you should be nice to yourself. If you had one of the other graphs in your head, you wouldn’t really have any good reason to be nice to yourself that isn’t arbitrary or purely instrumental.
But every so often, when you face down a new career decision with fresh eyes, it can help to remember that the world is not so nice.