To be very clear: I am not saying “this can never be changed.” I am saying that it would require changing the EA social scene—that is, to somehow decentralize it. I am not sure how to do that well (rather than doing it poorly, or doing it in name only). But I increasingly believe it is likely to be necessary.
TW123
Karma: 3,704
The problem goes beyond guardrails. Any attempts to reduce these conflicts of interest would have to contend with the extremely insular social scene in Berkeley. Since grantmakers frequently do not interact with many people outside of EA, and everyone in EA might end up applying for a grant from Open Phil, guardrails would significantly disrupt the social lives of grantmakers.
Let’s not forget that you can not just improperly favor romantic partners, but also just friends. The idea of stopping Open Phil from making grants to organizations where employees are close friends with (other) grantmakers is almost laughable because of how insular the social scene is—but that’s not at all normal for a grantmaking organization.
Even if Open Phil grantmakers separated themselves from the rest of the community, anyone who ever wanted to potentially become a grantmaker would have to do so as well because the community is so small. What if you become a grantmaker and your friend or romantic partner ends up applying for a grant?
In addition, many grants are socially evaluated at least partially, in my experience. Grantmakers have sometimes asked me what I think of people applying for grants, for example. This process will obviously favor friends of friends.
As such, the only way to fully remove conflicts of interest is likely to entail significant disruptions to the entire EA social scene (the one that involves everyone living/partying/working with the same very small group of people). I think that would be warranted, but that’s another post and I recognize I haven’t justified it fully here.
These dynamics are one reason (certainly not the only one) why I turned down an offer to be a part time grantmaker, choose not to live in Berkeley, and generally avoid dating within EA. Even if I cannot unilaterally remove these problems, I can avoid being part of them.
I’ve never felt comfortable in EA broadly construed, not since I encountered it about three years ago. And yet I continue to be involved to a certain extent. Why? Because I think that doing so is useful for doing good, and many of the issues that EA focuses on are sadly still far too neglected elsewhere. Many of the people who come closest to sharing my values are in EA, so even if I didn’t want to be “in EA,” it would be pretty difficult to remove myself entirely.
I also love my university EA group, which is (intentionally, in part by my design, in part by the design of others) different from many other groups I’ve encountered.
I work in AI safety, and so the benefit of staying plugged into EA for me is probably higher than it would be for somebody who wants to work in global health and development. But I could still be making a (potentially massive) miscalculation.
If you think that EA is not serving your aims of doing good (the whole point of EA), then remember to look out the window. And even if you run an “EA” group, you don’t need to feel tied to the brand. Do what you think will actually be good for the world. Best of luck.
“Living expenses while doing some of my early research” is one of the main purposes of the LTFF; to me Atlas feels like a roundabout way of getting that. LTFF asks you to have a specific high-impact project or educational opportunity for you to pursue, but as far as I know that wasn’t true of Atlas.
I think The Century Fellowship would make a better comparison to the Thiel Fellowship than Atlas would. It seems aimed at similar types to the Thiel Fellowship (college age people who are prepared to start projects and need to be financially independent to do so), while Atlas targets a slightly younger demographic and gives scholarships.
Atlas is posed as a talent search and development program, so I think any evaluation of Atlas should focus on how well it is searching for and developing talent that would not otherwise exist. I personally don’t know anything about how that has been turning out, or what the graduates have done/are doing with the money, so I don’t feel very qualified to evaluate it myself.
In the past two years, the technical alignment organisations which have received substantial funding include
Your post does not actually say this, but when I read it I thought you were saying that these are all the organizations that have received major funding in technical alignment. I think it would have been clearer if you had said “include the following organizations based in the San Francisco Bay Area:” to make it clearer you’re discussing a subset.
Anyway, here are the public numbers, for those curious, of $1 million+ grants in technical AI safety in 2021 and 2022 (ordered by total size) made by Open Philanthropy:
Redwood Research: $9.4 million, and then another grant for $10.7 million
Many professors at a lot of universities: $14.4 million
CHAI: $11.3 million
Aleksander Madry at MIT: $1.4 million
Hofvarpnir Studios: $1.4 million
Berkeley Existential Risk Initiative—CHAI collaboration: $1.1 million
Berkeley Existential Risk Initiative—SERI MATS Program: $1 million
The Alignment Research Center received much less: $265,000.
There isn’t actually any public grant saying that Open Phil funded Anthropic. However, that isn’t to say that they couldn’t have made a non-public grant. It was public that FTX funded Anthropic.
having strong or intimate connections with employees of Open Philanthropy greatly enhances the chances of having funding, and it seems almost necessary
Based on spending some time in Berkeley, I think a more accurate way to describe this is as follows:
People who care about AI safety and are involved in EA tend to move to Berkeley because that is where everyone else is. It really can increase your productivity if you can easily interact with others working in your field and know what is going on, or so the established wisdom goes. The people who have been around the longest are often leading research organizations or are grantmakers at Open Phil. They go to the same parties, have the same friends, work in the same offices, and often spend nearly all of their time working with little time to socialize with anyone outside their community. Unless they make a special effort to avoid dating anyone in their social community, they may end up dating a grantmaker.
If we want these conflicts of interest to go away, we could try simply saying it should be a norm for Open Phil not to grant to organizations with possible conflicts of interest. But knowing the Berkeley social scene, this means that many Open Phil grantmakers wouldn’t be able to date anyone in their social circles, since basically everyone in their social circles is receiving money from Open Phil.
The real question is as you say one of structure: whether so many of the EA-aligned AI safety organizations should be headquartered in close proximity and whether EAs should live together and be friends with basically only other EAs. That’s the dynamic that created the conflicts. I don’t think the answer to this is extremely obvious, but I don’t really feel like trying to argue both sides of it right now.
It’s possibly true that regrantors would reduce this effect in grantmaking, because you could designate regrantors in other places or who have different friends. But my suspicion would be that regrantors would by default be the same people who are already receiving grants.
GiveDirectly has a program for the US that you can donate to. I don’t really know how good it is, but the organization in general seems excellent.
For what it’s worth, I don’t think this needed a retraction. It’s true the original post was pretty overconfident about things. Instead of asserting something and defending it, it would probably have been better to assert it with the explicit aim of hearing criticism from people on here. That’s what happened anyway, but your framing was more “here is a thing I think is good.”
That would certainly be great if she would. I actually first heard about EA when I read Strangers Drowning in 2016! It’s very well written.
I’m a bit confused about crossposting, are you saying it was always available? I don’t remember seeing any crossposts a year ago, or being able to use the feature. In fact I used to crosspost a lot of things and specifically remember the first time I saw the crossposting feature. But maybe I just didn’t notice this before.
Didn’t know that about the dev teams, that’s useful to know!
I’m in favor of a clear separation between the forums. They are made for different audiences and not everything that is meant for one is meant for the other. As somebody who writes some pieces that are meant for both audiences, the cross posting feature is somewhat convenient for me (but not hugely so; I can just copy and paste). And as a reader, sometimes it’s nice to see a post is cross posted so that I can go see the comments on the other forum.
I’d be interested to see how much the easy cross posting has increased the number of cross posts, and if so what kinds of posts are now more likely to be cross posted. This seems like an analysis the forum team could do and is harder to do anecdotally.
The EA Forum and LessWrong have some of the best technical infrastructure on the internet, and I think the EA Forum derives huge benefit from that. However, it does make me a little uneasy that it’s made by the Lightcone team, who are in charge of LessWrong. I like the people on that team, but I expect probably some decisions that are good for LessWrong but not so good for the EA Forum might just end up propagating here by default. This is just a suspicion; I don’t have any particular examples.
A lot of separation does exist. LessWrong posts are moderated pretty differently, the commenter audiences are often very different, and the types of posts are mostly different. So the connection as is isn’t currently a huge concern of mine.
I still believe that there were significant problems with a section of the original statement from Max Tegmark, and they have been reinforced, not undermined, by this FAQ. To be clear, I am not referring to problems like “they left out detail x”; I am referring to the fact that a particular section was actively misleading. I understand FLI was under a lot of pressure to churn out a statement fast, so I’m not totally surprised the original statement wasn’t good quality. Still, I think FLI has a responsibility not to make misleading statements that they know, or should know, are misleading.
In this FAQ, FLI states the following as a main reason they rejected the grant:
we found the term “ethnopluralism” endorsed in Nya Dagbladet.
However, in their initial statement, they wrote:
We also point out that the claim by Expo.se that NDF is “pro-Nazi” [the lede in the article] is apparently not shared by the (center-left) former Swedish government, which not only certified the Foundation as charitable but granted $30,000 in government funding and support to Nya Dagbladet in 2021. This is exactly $30,000 more than the zero dollars FLI granted to them.
The invocation of Swedish government funding was never appropriate in the first place, as I wrote at the time in my only previous comment on this situation (and also there are some good replies). This is perhaps an understandable mistake for somebody might not know about the Swedish press support system or how it supports papers with essentially all political leanings. However, given the fact that FLI now states that they already knew that the Swedish government was supporting a newspaper that favors ethnopluralism (a view that wikipedia says has been linked to neo-fascist groups), surely they could not have thought that this funding was any kind of endorsement from a “center-left” government. As a result, this part of the statement appears even more misleading to me than it did when I originally pointed it out.
is apparently not shared by the (center-left) former Swedish government, which not only certified the Foundation as charitable but granted $30,000 in government funding and support to Nya Dagbladet in 2021
Disclaimer: I previously knew nothing about the Swedish press; I still know almost nothing. I just thought this seemed weird and spent about 20 minutes looking into it.
Some context which I think would be useful to evaluate this claim.
It appears that in Sweden the government subsidizes newspapers in the form of “press support.” From reading the Wikipedia page on press support, which is mostly actually about Norway not Sweden, it seems like support does not really constitute a government endorsement, but rather is provided to a lot of different newspapers and is mostly to ensure a healthy press. It’s possible this differs between Norway and Sweden though.
The $30,000 figure comes from the expo.se piece, which says:
Nya Dagbladet applied to the Swedish Press and Broadcasting Authority for public funding in 2020, but was turned down. The platform reacted angrily to the decision, and published a series of articles where specific officials at the agency were named and pictured. The publications caused distress among employees at the agency who felt menaced and pressured, as Dagens Nyheter reported at the time. The following year, Nya Dagbladet made another application for public funding; this time they were successful and received about $30,000 in various grants.
The article linked (archive here) Google translates the article as referring to the Norwegian press. I thought that was pretty weird, but from googling the Swedish (Myndigheten för press, radio och tv), I think the Google translate is wrong and it is indeed about the Swedish press (here is the website of the Swedish press agency). The expo.se piece might seem to imply that the government officials may have been intimidated into making the later grant, but I think that’s a bit less clear if press support is supposed to be widely distributed to newspapers in any case.
Regardless, to me it does not seem like the reception of this grant really indicates that the organization is not pro-Nazi, and certainly it doesn’t seem to imply endorsement of that claim from the Swedish government, at least as far as I can tell. A good understanding would require a better understanding of the Swedish press support system, which I neither I or presumably the vast majority of readers of this comment have.
You mention opportunity cost, but I think it’s worth further emphasizing. To do this well, you’d need somebody who has been around a while (or at least a lot of time and cooperation from people who have). You’d need them to manage different perspectives and opinions about various things that happened. You’d need them to be a very good writer. And you’d need the writer to be someone people trust—my perspective is “Open Phil hired this person” would probably not be sufficient for trust.
There are people who could do this: Kelsey Piper is one as you suggest. But these are all pretty unusual characteristics and the opportunity costs for the sort of person who could do this well just seem really massive. I might be wrong about this, but that’s my first thought when reading your post.
The latest formal iteration of which is here. You can always go through the content on your own time at the link above too.
I originally helped design the course and I ran the first iteration of a similar program. I’m not really involved with the course now but I think I’m qualified to answer. However, I did AGI safety fundamentals a long time ago and haven’t done MLAB, so my knowledge of those could be wrong (though I don’t think so).
In comparison to AGI Safety Fundamentals, this course a lot more technical and less conceptual. AGISF is not going to include the latest in machine learning on a technical level, and this course doesn’t include as many conceptual readings.
In comparison with MLAB, this course is more focused on reading papers and understanding research, and less focused on teaching particular frameworks or engineering skills.
There’s a bit of overlap between all, but it’s pretty minimal. I think anyone who has done any of these programs would learn something from doing the others. It mostly depends on what people want to take out of the course: knowledge of a lot of different conceptual research directions (AGISF), skills in engineering with Pytorch (MLAB), or knowledge of the frontier of ML safety research and paper reading skills (Intro to ML Safety).
I thought The Alignment Problem was pretty good at giving a high level history. Despite the name, only a pretty small portion is actually about the alignment problem and a lot is about ML history.
Thanks for writing. I will say this phenomenon isn’t specific to EA. I used to organize a large ish non-EA event and huge numbers of people would fail to show up and others would try to register the day before. After enough iterations, we knew the fraction of people who did each, and we just planned for that. I wonder if you could do something similar for these events? But if it’s extremely variable based on year/location, that would be harder.
Also, I realized I’ve been assuming that for virtual conferences, there is essentially zero downside to being a no show. But maybe this isn’t true? Do people have to pay per swapcard profile, or some other variable cost I’m not aware of? If anyone knows, this seems relevant. For an in person conference it is a lot more obvious that money can still get spent on you if you don’t show up.
Not hugely surprising, given the people at CEA have certainly thought about this more than random forum users. Still, it’s good to do diligence. Thank you Eli for responding; this is a model example of somebody graciously explaining non-obvious considerations in a decision.
This is one reason I haven’t wanted to live in Berkeley. Whenever I visited (which was very frequently at one point) it was pretty exhausting to have so many people running around trying to figure out and chase which things were “cool” (far from everyone in Berkeley was like this, but too many seemed to be).
Outside Berkeley I notice it much less. I still interact frequently online with a lot of people who live in Berkeley, and for me this is much more pleasant. It’s a shame, because there are lots of benefits of talking to people in person. But for me they don’t seem worth the costs.
I have not (yet) known myself to ever be negatively affected for speaking my mind in EA. However, I know others who have. Some possible reasons for the difference:
My fundamental ethical beliefs are pretty similar to the most senior people.
On the EA Forum, I make almost extreme effort to make tight claims and avoid overclaiming (though I don’t always succeed). If I have vibes-based criticisms (I have plenty) I tend to keep them to people I trust.
I “know my audience:” I am good at determining how to say things such that they won’t be received poorly. This doesn’t mean “rhetoric,” it means being aware of the most common ways my audience might misinterpret my words or the intent behind them, and making a conscious effort to clearly avoid those misinterpretations.
Related to the above, I tend to “listen before I speak” in new environments. I avoid making sweeping claims before I know my audience and understand their perspective inside and out.
I’m a techy white man working in AI safety and I’m not a leftist, so I’m less likely to be typed by people as an “outsider.” I suspect this is mostly subconscious, except for the leftist part, where I think there are some community members who will consciously think you are harmful to the epistemic environment if they think you’re a leftist and don’t know much else about you. Sometimes this is in a fair way, and sometimes it’s not.
I’m very junior, but in comparison to even more junior people I have more “f*** you social capital” and “f*** you concrete achievements you cannot ignore”.