Reflections 🤔 on EA & EAG following EAG London (2024):
I really liked the location this year. The venue itself was easy to get to on public transport, seemed sleek and clean, and having lots of natural light on the various floors made for a nice environment. We even got some decent sun (for London) on Saturday and Sunday. Thanks to all the organisers and volunteers involved, I know it’s lot of work setting up an event like this us and making it run smoothly.
It was good to meet people in person who I previous had only met or recognised from online interaction. I won’t single out individual 1-on-1s I had, but it was great to be able to put faces to names, and hearing peoples stories and visions in person was hugely inspiring. I talked to people involved in all sorts of cause areas and projects, and that combination of diversity, compassion, and moral seriousness is one of the best things about EA.
Listening to the two speakers from the Hibakusha Project at the closing talk was very moving, and clear case of how knowing something intellectually is not the same thing as hearing personal (and in-person) testimony. I think it would’ve been one of my conference highlights in the feedback form if we hadn’t already been asked to fill it out a few minutes beforehand!
I was going to make a point about a ‘lack of EA leadership’ turning up apart from Zach Robinson, but when I double-checked the event attendee list I think I was just wrong on this. Sure, a couple of big names didn’t turn up, and it may depend on what list of ‘EA leaders’ you’re using as a reference, but I want to admit I was directionally wrong here.
I thought Zach gave a good opening speech, but many people noted on the apparent dissonance between saying that CEA wanted to focus on ‘principles-first’ approach to EA, but that they also expected AI to be their area of most focus/highest priority and that they don’t expect that to change in the near future.
Finally, while I’m sure the people I spoke to (and those who wanted to speak to me) is strongly affected by selection-effects, and my own opinions on this are fairly strong, it did feel that there was consensus on there being a lack of trust/deference/shared beliefs from ‘Bay-Area EA’:[1]
Many people think that working on AI Safety and Governance is important and valuable, but not ‘overwhelmingly important’ or ‘the most important thing human has done/will ever do’. This included some fairly well-known names from those who attended, and basically nobody there (as far as I could tell) I interacted with held extremely ‘doomer’ beliefs about AI.
There was a lot of uncomfortable feeling at the community-building funding being directed to ‘longtermism’ and AI Safety in particular. This is definitely a topic I’m want to investigate more post-EAG, as I’m not sure what the truth of the matter is, but I’d certainly find it problematic if some of the anecdotes I heard were a fair representation of reality.
In any case, I think it’s clear that AI Safety is no longer ‘neglected’ within EA, and possibly outside of it.[2](Retracted this as, while it’s not true, commenters have pointed out that it’s not really the relevant metric to be tracking here)
On a personal level, it felt a bit odd to me that the LessOnline conference was held at exactly the same time as EAG. Feels like it could be a coincidence, but on the other hand this is not a coincidence because nothing is ever a coincidence. It feeds into my impression that the Bay is not very interested in what the rest of EA has to say.
One point which I didn’t get any clear answers to was ‘what are the feedback mechanisms in the community to push back on this’, and do such feedback mechanisms even exist?
In summary: It feels like, from my perspective, that the Bay Area/Exclusively Longtermist/AI Safety Maximalist version of EA has ‘lost of the mandate of heaven’, but nonetheless at the moment controls a lot of the community’s money and power. This, again, is a theme I want to explicitly explore in future posts.
I am old (over 30yo) and can’t party like the young EAs anymore 😔
In any case, I think it’s clear that AI Safety is no longer ‘neglected’ within EA, and possibly outside of it.
I think this can’t be clear based only on observing lots of people at EAG are into it. You have to include some kind of independent evaluation of how much attention the area “should” have. For example, if you believed that AI alignment should receive as much attention as climate change, then EAG being fully 100% about AI would still not be enough to make it no longer neglected.
(Maybe you implicitly do have a model of this, but then I’d like to hear more about it.)
FWIW I’m not sure what my model is, but it involves the fact that despite many people being interested in the field, the number actually working on it full time still seems kind of small, and in particular still dramatically smaller than the number of people working on advancing AI tech.
On a personal level, it felt a bit odd to me that the LessOnline conference was held at exactly the same time as EAG. Feels like it could be a coincidence, but on the other hand this is not a coincidence because nothing is ever a coincidence. It feeds into my impression that the Bay is not very interested in what the rest of EA has to say.
It was really 90% coincidence in that Manifest and MATS basically fully determined when LessOnline would happen. I do think in a world where I considered myself more interested in investing in EA, or being involved in EA community building, I would have felt more sadness and hesitation about scheduling it at the same time, though I think it’s unlikely that would have shifted the overall decision (~15% for this weird counterfactual).
As Phib also says, it is the case that at least historically very few people travel for EAG. I was surprised by this when I did the surveys and analytics for this when I ran EAG in 2015 and 2016.
at least historically very few people travel for EAG. I was surprised by this when I did the surveys and analytics for this when I ran EAG in 2015 and 2016.
Here are some numbers from Swapcard for EAG London 2024:
Country
COUNTA of Country
United Kingdom
608
United States
196
Germany
85
Netherlands
48
France
44
Switzerland
34
India
23
Sweden
21
Canada
21
Australia
21
Norway
17
Brazil
15
Belgium
13
Philippines
12
Austria
12
Spain
11
Poland
11
Czech Republic
11
Singapore
10
Nigeria
10
Italy
10
Denmark
10
South Africa
9
Kenya
9
Finland
8
Israel
7
Hungary
7
Mexico
5
Ireland
5
Hong Kong
5
Malaysia
4
Estonia
4
China
4
Turkey
3
Taiwan
3
Romania
3
Portugal
3
New Zealand
3
Chile
3
United Arab Emirates
2
Peru
2
Luxembourg
2
Latvia
2
Indonesia
2
Ghana
2
Colombia
2
Zambia
1
Uganda
1
Thailand
1
Slovakia
1
Russia
1
Morocco
1
Japan
1
Iceland
1
Georgia
1
Egypt
1
Ecuador
1
Cambodia
1
Bulgaria
1
Botswana
1
Argentina
1
55% of attendees were not from the UK, 14% of attendees were from the US, at least based on Swapcard data
London is a particularly easy city to travel to from the rest of Europe, but that’s still like 50% more than the baseline we had in 2015/2016/2017. The most relevant numbers here would be the people who would travel all the way from the U.S. and who would overlap with people who would want to attend LessOnline. My best guess is there are around 30-40 attendees for which there was a real conflict between the two events, though it wouldn’t surprise me if that’s off by a factor of 2-3 in either direction.
“I do think in a world where I considered myself more interested in investing in EA, or being involved in EA community building, I would have felt more sadness and hesitation about scheduling it at the same time, though I think it’s unlikely that would have shifted the overall decision (~15% for this weird counterfactual)”
I find this comment quite discouraging that you didn’t feel sadness and hesitation about scheduling it at the same time. I would have hoped that leaders like you who organised important events like LessOnline, Manifest and MATS, that have EA heritage and connection would have at least a little interest in doing what was best for EA and community building (even without having to “invest” in it yourself) and therefore at least trying to co-ordinate with the CEA events crew.
I also think your comment partially refutes your assessment that it was “90% coincidence” that Manifest and MATS rather than EAG determined when LessOnline would be. If you care about the other 2 conferences but not much about clashes with EAG, then its hardly completely coincidence that you clashed with EAG.…
I find this comment quite discouraging that you didn’t feel sadness and hesitation about scheduling it at the same time.
I didn’t say that I didn’t feel sadness or hesitation about scheduling it at the same time. Indeed, I think my comment directly implied that I did feel some sadness or hesitation, because I used the word “more”, implying there was indeed a baseline level of sadness or hesitation that’s non-zero.
Ignoring that detail, a bit of broader commentary on why I don’t feel that sad:
I at the moment think that most EA community building is net-negative for the world. I am still here as someone trying to hold people accountable and because I have contributed to a bunch of the harm this community has caused. I am in some important sense an “EA Leader” but I don’t seem to be on good terms with most of what you would call EA leadership, and honestly, I wish the EA community would disband and disappear and expect it to cause enormous harm in the future (or more ideally I wish it would undergo substantial reform, though my guess is the ship for that has sailed, which makes me deeply sad).
I have a lot of complicated opinions about what this implies about how I should relate to stuff like event scheduling, and helping with things like the EA Forum, that I would be happy to go into sometime (though this doesn’t seem like the right place for it). I care a lot about being cooperative despite my misgivings, and will continue to coordinate with people, but I feel like you should be aware that I do not wish the EA community to grow or gain more power in the world (though I am happy to engage in trade and to avoid dumb coordination problems that lose value for all parties involved).
If you care about the other 2 conferences but not much about clashes with EAG, then its hardly completely coincidence that you clashed with EAG....
MATS and Manifest pay us hundreds of thousands of dollars. It would have been obviously reckless and value destroying to pass on either of these contracts because of a scheduling conflict with a conference on the other side of the world, and also separately, a decision that would have substantially increased the risk of future bankruptcy of my organization. I do not consider myself under an obligation to make sacrifices at this level.
I’ve considered it! My guess is it would be bad for evaporative cooling reasons for people like me to just leave the positions from which they could potentially fix and improve things (and IMO, it seems like a bad pattern that when someone starts thinking that we are causing harm that the first thing we do is to downvote their comment expressing such sadness and ask them to resign, that really seems like a great recipe for evaporative cooling).
Also separately, I am importantly on the Long Term Future Fund, not the EA Infrastructure Fund. I would have likely left or called for very substantial reform of the EA Infrastructure Fund, but the LTFF seems like it’s probably still overall doing good things (though I am definitely not confident).
Precommitting to not posting more in this whole thread, but I thought Habryka’s thoughts deserved a response
IMO, it seems like a bad pattern that when someone starts thinking that we are causing harm that the first thing we do is to downvote their comment
I think this is a fair cop.[1] I appreciate the added context you’ve added to your comment and have removed the downvote. Reforming EA is certainly high on my list of things to write about/work on, so would appreciate your thoughts and takes here even if I suspect I’ll ending up disagreeing with diagnosis/solutions.[2]
My guess is it would be bad for evaporative cooling reasons for people like me to just leave the positions from which they could potentially fix and improve things
I guess that depends on the theory of change for improving things. If it’s using your influence and standing to suggest reforms and hold people accountable, sure. If it’s asking for the community to “disband and disappear”, I don’t know. Like, I don’t know in many other movements would that be tolerated with significant influence and funding power?[3] If one of the Lightcone Infrastructure team said “I think lightcone infrastructure in its entirety should shut down and disband, and return all funds” and then made decisions about funding and work that aligned with that goal and not yours, how long should they expect to remain part of the core team?
Maybe we’re disagreeing about what we mean by the ‘EA community’ implicitly here, and I feel that sometimes the ‘EA Community’ is used as a bit of a scapegoat, but when I see takes like this I think “Why should GWWC shut down and disband because of the actions of SBF/OpenAI?”—Like I think GWWC and its members definitely count as part of the EA Community, and your opinion seems to be pretty maximal without much room for exceptions.
(Also I think it’s important to note that your own Forum use seems to have contributed to instances of evaporative cooling, so that felt a little off to me.)
I am importantly on the Long Term Future Fund, not the EA Infrastructure Fund
This is true, but LTFF is part of EA Funds, and to me is clearly EA-run/affiliated/associated. It feels like its odd that you’re a grantmaker who decides where money to the community, from one of its most well-known and accessible funds, and you think that said community should disperse/disband/not grow/is net-negative for the world. That just seems rife for weird incentives/decisions unless, again, you’re explicitly red-teaming grant proposals and funding decisions. If you’re using it to “run interference” from the inside, to move funding away from the EA community and its causes, that feels a lot more sketchy to me.
FWIW Habryka, I appreciate all that I know you’ve done and expect there’s a lot more I don’t know about that I should be appreciative of too.
I would also appreciate if you’d write up these concerns? I guess I want to know if I should feel similarly even as I rather trust your judgment. Sorry to ask, and thanks again
Editing to note I‘ve now seen some of comments elsewhere
I wish the EA community would disband and disappear and expect it to cause enormous harm in the future.
I would be curious to hear you expand more on this:
What is your confidence level? (e.g. is it similar to the confidence you had in “very few people travel for EAG”, or is it something like 90%?)
What scenarios are you worried about? E.g. is it more about EA hastening the singularity by continuing to help research labs, or about EA making a government-caused slowdown less likely and less effective?
What is your confidence level? (e.g. is it similar to the confidence you had in “very few people travel for EAG”, or is it something like 90%?)
Extremely unconfident, both in overall probability and in robustness. It’s the kind of belief where I can easily imagine someone swaying me one way or another in a short period of time, and the kind of belief I’ve gone back and forth on a lot over the years.
On the question of confidence, I feel confused about how to talk about probabilities of expected value. My guess is EA is mostly irrelevant for the things that I care about in ~50% of worlds, is bad in like 30% of worlds and good in like 20% of worlds, but the exact operationalization here is quite messy. Also in the median world in which EA is bad, it seems likely to me that EA causes more harm than it makes up for in the median world where it is good.
What scenarios are you worried about? Hastening the singularity by continuing to help research labs or by making government intervention less like and less effective?
Those are two relatively concrete things I am worried about. More broadly, I am worried about EA generally having a deceptive and sanity-reducing relationship to the world, and for it to be in some sense a honeypot that lots of the world’s smartest and most moral people end up getting stuck in and where they lend their credibility to bad actors (SBF and Sam Altman being the obvious examples here, and Anthropic seems like the one I am betting on will be looked back on similarly).
My key motivation is mostly “make key decision makers better informed and help smart and moral people understand the state of the world better”.
I think an attitude that promotes truth-seeking and informedness above other things is more conductive to that than EA stuff. I also don’t think I would describe most of my work straightforwardly as “rationalist community building”. LessWrong is its own thing that’s quite different from a lot of the rationality community, and is trying to do something relatively specific.
OK your initial message makes more sense given your response here—Although I can’t quite now connect why MATS and Manifest would be net positive things under this framework while EA community building would be net negative.
My slight pushback would be that EAG London is the most near-term focused of the EAGs, so some of the long-termist potential net negatives you list might not apply so much with that conference.
My slight pushback would be that EAG London is the most near-term focused of the EAGs, so some of the long-termist potential net negatives you list might not apply so much with that conference.
Yeah this is probably my biggest disagreement with Oli on this issue.
I was going to make a point about a ‘lack of EA leadership’ turning up apart from Zach Robinson, but when I double-checked the event attendee list I think I was just wrong on this. Sure, a couple of big names didn’t turn up, and it may depend on what list of ‘EA leaders’ you’re using as a reference, but I want to admit I was directionally wrong here.
I talked to people involved in all sorts of cause areas and projects, and that combination of diversity, compassion, and moral seriousness is one of the best things about EA.
+1, I always leave these conferences filled with inspiration and gratitude 😊
it felt a bit odd to me that the LessOnline conference was held at exactly the same time as EAG
Yeah I was confused by this at first, but now I’m pretty sure this is a coincidence and there’s a good chance the organizers just didn’t think to check the dates of EAG.
what are the feedback mechanisms in the community to push back on this
I’m not sure I understand what “this” is referring to, but in general I think discussing things on the Forum is a reasonable way to provide feedback and push for change within EA. Stuff on the Forum does get around.
In any case, I think it’s clear that AI Safety is no longer ‘neglected’ within EA, and possibly outside of it.
Maybe a better question is how neglected is this within society? And AI technical research is a lot less neglected than before, but governance work is still looking extremely neglected AND we appear to be in a critical policy window.
Hi, I went to Lessonline after registering for EAG London, my impression of both events being held on the same weekend is something like:
Events around the weekend (Manifest being held the weekend after Lessonline) informed Lessonline’s dates (but why not the weekend after Manifest then?)
People don’t travel internationally as much for EAGs (someone cited to me ~10% of attendees but my opinion on reflection is that this seems an underestimate).
I imagine EAG Bay Area, Global Catastrophic Risks in early Feb also somewhat covered the motivation for “AI Safety/EA conference”.
I think you’re right that it’s not entirely* a coincidence that Lessonline conflicted with EAG Bay Area, but I’m thinking this was done somewhat more casually and probably reasonably.
I think it’s odd, and other’s have noted too, the most significant AI safety conference shares space with things unrelated on an object-level. I think it’s further odd to consider, I’ve heard people say, why bother going to a conference like this when I live in the same city as the people I’d most want to talk with (Berkeley/SF).
Finally, I feel weird about AI, since I think insiders are only becoming more convinced/confirmed of extreme event likelihoods (AI capabilities). I think it has only become more important by virtue of most people updating timelines earlier, not later, and this includes Open Phil’s version of this (Ajeya and Joe Carlsmith’s AI timelines). In fact, I’ve heard arguments that it’s actually less important by virtue of, “the cat’s out of the bag and not even Open Phil can influence trajectories here.” Maybe AI safety feels less neglected because it’s being advocated from large labs, but that may be both a result of EA/EA-adjacent efforts and not really enough to solve a unilateralizing problem.
In fairness you don’t need high p|doom to think AI safety should be the no.1 priority if you a) think that AI is a non-negligible extinction risk (say >0.1%), b) no other extinction risk has an equal degree of combined neglectedness and size and c) the expected value of the future conditional on us not going extinct in the next 100 years is astronomically high, and d) AI safety work makes a significant difference to how likely doom is to occur.. None of these are innocent or obvious assumptions, but I think a lot of people in the community hold all 3. I consider myself a critic of doomers in one sense, because I suspect p|doom is under 0.1%, and I think once you get down below that level, you should be nervous about taking expected value calculation that include your p|doom literally, because you probably don’t really know whether you should be 0.09% or several orders of magnitude lower. But even I am not *sure* that this is not swamped by c). (The Bostrom Pascal’s Mugging case involves probabilities way below 1 in a million, never mind 1 in a 1000.) Sometimes I get the impression though, that some people think of themselves as anti-doomer, when their p|doom is officially more like 1%. I think that’s a major error, if they really believe that figure. 1% is not low for human extinction. In fact, it’s not low even if you only care about currently existing people being murdered: in expectation that is 0.01x8 billion=80 million deaths(!). Insofar as what is going on is really just that people in their heart of hearts are much lower than 1%, but don’t want to say that because it feels extreme maybe this is ok. But if people actually mean figures like 1% or 5%, they ought to basically be on the doomers’ side, even if they think very high p|doom estimates given by some doomers are extremely implausible.
Going to merge replies into this one comment, rather than sending lots and flooding the forum. If I’ve @ you specifically and you don’t want to respond in the chain, feel free to DM:
On neglectedness—Yep, fair point that our relevant metric here is neglectedness in the world, not in EA. I think there is a point to make here but it was probably the wrong phrasing to use, I should have made it more about ‘AI Safety being too large a part of EA’ than ‘Lack of neglectedness in EA implies lower ITN returns overall’
On selection bias/other takes—These were only ever meant to be my takes and reflections, so I definitely think they’re only a very small part of the story. I guess @Stefan_Schubert would be interested to hear about your impression of ‘lack of leadership’ and any potential reasons why?
On the Bay/Insiders—It does seem like the Bay is convinced AI is the only game in town? (Aschenbrenner’s recent blog seems to validate this). @Phib would be interested to hear you say more on your last paragraph, I don’t think I entirely grok it but it sounds very interesting.
On the Object Level—I think this one for an upcoming sequence. Suffice to say that one can infer from my top level post that I have very different beliefs on this issue than many ‘insider EAs’, and I do work on AI/ML for my day job![1] But I think that while David sketches out a case for overall points, I think those points have been highly underargued and underscrutinised given their application in shaping the EA movement and its funding. So look it for a more specific sequence on the object level[2] maybe-soon-depending-on-writing-speed.
Yeah, thank you, I guess I was trying to say that the evidence only seems to be stronger over time that the Bay Area’s: ‘AI is the only game in town’, is accurate.
Insofar as, timelines for various AI capabilities have outperformed both superforecasters’ and AI insiders’ predictions; transformative AI timelines (at Open Phil, prediction markets, AI experts I think) have decreased significantly over the past few years; the performance of LLMs have increased at an extraordinary rate across benchmarks; and we expect the next decade to extrapolate this scaling to some extent (w/ essentially hundreds of billions if not tens of trillions to be invested).
Although, yeah, I think to some extent we can’t know if this continues to scale as prettily as we’d expect and it’s especially hard to predict categorically new futures like exponential growth (10%, 50%, etc, growth/year). Given the forecasting efforts and trends thus far it feels like there’s a decent chance of these wild futures, and people are kinda updating all the way? Maybe not Open Phil entirely (to the point that EA isn’t just AIS), since they are hedging their altruistic bets, in the face of some possibility this decade could be ‘the precipice’ or one of the most important ever.
Misuse and AI risk seem like the negative valence of AI’s transformational potential. I personally buy the arguments around transformational technologies needing more reasoned steering and safety, and I also buy that EA has probably been a positive influence, and that alignment research has been at least somewhat tractable. Finally I think that there’s more that could be done to safely navigate this transition.
Also, re David (Thorstad?) yeah I haven’t engaged with his stuff as I probably should, and I really don’t know how to reason for or against arguments around the singularity, exponential growth, and the potential of AI without deferring to people more knowledgeable/smarter than me. I do feel like I have seen the start and middle of trends they predicted, and predict will extrapolate-with my own personal use and some early reports on productivity increases.
I do look forward to your sequence and hope you do really well on it!
What do you mean by that? Presumably you do not mean it in any religious sense. Do you want to say that exclusively longtermist EA is much less popular among EAs than it used to be (i.e. the “heaven” is the opinion of the average EA)?
Ah sorry, it’s a bit of linguistic shortcut, I’ll try my best to explain more clearly:
As David says, it’s an idea from Chinese history. Rulers used the concept as a way of legitimising their hold on power, where Tian (Heaven) would bestow a ‘right to rule’ on the virtuous ruler. Conversely, rebellions/usurpations often used the same concept to justify their rebellion, often by claiming the current rulers had lost heaven’s mandate.
Roughly, I’m using this to analogise to the state of EA, where AI Safety and AI x-risk has become an increasingly large/well-funded/high-status[1] part of the movement, especially (at least apparently) amongst EA Leadership and the organisations that control most of the funding/community decisions.
My impression is that there was a consensus and ideological movement amongst EA leadership (as opposed to an already-held belief where they pulled a bait-and-switch), but many ‘rank-and-file’ EAs simply deferred to these people, rather than considering the arguments deeply.
I think that various amounts of scandals/bad outcomes/bad decisions/bad vibes around EA in recent years and at the moment can be linked to this turn towards the overwhelming importance of AI Safety, and as EffectiveAdvocate says below, I would like that part of EA to reduce its relative influence and power on the rest of it, and for rank-and-file EAs to stop deferring on this issue especially, but also in general.
Reflections 🤔 on EA & EAG following EAG London (2024):
I really liked the location this year. The venue itself was easy to get to on public transport, seemed sleek and clean, and having lots of natural light on the various floors made for a nice environment. We even got some decent sun (for London) on Saturday and Sunday. Thanks to all the organisers and volunteers involved, I know it’s lot of work setting up an event like this us and making it run smoothly.
It was good to meet people in person who I previous had only met or recognised from online interaction. I won’t single out individual 1-on-1s I had, but it was great to be able to put faces to names, and hearing peoples stories and visions in person was hugely inspiring. I talked to people involved in all sorts of cause areas and projects, and that combination of diversity, compassion, and moral seriousness is one of the best things about EA.
Listening to the two speakers from the Hibakusha Project at the closing talk was very moving, and clear case of how knowing something intellectually is not the same thing as hearing personal (and in-person) testimony. I think it would’ve been one of my conference highlights in the feedback form if we hadn’t already been asked to fill it out a few minutes beforehand!
I was going to make a point about a ‘lack of EA leadership’ turning up apart from Zach Robinson, but when I double-checked the event attendee list I think I was just wrong on this. Sure, a couple of big names didn’t turn up, and it may depend on what list of ‘EA leaders’ you’re using as a reference, but I want to admit I was directionally wrong here.
I thought Zach gave a good opening speech, but many people noted on the apparent dissonance between saying that CEA wanted to focus on ‘principles-first’ approach to EA, but that they also expected AI to be their area of most focus/highest priority and that they don’t expect that to change in the near future.
Finally, while I’m sure the people I spoke to (and those who wanted to speak to me) is strongly affected by selection-effects, and my own opinions on this are fairly strong, it did feel that there was consensus on there being a lack of trust/deference/shared beliefs from ‘Bay-Area EA’:[1]
Many people think that working on AI Safety and Governance is important and valuable, but not ‘overwhelmingly important’ or ‘the most important thing human has done/will ever do’. This included some fairly well-known names from those who attended, and basically nobody
there (as far as I could tell)I interacted with held extremely ‘doomer’ beliefs about AI.There was a lot of uncomfortable feeling at the community-building funding being directed to ‘longtermism’ and AI Safety in particular. This is definitely a topic I’m want to investigate more post-EAG, as I’m not sure what the truth of the matter is, but I’d certainly find it problematic if some of the anecdotes I heard were a fair representation of reality.
In any case, I think it’s clear that AI Safety is no longer ‘neglected’ within EA, and possibly outside of it.[2]On a personal level, it felt a bit odd to me that the LessOnline conference was held at exactly the same time as EAG. Feels like it could be a coincidence, but on the other hand this is not a coincidence because nothing is ever a coincidence. It feeds into my impression that the Bay is not very interested in what the rest of EA has to say.
One point which I didn’t get any clear answers to was ‘what are the feedback mechanisms in the community to push back on this’, and do such feedback mechanisms even exist?
In summary: It feels like, from my perspective, that the Bay Area/Exclusively Longtermist/AI Safety Maximalist version of EA has ‘lost of the mandate of heaven’, but nonetheless at the moment controls a lot of the community’s money and power. This, again, is a theme I want to explicitly explore in future posts.
I am old (over 30yo) and can’t party like the young EAs anymore 😔
I’m not sure I have a good name for this, or concrete dividing lines. But in discussions people seemed to understand what it was meant to capture.
To me it does seem like the case for the overwhelming importance of AI has been under-argued for and under-scrutinised.
I think this can’t be clear based only on observing lots of people at EAG are into it. You have to include some kind of independent evaluation of how much attention the area “should” have. For example, if you believed that AI alignment should receive as much attention as climate change, then EAG being fully 100% about AI would still not be enough to make it no longer neglected.
(Maybe you implicitly do have a model of this, but then I’d like to hear more about it.)
FWIW I’m not sure what my model is, but it involves the fact that despite many people being interested in the field, the number actually working on it full time still seems kind of small, and in particular still dramatically smaller than the number of people working on advancing AI tech.
It was really 90% coincidence in that Manifest and MATS basically fully determined when LessOnline would happen. I do think in a world where I considered myself more interested in investing in EA, or being involved in EA community building, I would have felt more sadness and hesitation about scheduling it at the same time, though I think it’s unlikely that would have shifted the overall decision (~15% for this weird counterfactual).
As Phib also says, it is the case that at least historically very few people travel for EAG. I was surprised by this when I did the surveys and analytics for this when I ran EAG in 2015 and 2016.
Here are some numbers from Swapcard for EAG London 2024:
55% of attendees were not from the UK, 14% of attendees were from the US, at least based on Swapcard data
London is a particularly easy city to travel to from the rest of Europe, but that’s still like 50% more than the baseline we had in 2015/2016/2017. The most relevant numbers here would be the people who would travel all the way from the U.S. and who would overlap with people who would want to attend LessOnline. My best guess is there are around 30-40 attendees for which there was a real conflict between the two events, though it wouldn’t surprise me if that’s off by a factor of 2-3 in either direction.
Raising my hand for an even more niche category: people who likely would have attended LessOnline had their partner not been attending EAG.
Detail, but afaict there were at least five Irish participants.
Thanks! I was using old data, I updated the table.
I’m surprised there were only five
“I do think in a world where I considered myself more interested in investing in EA, or being involved in EA community building, I would have felt more sadness and hesitation about scheduling it at the same time, though I think it’s unlikely that would have shifted the overall decision (~15% for this weird counterfactual)”
I find this comment quite discouraging that you didn’t feel sadness and hesitation about scheduling it at the same time. I would have hoped that leaders like you who organised important events like LessOnline, Manifest and MATS, that have EA heritage and connection would have at least a little interest in doing what was best for EA and community building (even without having to “invest” in it yourself) and therefore at least trying to co-ordinate with the CEA events crew.
I also think your comment partially refutes your assessment that it was “90% coincidence” that Manifest and MATS rather than EAG determined when LessOnline would be. If you care about the other 2 conferences but not much about clashes with EAG, then its hardly completely coincidence that you clashed with EAG.…
I didn’t say that I didn’t feel sadness or hesitation about scheduling it at the same time. Indeed, I think my comment directly implied that I did feel some sadness or hesitation, because I used the word “more”, implying there was indeed a baseline level of sadness or hesitation that’s non-zero.
Ignoring that detail, a bit of broader commentary on why I don’t feel that sad:
I at the moment think that most EA community building is net-negative for the world. I am still here as someone trying to hold people accountable and because I have contributed to a bunch of the harm this community has caused. I am in some important sense an “EA Leader” but I don’t seem to be on good terms with most of what you would call EA leadership, and honestly, I wish the EA community would disband and disappear and expect it to cause enormous harm in the future (or more ideally I wish it would undergo substantial reform, though my guess is the ship for that has sailed, which makes me deeply sad).
I have a lot of complicated opinions about what this implies about how I should relate to stuff like event scheduling, and helping with things like the EA Forum, that I would be happy to go into sometime (though this doesn’t seem like the right place for it). I care a lot about being cooperative despite my misgivings, and will continue to coordinate with people, but I feel like you should be aware that I do not wish the EA community to grow or gain more power in the world (though I am happy to engage in trade and to avoid dumb coordination problems that lose value for all parties involved).
MATS and Manifest pay us hundreds of thousands of dollars. It would have been obviously reckless and value destroying to pass on either of these contracts because of a scheduling conflict with a conference on the other side of the world, and also separately, a decision that would have substantially increased the risk of future bankruptcy of my organization. I do not consider myself under an obligation to make sacrifices at this level.
Feels like you should resign from EA Funds grantmaking then
I’ve considered it! My guess is it would be bad for evaporative cooling reasons for people like me to just leave the positions from which they could potentially fix and improve things (and IMO, it seems like a bad pattern that when someone starts thinking that we are causing harm that the first thing we do is to downvote their comment expressing such sadness and ask them to resign, that really seems like a great recipe for evaporative cooling).
Also separately, I am importantly on the Long Term Future Fund, not the EA Infrastructure Fund. I would have likely left or called for very substantial reform of the EA Infrastructure Fund, but the LTFF seems like it’s probably still overall doing good things (though I am definitely not confident).
Precommitting to not posting more in this whole thread, but I thought Habryka’s thoughts deserved a response
I think this is a fair cop.[1] I appreciate the added context you’ve added to your comment and have removed the downvote. Reforming EA is certainly high on my list of things to write about/work on, so would appreciate your thoughts and takes here even if I suspect I’ll ending up disagreeing with diagnosis/solutions.[2]
I guess that depends on the theory of change for improving things. If it’s using your influence and standing to suggest reforms and hold people accountable, sure. If it’s asking for the community to “disband and disappear”, I don’t know. Like, I don’t know in many other movements would that be tolerated with significant influence and funding power?[3] If one of the Lightcone Infrastructure team said “I think lightcone infrastructure in its entirety should shut down and disband, and return all funds” and then made decisions about funding and work that aligned with that goal and not yours, how long should they expect to remain part of the core team?
Maybe we’re disagreeing about what we mean by the ‘EA community’ implicitly here, and I feel that sometimes the ‘EA Community’ is used as a bit of a scapegoat, but when I see takes like this I think “Why should GWWC shut down and disband because of the actions of SBF/OpenAI?”—Like I think GWWC and its members definitely count as part of the EA Community, and your opinion seems to be pretty maximal without much room for exceptions.
(Also I think it’s important to note that your own Forum use seems to have contributed to instances of evaporative cooling, so that felt a little off to me.)
This is true, but LTFF is part of EA Funds, and to me is clearly EA-run/affiliated/associated. It feels like its odd that you’re a grantmaker who decides where money to the community, from one of its most well-known and accessible funds, and you think that said community should disperse/disband/not grow/is net-negative for the world. That just seems rife for weird incentives/decisions unless, again, you’re explicitly red-teaming grant proposals and funding decisions. If you’re using it to “run interference” from the inside, to move funding away from the EA community and its causes, that feels a lot more sketchy to me.
Never downvote while upset I guess
I think I’ve noted before that there’s a very large inferential difference between us, as we’re two very different people
Unless it was specifically for red-teaming
FWIW Habryka, I appreciate all that I know you’ve done and expect there’s a lot more I don’t know about that I should be appreciative of too.
I would also appreciate if you’d write up these concerns? I guess I want to know if I should feel similarly even as I rather trust your judgment. Sorry to ask, and thanks again
Editing to note I‘ve now seen some of comments elsewhere
I would be curious to hear you expand more on this:
What is your confidence level? (e.g. is it similar to the confidence you had in “very few people travel for EAG”, or is it something like 90%?)
What scenarios are you worried about? E.g. is it more about EA hastening the singularity by continuing to help research labs, or about EA making a government-caused slowdown less likely and less effective?
What is your main theory of change at the moment with rationalist community building, and how is it different from EA community building? Is it mostly focused on “slowing down AI progress, pivotal acts, intelligence enhancement”?
Extremely unconfident, both in overall probability and in robustness. It’s the kind of belief where I can easily imagine someone swaying me one way or another in a short period of time, and the kind of belief I’ve gone back and forth on a lot over the years.
On the question of confidence, I feel confused about how to talk about probabilities of expected value. My guess is EA is mostly irrelevant for the things that I care about in ~50% of worlds, is bad in like 30% of worlds and good in like 20% of worlds, but the exact operationalization here is quite messy. Also in the median world in which EA is bad, it seems likely to me that EA causes more harm than it makes up for in the median world where it is good.
Those are two relatively concrete things I am worried about. More broadly, I am worried about EA generally having a deceptive and sanity-reducing relationship to the world, and for it to be in some sense a honeypot that lots of the world’s smartest and most moral people end up getting stuck in and where they lend their credibility to bad actors (SBF and Sam Altman being the obvious examples here, and Anthropic seems like the one I am betting on will be looked back on similarly).
My key motivation is mostly “make key decision makers better informed and help smart and moral people understand the state of the world better”.
I think an attitude that promotes truth-seeking and informedness above other things is more conductive to that than EA stuff. I also don’t think I would describe most of my work straightforwardly as “rationalist community building”. LessWrong is its own thing that’s quite different from a lot of the rationality community, and is trying to do something relatively specific.
OK your initial message makes more sense given your response here—Although I can’t quite now connect why MATS and Manifest would be net positive things under this framework while EA community building would be net negative.
My slight pushback would be that EAG London is the most near-term focused of the EAGs, so some of the long-termist potential net negatives you list might not apply so much with that conference.
Yeah this is probably my biggest disagreement with Oli on this issue.
I presume the person doesn’t realise those events are hosted at your venue
Fwiw I think there was such a tendency.
+1, I always leave these conferences filled with inspiration and gratitude 😊
Yeah I was confused by this at first, but now I’m pretty sure this is a coincidence and there’s a good chance the organizers just didn’t think to check the dates of EAG.
I’m not sure I understand what “this” is referring to, but in general I think discussing things on the Forum is a reasonable way to provide feedback and push for change within EA. Stuff on the Forum does get around.
Maybe a better question is how neglected is this within society? And AI technical research is a lot less neglected than before, but governance work is still looking extremely neglected AND we appear to be in a critical policy window.
Hi, I went to Lessonline after registering for EAG London, my impression of both events being held on the same weekend is something like:
Events around the weekend (Manifest being held the weekend after Lessonline) informed Lessonline’s dates (but why not the weekend after Manifest then?)
People don’t travel internationally as much for EAGs (someone cited to me ~10% of attendees but my opinion on reflection is that this seems an underestimate).
I imagine EAG Bay Area, Global Catastrophic Risks in early Feb also somewhat covered the motivation for “AI Safety/EA conference”.
I think you’re right that it’s not entirely* a coincidence that Lessonline conflicted with EAG Bay Area, but I’m thinking this was done somewhat more casually and probably reasonably.
I think it’s odd, and other’s have noted too, the most significant AI safety conference shares space with things unrelated on an object-level. I think it’s further odd to consider, I’ve heard people say, why bother going to a conference like this when I live in the same city as the people I’d most want to talk with (Berkeley/SF).
Finally, I feel weird about AI, since I think insiders are only becoming more convinced/confirmed of extreme event likelihoods (AI capabilities). I think it has only become more important by virtue of most people updating timelines earlier, not later, and this includes Open Phil’s version of this (Ajeya and Joe Carlsmith’s AI timelines). In fact, I’ve heard arguments that it’s actually less important by virtue of, “the cat’s out of the bag and not even Open Phil can influence trajectories here.” Maybe AI safety feels less neglected because it’s being advocated from large labs, but that may be both a result of EA/EA-adjacent efforts and not really enough to solve a unilateralizing problem.
MATS is happening one week after Manifest.
In fairness you don’t need high p|doom to think AI safety should be the no.1 priority if you a) think that AI is a non-negligible extinction risk (say >0.1%), b) no other extinction risk has an equal degree of combined neglectedness and size and c) the expected value of the future conditional on us not going extinct in the next 100 years is astronomically high, and d) AI safety work makes a significant difference to how likely doom is to occur.. None of these are innocent or obvious assumptions, but I think a lot of people in the community hold all 3. I consider myself a critic of doomers in one sense, because I suspect p|doom is under 0.1%, and I think once you get down below that level, you should be nervous about taking expected value calculation that include your p|doom literally, because you probably don’t really know whether you should be 0.09% or several orders of magnitude lower. But even I am not *sure* that this is not swamped by c). (The Bostrom Pascal’s Mugging case involves probabilities way below 1 in a million, never mind 1 in a 1000.) Sometimes I get the impression though, that some people think of themselves as anti-doomer, when their p|doom is officially more like 1%. I think that’s a major error, if they really believe that figure. 1% is not low for human extinction. In fact, it’s not low even if you only care about currently existing people being murdered: in expectation that is 0.01x8 billion=80 million deaths(!). Insofar as what is going on is really just that people in their heart of hearts are much lower than 1%, but don’t want to say that because it feels extreme maybe this is ok. But if people actually mean figures like 1% or 5%, they ought to basically be on the doomers’ side, even if they think very high p|doom estimates given by some doomers are extremely implausible.
Going to merge replies into this one comment, rather than sending lots and flooding the forum. If I’ve @ you specifically and you don’t want to respond in the chain, feel free to DM:
On neglectedness—Yep, fair point that our relevant metric here is neglectedness in the world, not in EA. I think there is a point to make here but it was probably the wrong phrasing to use, I should have made it more about ‘AI Safety being too large a part of EA’ than ‘Lack of neglectedness in EA implies lower ITN returns overall’
On selection bias/other takes—These were only ever meant to be my takes and reflections, so I definitely think they’re only a very small part of the story. I guess @Stefan_Schubert would be interested to hear about your impression of ‘lack of leadership’ and any potential reasons why?
On the Bay/Insiders—It does seem like the Bay is convinced AI is the only game in town? (Aschenbrenner’s recent blog seems to validate this). @Phib would be interested to hear you say more on your last paragraph, I don’t think I entirely grok it but it sounds very interesting.
On the Object Level—I think this one for an upcoming sequence. Suffice to say that one can infer from my top level post that I have very different beliefs on this issue than many ‘insider EAs’, and I do work on AI/ML for my day job![1] But I think that while David sketches out a case for overall points, I think those points have been highly underargued and underscrutinised given their application in shaping the EA movement and its funding. So look it for a more specific sequence on the object level[2] maybe-soon-depending-on-writing-speed.
Which I have recently left to do some AI research and see if it’s the right fit for me.
Currently tentatively titled “Against the overwhelming importance of AI x-risk reduction”
Yeah, thank you, I guess I was trying to say that the evidence only seems to be stronger over time that the Bay Area’s: ‘AI is the only game in town’, is accurate.
Insofar as, timelines for various AI capabilities have outperformed both superforecasters’ and AI insiders’ predictions; transformative AI timelines (at Open Phil, prediction markets, AI experts I think) have decreased significantly over the past few years; the performance of LLMs have increased at an extraordinary rate across benchmarks; and we expect the next decade to extrapolate this scaling to some extent (w/ essentially hundreds of billions if not tens of trillions to be invested).
Although, yeah, I think to some extent we can’t know if this continues to scale as prettily as we’d expect and it’s especially hard to predict categorically new futures like exponential growth (10%, 50%, etc, growth/year). Given the forecasting efforts and trends thus far it feels like there’s a decent chance of these wild futures, and people are kinda updating all the way? Maybe not Open Phil entirely (to the point that EA isn’t just AIS), since they are hedging their altruistic bets, in the face of some possibility this decade could be ‘the precipice’ or one of the most important ever.
Misuse and AI risk seem like the negative valence of AI’s transformational potential. I personally buy the arguments around transformational technologies needing more reasoned steering and safety, and I also buy that EA has probably been a positive influence, and that alignment research has been at least somewhat tractable. Finally I think that there’s more that could be done to safely navigate this transition.
Also, re David (Thorstad?) yeah I haven’t engaged with his stuff as I probably should, and I really don’t know how to reason for or against arguments around the singularity, exponential growth, and the potential of AI without deferring to people more knowledgeable/smarter than me. I do feel like I have seen the start and middle of trends they predicted, and predict will extrapolate-with my own personal use and some early reports on productivity increases.
I do look forward to your sequence and hope you do really well on it!
I think this is basically entirely selection effects. Almost all the people I spoke to were “doomers” to some extent.
What do you mean by that? Presumably you do not mean it in any religious sense. Do you want to say that exclusively longtermist EA is much less popular among EAs than it used to be (i.e. the “heaven” is the opinion of the average EA)?
Ah sorry, it’s a bit of linguistic shortcut, I’ll try my best to explain more clearly:
As David says, it’s an idea from Chinese history. Rulers used the concept as a way of legitimising their hold on power, where Tian (Heaven) would bestow a ‘right to rule’ on the virtuous ruler. Conversely, rebellions/usurpations often used the same concept to justify their rebellion, often by claiming the current rulers had lost heaven’s mandate.
Roughly, I’m using this to analogise to the state of EA, where AI Safety and AI x-risk has become an increasingly large/well-funded/high-status[1] part of the movement, especially (at least apparently) amongst EA Leadership and the organisations that control most of the funding/community decisions.
My impression is that there was a consensus and ideological movement amongst EA leadership (as opposed to an already-held belief where they pulled a bait-and-switch), but many ‘rank-and-file’ EAs simply deferred to these people, rather than considering the arguments deeply.
I think that various amounts of scandals/bad outcomes/bad decisions/bad vibes around EA in recent years and at the moment can be linked to this turn towards the overwhelming importance of AI Safety, and as EffectiveAdvocate says below, I would like that part of EA to reduce its relative influence and power on the rest of it, and for rank-and-file EAs to stop deferring on this issue especially, but also in general.
I don’t like this term but again, I think people know what I mean when I say this
It’s a reference to an idea from old Chinese political thinking: https://en.wikipedia.org/wiki/Mandate_of_Heaven
That does not help me understand what is meant there. I fail to see relevant analogies to AI Safety.
I am fairly sure that the JWS means to say that these subgroups are about to / should lose some of their dominance in the EA movement.
Agree