I had considered calling the third wave of EA âEA in a World Where People Actually Listen to Usâ.
Leopoldâs situational awareness memo has become a salient example of this for me. I used to sometimes think that arguments about whether we should avoid discussing the power of AI in order to avoid triggering an arms race were a bit silly and self important because obviously defense leaders arenât going to be listening to some random internet charity nerds and changing policy as a result.
Well, they are and they are. Letâs hope itâs for the better.
Seems plausible the impact of that single individual act is so negative that aggregate impact of EA is negative.
I think people should reflect seriously upon this possibility and not fall prey to wishful thinking (letâs hope speeding up the AI race and making it superpower powered is the best intervention! itâs better if everyone warning about this was wrong and Leopold is right!).
The broader story here is that EA prioritization methodology is really good for finding highly leveraged spots in the world, but there isnât a good methodology for figuring out what to do in such places, and there also isnât a robust pipeline for promoting virtues and virtuous actors to such places.
Iâm not sure to what extent the Situational Awareness Memo or Leopold himself are representatives of âEAâ
In the pro-side:
Leopold thinks AGI is coming soon, will be a big deal, and that solving the alignment problem is one of the worldâs most important priorities
He used to work at GPI & FTX, and formerly identified with EA
He (probably almost certainly) personally knows lots of EA people in the Bay
On the con-side:
EA isnât just AI Safety (yet), so having short timelines/âhigh importance on AI shouldnât be sufficient to make someone an EA?[1]
EA shouldnât also just refer to a specific subset of the Bay Culture (please), or at least we need some more labels to distinguish different parts of it in that case
Many EAs have disagreed with various parts of the memo, e.g. Gideonâs well received post here
Since his EA institutional history he moved to OpenAI (mixed)[2] and now runs an AGI investment firm.
By self-identification, Iâm not sure Iâve seen Leopold identify as an EA at all recently.
This again comes down to the nebulousness of what âbeing an EAâ means.[3] I have no doubts at all that, given what Leopold thinks is the way to have the most impact heâll be very effective at achieving that.
Further, on your point, I think thereâs a reason to suspect that something like situational awareness went viral in a way that, say, Rethink Priorities Moral Weight project didnâtâthe promise many people see in powerful AI is power itself, and thatâs always going to be interesting for people to follow, so Iâm not sure that situational awareness becoming influential makes it more likely that other âEAâ ideas will
I view OpenAI as tending implicitly/âexplicitly anti-EA, though I donât think there was an explicit âpurgeâ, I think the culture/âvision of the company was changed such that card-carrying EAs didnât want to work there any more
I think he is pretty clearly an EA given he used to help run the Future Fund, or at most an only very recently ex-EA. Having said that, itâs not clear to me this means that âEAsâ are at fault for everything he does.
Yeah again I just think this depends on oneâs definition of EA, which is the point I was trying to make above.
Many people have turned away from EA, both the beliefs, institutions, and community in the aftermath of the FTX collapse. Even Ben Todd seems to not be an EA by some definitions any more, be that via association or identification. Who is to say Leopold is any different, or has not gone further? What then is the use of calling him EA, or using his views to represent the âThird Waveâ of EA?
I guess from my PoV what Iâm saying is that Iâm not sure thereâs much âconnective tissueâ between Leopold and myself, so when people use phrases like âlisten to usâ or âHow could we have doneâ I end up thinking âwho the heck is we/âus?â
In my post, I suggested that one possible future is that we stay at the âforefront of weirdness.â Calculating moral weights, to use your example.
I could imagine though that the fact that our opinions might be read by someone with access to the nuclear codes changes how we do things.
I wish there was more debate about which of these futures is more desirable.
(This is what I was trying to get out with my original post. Iâm not trying to make any strong claims about whether any individual person counts as âEAâ.)
I donât want to claim all EAs believe the same things, but if the congressional commission had listened to what you might call the âcentralâ EA position, it would not be recommending an arms race because it would be much more concerned about misalignment risk. The overwhelming majority of EAs involved in AI safety seem to agree that arms races are bad and misalignment risk is the biggest concern (within AI safety). So if anything this is a problem of the commission not listening to EAs, or at least selectively listening to only the parts they want to hear.
In most cases this is a rumors based thing, but I have heard that a substantial chunk of the OP-adjacent EA-policy space has been quite hawkish for many years, and at least the things I have heard is that a bunch of key leaders âbasically agreed with the China part of situational awarenessâ.
Again, people should really take this with a double-dose of salt, I am personally at like 50â50 of this being true, and I would love people like lukeprog or Holden or Jason Matheny or others high up at RAND to clarify their positions here. I am not attached to what I believe, but I have heard these rumors from sources that didnât seem crazy (but also various things could have been lost in a game of telephone, and being very concerned about China doesnât result in endorsing a âManhattan project to AGIâ, though the rumors that I have heard did sound like they would endorse that)
Less rumor-based, I also know that Dario has historically been very hawkish, and âneeding to beat Chinaâ was one of the top justifications historically given for why Anthropic does capability research. I have heard this from many people, so feel more comfortable saying it with fewer disclaimers, but am still only like 80% on it being true.
Overall, my current guess is that indeed, a large-ish fraction of the EA policy people would have pushed for things like this, and at least didnât seem like they would push back on it that much. My guess is âweâ are at least somewhat responsible for this, and there is much less of a consensus against a U.S. china arms race in US governance among EAs than one might think, and so the above is not much evidence that there was no listening or only very selective listening to EAs.
I looked thru the congressional commission reportâs list of testimonies for plausibly EA-adjacent people. The only EA-adjacent org I saw was CSET, which had two testimonies (1, 2). From a brief skim, neither one looked clearly pro- or anti-arms race. They seemed vaguely pro-arms race on vibes but I didnât see any claims that look like they were clearly encouraging an arms raceâbut like I said, I only briefly skimmed them, so I could have missed a lot.
This is inconsistent with my impressions and recollections. Most clearly, my sense is that CSET was (maybe still is, not sure) known for being very anti-escalatory towards China, and did substantial early research debunking hawkish views about AI progress in China, demonstrating it was less far along than ways widely believed in DC (and that EAs were involved in this, because they thought it was true and important, because they thought current false fears in the greater natsec community were enhancing arms race risks) (and this was when Jason was leading CSET, and OP supporting its founding). Some of the same people were also supportive of export controls, which are more ambiguous-sign here.
The export controls seemed like a pretty central example of hawkishness towards China and a reasonable precursor to this report. The central motivation in all that I have written related to them was about beating China in AI capabilities development.
Of course no one likes a symmetric arms race, but the question is did people favor the âquickly establish overwhelming dominance towards China by investing heavily in AIâ or the âtry to negotiate with China and not set an example of racing towards AGIâ strategy. My sense is many people favored the former (though definitely not all, and I am not saying that there is anything like consensus, my sense is itâs a quite divisive topic).
To support your point, I have seen much writing from Helen Toner on trying to dispel hawkishness towards China, and have been grateful for that. Against your point, at the recent âAI Security Forumâ in Vegas, many x-risk concerned people expressed very hawkish opinions.
Yeah re the export controls, I was trying to say âI think CSET was generally anti-escalatory, but in contrast, the effect of their export controls work was less soâ (though I used the word âambiguousâ because my impression was that some relevant people saw a pro of that work that it also mostly didnât directly advance AI progress in the US, i.e. it set China back without necessarily bringing the US forward towards AGI). To use your terminology, my impression is some of those people were âtrying to establish overwhelming dominance over Chinaâ but not by âinvesting heavily in AIâ.
It looks to me like the online EA community, and the EAs I know IRL, have a fairly strong consensus that arms races are bad. Perhaps thereâs a divide in opinions with most self-identified EAs on one side, and policy people /â company leaders on the other sideâwhich in my view is unfortunate since the people holding the most power are also the most wrong.
(Is there some systematic reason why this would be true? At least one part of it makes sense: people who start AGI companies must believe that building AGI is the right move. It could also be that power corrupts, or something.)
So maybe I should say the congressional commission shouldâve spent less time listening to EA policy people and more time reading the EA Forum. Which obviously was never going to happen but it wouldâve been nice.
Slightly independent to the point Habryka is making, which may well also be true, my anecdotal impression is that the online EA community /â EAs I know IRL were much bigger on âwe need to beat Chinaâ arguments 2-4 years ago. If so, simple lag can also be part of the story here. In particular I think it was the mainstream position just before ChatGPT was released, and partly as a result I doubt an âoverwhelming majority of EAs involved in AI safetyâ disagree with it even now.
So maybe (the argument goes) we should take a cue from the environmental activists, and be hostile towards AI companies...
This is the most common question I get on AI safety posts: why isnât the rationalist /â EA /â AI safety movement doing this more? Itâs a great question, and itâs one that the movement asks itself a lot...
Still, most people arenât doing this. Why not?
Later, talking about why attempting a regulatory approach to avoiding a race is futile:
The biggest problem is China. US regulations donât affect China. China says that AI leadership is a cornerstone of their national securityâboth as a massive boon to their surveillance state, and because it would boost their national pride if they could beat America in something so cutting-edge.
So the real question is: which would we prefer? OpenAI gets superintelligence in 2040? Or Facebook gets superintelligence in 2044? Or China gets superintelligence in 2048?
Might we be able to strike an agreement with China on AI, much as countries have previously made arms control or climate change agreements? This is . . . not technically prevented by the laws of physics, but it sounds really hard. When I bring this challenge up with AI policy people, they ask âHarder than the technical AI alignment problem?â Okay, fine, you win this one.
I feel like a generic non-EA policy person reading that post could well end up where the congressional commission landed? Itâs right there in the section that most explicitly talks about policy.
Huh, fwiw this is not my anecdotal experience. I would suggest that this is because I spend more time around doomers than you and doomers are very influenced by Yudkowskyâs âdonât fight over which monkey gets to eat the poison banana firstâ framing, but that seems contradicted by your example being ACX, who is also quite doomer-adjacent.
That sounds plausible. I do think of ACX as much more âaccelerationistâ than the doomer circles, for lack of a better term. Hereâs a more recent post from October 2023 informing that impression, below probably does a better job than I can do of adding nuance to Scottâs position.
Second, if we never get AI, I expect the future to be short and grim. Most likely we kill ourselves with synthetic biology. If not, some combination of technological and economic stagnation, rising totalitarianism + illiberalism+mobocracy, fertility collapse and dysgenics will impoverish the world and accelerate its decaying institutional quality. I donât spend much time worrying about any of these, because I think theyâll take a few generations to reach crisis level, and I expect technology to flip the gameboard well before then. But if we ban all gameboard-flipping technologies (the only other one I know is genetic enhancement, which is even more bannable), then we do end up with bioweapon catastrophe or social collapse. Iâve said before I think thereâs a ~20% chance of AI destroying the world. But if we donât get AI, I think thereâs a 50%+ chance in the next 100 years we end up dead or careening towards Venezuela. That doesnât mean I have to support AI accelerationism because 20% is smaller than 50%. Short, carefully-tailored pauses could improve the chance of AI going well by a lot, without increasing the risk of social collapse too much. But itâs something on my mind.
Scottâs last sentence seems to be claiming that avoiding an arms race is easier than solving alignment (and it would seem to follow from that that we shouldnât race). But I can see how a politician reading this article wouldnât see that implication.
Yep, my impression is that this is an opinion that people mostly adopted after spending a bunch of time in DC and engaging with governance stuff, and so is not something represented in the broader EA population.
My best explanation is that when working in governance, being pro-China is just very costly, and especially combining the belief that AI will be very powerful, and there is no urgency to beat China to it, seems very anti-memetic in DC, and so people working in the space started adopting those stances.
But I am not sure. There are also non-terrible arguments for beating China being really important (though they are mostly premised on alignment being relatively easy, which seems very wrong to me).
(though they are mostly premised on alignment being relatively easy, which seems very wrong to me)
Not just alignment being easy, but alignment being easy with overwhelmingly high probability. It seems to me that pushing for an arms race is bad even if thereâs only a 5% chance that alignment is hard.
I think most of those people believe that âhaving an AI aligned to âChinaâs valuesââ would be comparably bad to a catastrophic misalignment failure, and if you believe that, 5% is not sufficient, if you think there is a greater than 5% of China ending up with âaligned AIâ instead.
I think thatâs not a reasonable position to hold but I donât know how to constructively argue against it in a short comment so Iâll just register my disagreement.
Like, presumably Chinaâs values include humans existing and having mostly good experiences.
The Biden administrationâs decision, in October of last year, to impose drastic export controls on semiconductors, stands as one of its most substantial policy changes so far. As Jacobinâs Branko Marcetic wrote at the time, the controls were likely the first shot in a new economic Cold War between the United States and China, in which both superpowers (not to mention the rest of the world) will feel the hurt for years or decades, if not permanently.
[...]
The idea behind the policy, however, did not emerge from the ether. Three years before the current administration issued the rule, Congress was already receiving extensive testimony in favor of something much like it. The lengthy 2019 report from the National Security Commission on Artificial Intelligence suggests unambiguously that the âUnited States should commit to a strategy to stay at least two generations ahead of China in state-of-the-art microelectronicsâ and
The commission report makes repeated references to the risks posed by AI development in âauthoritarianâ regimes like Chinaâs, predicting dire consequences as compared with similar research and development carried out under the auspices of liberal democracy. (Its hand-wringing in particular about AI-powered, authoritarian Chinese surveillance is ironic, as it also ominously exhorts, âThe [US] Intelligence Community (IC) should adopt and integrate AI-enabled capabilities across all aspects of its work, from collection to analysis.â)
These emphases on the dangers of morally misinformed AI are no accident. The commission head was Eric Schmidt, tech billionaire and contributor to Future Forward, whose philanthropic venture Schmidt Futures has both deep ties with the longtermist community and a record of shady influence over the White House on science policy. Schmidt himself has voiced measured concern about AI safety, albeit tinged with optimism, opining that âdoomsday scenariosâ of AI run amok deserve âthoughtful consideration.â He has also coauthored a book on the future risks of AI, with no lesser an expert on morally unchecked threats to human life than notorious war criminal Henry Kissinger.
Also of note is commission member Jason Matheny, CEO of the RAND Corporation. Matheny is an alum of the longtermist Future of Humanity Institute (FHI) at the University of Oxford, who has claimed existential risk and machine intelligence are more dangerous than any historical pandemics and âa neglected topic in both the scientific and governmental communities, but itâs hard to think of a topic more important than human survival.â This commission report was not his last testimony to Congress on the subject, either: in September 2020, he would individually speak before the House Budget Committee urging âmultilateral export controls on the semiconductor manufacturing equipment needed to produce advanced chips,â the better to preserve American dominance in AI.
Congressional testimony and his position at the RAND Corporation, moreover, were not Mathenyâs only channels for influencing US policy on the matter. In 2021 and 2022, he served in the White Houseâs Office of Science and Technology Policy (OSTP) as deputy assistant to the president for technology and national security and as deputy director for national security (the head of the OSTP national security division). As a senior figure in the Office â to which Biden has granted âunprecedented access and powerâ â advice on policies like the October export controls would have fallen squarely within his professional mandate.
The most significant restrictions advocates (aside from Matheny) to emerge from CSET, however, have been Saif Khan and Kevin Wolf. The former is an alum from the Center and, since April 2021, the director for technology and national security at the White House National Security Council. The latter has been a senior fellow at CSET since February 2022 and has a long history of service in and connections with US export policy. He served as assistant secretary of commerce for export administration from 2010â17 (among other work in the field, both private and public), and his extensive familiarity with the US export regulation system would be valuable to anyone aspiring to influence policy on the subject. Both would, before and after October, champion the semiconductor controls.
At CSET, Khan published repeatedly on the topic, time and again calling for the United States to implement semiconductor export controls to curb Chinese progress on AI. In March 2021, he testified before the Senate, arguing that the United States must impose such controls âto ensure that democracies lead in advanced chips and that they are used for good.â (Paradoxically, in the same breath the address calls on the United States to both âidentify opportunities to collaborate with competitors, including China, to build confidence and avoid races to the bottomâ and to âtightly control exports of American technology to human rights abusers,â such as⊠China.)
Among Khanâs coauthors was aforementioned former congressional hopeful and longtermist Carrick Flynn, previously assistant director of the Center for the Governance of AI at FHI. Flynn himself individually authored a CSET issue brief, âRecommendations on Export Controls for Artificial Intelligence,â in February 2020. The brief, unsurprisingly, argues for tightened semiconductor export regulation much like Khan and Matheny.
This February, Wolf too provided a congressional address on âAdvancing National Security and Foreign Policy Through Sanctions, Export Controls, and Other Economic Tools,â praising the October controls and urging further policy in the same vein. In it, he claims knowledge of the specific motivations of the controlsâ writers:
BIS did not rely on ECRAâs emerging and foundational technology provisions when publishing this rule so that it would not need to seek public comments before publishing it.
These motivations also clearly included exactly the sorts of AI concerns Matheny, Khan, Flynn, and other longtermists had long raised in this connection. In its background summary, the text of one rule explicitly links the controls with hopes of retarding Chinaâs AI development. Using language that could easily have been ripped from a CSET paper on the topic, the summary warns that ââsupercomputersâ are being used by the PRC to improve calculations in weapons design and testing including for WMD, such as nuclear weapons, hypersonics and other advanced missile systems, and to analyze battlefield effects,â as well as bolster citizen surveillance.
[...]
Longtermists, in short, have since at least 2019 exerted a strong influence over what would become the Biden White Houseâs October 2022 semiconductor export rules. If the policy is not itself the direct product of institutional longtermists, it at the very least bears the stamp of their enthusiastic approval and close monitoring.
Just as it would be a mistake to restrict interest in longtermismâs political ambitions exclusively to election campaigns, it would be shortsighted to treat its work on semiconductor infrastructure as a one-off incident. Khan and Matheny, among others, remain in positions of considerable influence, and have demonstrated a commitment to bringing longtermist concerns to bear on matters of high policy. The policy sophistication, political reach, and fresh-faced enthusiasm on display in its semiconductor export maneuvering should earn the AI doomsday lobby its fair share of critical attention in the years to come.
The article seems quite biased to me, but I do think some of the basics here make sense and match with things I have heard (but also, some of it seems wrong).
Maybe instead of âwhere people actually listen to usâ itâs more like âEA in a world where people filter the most memetically fit of our ideas through their preconceived notions into something that only vaguely resembles what the median EA cares about but is importantly different from the world in which EA didnât exist.â
Call me a hater, and believe me, I am, but maybe someone who went to university at 16 and clearly spent most of their time immersed in books is not the most socially developed.
Maybe after they are implicated in a huge scandal that destroyed our movementâs reputation we should gently nudge them to not go on popular podcasts and talk fantastically and almost giddily about how world war 3 is just around the corner. Especially when they are working in a financialcapacity in which they would benefit from said war.
Many of the people we have let be in charge of our movement and speak on behalf of it donât know the first thing about optics or leadership or politics. I donât think Elizier Yudowsky could win a middle school class president race with a million dollars.
I know your point was specifically tailored toward optics and thinking carefully about what we say when we have a large platform, but I think looking back and forward bad optics and a lack of real politik messaging are pretty obvious failure modes of a movement filled with chronically online young males who worship intelligence and research output above all else. Iâm not trying to sh*t on Leopold and I donât claim I was out here beating a drum about the risks of these specific papers but yea I do think this is one symptom of a larger problem. I can barely think of anyone high up (publicly) in this movement who has risen via organizing.
(I think the issue with Leopold is somewhat precisely that he seems to be quite politically savvy in a way that seems likely to make him a deca-multi-millionaire and politically influental, possibly at the cost of all of humanity. I agree Eliezer is not the best presenter, but his error modes are clearly enormously different)
I donât think I was claiming they have the exact same failure modesâdo you want to point out where I did that? Rather they both have failure modes that I would expect to happen as a result of selecting them to be talking heads on the basis of wits and research output. Also I feel like you are implying Leopold is evil or something like that and I donât agree but maybe Iâm misinterpretting.
He seems like a smooth operator in some ways and certainly is quite different than Elizier. That being said I showed my dad (who has become an oddly good litmus test for a lot of this stuff for me as someone who is somewhat sympathethic to our movement but also a pretty normal 60 year old man in a completely different headspace) the Dwarkesh episode and he thought Leopold was very, very, very weird (and not because of his ideas). He kind of reminds me of Peter Thiel. Iâll completely admit I wasnât especially clear in my points and that mostly reflects my own lack of clarity on the exact point I was trying to getting across.
I think I take back like 20% of what I said (basically to the extent I was making a very direct stab at what exactly that failure mode is) but mostly still stand by the original comment, which again I see as being approximately ~ âSelecting people to be the public figureheads of our movement on the basis wits and research output is likely to be bad for usâ.
The thing about Yudkowsky is that, yes, on the one hand, every time I read him, I think he surely must be coming across as super-weird and dodgy to ânormalâ people. But on the other hand, actually, it seems like he HAS done really well in getting people to take his ideas seriously? Sam Altman was trolling Yudkowsky on twitter a while back about how many of the people running/âfounding AGI labs had been inspired to do so by his work. He got invited to write on AI governance for TIME despite having no formal qualifications or significant scientific achievements whatsoever. I think if we actually look at his track record, he has done pretty well at convincing influential people to adopt what were once extremely fringe views, whilst also succeeding in being seen by the wider world as one of the most important proponents of those views, despite an almost complete lack of mainstream, legible credentials.
Hmm, I hear what you are saying but that could easily be attributed to some mix of
(1) he has really good/âconvincing ideas
(2) he seems to be a a public representative for the EA/âLW community for a journalist on the outside.
And Iâm responding to someone saying that we are in âphase 3ââthat is to say people in the public are listening to usâso I guess Iâm not extremely concerned about him not being able to draw attention or convince people. Iâm more just generally worried that people like him are not who we should be promoting to positions of power, even if those are de jure positions.
Yeah, Iâm not a Yudkowsky fan. But I think the fact that he mostly hasnât been a PR disaster is striking, surprising and not much remarked upon, including by people who are big fans.
I guess in thinking about this I realize itâs so hard to even know if someone is a âPR disasterâ that I probably have just been confirming my biases. What makes you say that he hasnât been?
Just the stuff I already said about the success he seems to have had. It is also true that many people hate him and think heâs ridiculous, but I think that makes him polarizing rather than disastrous. I suppose you could phrase it as âhe was a disaster in some ways but a success in othersâ if you want to.
How do you know Leopold or anyone else actually influenced the commissionâs report? Not that that seems particularly unlikely to me, but is there any hard evidence? EDIT: I text-searched the report and he is not mentioned by name, although obviously that doesnât prove much on its own.
Hi Ben! You might be interested to know I literally had a meeting with the Assistant Defence Minister in Australia about 10 months ago off the back of one email. I wrote about it here. AI Safety advocacy is IMO still low extremely hanging fruit. My best theory is EAs donât want to do it because EAs are drawn to spreadsheets etc (it isnât their comparative advantage).
It seems to plausible that, much like Environmental Political Orthodoxy (reverence for simple rural living as expressed through localism, anti-nuclear sentiment, etc.) ultimately led the environmental movement to be harmful for its own professed goals, EA Political Orthodoxy (technocratic liberalism, âmistake theoryâ, general disdain for social science) could (and maybe already had, with the creation of OpenAI) ultimately lead EA efforts on AI to be a net negative by its own standards.
EA in a World Where People Actually Listen to Us
I had considered calling the third wave of EA âEA in a World Where People Actually Listen to Usâ.
Leopoldâs situational awareness memo has become a salient example of this for me. I used to sometimes think that arguments about whether we should avoid discussing the power of AI in order to avoid triggering an arms race were a bit silly and self important because obviously defense leaders arenât going to be listening to some random internet charity nerds and changing policy as a result.
Well, they are and they are. Letâs hope itâs for the better.
Seems plausible the impact of that single individual act is so negative that aggregate impact of EA is negative.
I think people should reflect seriously upon this possibility and not fall prey to wishful thinking (letâs hope speeding up the AI race and making it superpower powered is the best intervention! itâs better if everyone warning about this was wrong and Leopold is right!).
The broader story here is that EA prioritization methodology is really good for finding highly leveraged spots in the world, but there isnât a good methodology for figuring out what to do in such places, and there also isnât a robust pipeline for promoting virtues and virtuous actors to such places.
I spent all day in tears when I read the congressional report. This is a nightmare. I was literally hoping to wake up from a bad dream.
I really hope people donât suffer for our sins.
How could we have done something so terrible. Starting an arms race and making literal war more likely.
| and there also isnât a robust pipeline for promoting virtues and virtuous actors to such places.
this ^
Iâm not sure to what extent the Situational Awareness Memo or Leopold himself are representatives of âEAâ
In the pro-side:
Leopold thinks AGI is coming soon, will be a big deal, and that solving the alignment problem is one of the worldâs most important priorities
He used to work at GPI & FTX, and formerly identified with EA
He (
probablyalmost certainly) personally knows lots of EA people in the BayOn the con-side:
EA isnât just AI Safety (yet), so having short timelines/âhigh importance on AI shouldnât be sufficient to make someone an EA?[1]
EA shouldnât also just refer to a specific subset of the Bay Culture (please), or at least we need some more labels to distinguish different parts of it in that case
Many EAs have disagreed with various parts of the memo, e.g. Gideonâs well received post here
Since his EA institutional history he moved to OpenAI (mixed)[2] and now runs an AGI investment firm.
By self-identification, Iâm not sure Iâve seen Leopold identify as an EA at all recently.
This again comes down to the nebulousness of what âbeing an EAâ means.[3] I have no doubts at all that, given what Leopold thinks is the way to have the most impact heâll be very effective at achieving that.
Further, on your point, I think thereâs a reason to suspect that something like situational awareness went viral in a way that, say, Rethink Priorities Moral Weight project didnâtâthe promise many people see in powerful AI is power itself, and thatâs always going to be interesting for people to follow, so Iâm not sure that situational awareness becoming influential makes it more likely that other âEAâ ideas will
Plenty of e/âaccs have these two beliefs as well, they just expect alignment by default, for instance
I view OpenAI as tending implicitly/âexplicitly anti-EA, though I donât think there was an explicit âpurgeâ, I think the culture/âvision of the company was changed such that card-carrying EAs didnât want to work there any more
The 3 big defintions I have (self-identification, beliefs, actions) could all easily point in different directions for Leopold
I think he is pretty clearly an EA given he used to help run the Future Fund, or at most an only very recently ex-EA. Having said that, itâs not clear to me this means that âEAsâ are at fault for everything he does.
Yeah again I just think this depends on oneâs definition of EA, which is the point I was trying to make above.
Many people have turned away from EA, both the beliefs, institutions, and community in the aftermath of the FTX collapse. Even Ben Todd seems to not be an EA by some definitions any more, be that via association or identification. Who is to say Leopold is any different, or has not gone further? What then is the use of calling him EA, or using his views to represent the âThird Waveâ of EA?
I guess from my PoV what Iâm saying is that Iâm not sure thereâs much âconnective tissueâ between Leopold and myself, so when people use phrases like âlisten to usâ or âHow could we have doneâ I end up thinking âwho the heck is we/âus?â
In my post, I suggested that one possible future is that we stay at the âforefront of weirdness.â Calculating moral weights, to use your example.
I could imagine though that the fact that our opinions might be read by someone with access to the nuclear codes changes how we do things.
I wish there was more debate about which of these futures is more desirable.
(This is what I was trying to get out with my original post. Iâm not trying to make any strong claims about whether any individual person counts as âEAâ.)
I donât want to claim all EAs believe the same things, but if the congressional commission had listened to what you might call the âcentralâ EA position, it would not be recommending an arms race because it would be much more concerned about misalignment risk. The overwhelming majority of EAs involved in AI safety seem to agree that arms races are bad and misalignment risk is the biggest concern (within AI safety). So if anything this is a problem of the commission not listening to EAs, or at least selectively listening to only the parts they want to hear.
In most cases this is a rumors based thing, but I have heard that a substantial chunk of the OP-adjacent EA-policy space has been quite hawkish for many years, and at least the things I have heard is that a bunch of key leaders âbasically agreed with the China part of situational awarenessâ.
Again, people should really take this with a double-dose of salt, I am personally at like 50â50 of this being true, and I would love people like lukeprog or Holden or Jason Matheny or others high up at RAND to clarify their positions here. I am not attached to what I believe, but I have heard these rumors from sources that didnât seem crazy (but also various things could have been lost in a game of telephone, and being very concerned about China doesnât result in endorsing a âManhattan project to AGIâ, though the rumors that I have heard did sound like they would endorse that)
Less rumor-based, I also know that Dario has historically been very hawkish, and âneeding to beat Chinaâ was one of the top justifications historically given for why Anthropic does capability research. I have heard this from many people, so feel more comfortable saying it with fewer disclaimers, but am still only like 80% on it being true.
Overall, my current guess is that indeed, a large-ish fraction of the EA policy people would have pushed for things like this, and at least didnât seem like they would push back on it that much. My guess is âweâ are at least somewhat responsible for this, and there is much less of a consensus against a U.S. china arms race in US governance among EAs than one might think, and so the above is not much evidence that there was no listening or only very selective listening to EAs.
I looked thru the congressional commission reportâs list of testimonies for plausibly EA-adjacent people. The only EA-adjacent org I saw was CSET, which had two testimonies (1, 2). From a brief skim, neither one looked clearly pro- or anti-arms race. They seemed vaguely pro-arms race on vibes but I didnât see any claims that look like they were clearly encouraging an arms raceâbut like I said, I only briefly skimmed them, so I could have missed a lot.
This is inconsistent with my impressions and recollections. Most clearly, my sense is that CSET was (maybe still is, not sure) known for being very anti-escalatory towards China, and did substantial early research debunking hawkish views about AI progress in China, demonstrating it was less far along than ways widely believed in DC (and that EAs were involved in this, because they thought it was true and important, because they thought current false fears in the greater natsec community were enhancing arms race risks) (and this was when Jason was leading CSET, and OP supporting its founding). Some of the same people were also supportive of export controls, which are more ambiguous-sign here.
The export controls seemed like a pretty central example of hawkishness towards China and a reasonable precursor to this report. The central motivation in all that I have written related to them was about beating China in AI capabilities development.
Of course no one likes a symmetric arms race, but the question is did people favor the âquickly establish overwhelming dominance towards China by investing heavily in AIâ or the âtry to negotiate with China and not set an example of racing towards AGIâ strategy. My sense is many people favored the former (though definitely not all, and I am not saying that there is anything like consensus, my sense is itâs a quite divisive topic).
To support your point, I have seen much writing from Helen Toner on trying to dispel hawkishness towards China, and have been grateful for that. Against your point, at the recent âAI Security Forumâ in Vegas, many x-risk concerned people expressed very hawkish opinions.
Yeah re the export controls, I was trying to say âI think CSET was generally anti-escalatory, but in contrast, the effect of their export controls work was less soâ (though I used the word âambiguousâ because my impression was that some relevant people saw a pro of that work that it also mostly didnât directly advance AI progress in the US, i.e. it set China back without necessarily bringing the US forward towards AGI). To use your terminology, my impression is some of those people were âtrying to establish overwhelming dominance over Chinaâ but not by âinvesting heavily in AIâ.
It looks to me like the online EA community, and the EAs I know IRL, have a fairly strong consensus that arms races are bad. Perhaps thereâs a divide in opinions with most self-identified EAs on one side, and policy people /â company leaders on the other sideâwhich in my view is unfortunate since the people holding the most power are also the most wrong.
(Is there some systematic reason why this would be true? At least one part of it makes sense: people who start AGI companies must believe that building AGI is the right move. It could also be that power corrupts, or something.)
So maybe I should say the congressional commission shouldâve spent less time listening to EA policy people and more time reading the EA Forum. Which obviously was never going to happen but it wouldâve been nice.
Slightly independent to the point Habryka is making, which may well also be true, my anecdotal impression is that the online EA community /â EAs I know IRL were much bigger on âwe need to beat Chinaâ arguments 2-4 years ago. If so, simple lag can also be part of the story here. In particular I think it was the mainstream position just before ChatGPT was released, and partly as a result I doubt an âoverwhelming majority of EAs involved in AI safetyâ disagree with it even now.
Example from August 2022:
https://ââwww.astralcodexten.com/ââp/ââwhy-not-slow-ai-progress
Later, talking about why attempting a regulatory approach to avoiding a race is futile:
I feel like a generic non-EA policy person reading that post could well end up where the congressional commission landed? Itâs right there in the section that most explicitly talks about policy.
Huh, fwiw this is not my anecdotal experience. I would suggest that this is because I spend more time around doomers than you and doomers are very influenced by Yudkowskyâs âdonât fight over which monkey gets to eat the poison banana firstâ framing, but that seems contradicted by your example being ACX, who is also quite doomer-adjacent.
That sounds plausible. I do think of ACX as much more âaccelerationistâ than the doomer circles, for lack of a better term. Hereâs a more recent post from October 2023 informing that impression, below probably does a better job than I can do of adding nuance to Scottâs position.
https://ââwww.astralcodexten.com/ââp/ââpause-for-thought-the-ai-pause-debate
Scottâs last sentence seems to be claiming that avoiding an arms race is easier than solving alignment (and it would seem to follow from that that we shouldnât race). But I can see how a politician reading this article wouldnât see that implication.
Yep, my impression is that this is an opinion that people mostly adopted after spending a bunch of time in DC and engaging with governance stuff, and so is not something represented in the broader EA population.
My best explanation is that when working in governance, being pro-China is just very costly, and especially combining the belief that AI will be very powerful, and there is no urgency to beat China to it, seems very anti-memetic in DC, and so people working in the space started adopting those stances.
But I am not sure. There are also non-terrible arguments for beating China being really important (though they are mostly premised on alignment being relatively easy, which seems very wrong to me).
Not just alignment being easy, but alignment being easy with overwhelmingly high probability. It seems to me that pushing for an arms race is bad even if thereâs only a 5% chance that alignment is hard.
I think most of those people believe that âhaving an AI aligned to âChinaâs valuesââ would be comparably bad to a catastrophic misalignment failure, and if you believe that, 5% is not sufficient, if you think there is a greater than 5% of China ending up with âaligned AIâ instead.
I think thatâs not a reasonable position to hold but I donât know how to constructively argue against it in a short comment so Iâll just register my disagreement.
Like, presumably Chinaâs values include humans existing and having mostly good experiences.
Yep, I agree with this, but it appears nevertheless a relatively prevalent opinion among many EAs working in AI policy.
A somewhat relevant article that I discovered while researching this: Longtermists Are Pushing a New Cold War With ChinaâJacobin
The article seems quite biased to me, but I do think some of the basics here make sense and match with things I have heard (but also, some of it seems wrong).
Maybe instead of âwhere people actually listen to usâ itâs more like âEA in a world where people filter the most memetically fit of our ideas through their preconceived notions into something that only vaguely resembles what the median EA cares about but is importantly different from the world in which EA didnât exist.â
On that framing, I agree that thatâs something that happens and that we should be able to anticipate will happen.
Call me a hater, and believe me, I am, but maybe someone who went to university at 16 and clearly spent most of their time immersed in books is not the most socially developed.
Maybe after they are implicated in a huge scandal that destroyed our movementâs reputation we should gently nudge them to not go on popular podcasts and talk fantastically and almost giddily about how world war 3 is just around the corner. Especially when they are working in a financial capacity in which they would benefit from said war.
Many of the people we have let be in charge of our movement and speak on behalf of it donât know the first thing about optics or leadership or politics. I donât think Elizier Yudowsky could win a middle school class president race with a million dollars.
I know your point was specifically tailored toward optics and thinking carefully about what we say when we have a large platform, but I think looking back and forward bad optics and a lack of real politik messaging are pretty obvious failure modes of a movement filled with chronically online young males who worship intelligence and research output above all else. Iâm not trying to sh*t on Leopold and I donât claim I was out here beating a drum about the risks of these specific papers but yea I do think this is one symptom of a larger problem. I can barely think of anyone high up (publicly) in this movement who has risen via organizing.
(I think the issue with Leopold is somewhat precisely that he seems to be quite politically savvy in a way that seems likely to make him a deca-multi-millionaire and politically influental, possibly at the cost of all of humanity. I agree Eliezer is not the best presenter, but his error modes are clearly enormously different)
I donât think I was claiming they have the exact same failure modesâdo you want to point out where I did that? Rather they both have failure modes that I would expect to happen as a result of selecting them to be talking heads on the basis of wits and research output. Also I feel like you are implying Leopold is evil or something like that and I donât agree but maybe Iâm misinterpretting.
He seems like a smooth operator in some ways and certainly is quite different than Elizier. That being said I showed my dad (who has become an oddly good litmus test for a lot of this stuff for me as someone who is somewhat sympathethic to our movement but also a pretty normal 60 year old man in a completely different headspace) the Dwarkesh episode and he thought Leopold was very, very, very weird (and not because of his ideas). He kind of reminds me of Peter Thiel. Iâll completely admit I wasnât especially clear in my points and that mostly reflects my own lack of clarity on the exact point I was trying to getting across.
I think I take back like 20% of what I said (basically to the extent I was making a very direct stab at what exactly that failure mode is) but mostly still stand by the original comment, which again I see as being approximately ~ âSelecting people to be the public figureheads of our movement on the basis wits and research output is likely to be bad for usâ.
The thing about Yudkowsky is that, yes, on the one hand, every time I read him, I think he surely must be coming across as super-weird and dodgy to ânormalâ people. But on the other hand, actually, it seems like he HAS done really well in getting people to take his ideas seriously? Sam Altman was trolling Yudkowsky on twitter a while back about how many of the people running/âfounding AGI labs had been inspired to do so by his work. He got invited to write on AI governance for TIME despite having no formal qualifications or significant scientific achievements whatsoever. I think if we actually look at his track record, he has done pretty well at convincing influential people to adopt what were once extremely fringe views, whilst also succeeding in being seen by the wider world as one of the most important proponents of those views, despite an almost complete lack of mainstream, legible credentials.
Hmm, I hear what you are saying but that could easily be attributed to some mix of
(1) he has really good/âconvincing ideas
(2) he seems to be a a public representative for the EA/âLW community for a journalist on the outside.
And Iâm responding to someone saying that we are in âphase 3ââthat is to say people in the public are listening to usâso I guess Iâm not extremely concerned about him not being able to draw attention or convince people. Iâm more just generally worried that people like him are not who we should be promoting to positions of power, even if those are de jure positions.
Yeah, Iâm not a Yudkowsky fan. But I think the fact that he mostly hasnât been a PR disaster is striking, surprising and not much remarked upon, including by people who are big fans.
I guess in thinking about this I realize itâs so hard to even know if someone is a âPR disasterâ that I probably have just been confirming my biases. What makes you say that he hasnât been?
Just the stuff I already said about the success he seems to have had. It is also true that many people hate him and think heâs ridiculous, but I think that makes him polarizing rather than disastrous. I suppose you could phrase it as âhe was a disaster in some ways but a success in othersâ if you want to.
How do you know Leopold or anyone else actually influenced the commissionâs report? Not that that seems particularly unlikely to me, but is there any hard evidence? EDIT: I text-searched the report and he is not mentioned by name, although obviously that doesnât prove much on its own.
Hi Ben! You might be interested to know I literally had a meeting with the Assistant Defence Minister in Australia about 10 months ago off the back of one email. I wrote about it here. AI Safety advocacy is IMO still low extremely hanging fruit. My best theory is EAs donât want to do it because EAs are drawn to spreadsheets etc (it isnât their comparative advantage).
It seems to plausible that, much like Environmental Political Orthodoxy (reverence for simple rural living as expressed through localism, anti-nuclear sentiment, etc.) ultimately led the environmental movement to be harmful for its own professed goals, EA Political Orthodoxy (technocratic liberalism, âmistake theoryâ, general disdain for social science) could (and maybe already had, with the creation of OpenAI) ultimately lead EA efforts on AI to be a net negative by its own standards.