I had considered calling the third wave of EA âEA in a World Where People Actually Listen to Usâ.
Leopoldâs situational awareness memo has become a salient example of this for me. I used to sometimes think that arguments about whether we should avoid discussing the power of AI in order to avoid triggering an arms race were a bit silly and self important because obviously defense leaders arenât going to be listening to some random internet charity nerds and changing policy as a result.
Well, they are and they are. Letâs hope itâs for the better.
Seems plausible the impact of that single individual act is so negative that aggregate impact of EA is negative.
I think people should reflect seriously upon this possibility and not fall prey to wishful thinking (letâs hope speeding up the AI race and making it superpower powered is the best intervention! itâs better if everyone warning about this was wrong and Leopold is right!).
The broader story here is that EA prioritization methodology is really good for finding highly leveraged spots in the world, but there isnât a good methodology for figuring out what to do in such places, and there also isnât a robust pipeline for promoting virtues and virtuous actors to such places.
Iâm not sure to what extent the Situational Awareness Memo or Leopold himself are representatives of âEAâ
In the pro-side:
Leopold thinks AGI is coming soon, will be a big deal, and that solving the alignment problem is one of the worldâs most important priorities
He used to work at GPI & FTX, and formerly identified with EA
He (probably almost certainly) personally knows lots of EA people in the Bay
On the con-side:
EA isnât just AI Safety (yet), so having short timelines/âhigh importance on AI shouldnât be sufficient to make someone an EA?[1]
EA shouldnât also just refer to a specific subset of the Bay Culture (please), or at least we need some more labels to distinguish different parts of it in that case
Many EAs have disagreed with various parts of the memo, e.g. Gideonâs well received post here
Since his EA institutional history he moved to OpenAI (mixed)[2] and now runs an AGI investment firm.
By self-identification, Iâm not sure Iâve seen Leopold identify as an EA at all recently.
This again comes down to the nebulousness of what âbeing an EAâ means.[3] I have no doubts at all that, given what Leopold thinks is the way to have the most impact heâll be very effective at achieving that.
Further, on your point, I think thereâs a reason to suspect that something like situational awareness went viral in a way that, say, Rethink Priorities Moral Weight project didnâtâthe promise many people see in powerful AI is power itself, and thatâs always going to be interesting for people to follow, so Iâm not sure that situational awareness becoming influential makes it more likely that other âEAâ ideas will
I view OpenAI as tending implicitly/âexplicitly anti-EA, though I donât think there was an explicit âpurgeâ, I think the culture/âvision of the company was changed such that card-carrying EAs didnât want to work there any more
I think he is pretty clearly an EA given he used to help run the Future Fund, or at most an only very recently ex-EA. Having said that, itâs not clear to me this means that âEAsâ are at fault for everything he does.
Yeah again I just think this depends on oneâs definition of EA, which is the point I was trying to make above.
Many people have turned away from EA, both the beliefs, institutions, and community in the aftermath of the FTX collapse. Even Ben Todd seems to not be an EA by some definitions any more, be that via association or identification. Who is to say Leopold is any different, or has not gone further? What then is the use of calling him EA, or using his views to represent the âThird Waveâ of EA?
I guess from my PoV what Iâm saying is that Iâm not sure thereâs much âconnective tissueâ between Leopold and myself, so when people use phrases like âlisten to usâ or âHow could we have doneâ I end up thinking âwho the heck is we/âus?â
In my post, I suggested that one possible future is that we stay at the âforefront of weirdness.â Calculating moral weights, to use your example.
I could imagine though that the fact that our opinions might be read by someone with access to the nuclear codes changes how we do things.
I wish there was more debate about which of these futures is more desirable.
(This is what I was trying to get out with my original post. Iâm not trying to make any strong claims about whether any individual person counts as âEAâ.)
I donât want to claim all EAs believe the same things, but if the congressional commission had listened to what you might call the âcentralâ EA position, it would not be recommending an arms race because it would be much more concerned about misalignment risk. The overwhelming majority of EAs involved in AI safety seem to agree that arms races are bad and misalignment risk is the biggest concern (within AI safety). So if anything this is a problem of the commission not listening to EAs, or at least selectively listening to only the parts they want to hear.
In most cases this is a rumors based thing, but I have heard that a substantial chunk of the OP-adjacent EA-policy space has been quite hawkish for many years, and at least the things I have heard is that a bunch of key leaders âbasically agreed with the China part of situational awarenessâ.
Again, people should really take this with a double-dose of salt, I am personally at like 50â50 of this being true, and I would love people like lukeprog or Holden or Jason Matheny or others high up at RAND to clarify their positions here. I am not attached to what I believe, but I have heard these rumors from sources that didnât seem crazy (but also various things could have been lost in a game of telephone, and being very concerned about China doesnât result in endorsing a âManhattan project to AGIâ, though the rumors that I have heard did sound like they would endorse that)
Less rumor-based, I also know that Dario has historically been very hawkish, and âneeding to beat Chinaâ was one of the top justifications historically given for why Anthropic does capability research. I have heard this from many people, so feel more comfortable saying it with fewer disclaimers, but am still only like 80% on it being true.
Overall, my current guess is that indeed, a large-ish fraction of the EA policy people would have pushed for things like this, and at least didnât seem like they would push back on it that much. My guess is âweâ are at least somewhat responsible for this, and there is much less of a consensus against a U.S. china arms race in US governance among EAs than one might think, and so the above is not much evidence that there was no listening or only very selective listening to EAs.
I looked thru the congressional commission reportâs list of testimonies for plausibly EA-adjacent people. The only EA-adjacent org I saw was CSET, which had two testimonies (1, 2). From a brief skim, neither one looked clearly pro- or anti-arms race. They seemed vaguely pro-arms race on vibes but I didnât see any claims that look like they were clearly encouraging an arms raceâbut like I said, I only briefly skimmed them, so I could have missed a lot.
It looks to me like the online EA community, and the EAs I know IRL, have a fairly strong consensus that arms races are bad. Perhaps thereâs a divide in opinions with most self-identified EAs on one side, and policy people /â company leaders on the other sideâwhich in my view is unfortunate since the people holding the most power are also the most wrong.
(Is there some systematic reason why this would be true? At least one part of it makes sense: people who start AGI companies must believe that building AGI is the right move. It could also be that power corrupts, or something.)
So maybe I should say the congressional commission shouldâve spent less time listening to EA policy people and more time reading the EA Forum. Which obviously was never going to happen but it wouldâve been nice.
Slightly independent to the point Habryka is making, which may well also be true, my anecdotal impression is that the online EA community /â EAs I know IRL were much bigger on âwe need to beat Chinaâ arguments 2-4 years ago. If so, simple lag can also be part of the story here. In particular I think it was the mainstream position just before ChatGPT was released, and partly as a result I doubt an âoverwhelming majority of EAs involved in AI safetyâ disagree with it even now.
So maybe (the argument goes) we should take a cue from the environmental activists, and be hostile towards AI companies...
This is the most common question I get on AI safety posts: why isnât the rationalist /â EA /â AI safety movement doing this more? Itâs a great question, and itâs one that the movement asks itself a lot...
Still, most people arenât doing this. Why not?
Later, talking about why attempting a regulatory approach to avoiding a race is futile:
The biggest problem is China. US regulations donât affect China. China says that AI leadership is a cornerstone of their national securityâboth as a massive boon to their surveillance state, and because it would boost their national pride if they could beat America in something so cutting-edge.
So the real question is: which would we prefer? OpenAI gets superintelligence in 2040? Or Facebook gets superintelligence in 2044? Or China gets superintelligence in 2048?
Might we be able to strike an agreement with China on AI, much as countries have previously made arms control or climate change agreements? This is . . . not technically prevented by the laws of physics, but it sounds really hard. When I bring this challenge up with AI policy people, they ask âHarder than the technical AI alignment problem?â Okay, fine, you win this one.
I feel like a generic non-EA policy person reading that post could well end up where the congressional commission landed? Itâs right there in the section that most explicitly talks about policy.
Scottâs last sentence seems to be claiming that avoiding an arms race is easier than solving alignment (and it would seem to follow from that that we shouldnât race). But I can see how a politician reading this article wouldnât see that implication.
Yep, my impression is that this is an opinion that people mostly adopted after spending a bunch of time in DC and engaging with governance stuff, and so is not something represented in the broader EA population.
My best explanation is that when working in governance, being pro-China is just very costly, and especially combining the belief that AI will be very powerful, and there is no urgency to beat China to it, seems very anti-memetic in DC, and so people working in the space started adopting those stances.
But I am not sure. There are also non-terrible arguments for beating China being really important (though they are mostly premised on alignment being relatively easy, which seems very wrong to me).
(though they are mostly premised on alignment being relatively easy, which seems very wrong to me)
Not just alignment being easy, but alignment being easy with overwhelmingly high probability. It seems to me that pushing for an arms race is bad even if thereâs only a 5% chance that alignment is hard.
I think most of those people believe that âhaving an AI aligned to âChinaâs valuesââ would be comparably bad to a catastrophic misalignment failure, and if you believe that, 5% is not sufficient, if you think there is a greater than 5% of China ending up with âaligned AIâ instead.
I think thatâs not a reasonable position to hold but I donât know how to constructively argue against it in a short comment so Iâll just register my disagreement.
Like, presumably Chinaâs values include humans existing and having mostly good experiences.
This is inconsistent with my impressions and recollections. Most clearly, my sense is that CSET was (maybe still is, not sure) known for being very anti-escalatory towards China, and did substantial early research debunking hawkish views about AI progress in China, demonstrating it was less far along than ways widely believed in DC (and that EAs were involved in this, because they thought it was true and important, because they thought current false fears in the greater natsec community were enhancing arms race risks) (and this was when Jason was leading CSET, and OP supporting its founding). Some of the same people were also supportive of export controls, which are more ambiguous-sign here.
The export controls seemed like a pretty central example of hawkishness towards China and a reasonable precursor to this report. The central motivation in all that I have written related to them was about beating China in AI capabilities development.
Of course no one likes a symmetric arms race, but the question is did people favor the âquickly establish overwhelming dominance towards China by investing heavily in AIâ or the âtry to negotiate with China and not set an example of racing towards AGIâ strategy. My sense is many people favored the former (though definitely not all, and I am not saying that there is anything like consensus, my sense is itâs a quite divisive topic).
To support your point, I have seen much writing from Helen Toner on trying to dispel hawkishness towards China, and have been grateful for that. Against your point, at the recent âAI Security Forumâ in Vegas, many x-risk concerned people expressed very hawkish opinions.
Yeah re the export controls, I was trying to say âI think CSET was generally anti-escalatory, but in contrast, the effect of their export controls work was less soâ (though I used the word âambiguousâ because my impression was that some relevant people saw a pro of that work that it also mostly didnât directly advance AI progress in the US, i.e. it set China back without necessarily bringing the US forward towards AGI). To use your terminology, my impression is some of those people were âtrying to establish overwhelming dominance over Chinaâ but not by âinvesting heavily in AIâ.
Maybe instead of âwhere people actually listen to usâ itâs more like âEA in a world where people filter the most memetically fit of our ideas through their preconceived notions into something that only vaguely resembles what the median EA cares about but is importantly different from the world in which EA didnât exist.â
Call me a hater, and believe me, I am, but maybe someone who went to university at 16 and clearly spent most of their time immersed in books is not the most socially developed.
Maybe after they are implicated in a huge scandal that destroyed our movementâs reputation we should gently nudge them to not go on popular podcasts and talk fantastically and almost giddily about how world war 3 is just around the corner. Especially when they are working in a financialcapacity in which they would benefit from said war.
Many of the people we have let be in charge of our movement and speak on behalf of it donât know the first thing about optics or leadership or politics. I donât think Elizier Yudowsky could win a middle school class president race with a million dollars.
I know your point was specifically tailored toward optics and thinking carefully about what we say when we have a large platform, but I think looking back and forward bad optics and a lack of real politik messaging are pretty obvious failure modes of a movement filled with chronically online young males who worship intelligence and research output above all else. Iâm not trying to sh*t on Leopold and I donât claim I was out here beating a drum about the risks of these specific papers but yea I do think this is one symptom of a larger problem. I can barely think of anyone high up (publicly) in this movement who has risen via organizing.
(I think the issue with Leopold is somewhat precisely that he seems to be quite politically savvy in a way that seems likely to make him a deca-multi-millionaire and politically influental, possibly at the cost of all of humanity. I agree Eliezer is not the best presenter, but his error modes are clearly enormously different)
I donât think I was claiming they have the exact same failure modesâdo you want to point out where I did that? Rather they both have failure modes that I would expect to happen as a result of selecting them to be talking heads on the basis of wits and research output. Also I feel like you are implying Leopold is evil or something like that and I donât agree but maybe Iâm misinterpretting.
He seems like a smooth operator in some ways and certainly is quite different than Elizier. That being said I showed my dad (who has become an oddly good litmus test for a lot of this stuff for me as someone who is somewhat sympathethic to our movement but also a pretty normal 60 year old man in a completely different headspace) the Dwarkesh episode and he thought Leopold was very, very, very weird (and not because of his ideas). He kind of reminds me of Peter Thiel. Iâll completely admit I wasnât especially clear in my points and that mostly reflects my own lack of clarity on the exact point I was trying to getting across.
I think I take back like 20% of what I said (basically to the extent I was making a very direct stab at what exactly that failure mode is) but mostly still stand by the original comment, which again I see as being approximately ~ âSelecting people to be the public figureheads of our movement on the basis wits and research output is likely to be bad for usâ.
The thing about Yudkowsky is that, yes, on the one hand, every time I read him, I think he surely must be coming across as super-weird and dodgy to ânormalâ people. But on the other hand, actually, it seems like he HAS done really well in getting people to take his ideas seriously? Sam Altman was trolling Yudkowsky on twitter a while back about how many of the people running/âfounding AGI labs had been inspired to do so by his work. He got invited to write on AI governance for TIME despite having no formal qualifications or significant scientific achievements whatsoever. I think if we actually look at his track record, he has done pretty well at convincing influential people to adopt what were once extremely fringe views, whilst also succeeding in being seen by the wider world as one of the most important proponents of those views, despite an almost complete lack of mainstream, legible credentials.
Hmm, I hear what you are saying but that could easily be attributed to some mix of
(1) he has really good/âconvincing ideas
(2) he seems to be a a public representative for the EA/âLW community for a journalist on the outside.
And Iâm responding to someone saying that we are in âphase 3ââthat is to say people in the public are listening to usâso I guess Iâm not extremely concerned about him not being able to draw attention or convince people. Iâm more just generally worried that people like him are not who we should be promoting to positions of power, even if those are de jure positions.
Yeah, Iâm not a Yudkowsky fan. But I think the fact that he mostly hasnât been a PR disaster is striking, surprising and not much remarked upon, including by people who are big fans.
I guess in thinking about this I realize itâs so hard to even know if someone is a âPR disasterâ that I probably have just been confirming my biases. What makes you say that he hasnât been?
Just the stuff I already said about the success he seems to have had. It is also true that many people hate him and think heâs ridiculous, but I think that makes him polarizing rather than disastrous. I suppose you could phrase it as âhe was a disaster in some ways but a success in othersâ if you want to.
How do you know Leopold or anyone else actually influenced the commissionâs report? Not that that seems particularly unlikely to me, but is there any hard evidence? EDIT: I text-searched the report and he is not mentioned by name, although obviously that doesnât prove much on its own.
Hi Ben! You might be interested to know I literally had a meeting with the Assistant Defence Minister in Australia about 10 months ago off the back of one email. I wrote about it here. AI Safety advocacy is IMO still low extremely hanging fruit. My best theory is EAs donât want to do it because EAs are drawn to spreadsheets etc (it isnât their comparative advantage).
EA in a World Where People Actually Listen to Us
I had considered calling the third wave of EA âEA in a World Where People Actually Listen to Usâ.
Leopoldâs situational awareness memo has become a salient example of this for me. I used to sometimes think that arguments about whether we should avoid discussing the power of AI in order to avoid triggering an arms race were a bit silly and self important because obviously defense leaders arenât going to be listening to some random internet charity nerds and changing policy as a result.
Well, they are and they are. Letâs hope itâs for the better.
Seems plausible the impact of that single individual act is so negative that aggregate impact of EA is negative.
I think people should reflect seriously upon this possibility and not fall prey to wishful thinking (letâs hope speeding up the AI race and making it superpower powered is the best intervention! itâs better if everyone warning about this was wrong and Leopold is right!).
The broader story here is that EA prioritization methodology is really good for finding highly leveraged spots in the world, but there isnât a good methodology for figuring out what to do in such places, and there also isnât a robust pipeline for promoting virtues and virtuous actors to such places.
I spent all day in tears when I read the congressional report. This is a nightmare. I was literally hoping to wake up from a bad dream.
I really hope people donât suffer for our sins.
How could we have done something so terrible. Starting an arms race and making literal war more likely.
| and there also isnât a robust pipeline for promoting virtues and virtuous actors to such places.
this ^
Iâm not sure to what extent the Situational Awareness Memo or Leopold himself are representatives of âEAâ
In the pro-side:
Leopold thinks AGI is coming soon, will be a big deal, and that solving the alignment problem is one of the worldâs most important priorities
He used to work at GPI & FTX, and formerly identified with EA
He (
probablyalmost certainly) personally knows lots of EA people in the BayOn the con-side:
EA isnât just AI Safety (yet), so having short timelines/âhigh importance on AI shouldnât be sufficient to make someone an EA?[1]
EA shouldnât also just refer to a specific subset of the Bay Culture (please), or at least we need some more labels to distinguish different parts of it in that case
Many EAs have disagreed with various parts of the memo, e.g. Gideonâs well received post here
Since his EA institutional history he moved to OpenAI (mixed)[2] and now runs an AGI investment firm.
By self-identification, Iâm not sure Iâve seen Leopold identify as an EA at all recently.
This again comes down to the nebulousness of what âbeing an EAâ means.[3] I have no doubts at all that, given what Leopold thinks is the way to have the most impact heâll be very effective at achieving that.
Further, on your point, I think thereâs a reason to suspect that something like situational awareness went viral in a way that, say, Rethink Priorities Moral Weight project didnâtâthe promise many people see in powerful AI is power itself, and thatâs always going to be interesting for people to follow, so Iâm not sure that situational awareness becoming influential makes it more likely that other âEAâ ideas will
Plenty of e/âaccs have these two beliefs as well, they just expect alignment by default, for instance
I view OpenAI as tending implicitly/âexplicitly anti-EA, though I donât think there was an explicit âpurgeâ, I think the culture/âvision of the company was changed such that card-carrying EAs didnât want to work there any more
The 3 big defintions I have (self-identification, beliefs, actions) could all easily point in different directions for Leopold
I think he is pretty clearly an EA given he used to help run the Future Fund, or at most an only very recently ex-EA. Having said that, itâs not clear to me this means that âEAsâ are at fault for everything he does.
Yeah again I just think this depends on oneâs definition of EA, which is the point I was trying to make above.
Many people have turned away from EA, both the beliefs, institutions, and community in the aftermath of the FTX collapse. Even Ben Todd seems to not be an EA by some definitions any more, be that via association or identification. Who is to say Leopold is any different, or has not gone further? What then is the use of calling him EA, or using his views to represent the âThird Waveâ of EA?
I guess from my PoV what Iâm saying is that Iâm not sure thereâs much âconnective tissueâ between Leopold and myself, so when people use phrases like âlisten to usâ or âHow could we have doneâ I end up thinking âwho the heck is we/âus?â
In my post, I suggested that one possible future is that we stay at the âforefront of weirdness.â Calculating moral weights, to use your example.
I could imagine though that the fact that our opinions might be read by someone with access to the nuclear codes changes how we do things.
I wish there was more debate about which of these futures is more desirable.
(This is what I was trying to get out with my original post. Iâm not trying to make any strong claims about whether any individual person counts as âEAâ.)
I donât want to claim all EAs believe the same things, but if the congressional commission had listened to what you might call the âcentralâ EA position, it would not be recommending an arms race because it would be much more concerned about misalignment risk. The overwhelming majority of EAs involved in AI safety seem to agree that arms races are bad and misalignment risk is the biggest concern (within AI safety). So if anything this is a problem of the commission not listening to EAs, or at least selectively listening to only the parts they want to hear.
In most cases this is a rumors based thing, but I have heard that a substantial chunk of the OP-adjacent EA-policy space has been quite hawkish for many years, and at least the things I have heard is that a bunch of key leaders âbasically agreed with the China part of situational awarenessâ.
Again, people should really take this with a double-dose of salt, I am personally at like 50â50 of this being true, and I would love people like lukeprog or Holden or Jason Matheny or others high up at RAND to clarify their positions here. I am not attached to what I believe, but I have heard these rumors from sources that didnât seem crazy (but also various things could have been lost in a game of telephone, and being very concerned about China doesnât result in endorsing a âManhattan project to AGIâ, though the rumors that I have heard did sound like they would endorse that)
Less rumor-based, I also know that Dario has historically been very hawkish, and âneeding to beat Chinaâ was one of the top justifications historically given for why Anthropic does capability research. I have heard this from many people, so feel more comfortable saying it with fewer disclaimers, but am still only like 80% on it being true.
Overall, my current guess is that indeed, a large-ish fraction of the EA policy people would have pushed for things like this, and at least didnât seem like they would push back on it that much. My guess is âweâ are at least somewhat responsible for this, and there is much less of a consensus against a U.S. china arms race in US governance among EAs than one might think, and so the above is not much evidence that there was no listening or only very selective listening to EAs.
I looked thru the congressional commission reportâs list of testimonies for plausibly EA-adjacent people. The only EA-adjacent org I saw was CSET, which had two testimonies (1, 2). From a brief skim, neither one looked clearly pro- or anti-arms race. They seemed vaguely pro-arms race on vibes but I didnât see any claims that look like they were clearly encouraging an arms raceâbut like I said, I only briefly skimmed them, so I could have missed a lot.
It looks to me like the online EA community, and the EAs I know IRL, have a fairly strong consensus that arms races are bad. Perhaps thereâs a divide in opinions with most self-identified EAs on one side, and policy people /â company leaders on the other sideâwhich in my view is unfortunate since the people holding the most power are also the most wrong.
(Is there some systematic reason why this would be true? At least one part of it makes sense: people who start AGI companies must believe that building AGI is the right move. It could also be that power corrupts, or something.)
So maybe I should say the congressional commission shouldâve spent less time listening to EA policy people and more time reading the EA Forum. Which obviously was never going to happen but it wouldâve been nice.
Slightly independent to the point Habryka is making, which may well also be true, my anecdotal impression is that the online EA community /â EAs I know IRL were much bigger on âwe need to beat Chinaâ arguments 2-4 years ago. If so, simple lag can also be part of the story here. In particular I think it was the mainstream position just before ChatGPT was released, and partly as a result I doubt an âoverwhelming majority of EAs involved in AI safetyâ disagree with it even now.
Example from August 2022:
https://ââwww.astralcodexten.com/ââp/ââwhy-not-slow-ai-progress
Later, talking about why attempting a regulatory approach to avoiding a race is futile:
I feel like a generic non-EA policy person reading that post could well end up where the congressional commission landed? Itâs right there in the section that most explicitly talks about policy.
Scottâs last sentence seems to be claiming that avoiding an arms race is easier than solving alignment (and it would seem to follow from that that we shouldnât race). But I can see how a politician reading this article wouldnât see that implication.
Yep, my impression is that this is an opinion that people mostly adopted after spending a bunch of time in DC and engaging with governance stuff, and so is not something represented in the broader EA population.
My best explanation is that when working in governance, being pro-China is just very costly, and especially combining the belief that AI will be very powerful, and there is no urgency to beat China to it, seems very anti-memetic in DC, and so people working in the space started adopting those stances.
But I am not sure. There are also non-terrible arguments for beating China being really important (though they are mostly premised on alignment being relatively easy, which seems very wrong to me).
Not just alignment being easy, but alignment being easy with overwhelmingly high probability. It seems to me that pushing for an arms race is bad even if thereâs only a 5% chance that alignment is hard.
I think most of those people believe that âhaving an AI aligned to âChinaâs valuesââ would be comparably bad to a catastrophic misalignment failure, and if you believe that, 5% is not sufficient, if you think there is a greater than 5% of China ending up with âaligned AIâ instead.
I think thatâs not a reasonable position to hold but I donât know how to constructively argue against it in a short comment so Iâll just register my disagreement.
Like, presumably Chinaâs values include humans existing and having mostly good experiences.
Yep, I agree with this, but it appears nevertheless a relatively prevalent opinion among many EAs working in AI policy.
This is inconsistent with my impressions and recollections. Most clearly, my sense is that CSET was (maybe still is, not sure) known for being very anti-escalatory towards China, and did substantial early research debunking hawkish views about AI progress in China, demonstrating it was less far along than ways widely believed in DC (and that EAs were involved in this, because they thought it was true and important, because they thought current false fears in the greater natsec community were enhancing arms race risks) (and this was when Jason was leading CSET, and OP supporting its founding). Some of the same people were also supportive of export controls, which are more ambiguous-sign here.
The export controls seemed like a pretty central example of hawkishness towards China and a reasonable precursor to this report. The central motivation in all that I have written related to them was about beating China in AI capabilities development.
Of course no one likes a symmetric arms race, but the question is did people favor the âquickly establish overwhelming dominance towards China by investing heavily in AIâ or the âtry to negotiate with China and not set an example of racing towards AGIâ strategy. My sense is many people favored the former (though definitely not all, and I am not saying that there is anything like consensus, my sense is itâs a quite divisive topic).
To support your point, I have seen much writing from Helen Toner on trying to dispel hawkishness towards China, and have been grateful for that. Against your point, at the recent âAI Security Forumâ in Vegas, many x-risk concerned people expressed very hawkish opinions.
Yeah re the export controls, I was trying to say âI think CSET was generally anti-escalatory, but in contrast, the effect of their export controls work was less soâ (though I used the word âambiguousâ because my impression was that some relevant people saw a pro of that work that it also mostly didnât directly advance AI progress in the US, i.e. it set China back without necessarily bringing the US forward towards AGI). To use your terminology, my impression is some of those people were âtrying to establish overwhelming dominance over Chinaâ but not by âinvesting heavily in AIâ.
Maybe instead of âwhere people actually listen to usâ itâs more like âEA in a world where people filter the most memetically fit of our ideas through their preconceived notions into something that only vaguely resembles what the median EA cares about but is importantly different from the world in which EA didnât exist.â
On that framing, I agree that thatâs something that happens and that we should be able to anticipate will happen.
Call me a hater, and believe me, I am, but maybe someone who went to university at 16 and clearly spent most of their time immersed in books is not the most socially developed.
Maybe after they are implicated in a huge scandal that destroyed our movementâs reputation we should gently nudge them to not go on popular podcasts and talk fantastically and almost giddily about how world war 3 is just around the corner. Especially when they are working in a financial capacity in which they would benefit from said war.
Many of the people we have let be in charge of our movement and speak on behalf of it donât know the first thing about optics or leadership or politics. I donât think Elizier Yudowsky could win a middle school class president race with a million dollars.
I know your point was specifically tailored toward optics and thinking carefully about what we say when we have a large platform, but I think looking back and forward bad optics and a lack of real politik messaging are pretty obvious failure modes of a movement filled with chronically online young males who worship intelligence and research output above all else. Iâm not trying to sh*t on Leopold and I donât claim I was out here beating a drum about the risks of these specific papers but yea I do think this is one symptom of a larger problem. I can barely think of anyone high up (publicly) in this movement who has risen via organizing.
(I think the issue with Leopold is somewhat precisely that he seems to be quite politically savvy in a way that seems likely to make him a deca-multi-millionaire and politically influental, possibly at the cost of all of humanity. I agree Eliezer is not the best presenter, but his error modes are clearly enormously different)
I donât think I was claiming they have the exact same failure modesâdo you want to point out where I did that? Rather they both have failure modes that I would expect to happen as a result of selecting them to be talking heads on the basis of wits and research output. Also I feel like you are implying Leopold is evil or something like that and I donât agree but maybe Iâm misinterpretting.
He seems like a smooth operator in some ways and certainly is quite different than Elizier. That being said I showed my dad (who has become an oddly good litmus test for a lot of this stuff for me as someone who is somewhat sympathethic to our movement but also a pretty normal 60 year old man in a completely different headspace) the Dwarkesh episode and he thought Leopold was very, very, very weird (and not because of his ideas). He kind of reminds me of Peter Thiel. Iâll completely admit I wasnât especially clear in my points and that mostly reflects my own lack of clarity on the exact point I was trying to getting across.
I think I take back like 20% of what I said (basically to the extent I was making a very direct stab at what exactly that failure mode is) but mostly still stand by the original comment, which again I see as being approximately ~ âSelecting people to be the public figureheads of our movement on the basis wits and research output is likely to be bad for usâ.
The thing about Yudkowsky is that, yes, on the one hand, every time I read him, I think he surely must be coming across as super-weird and dodgy to ânormalâ people. But on the other hand, actually, it seems like he HAS done really well in getting people to take his ideas seriously? Sam Altman was trolling Yudkowsky on twitter a while back about how many of the people running/âfounding AGI labs had been inspired to do so by his work. He got invited to write on AI governance for TIME despite having no formal qualifications or significant scientific achievements whatsoever. I think if we actually look at his track record, he has done pretty well at convincing influential people to adopt what were once extremely fringe views, whilst also succeeding in being seen by the wider world as one of the most important proponents of those views, despite an almost complete lack of mainstream, legible credentials.
Hmm, I hear what you are saying but that could easily be attributed to some mix of
(1) he has really good/âconvincing ideas
(2) he seems to be a a public representative for the EA/âLW community for a journalist on the outside.
And Iâm responding to someone saying that we are in âphase 3ââthat is to say people in the public are listening to usâso I guess Iâm not extremely concerned about him not being able to draw attention or convince people. Iâm more just generally worried that people like him are not who we should be promoting to positions of power, even if those are de jure positions.
Yeah, Iâm not a Yudkowsky fan. But I think the fact that he mostly hasnât been a PR disaster is striking, surprising and not much remarked upon, including by people who are big fans.
I guess in thinking about this I realize itâs so hard to even know if someone is a âPR disasterâ that I probably have just been confirming my biases. What makes you say that he hasnât been?
Just the stuff I already said about the success he seems to have had. It is also true that many people hate him and think heâs ridiculous, but I think that makes him polarizing rather than disastrous. I suppose you could phrase it as âhe was a disaster in some ways but a success in othersâ if you want to.
How do you know Leopold or anyone else actually influenced the commissionâs report? Not that that seems particularly unlikely to me, but is there any hard evidence? EDIT: I text-searched the report and he is not mentioned by name, although obviously that doesnât prove much on its own.
Hi Ben! You might be interested to know I literally had a meeting with the Assistant Defence Minister in Australia about 10 months ago off the back of one email. I wrote about it here. AI Safety advocacy is IMO still low extremely hanging fruit. My best theory is EAs donât want to do it because EAs are drawn to spreadsheets etc (it isnât their comparative advantage).