Slightly independent to the point Habryka is making, which may well also be true, my anecdotal impression is that the online EA community / EAs I know IRL were much bigger on ‘we need to beat China’ arguments 2-4 years ago. If so, simple lag can also be part of the story here. In particular I think it was the mainstream position just before ChatGPT was released, and partly as a result I doubt an ‘overwhelming majority of EAs involved in AI safety’ disagree with it even now.
So maybe (the argument goes) we should take a cue from the environmental activists, and be hostile towards AI companies...
This is the most common question I get on AI safety posts: why isn’t the rationalist / EA / AI safety movement doing this more? It’s a great question, and it’s one that the movement asks itself a lot...
Still, most people aren’t doing this. Why not?
Later, talking about why attempting a regulatory approach to avoiding a race is futile:
The biggest problem is China. US regulations don’t affect China. China says that AI leadership is a cornerstone of their national security—both as a massive boon to their surveillance state, and because it would boost their national pride if they could beat America in something so cutting-edge.
So the real question is: which would we prefer? OpenAI gets superintelligence in 2040? Or Facebook gets superintelligence in 2044? Or China gets superintelligence in 2048?
Might we be able to strike an agreement with China on AI, much as countries have previously made arms control or climate change agreements? This is . . . not technically prevented by the laws of physics, but it sounds really hard. When I bring this challenge up with AI policy people, they ask “Harder than the technical AI alignment problem?” Okay, fine, you win this one.
I feel like a generic non-EA policy person reading that post could well end up where the congressional commission landed? It’s right there in the section that most explicitly talks about policy.
Huh, fwiw this is not my anecdotal experience. I would suggest that this is because I spend more time around doomers than you and doomers are very influenced by Yudkowsky’s “don’t fight over which monkey gets to eat the poison banana first” framing, but that seems contradicted by your example being ACX, who is also quite doomer-adjacent.
That sounds plausible. I do think of ACX as much more ‘accelerationist’ than the doomer circles, for lack of a better term. Here’s a more recent post from October 2023 informing that impression, below probably does a better job than I can do of adding nuance to Scott’s position.
Second, if we never get AI, I expect the future to be short and grim. Most likely we kill ourselves with synthetic biology. If not, some combination of technological and economic stagnation, rising totalitarianism + illiberalism+mobocracy, fertility collapse and dysgenics will impoverish the world and accelerate its decaying institutional quality. I don’t spend much time worrying about any of these, because I think they’ll take a few generations to reach crisis level, and I expect technology to flip the gameboard well before then. But if we ban all gameboard-flipping technologies (the only other one I know is genetic enhancement, which is even more bannable), then we do end up with bioweapon catastrophe or social collapse. I’ve said before I think there’s a ~20% chance of AI destroying the world. But if we don’t get AI, I think there’s a 50%+ chance in the next 100 years we end up dead or careening towards Venezuela. That doesn’t mean I have to support AI accelerationism because 20% is smaller than 50%. Short, carefully-tailored pauses could improve the chance of AI going well by a lot, without increasing the risk of social collapse too much. But it’s something on my mind.
Scott’s last sentence seems to be claiming that avoiding an arms race is easier than solving alignment (and it would seem to follow from that that we shouldn’t race). But I can see how a politician reading this article wouldn’t see that implication.
Slightly independent to the point Habryka is making, which may well also be true, my anecdotal impression is that the online EA community / EAs I know IRL were much bigger on ‘we need to beat China’ arguments 2-4 years ago. If so, simple lag can also be part of the story here. In particular I think it was the mainstream position just before ChatGPT was released, and partly as a result I doubt an ‘overwhelming majority of EAs involved in AI safety’ disagree with it even now.
Example from August 2022:
https://www.astralcodexten.com/p/why-not-slow-ai-progress
Later, talking about why attempting a regulatory approach to avoiding a race is futile:
I feel like a generic non-EA policy person reading that post could well end up where the congressional commission landed? It’s right there in the section that most explicitly talks about policy.
Huh, fwiw this is not my anecdotal experience. I would suggest that this is because I spend more time around doomers than you and doomers are very influenced by Yudkowsky’s “don’t fight over which monkey gets to eat the poison banana first” framing, but that seems contradicted by your example being ACX, who is also quite doomer-adjacent.
That sounds plausible. I do think of ACX as much more ‘accelerationist’ than the doomer circles, for lack of a better term. Here’s a more recent post from October 2023 informing that impression, below probably does a better job than I can do of adding nuance to Scott’s position.
https://www.astralcodexten.com/p/pause-for-thought-the-ai-pause-debate
Scott’s last sentence seems to be claiming that avoiding an arms race is easier than solving alignment (and it would seem to follow from that that we shouldn’t race). But I can see how a politician reading this article wouldn’t see that implication.