Thanks for writing this up! I thought it was really interesting (and this seems a really excellent talk to be doing at student groups :) ). Especially the arguments about the economic impact of AGI, and the focus on what it costs—that’s an interesting perspective I haven’t heard emphasised elsewhere.
The parts I feel most unconvinced by:
The content in Crux 1 seems to argue that AGI will be important when it scales and becomes cheap, because of the economic impact. But the argument for the actual research being done seem more focused on AGI as a single monolithic thing, eg framings like a safety tax/arms race, comparing costs of building an unaligned AGI vs an aligned AGI.
My best guess for what you mean is that “If AGI goes well, for economic reasons, the world will look very different and so any future plans will be suspect. But the threat from AGI comes the first time one is made”, ie that Crux 1 is an argument for prioritising AGI work over other work, but unrelated to the severity of the threat of AGI—is this correct?
The claim that good alignment solutions would be put to use. The fact that so many computer systems put minimal effort into security today seems a very compelling counter-argument.
I’m especially concerned if the problems are subtle—my impression is that especially a lot of what MIRI thinks about sounds weird and “I could maybe buy this”, but could maybe not buy it. And I have much lower confidence that companies would invest heavily in security for more speculative, abstract concerns
This seems bad, because intuitively AI Safety research seems more counterfactually useful the more subtle the problems are—I’d expect people to solve obvious problems before deploying AGI even without AI Safety as a field.
Related to the first point, I have much higher confidence AGI would be safe if it’s a single, large project eg a major $100 billion deployment, that people put a lot of thought into, than if it’s cheap and used ubiquitously.
Thanks for writing this up! I thought it was really interesting (and this seems a really excellent talk to be doing at student groups :) ). Especially the arguments about the economic impact of AGI, and the focus on what it costs—that’s an interesting perspective I haven’t heard emphasised elsewhere.
The parts I feel most unconvinced by:
The content in Crux 1 seems to argue that AGI will be important when it scales and becomes cheap, because of the economic impact. But the argument for the actual research being done seem more focused on AGI as a single monolithic thing, eg framings like a safety tax/arms race, comparing costs of building an unaligned AGI vs an aligned AGI.
My best guess for what you mean is that “If AGI goes well, for economic reasons, the world will look very different and so any future plans will be suspect. But the threat from AGI comes the first time one is made”, ie that Crux 1 is an argument for prioritising AGI work over other work, but unrelated to the severity of the threat of AGI—is this correct?
The claim that good alignment solutions would be put to use. The fact that so many computer systems put minimal effort into security today seems a very compelling counter-argument.
I’m especially concerned if the problems are subtle—my impression is that especially a lot of what MIRI thinks about sounds weird and “I could maybe buy this”, but could maybe not buy it. And I have much lower confidence that companies would invest heavily in security for more speculative, abstract concerns
This seems bad, because intuitively AI Safety research seems more counterfactually useful the more subtle the problems are—I’d expect people to solve obvious problems before deploying AGI even without AI Safety as a field.
Related to the first point, I have much higher confidence AGI would be safe if it’s a single, large project eg a major $100 billion deployment, that people put a lot of thought into, than if it’s cheap and used ubiquitously.