I’m not sure which is the better place to have this discussion, so I’m trying both. Copied from my comment on Less Wrong:
That all makes sense. To expand a little more on some of the logic:
It seems like the outcome of a partial pause rests in part on whether that would tend to put people in the lead of the AGI race who are more or less safety-concerned.
I think it’s nontrivial that we currently have three teams in the lead who all appear to honestly take the risks very seriously, and changing that might be a very bad idea.
On the other hand, the argument for alignment risks is quite strong, and we might expect more people to take the risks more seriously as those arguments diffuse. This might not happen if polarization becomes a large factor in beliefs on AGI risk. The evidence for climate change was also pretty strong, but we saw half of America believe in it less, not more, as evidence mounted. The lines of polarization would be different in this case, but I’m afraid it could happen. I outlined that case a little in AI scares and changing public beliefs
In that case, I think a partial pause would have a negative expected value, as the current lead decayed, and more people who believe in risks less get into the lead by circumventing the pause.
This makes me highly unsure if a pause would be net-positive. Having alignment solutions won’t help if they’re not implemented because the taxes are too high.
The creation of compute overhang is another reason to worry about a pause. It’s highly uncertain how far we are from making adequate compute for AGI affordable to individuals. Algorithms and compute will keep getting better during a pause. So will theory of AGI, along with theory of alignment.
This puts me, and I think the alignment community at large, in a very uncomfortable position of not knowing whether a realistic pause would be helpful.
It does seem clear that creating mechanisms and political will for a pause are a good idea.
Advocating for more safety work also seems clear cut.
To this end, I think it’s true that you create more political capitol by successfully pushing for policy.
A pause now would create even more capitol, but it’s also less likely to be a win, and it could wind up creating polarization and so costing rather than creating capitol. It’s harder to argue for a pause now when even most alignment folks think we’re years from AGI.
So perhaps the low-hanging fruit is pushing for voluntary RSPs, and government funding for safety work. These are clear improvements, and likely to be wins that create capitol for a pause as we get closer to AGI.
There’s a lot of uncertainty here, and that’s uncomfortable. More discussion like this should help resolve that uncertainty, and thereby help clarify and unify the collective will of the safety community.
Seth—you mentioned that ‘we currently have three teams in the lead who all appear to honestly take the risks very seriously, and changing that might be a very bad idea.’
I assume you’re referring to OpenAI, DeepMind, and Anthropic.
Yes, they all give lip service to AI safety, and they hire safety researchers, and they safety-wash their capabilities development.
But I see no evidence that they would actually stop their AGI development under any circumstances, no matter how risky it started to seem.
Maybe you trust their leadership. I do not. And I don’t think the 8 billion people in the world should have their fates left in the hands of a tiny set of AI industry leaders—no matter how benevolent they seem, or how many times they talk about AI safety in interviews.
I agree that those teams aren’t completely trustworthy, and in an ideal world, we should be making this decision by including everyone on earth. But with a partial pause, do you expect to have better or worse teams in the lead for achieving AGI? That was my point.
Well from an AI safety viewpoint, the very worst teams to be leading the AGI rush would be those that (1) are very competent, well-funded, well-run, and full of idealistic talent, and (2) don’t actually care about reducing extinction risk—however much lip service they pay to AI safety.
From that perspective, OpenAI is the worst team, and they’re in the lead.
I think that’s quite a pessimistic take. I take Altman seriously on caring about x-risk, although I’m not sure he takes it quite seriously enough. This is based on public comments to that effect around 2013, before he started running OpenAI. And Sutskever definitely seems properly concerned.
I’m not sure which is the better place to have this discussion, so I’m trying both. Copied from my comment on Less Wrong:
That all makes sense. To expand a little more on some of the logic:
It seems like the outcome of a partial pause rests in part on whether that would tend to put people in the lead of the AGI race who are more or less safety-concerned.
I think it’s nontrivial that we currently have three teams in the lead who all appear to honestly take the risks very seriously, and changing that might be a very bad idea.
On the other hand, the argument for alignment risks is quite strong, and we might expect more people to take the risks more seriously as those arguments diffuse. This might not happen if polarization becomes a large factor in beliefs on AGI risk. The evidence for climate change was also pretty strong, but we saw half of America believe in it less, not more, as evidence mounted. The lines of polarization would be different in this case, but I’m afraid it could happen. I outlined that case a little in AI scares and changing public beliefs
In that case, I think a partial pause would have a negative expected value, as the current lead decayed, and more people who believe in risks less get into the lead by circumventing the pause.
This makes me highly unsure if a pause would be net-positive. Having alignment solutions won’t help if they’re not implemented because the taxes are too high.
The creation of compute overhang is another reason to worry about a pause. It’s highly uncertain how far we are from making adequate compute for AGI affordable to individuals. Algorithms and compute will keep getting better during a pause. So will theory of AGI, along with theory of alignment.
This puts me, and I think the alignment community at large, in a very uncomfortable position of not knowing whether a realistic pause would be helpful.
It does seem clear that creating mechanisms and political will for a pause are a good idea.
Advocating for more safety work also seems clear cut.
To this end, I think it’s true that you create more political capitol by successfully pushing for policy.
A pause now would create even more capitol, but it’s also less likely to be a win, and it could wind up creating polarization and so costing rather than creating capitol. It’s harder to argue for a pause now when even most alignment folks think we’re years from AGI.
So perhaps the low-hanging fruit is pushing for voluntary RSPs, and government funding for safety work. These are clear improvements, and likely to be wins that create capitol for a pause as we get closer to AGI.
There’s a lot of uncertainty here, and that’s uncomfortable. More discussion like this should help resolve that uncertainty, and thereby help clarify and unify the collective will of the safety community.
Seth—you mentioned that ‘we currently have three teams in the lead who all appear to honestly take the risks very seriously, and changing that might be a very bad idea.’
I assume you’re referring to OpenAI, DeepMind, and Anthropic.
Yes, they all give lip service to AI safety, and they hire safety researchers, and they safety-wash their capabilities development.
But I see no evidence that they would actually stop their AGI development under any circumstances, no matter how risky it started to seem.
Maybe you trust their leadership. I do not. And I don’t think the 8 billion people in the world should have their fates left in the hands of a tiny set of AI industry leaders—no matter how benevolent they seem, or how many times they talk about AI safety in interviews.
I agree that those teams aren’t completely trustworthy, and in an ideal world, we should be making this decision by including everyone on earth. But with a partial pause, do you expect to have better or worse teams in the lead for achieving AGI? That was my point.
Well from an AI safety viewpoint, the very worst teams to be leading the AGI rush would be those that (1) are very competent, well-funded, well-run, and full of idealistic talent, and (2) don’t actually care about reducing extinction risk—however much lip service they pay to AI safety.
From that perspective, OpenAI is the worst team, and they’re in the lead.
I think that’s quite a pessimistic take. I take Altman seriously on caring about x-risk, although I’m not sure he takes it quite seriously enough. This is based on public comments to that effect around 2013, before he started running OpenAI. And Sutskever definitely seems properly concerned.