I disagree that “we know what we need to know”. To me, the proper conversation about timelines isn’t just “when AGI”, but rather, “at what times will a number of things happen”, including various stages of post-AGI technology, and AI’s dynamics with the world as a whole. It incorporates questions like “what kinds of AIs will be present”.
See I think forecasts like that don’t really give us useful enough information about how to plan for future contingencies. I think we are deluded if we think we can make important moves based, for example, on the kinds of AIs that we project could be present in the future. The actual state of our knowledge is very coarse and we need to act accordingly. I really think the only prospective chance for impact is to do things that slow development and create real human and democratic oversight, and we have almost no chance of nudging the trajectory of development in a technical direction that works for us from here. (Maybe we will after we’ve secured the time and will to do so!)
Making the writing easy for myself: What’s your response to Carl Shulman’s take? Which is, that pushing for a pause too early might spoil the chance of getting people to agree to a pause when it would matter the most: at pivotal points where AI improvement is happening tremendously quickly. Carl Shulman on the 80k podcast.
Short answer I think trying to time this is too galaxy-brained. I think getting the meme of Pause out there ASAP is good because it pushes the Overton window and it gives people longer to chew on it. If and when warning shots occur, they will mainly advance Pause if people already had the idea that Pause would combat things like the warning shot happening before they happened.
I think takes that rely on saving up some kind of political capital and deploying it at the perfect time are generally wrong. PauseAI will gain more capital with more time and conversation, not use it up.
I’m not Holly, but my response is that getting a pause now is likely to increase, rather than decrease, the chance of getting future pauses. Quoting Evan Hubinger (2022):
In the theory of political capital, it is a fairly well-established fact that ‘Everybody Loves a Winner.’ That is: the more you succeed at leveraging your influence to get things done, the more influence you get in return. This phenomenon is most thoroughlystudied in the context of the ability of U.S. presidents to get their agendas through Congress—contrary to a naive model that might predict that legislative success uses up a president’s influence, what is actually found is the opposite: legislative success engenders future legislative success, greater presidential approval, and long-term gains for the president’s party.
I think many people who think about the mechanics of leveraging influence don’t really understand this phenomenon and conceptualize their influence as a finite resource to be saved up over time so it can all be spent down when it matters most. But I think that is just not how it works: if people see you successfully leveraging influence to change things, you become seen as a person who has influence, has the ability to change things, can get things done, etc. in a way that gives you more influence in the future, not less.
My sense is that this is a pretty major crux between my and Carl’s views.
Also not Holly, but another response might be the following:
Pausing in the very near future without a rise in political salience is just very very unlikely. The pause movement getting large influence is unlikely without a similar rise in political salience.
If a future rise in political salience occurs, this is likely an approximation of a ‘pivotal point’ (and if its not, well, policymakers are unlikely to agree to a pause at a pivotal point anyway)
Thus, what advocacy now is actually doing predominantly is creating the groundwork for a movement/idea that can be influential when the time comes.
I think this approach runs real risks, which I’d be happy to discuss, but also strikes me as an important response to the Shulman take.
See I think forecasts like that don’t really give us useful enough information about how to plan for future contingencies. I think we are deluded if we think we can make important moves based, for example, on the kinds of AIs that we project could be present in the future. The actual state of our knowledge is very coarse and we need to act accordingly. I really think the only prospective chance for impact is to do things that slow development and create real human and democratic oversight, and we have almost no chance of nudging the trajectory of development in a technical direction that works for us from here. (Maybe we will after we’ve secured the time and will to do so!)
Making the writing easy for myself: What’s your response to Carl Shulman’s take? Which is, that pushing for a pause too early might spoil the chance of getting people to agree to a pause when it would matter the most: at pivotal points where AI improvement is happening tremendously quickly. Carl Shulman on the 80k podcast.
You may have responded to this before. Feel free to provide a link.
This page is giving me a 404 right now: https://pauseai.info/mitigating-pause-failures
Short answer I think trying to time this is too galaxy-brained. I think getting the meme of Pause out there ASAP is good because it pushes the Overton window and it gives people longer to chew on it. If and when warning shots occur, they will mainly advance Pause if people already had the idea that Pause would combat things like the warning shot happening before they happened.
I think takes that rely on saving up some kind of political capital and deploying it at the perfect time are generally wrong. PauseAI will gain more capital with more time and conversation, not use it up.
I’m not Holly, but my response is that getting a pause now is likely to increase, rather than decrease, the chance of getting future pauses. Quoting Evan Hubinger (2022):
My sense is that this is a pretty major crux between my and Carl’s views.
Also not Holly, but another response might be the following:
Pausing in the very near future without a rise in political salience is just very very unlikely. The pause movement getting large influence is unlikely without a similar rise in political salience.
If a future rise in political salience occurs, this is likely an approximation of a ‘pivotal point’ (and if its not, well, policymakers are unlikely to agree to a pause at a pivotal point anyway)
Thus, what advocacy now is actually doing predominantly is creating the groundwork for a movement/idea that can be influential when the time comes.
I think this approach runs real risks, which I’d be happy to discuss, but also strikes me as an important response to the Shulman take.