Thanks Holly. I agree that fixating on just trying to answer the “AI timelines” question won’t be productive for most people. Though, we all need to come to terms with it somehow. I like your callout for “timeline-robust interventions”. I think that’s a very important point. Though I’m not sure that implies calling your representatives.
I disagree that “we know what we need to know”. To me, the proper conversation about timelines isn’t just “when AGI”, but rather, “at what times will a number of things happen”, including various stages of post-AGI technology, and AI’s dynamics with the world as a whole. It incorporates questions like “what kinds of AIs will be present”.
This allows us to make more prudent interventions: What technical AI safety and AI governance you need depends on the nature of the AI that will be built. Important AI to address isn’t just orthogonality thesis-driven paperclip maximizers.
I think seeing the way AI is emerging, that it’s clear some classic AI safety challenges are not as relevant anymore. For example, it seems to me that “value learning” is looking much easier than classic AI safety advocates thought. But versions of many classic AI safety challenges are still relevant. The same issue remains: if we can’t verify that something vastly more intelligent than us is acting in our interests, then we are in peril.
I don’t think it would be right if everyone would be occupied with such AI timelines and AI scenarios questions, but I think they deserve very strong efforts. If you are trying to solve a problem, the most important thing to get right is what problem you’re trying to solve. And what is the problem of AI safety? That depends on what kind of AI will be present in the world and what humans will be doing with it.
I agree that better understanding of progress and which problems are more or less challenging is valuable, but it seems clear that timelines get fare more attention than needed in places where they aren’t decision relevant.
I thought that David meant “intellectual areas” as opposed to physical/digital venues. Seems like lots of discussion in those venues isn’t particularly decision-relevant, and timelines aren’t really an unusually decision-irrelevant case.
Yeah, I was mostly thinking about policy—if we’re facing 90% unemployment, or existential risk, and need policy solutions, the difference between 5 and 7 years is immaterial. (There are important political differences, but the needed policies are identical.)
I disagree that “we know what we need to know”. To me, the proper conversation about timelines isn’t just “when AGI”, but rather, “at what times will a number of things happen”, including various stages of post-AGI technology, and AI’s dynamics with the world as a whole. It incorporates questions like “what kinds of AIs will be present”.
See I think forecasts like that don’t really give us useful enough information about how to plan for future contingencies. I think we are deluded if we think we can make important moves based, for example, on the kinds of AIs that we project could be present in the future. The actual state of our knowledge is very coarse and we need to act accordingly. I really think the only prospective chance for impact is to do things that slow development and create real human and democratic oversight, and we have almost no chance of nudging the trajectory of development in a technical direction that works for us from here. (Maybe we will after we’ve secured the time and will to do so!)
Making the writing easy for myself: What’s your response to Carl Shulman’s take? Which is, that pushing for a pause too early might spoil the chance of getting people to agree to a pause when it would matter the most: at pivotal points where AI improvement is happening tremendously quickly. Carl Shulman on the 80k podcast.
Short answer I think trying to time this is too galaxy-brained. I think getting the meme of Pause out there ASAP is good because it pushes the Overton window and it gives people longer to chew on it. If and when warning shots occur, they will mainly advance Pause if people already had the idea that Pause would combat things like the warning shot happening before they happened.
I think takes that rely on saving up some kind of political capital and deploying it at the perfect time are generally wrong. PauseAI will gain more capital with more time and conversation, not use it up.
I’m not Holly, but my response is that getting a pause now is likely to increase, rather than decrease, the chance of getting future pauses. Quoting Evan Hubinger (2022):
In the theory of political capital, it is a fairly well-established fact that ‘Everybody Loves a Winner.’ That is: the more you succeed at leveraging your influence to get things done, the more influence you get in return. This phenomenon is most thoroughlystudied in the context of the ability of U.S. presidents to get their agendas through Congress—contrary to a naive model that might predict that legislative success uses up a president’s influence, what is actually found is the opposite: legislative success engenders future legislative success, greater presidential approval, and long-term gains for the president’s party.
I think many people who think about the mechanics of leveraging influence don’t really understand this phenomenon and conceptualize their influence as a finite resource to be saved up over time so it can all be spent down when it matters most. But I think that is just not how it works: if people see you successfully leveraging influence to change things, you become seen as a person who has influence, has the ability to change things, can get things done, etc. in a way that gives you more influence in the future, not less.
My sense is that this is a pretty major crux between my and Carl’s views.
Also not Holly, but another response might be the following:
Pausing in the very near future without a rise in political salience is just very very unlikely. The pause movement getting large influence is unlikely without a similar rise in political salience.
If a future rise in political salience occurs, this is likely an approximation of a ‘pivotal point’ (and if its not, well, policymakers are unlikely to agree to a pause at a pivotal point anyway)
Thus, what advocacy now is actually doing predominantly is creating the groundwork for a movement/idea that can be influential when the time comes.
I think this approach runs real risks, which I’d be happy to discuss, but also strikes me as an important response to the Shulman take.
Thanks Holly. I agree that fixating on just trying to answer the “AI timelines” question won’t be productive for most people. Though, we all need to come to terms with it somehow. I like your callout for “timeline-robust interventions”. I think that’s a very important point. Though I’m not sure that implies calling your representatives.
I disagree that “we know what we need to know”. To me, the proper conversation about timelines isn’t just “when AGI”, but rather, “at what times will a number of things happen”, including various stages of post-AGI technology, and AI’s dynamics with the world as a whole. It incorporates questions like “what kinds of AIs will be present”.
This allows us to make more prudent interventions: What technical AI safety and AI governance you need depends on the nature of the AI that will be built. Important AI to address isn’t just orthogonality thesis-driven paperclip maximizers.
I think seeing the way AI is emerging, that it’s clear some classic AI safety challenges are not as relevant anymore. For example, it seems to me that “value learning” is looking much easier than classic AI safety advocates thought. But versions of many classic AI safety challenges are still relevant. The same issue remains: if we can’t verify that something vastly more intelligent than us is acting in our interests, then we are in peril.
I don’t think it would be right if everyone would be occupied with such AI timelines and AI scenarios questions, but I think they deserve very strong efforts. If you are trying to solve a problem, the most important thing to get right is what problem you’re trying to solve. And what is the problem of AI safety? That depends on what kind of AI will be present in the world and what humans will be doing with it.
I agree that better understanding of progress and which problems are more or less challenging is valuable, but it seems clear that timelines get fare more attention than needed in places where they aren’t decision relevant.
In what places do timelines get a lot of attention despite not being very decision-relevant?
AI Safety Twitter, this Forum, Bay Area parties…
I thought that David meant “intellectual areas” as opposed to physical/digital venues. Seems like lots of discussion in those venues isn’t particularly decision-relevant, and timelines aren’t really an unusually decision-irrelevant case.
I thought you were taking issue with the claim they were overdiscussed and asking where.
The areas where timelines are overdiscussed are numerous. Policy and technical safety career advice are the biggest ime.
Yeah, I was mostly thinking about policy—if we’re facing 90% unemployment, or existential risk, and need policy solutions, the difference between 5 and 7 years is immaterial. (There are important political differences, but the needed policies are identical.)
See I think forecasts like that don’t really give us useful enough information about how to plan for future contingencies. I think we are deluded if we think we can make important moves based, for example, on the kinds of AIs that we project could be present in the future. The actual state of our knowledge is very coarse and we need to act accordingly. I really think the only prospective chance for impact is to do things that slow development and create real human and democratic oversight, and we have almost no chance of nudging the trajectory of development in a technical direction that works for us from here. (Maybe we will after we’ve secured the time and will to do so!)
Making the writing easy for myself: What’s your response to Carl Shulman’s take? Which is, that pushing for a pause too early might spoil the chance of getting people to agree to a pause when it would matter the most: at pivotal points where AI improvement is happening tremendously quickly. Carl Shulman on the 80k podcast.
You may have responded to this before. Feel free to provide a link.
This page is giving me a 404 right now: https://pauseai.info/mitigating-pause-failures
Short answer I think trying to time this is too galaxy-brained. I think getting the meme of Pause out there ASAP is good because it pushes the Overton window and it gives people longer to chew on it. If and when warning shots occur, they will mainly advance Pause if people already had the idea that Pause would combat things like the warning shot happening before they happened.
I think takes that rely on saving up some kind of political capital and deploying it at the perfect time are generally wrong. PauseAI will gain more capital with more time and conversation, not use it up.
I’m not Holly, but my response is that getting a pause now is likely to increase, rather than decrease, the chance of getting future pauses. Quoting Evan Hubinger (2022):
My sense is that this is a pretty major crux between my and Carl’s views.
Also not Holly, but another response might be the following:
Pausing in the very near future without a rise in political salience is just very very unlikely. The pause movement getting large influence is unlikely without a similar rise in political salience.
If a future rise in political salience occurs, this is likely an approximation of a ‘pivotal point’ (and if its not, well, policymakers are unlikely to agree to a pause at a pivotal point anyway)
Thus, what advocacy now is actually doing predominantly is creating the groundwork for a movement/idea that can be influential when the time comes.
I think this approach runs real risks, which I’d be happy to discuss, but also strikes me as an important response to the Shulman take.