The tl;dr is that trying to ban AI progress will increase the hardware overhang, and risk the ban getting lifted all of a sudden in a way that causes a dangerous jump in capabilities.
Background reading: this summary will rely on an understanding of hardware overhangs (second link), which is a somewhat slippery concept, and I myself wish I understood at a deeper level.
***
Barnett Against Model Scaling Bans
Effectiveness of regulation and the counterfactual
It is hard to prevent AI progress. Thereâs a large monetary incentive to make progress in AI, and companies can make algorithmic progress on smaller models. âLarger experiments donât appear vastly more informative than medium sized experiments.â[2] The current proposals on the table on ban the largest runs.
Your only other option is draconian regulation, which will be hard to do well and will unpredictable and bad effects.
Conversely, by default, Matthew is optimistic about companies putting lots of effort into alignment. Itâs economically incentivized. And we can see this happening: OpenAI has put more effort into aligning its models over time, and GPT-4 seems more aligned than GPT-2.
But maybe some delay on the margin will have good effects anyway? Not necessarily:
Overhang
Matthewâs arguments above about algorithmic progress still occurring imply that AI progress will occur during a ban.[3] Given that, the amount of AI power that can be wrung out humanityâs hardware stock will be higher at the end of the ban than at the start. What are these consequences of that? Nothing good, says Matthew:
First, we need to account for the sudden jump in capabilities when the ban is relaxed. Companies will suddenly train up to the economically incentivized levels, leading to a discontinuous jump in capabilities. Discontinuous jumps in capabilities are more dangerous than incremental improvements. This is the core of the argument, according to my read.
Maybe we can continue the ban indefinitely? Sounds extremely risky to Matthew. Matthew is worried that we will try, and then the overhang will get worse and worse until for some reason (global power conflict?) regulators decide to relax the ban, and a more massive discontinuous jump occurs.
As an additional consideration, there will now be more competitive pressure, as the regulation has evened the playing field by holding back the largest labs, leading to more incentive to cut corners in safety.
Fixed budget of delays
[Caveat lector: It seems likely to me that Iâm not grokking Matthew here, this section is rougher and more likely to mislead you]: Matthew claims that the thereâs only so much budget of delays that humanity has. He argues that humanity should spend that budget later, when AI is closer to being an existential threat, rather than now.
Matthew considers the argument that we will wait too long to spend our budget and reach AGI before we do, but rejects that because he believes it will be obvious when AI is getting dangerous.
***
I consider the arguments presented here to be overall quite strong. I may come back and write up a response, but this is basically just a passthrough of Matthew, if Iâve done it right. Which brings me to:
Disclaimer: I say âMatthew is worriedâ, and other such statements. I have not run this by Matthew. I am writing this based on my understand from his twitter thread, see the original for the ground truth of what he actually said.
Matthew does not mention the ongoing march of compute progress. I donât understand why. It seems to me to make his argument stronger. Compute progress will lead to overhang just as surely as algorithmic progress, and seems about as hard to stop.
Thanks. I think that your summary is great and touched on basically all the major points that I meant to address in my original Twitter thread. I just have one clarification.
[Caveat lector: It seems likely to me that Iâm not grokking Matthew here, this section is rougher and more likely to mislead you]: Matthew claims that the thereâs only so much budget of delays that humanity has. He argues that humanity should spend that budget later, when AI is closer to being an existential threat, rather than now.
I want to note that this argument was not spelled out very well, so I can understand why it might be confusing. I didnât mean to make a strong claim about whether we have a fixed budget of delays; only that itâs possible. In fact, there seems to be a reason why we wouldnât have a fixed budget, since delaying now might help us delay later.
Nonetheless, delaying now means that we get more overhang, which means that we might not be able to âstopâ and move through later stages as slowly as weâre moving through our current stage of AI development. Even if the moratorium is lifted slowly, I think weâd still get less incremental progress later than if we had never had the moratorium to begin with, although justifying this claim would take a while, and likely requires a mathematical model of the situation. But this is essentially just a repetition of the point about overhangs stated in a different way, rather than a separate strong claim that we only get a fixed budget of delays.
Iâd like to try my hand at summarizing /â paraphrasing Matthew Barnettâs interesting twitter thread on the FLI letter.[1]
The tl;dr is that trying to ban AI progress will increase the hardware overhang, and risk the ban getting lifted all of a sudden in a way that causes a dangerous jump in capabilities.
Background reading: this summary will rely on an understanding of hardware overhangs (second link), which is a somewhat slippery concept, and I myself wish I understood at a deeper level.
***
Barnett Against Model Scaling Bans
Effectiveness of regulation and the counterfactual
It is hard to prevent AI progress. Thereâs a large monetary incentive to make progress in AI, and companies can make algorithmic progress on smaller models. âLarger experiments donât appear vastly more informative than medium sized experiments.â[2] The current proposals on the table on ban the largest runs.
Your only other option is draconian regulation, which will be hard to do well and will unpredictable and bad effects.
Conversely, by default, Matthew is optimistic about companies putting lots of effort into alignment. Itâs economically incentivized. And we can see this happening: OpenAI has put more effort into aligning its models over time, and GPT-4 seems more aligned than GPT-2.
But maybe some delay on the margin will have good effects anyway? Not necessarily:
Overhang
Matthewâs arguments above about algorithmic progress still occurring imply that AI progress will occur during a ban.[3] Given that, the amount of AI power that can be wrung out humanityâs hardware stock will be higher at the end of the ban than at the start. What are these consequences of that? Nothing good, says Matthew:
First, we need to account for the sudden jump in capabilities when the ban is relaxed. Companies will suddenly train up to the economically incentivized levels, leading to a discontinuous jump in capabilities. Discontinuous jumps in capabilities are more dangerous than incremental improvements. This is the core of the argument, according to my read.
Maybe we can continue the ban indefinitely? Sounds extremely risky to Matthew. Matthew is worried that we will try, and then the overhang will get worse and worse until for some reason (global power conflict?) regulators decide to relax the ban, and a more massive discontinuous jump occurs.
As an additional consideration, there will now be more competitive pressure, as the regulation has evened the playing field by holding back the largest labs, leading to more incentive to cut corners in safety.
Fixed budget of delays
[Caveat lector: It seems likely to me that Iâm not grokking Matthew here, this section is rougher and more likely to mislead you]: Matthew claims that the thereâs only so much budget of delays that humanity has. He argues that humanity should spend that budget later, when AI is closer to being an existential threat, rather than now.
Matthew considers the argument that we will wait too long to spend our budget and reach AGI before we do, but rejects that because he believes it will be obvious when AI is getting dangerous.
***
I consider the arguments presented here to be overall quite strong. I may come back and write up a response, but this is basically just a passthrough of Matthew, if Iâve done it right. Which brings me to:
Disclaimer: I say âMatthew is worriedâ, and other such statements. I have not run this by Matthew. I am writing this based on my understand from his twitter thread, see the original for the ground truth of what he actually said.
Thanks to Lizka for pointing me to it.
Seems like a controversial claim, Iâd be curious for someone more knowledgable than I to weigh in.
Itâs not obvious that this is crux-y for me though. See next bullet point.
Matthew does not mention the ongoing march of compute progress. I donât understand why. It seems to me to make his argument stronger. Compute progress will lead to overhang just as surely as algorithmic progress, and seems about as hard to stop.
Thanks. I think that your summary is great and touched on basically all the major points that I meant to address in my original Twitter thread. I just have one clarification.
I want to note that this argument was not spelled out very well, so I can understand why it might be confusing. I didnât mean to make a strong claim about whether we have a fixed budget of delays; only that itâs possible. In fact, there seems to be a reason why we wouldnât have a fixed budget, since delaying now might help us delay later.
Nonetheless, delaying now means that we get more overhang, which means that we might not be able to âstopâ and move through later stages as slowly as weâre moving through our current stage of AI development. Even if the moratorium is lifted slowly, I think weâd still get less incremental progress later than if we had never had the moratorium to begin with, although justifying this claim would take a while, and likely requires a mathematical model of the situation. But this is essentially just a repetition of the point about overhangs stated in a different way, rather than a separate strong claim that we only get a fixed budget of delays.