An argument in favor of slow takeoff scenarios being generally safer is that we will get to see and experiment with the precursor AIs before they become capable of causing x-risks. Even if the behavior of this precursor AI is predictive of the superhuman AI’s, our ability to use it depends on the reaction to the potential dangers of this precursor AI. A society confident that there is no danger from increasing the capabilities of the machine that has been successfully running its electrical grid gains much less of an advantage from a slow takeoff (as opposed to the classic hard takeoff) than one with an awareness of its potential dangers.
Personally, I would expect a shift in attitudes towards AI as it becomes obviously more capable than humans in many domains. However, whether this shift involves being more careful or instead abdicating decisions to the AI entirely seems unclear to me. The way I play chess with a much stronger opponent is very different from how I play chess with a weaker or equally matched one. With the stronger opponent I am far more likely to expect obvious-looking blunders to actually be a set-up, for instance, and spend more time trying to figure out what advantage they might gain from it. On the other hand, I never bother to check my calculator’s math by hand, because the odds that it’s wrong is far lower than the chance that I will mess up somewhere in my arithmetic. If someone came up with an AI-calculator that gave occasional subtly wrong answers, I certainly wouldn’t notice.
Taking advantage of the benefits of a slow takeoff also requires the ability to have institutions capable of noticing and preventing problems. In a fast takeoff scenario, it is much easier for a single, relatively small project to unilaterally take off. This is, essentially, a gamble on that particular team’s capabilities. In a slow takeoff, it will be rapidly obvious that some project(s) seem to be trending in that direction, which makes it more likely that if the project seems unsafe there will be time to impose external control on it. How much of an advantage this is depends on how much you trust whichever institutions will be needed to impose those controls. Humanity’s track record in this respect seems to me to be mixed. Some historical precedents for cooperation (or lack thereof) in controlling dangerous technologies and their side-effects are the Asilomar Conference, nuclear proliferation treaties, and various pollution agreements. Asilomar, which seems to me the most successful of these, involved a relatively small scientific field voluntarily adhering to some limits on potentially dangerous research until more information could be gathered. Nuclear proliferation treaties reduce the cost of a zero-sum arms race, but it isn’t clear to me if they significantly reduced the risk of nuclear war. Pollution regulations have had very mixed results, with some major successes (eg acid rain) but on the whole failing to avert massive global change. Somewhat closer to home, the response to Covid-19 hasn’t been particularly encouraging. It is unclear to me which, if any, of these present a fair comparison, but our track record in cooperating seems decidedly mixed.
I found this interesting, and I think it would be worth expanding into a full post if you felt like it!
I don’t think you’d need more content: just a few more paragraph breaks, maybe a brief summary, and maybe a few questions to guide responses. If you have questions you’d want readers to tackle, consider including them as comments after the post.
An argument in favor of slow takeoff scenarios being generally safer is that we will get to see and experiment with the precursor AIs before they become capable of causing x-risks. Even if the behavior of this precursor AI is predictive of the superhuman AI’s, our ability to use it depends on the reaction to the potential dangers of this precursor AI. A society confident that there is no danger from increasing the capabilities of the machine that has been successfully running its electrical grid gains much less of an advantage from a slow takeoff (as opposed to the classic hard takeoff) than one with an awareness of its potential dangers.
Personally, I would expect a shift in attitudes towards AI as it becomes obviously more capable than humans in many domains. However, whether this shift involves being more careful or instead abdicating decisions to the AI entirely seems unclear to me. The way I play chess with a much stronger opponent is very different from how I play chess with a weaker or equally matched one. With the stronger opponent I am far more likely to expect obvious-looking blunders to actually be a set-up, for instance, and spend more time trying to figure out what advantage they might gain from it. On the other hand, I never bother to check my calculator’s math by hand, because the odds that it’s wrong is far lower than the chance that I will mess up somewhere in my arithmetic. If someone came up with an AI-calculator that gave occasional subtly wrong answers, I certainly wouldn’t notice.
Taking advantage of the benefits of a slow takeoff also requires the ability to have institutions capable of noticing and preventing problems. In a fast takeoff scenario, it is much easier for a single, relatively small project to unilaterally take off. This is, essentially, a gamble on that particular team’s capabilities. In a slow takeoff, it will be rapidly obvious that some project(s) seem to be trending in that direction, which makes it more likely that if the project seems unsafe there will be time to impose external control on it. How much of an advantage this is depends on how much you trust whichever institutions will be needed to impose those controls. Humanity’s track record in this respect seems to me to be mixed. Some historical precedents for cooperation (or lack thereof) in controlling dangerous technologies and their side-effects are the Asilomar Conference, nuclear proliferation treaties, and various pollution agreements. Asilomar, which seems to me the most successful of these, involved a relatively small scientific field voluntarily adhering to some limits on potentially dangerous research until more information could be gathered. Nuclear proliferation treaties reduce the cost of a zero-sum arms race, but it isn’t clear to me if they significantly reduced the risk of nuclear war. Pollution regulations have had very mixed results, with some major successes (eg acid rain) but on the whole failing to avert massive global change. Somewhat closer to home, the response to Covid-19 hasn’t been particularly encouraging. It is unclear to me which, if any, of these present a fair comparison, but our track record in cooperating seems decidedly mixed.
I found this interesting, and I think it would be worth expanding into a full post if you felt like it!
I don’t think you’d need more content: just a few more paragraph breaks, maybe a brief summary, and maybe a few questions to guide responses. If you have questions you’d want readers to tackle, consider including them as comments after the post.
Thanks! I ended up expanding it significantly and posting the full version here.