To push back a bit on the fast software-driven takeoff (i.e. a fast takeoff driven primarily by innovations in software):
Common objections to this narrative [of a fast software-driven takeoff] are that there won’t be enough compute, or data, for this to happen. These don’t hold water after a cursory examination of our situation. We are nowhere near close to the physical limits to computation …
While we’re nowhere near the physical limits to computation, it’s still true that hardware progress has slowed down considerably on various measures. I think the steelman of the compute-based argument against a fast software-driven takeoff is not that the ultimate limits to computation are near, but rather that the pace of hardware progress is unlikely to be explosively fast (e.g. in light of recent trends that arguably point in the opposite direction, and because software progress per se seems insufficient for driving explosive hardware progress).
That actors can afford to create this next generation of AIs does not imply that those AIs will in turn lead to a hard takeoff in capabilities. From my perspective at least, that seems like an unargued assumption here.
Data is not a show-stopper either. Sure, ~all the text on the internet might’ve already been digested, but Google could readily record more words per day via phone mics[16] than the number used to train GPT-4[17]. These may or may not be as high quality as text, but 1000x as many as all the text on the internet could be gathered within months[18]. Then there are all the billions of high-res video cameras (phones and CCTV), and sensors in the world[19]. And if that is not enough, there is already a fast-growing synthetic data industry serving the ML community’s ever growing thirst for data to train their models on.
A key question is whether this extra data would be all that valuable to the main tasks of concern. For example, it seems unclear whether low-quality data from phone conversations, video cameras, etc. would give that much of a boost to a model’s ability to write code. So I don’t think the point made above, as it stands, is a strong rebuttal to the claim that data will soon be a limiting bottleneck to significant capability gains. (Somerelatedposts.)
The time for talking politely to (and working with) big AI is over.
This is another claim I would push back against. For instance, from a perspective concerned with the reduction of s-risks, one could argue that talking politely to, and working with, leading AI companies is in fact the most responsible thing to do, and that taking a less cooperative stance is unduly risky and irresponsible. To be clear, I’m not saying that this is obviously the case, but I’m trying to say that it’s not clear-cut either way. Good arguments can be made for a different approach, and this seems true for a wide range of altruistic values.
Current scaling “laws” are not laws of nature. And there are already worrying signs that things like dataset optimization/pruning, curriculum learning and synthetic data might well break them—It seems likely to me that LLMs will be useful in all three. I would still be worried even if LLMs prove useless in enhancing architecture search.
Current scaling “laws” are not laws of nature. And there are already worrying signs that things like dataset optimization/pruning, curriculum learning and synthetic data might well break them
Thanks for writing this—it was useful to read the pushbacks!
As I said below, I want more synthesis of these sorts of arguments. I know that some academic groups are preparing literature reviews of the key arguments for and against AGI risk.
I really think that we should be doing that for ourselves as a community and to make sure that we are able to present busy smart people with more compelling content than a range of arguments spread across many different forum posts.
I don’t think that that is going to cut it for many people in the policy space.
Agree. But at the same time, we need to do this fast! The typical academic paper review cycle is far too slow for this. We probably need groups like SAGE (and Independent SAGE?) to step in. In fact, I’ll try and get hold of them.. (they are for “emergencies” in general, not just Covid[1])
hardware progress has slowed down considerably on various measures
I don’t think this matters, as per the next point about there already being enough compute for doom [Edit: I’ve relegated the “nowhere near close to the physical limits to computation” sentence to a footnote and added Magnus’ reference on slowdown to it].
That actors can afford to create this next generation of AIs does not imply that those AIs will in turn lead to a hard takeoff in capabilities. From my perspective at least, that seems like an unargued assumption here.
I think the burden of proof here needs to shift to those willing to gamble on the safety of 100x larger systems. All I’m really saying here is that the risk is way too high for comfort (given the jumps in capabilities we’ve seen so far going from GPT-3->GPT3.5->GPT-4).
[Meta: would appreciate separate points being made in separate comments]. Will look into your links re data and respond later.
from a perspective concerned with the reduction of s-risks, one could argue that talking politely to, and working with, leading AI companies is in fact the most responsible thing to do, and that taking a less cooperative stance is unduly risky and irresponsible.
I’m not sure what you are saying here? Do you think there is a risk of AI companies deliberately causing s-risks (e.g. releasing a basilisk) if we don’t play nice!? They may be crazy in a sense of being reckless with the fate of billions of people’s lives, but I don’t think they are that crazy (in a sense of being sadistically malicious and spiteful toward their opponents)!
I’m not sure what you are saying here? Do you think there is a risk of AI companies deliberately causing s-risks (e.g. releasing a basilisk) if we don’t play nice!?
No, I didn’t mean anything like that (although such crazy unlikely risks might also be marginally better reduced through cooperation with these actors). I was simply suggesting that cooperation could be a more effective way to reduce risks of worst-case outcomes that might occur in the absence of cooperative work to prevent them, i.e. work of the directional kind gestured at in my other comment (e.g. because ensuring the inclusion of certain measures to avoid worst-case outcomes has higher EV than does work to slow down AI). Again, I’m not saying that this is definitely the case, but it could well be. It’s fairly unclear, in my view.
Ok. I don’t put much weight on s-risks being a likely outcome. Far more likely seems to be just that the solar system (and beyond) will be arranged in some (to us) arbitrary way, and all carbon-based life will be lost as collateral damage.
Although I guess if you are looking a bit nearer term, then s-risk from misuse could be quite high. But I don’t think any of the major players (OpenAI, Deepmind, Anthropic) are even really working on trying to prevent misuse at all as part of their strategy (their core AI Alignment work is on aligning the AIs, rather than the humans using them!) So actually, this is just another reason to shut it all down.
I suspect that a different framing might be more realistic and more apt from our perspective. In terms of helpful actions we can take, I more see the choice before us as one between trying to slow down development vs. trying to steer future development in better (or less bad) directions conditional on the current pace of development continuing (of course, one could dedicate resources to both, but one would still need to prioritize between them). Both of those choices (as well as graded allocations between them) seem to come with a lot of risks, and they both strike me as gambles with potentially serious downsides. I don’t think there’s really a “safe” choice here.
All I’m really saying here is that the risk is way too high for comfort
I’d agree with that, but that seems different from saying that a fast software-driven takeoff is the most likely scenario, or that trying to slow down development is the most important or effective thing to do (e.g. compared to the alternative option mentioned above).
both strike me as gambles with potentially serious downsides.
What are the downsides from slowing down? Things like not curing diseases and ageing? Eliminating wild animal suffering? I address that here: “it’s a rather depressing thought. We may be far closer to the Dune universe than the Culture one (the worry driving a future Butlerian Jihad will be the advancement of AGI algorithms to the point of individual laptops and phones being able to end the world). For those who may worry about the loss of the “glorious transhumanist future”, and in particular, radical life extension and cryonic reanimation (I’m in favour of these things), I think there is some consolation in thinking that if a really strong taboo emerges around AGI, to the point of stopping all algorithm advancement, we can still achieve these ends using standard supercomputers, bioinformatics and human scientists. I hope so.”
To be clear, I’ll also say that it’s far too late to only steer future development better. For that, Alignment needs to be 10 years ahead of where it is now!
a fast software-driven takeoff is the most likely scenario
I don’t think you need to believe this to want to be slamming on the brakes now. As mentioned in the OP, is the prospect of mere imminent global catastrophe not enough?
I’d again prefer to frame the issue as “what are the downsides from spending marginal resources on efforts to slow down?” I think the main downside, from this marginal perspective, is opportunity costs in terms of other efforts to reduce future risks, e.g. trying to implement “fail-safe measures”/”separation from hyperexistential risk” in case a slowdown is insufficiently likely to be successful. There are various ideas that one could try to implement.
In other words, a serious downside of betting chiefly on efforts to slow down over these alternative options could be that these s-risks/hyperexistential risks would end up being significantly greater in counterfactual terms (again, not saying this is clearly the case, but, FWIW, I doubt that efforts to slow down are among the most effective ways to reduce risks like these).
a fast software-driven takeoff is the most likely scenario
I don’t think you need to believe this to want to be slamming on the breaks on now.
Didn’t mean to say that that’s a necessary condition for wanting to slow down. But again, I still think it’s highly unclear whether efforts that push for slower progress are more beneficial than alternative efforts.
I think it’s a very hard sell to try and get people to sacrifice themselves (and the whole world) for the sake of preventing “fates worse than death”. At that point most people would probably just be pretty nihilistic. It also feels like it’s not far off basically just giving up hope: the future is, at best, non-existence for sentient life; but we should still focus our efforts on avoiding hell. Nope. We should be doing all we can now to avoid having to face such a predicament! Global moratorium on AGI, now.
I think it’s a very hard sell to try and get people to sacrifice themselves (and the whole world) for the sake of preventing “fates worse than death”.
I’m not talking about people sacrificing themselves or the whole world. Even if we were to adopt a purely survivalist perspective, I think it’s still far from obvious that trying to slow things down is more effective than is focusing on other aims. After all, the space of alternative aims that one could focus on is vast, and trying to slow things down comes with non-trivial risks of its own (e.g. risks of backlash from tech-accelerationists). Again, I’m not saying it’s clear; I’m saying that it seems to me unclear either way.
We should be doing all we can now to avoid having to face such a predicament!
But, as I see it, what’s at issue is what the best way is to avoid such a predicament/how to best navigate given our current all-too risky predicament.
FWIW, I think that a lot of the discussion around this issue appears strongly fear-driven, to such an extent that it seems to get in the way of sober and helpful analysis. This is, to be sure, extremely understandable. But I also suspect that it is not the optimal way to figure out how to best achieve our aims, nor an effective way to persuade readers on this forum. Likewise, I suspect that rallying calls along the lines of “Global moratorium on AGI, now” might generally be received less well than would, say, a deeper analysis of the reasons for and against attempts to institute that policy.
I feel like I’m one of the main characters in the film Don’t Look Up here.
the space of alternative aims that one could focus on is vast
Please can you name 10? The way I see it is—either alignment is solved in time with business as usual[1], or we Pause to allow time for alignment to be solved (or establish it’s impossibility). It is not a complicated situation. No need to be worrying about “fates worse than death” at this juncture.
I didn’t claim that there isn’t plenty more data. But a relevant question is: plenty more data for what? He says that the data situation looks pretty good, which I trust is true in many domains (e.g. video data), and that data would probably in turn improve performance in those domains. But I don’t see him claiming that the data situation looks good in terms of ensuring significant performance gains across all domains, which would be a more specific and stronger claim.
Moreover, the deference question could be posed in the other direction as well, e.g. do you not trust the careful data collection and projections of Epoch? (Though again, Ilya saying that the data situation looks pretty good is arguably not in conflict with Epoch’s projections — nor with any claim I made above — mostly because his brief “pretty good” remark is quite vague.)
Note also that, at least in some domains, OpenAI could end up having less data to train their models with going forward, as they might have been using data illegally.
Coming back to the point about data. Whilst Epoch gathered some data showing that the stock high quality text data might soon be exhausted, their overall conclusion is that there is only a “20% chance that the scaling (as measured in training compute) of ML models will significantly slow down by 2040 due to a lack of training data.”. Regarding Jacob Buckman’s point about chess, he actually outlines a way around that (training data provided by narrow AI). As a counter to the wider point about the need for active learning, see DeepMind’s Adaptive Agent and the Voyager “lifelong learning” Minecraft agent, both of which seem like impressive steps in this direction.
To push back a bit on the fast software-driven takeoff (i.e. a fast takeoff driven primarily by innovations in software):
While we’re nowhere near the physical limits to computation, it’s still true that hardware progress has slowed down considerably on various measures. I think the steelman of the compute-based argument against a fast software-driven takeoff is not that the ultimate limits to computation are near, but rather that the pace of hardware progress is unlikely to be explosively fast (e.g. in light of recent trends that arguably point in the opposite direction, and because software progress per se seems insufficient for driving explosive hardware progress).
That actors can afford to create this next generation of AIs does not imply that those AIs will in turn lead to a hard takeoff in capabilities. From my perspective at least, that seems like an unargued assumption here.
A key question is whether this extra data would be all that valuable to the main tasks of concern. For example, it seems unclear whether low-quality data from phone conversations, video cameras, etc. would give that much of a boost to a model’s ability to write code. So I don’t think the point made above, as it stands, is a strong rebuttal to the claim that data will soon be a limiting bottleneck to significant capability gains. (Some related posts.)
This is another claim I would push back against. For instance, from a perspective concerned with the reduction of s-risks, one could argue that talking politely to, and working with, leading AI companies is in fact the most responsible thing to do, and that taking a less cooperative stance is unduly risky and irresponsible. To be clear, I’m not saying that this is obviously the case, but I’m trying to say that it’s not clear-cut either way. Good arguments can be made for a different approach, and this seems true for a wide range of altruistic values.
Current scaling “laws” are not laws of nature. And there are already worrying signs that things like dataset optimization/pruning, curriculum learning and synthetic data might well break them—It seems likely to me that LLMs will be useful in all three. I would still be worried even if LLMs prove useless in enhancing architecture search.
Interesting—can you provide some citations?
Thanks for writing this—it was useful to read the pushbacks!
As I said below, I want more synthesis of these sorts of arguments. I know that some academic groups are preparing literature reviews of the key arguments for and against AGI risk.
I really think that we should be doing that for ourselves as a community and to make sure that we are able to present busy smart people with more compelling content than a range of arguments spread across many different forum posts.
I don’t think that that is going to cut it for many people in the policy space.
Agree. But at the same time, we need to do this fast! The typical academic paper review cycle is far too slow for this. We probably need groups like SAGE (and Independent SAGE?) to step in. In fact, I’ll try and get hold of them.. (they are for “emergencies” in general, not just Covid[1])
Although it looks like they are highly specialised on viral threats. They would need totally new teams to be formed for AI. Maybe Hinton should chair?
I don’t think this matters, as per the next point about there already being enough compute for doom [Edit: I’ve relegated the “nowhere near close to the physical limits to computation” sentence to a footnote and added Magnus’ reference on slowdown to it].
I think the burden of proof here needs to shift to those willing to gamble on the safety of 100x larger systems. All I’m really saying here is that the risk is way too high for comfort (given the jumps in capabilities we’ve seen so far going from GPT-3->GPT3.5->GPT-4).
[Meta: would appreciate separate points being made in separate comments].
Will look into your links re data and respond later.
I’m not sure what you are saying here? Do you think there is a risk of AI companies deliberately causing s-risks (e.g. releasing a basilisk) if we don’t play nice!? They may be crazy in a sense of being reckless with the fate of billions of people’s lives, but I don’t think they are that crazy (in a sense of being sadistically malicious and spiteful toward their opponents)!
No, I didn’t mean anything like that (although such crazy unlikely risks might also be marginally better reduced through cooperation with these actors). I was simply suggesting that cooperation could be a more effective way to reduce risks of worst-case outcomes that might occur in the absence of cooperative work to prevent them, i.e. work of the directional kind gestured at in my other comment (e.g. because ensuring the inclusion of certain measures to avoid worst-case outcomes has higher EV than does work to slow down AI). Again, I’m not saying that this is definitely the case, but it could well be. It’s fairly unclear, in my view.
Ok. I don’t put much weight on s-risks being a likely outcome. Far more likely seems to be just that the solar system (and beyond) will be arranged in some (to us) arbitrary way, and all carbon-based life will be lost as collateral damage.
Although I guess if you are looking a bit nearer term, then s-risk from misuse could be quite high. But I don’t think any of the major players (OpenAI, Deepmind, Anthropic) are even really working on trying to prevent misuse at all as part of their strategy (their core AI Alignment work is on aligning the AIs, rather than the humans using them!) So actually, this is just another reason to shut it all down.
Thanks for your reply, Greg :)
That is what I did not find adequately justified or argued for in the post.
I suspect that a different framing might be more realistic and more apt from our perspective. In terms of helpful actions we can take, I more see the choice before us as one between trying to slow down development vs. trying to steer future development in better (or less bad) directions conditional on the current pace of development continuing (of course, one could dedicate resources to both, but one would still need to prioritize between them). Both of those choices (as well as graded allocations between them) seem to come with a lot of risks, and they both strike me as gambles with potentially serious downsides. I don’t think there’s really a “safe” choice here.
I’d agree with that, but that seems different from saying that a fast software-driven takeoff is the most likely scenario, or that trying to slow down development is the most important or effective thing to do (e.g. compared to the alternative option mentioned above).
What are the downsides from slowing down? Things like not curing diseases and ageing? Eliminating wild animal suffering? I address that here: “it’s a rather depressing thought. We may be far closer to the Dune universe than the Culture one (the worry driving a future Butlerian Jihad will be the advancement of AGI algorithms to the point of individual laptops and phones being able to end the world). For those who may worry about the loss of the “glorious transhumanist future”, and in particular, radical life extension and cryonic reanimation (I’m in favour of these things), I think there is some consolation in thinking that if a really strong taboo emerges around AGI, to the point of stopping all algorithm advancement, we can still achieve these ends using standard supercomputers, bioinformatics and human scientists. I hope so.”
To be clear, I’ll also say that it’s far too late to only steer future development better. For that, Alignment needs to be 10 years ahead of where it is now!
I don’t think you need to believe this to want to be slamming on the brakes now. As mentioned in the OP, is the prospect of mere imminent global catastrophe not enough?
I’d again prefer to frame the issue as “what are the downsides from spending marginal resources on efforts to slow down?” I think the main downside, from this marginal perspective, is opportunity costs in terms of other efforts to reduce future risks, e.g. trying to implement “fail-safe measures”/”separation from hyperexistential risk” in case a slowdown is insufficiently likely to be successful. There are various ideas that one could try to implement.
In other words, a serious downside of betting chiefly on efforts to slow down over these alternative options could be that these s-risks/hyperexistential risks would end up being significantly greater in counterfactual terms (again, not saying this is clearly the case, but, FWIW, I doubt that efforts to slow down are among the most effective ways to reduce risks like these).
Didn’t mean to say that that’s a necessary condition for wanting to slow down. But again, I still think it’s highly unclear whether efforts that push for slower progress are more beneficial than alternative efforts.
I think it’s a very hard sell to try and get people to sacrifice themselves (and the whole world) for the sake of preventing “fates worse than death”. At that point most people would probably just be pretty nihilistic. It also feels like it’s not far off basically just giving up hope: the future is, at best, non-existence for sentient life; but we should still focus our efforts on avoiding hell. Nope. We should be doing all we can now to avoid having to face such a predicament! Global moratorium on AGI, now.
I’m not talking about people sacrificing themselves or the whole world. Even if we were to adopt a purely survivalist perspective, I think it’s still far from obvious that trying to slow things down is more effective than is focusing on other aims. After all, the space of alternative aims that one could focus on is vast, and trying to slow things down comes with non-trivial risks of its own (e.g. risks of backlash from tech-accelerationists). Again, I’m not saying it’s clear; I’m saying that it seems to me unclear either way.
But, as I see it, what’s at issue is what the best way is to avoid such a predicament/how to best navigate given our current all-too risky predicament.
FWIW, I think that a lot of the discussion around this issue appears strongly fear-driven, to such an extent that it seems to get in the way of sober and helpful analysis. This is, to be sure, extremely understandable. But I also suspect that it is not the optimal way to figure out how to best achieve our aims, nor an effective way to persuade readers on this forum. Likewise, I suspect that rallying calls along the lines of “Global moratorium on AGI, now” might generally be received less well than would, say, a deeper analysis of the reasons for and against attempts to institute that policy.
I feel like I’m one of the main characters in the film Don’t Look Up here.
Please can you name 10? The way I see it is—either alignment is solved in time with business as usual[1], or we Pause to allow time for alignment to be solved (or establish it’s impossibility). It is not a complicated situation. No need to be worrying about “fates worse than death” at this juncture.
seems highly unlikely, but please say if you think there are promising solutions here
Do you not trust Ilya when he says they have plenty more data?
https://youtu.be/Yf1o0TQzry8?t=656
I didn’t claim that there isn’t plenty more data. But a relevant question is: plenty more data for what? He says that the data situation looks pretty good, which I trust is true in many domains (e.g. video data), and that data would probably in turn improve performance in those domains. But I don’t see him claiming that the data situation looks good in terms of ensuring significant performance gains across all domains, which would be a more specific and stronger claim.
Moreover, the deference question could be posed in the other direction as well, e.g. do you not trust the careful data collection and projections of Epoch? (Though again, Ilya saying that the data situation looks pretty good is arguably not in conflict with Epoch’s projections — nor with any claim I made above — mostly because his brief “pretty good” remark is quite vague.)
Note also that, at least in some domains, OpenAI could end up having less data to train their models with going forward, as they might have been using data illegally.
Let’s hope that OpenAI is forced to pull GPT-4 over the illegal data harvesting used to create it.
Coming back to the point about data. Whilst Epoch gathered some data showing that the stock high quality text data might soon be exhausted, their overall conclusion is that there is only a “20% chance that the scaling (as measured in training compute) of ML models will significantly slow down by 2040 due to a lack of training data.”. Regarding Jacob Buckman’s point about chess, he actually outlines a way around that (training data provided by narrow AI). As a counter to the wider point about the need for active learning, see DeepMind’s Adaptive Agent and the Voyager “lifelong learning” Minecraft agent, both of which seem like impressive steps in this direction.