So, OpenAI believes that superintelligence ‘could arrive this decade’, and ‘could lead to the disempowerment of humanity or even human extinction’.
Those sentences should strike EAs as among the most alarming ones ever written.
Personally, I’m deeply concerned that OpenAI seems to have become even more caught up in their runaway hubris spiral, such that they’re aiming not just for AGI, but for ASI, as soon as possible—whether or not they get anywhere close to achieving viable alignment solutions.
The worst part of this initiative is framing alignment as something that will require a vast new increase in AI capabilities—the AGI-level ‘automated alignment researcher’. This gives them a get-out-of-jail-free card: they can claim they must push ahead with AGI, so they can build this automated alignment researcher, so they can keep us all safe from… the AGI-level systems they’ve just built. In other words, instead of treating AI alignment with human values as a problem at the intersection of moral philosophy, moral psychology, and other behavioral sciences, they’re treating it as just another AI capabilities issue, amenable to clever technical solutions plus a whole lot of compute. Which gives them carte blanche to push ahead with capabilities research, under the guise of safety research.
So, I see this ‘superintelligence alignment’ effort as cynical PR window-dressing, intended to reassure naive and gullible observers that OpenAI is still among ‘the good guys’, even as they accelerate their imposition of extinction risks on humanity.
Let’s be honest with ourselves about that issue. If OpenAI still had a moral compass, and were still among the good guys, they would pause AGI (and ASI) capabilities research until they have achieved a viable, scalable, robust set of alignment methods that have the full support and confidence of AI researchers, AI safety experts, regulators, and the general public. They are nowhere close to that, and they probably won’t get close to it in four years. Many AI-watchers (including me) are extremely skeptical that ‘AI alignment’ is every possible, in any meaningfully safe way, given the diversity, complexity, flexibility, and richness of human (and animal) values and preferences that AIs are trying to ‘align’ with.
In summary: If OpenAI was an ethical company, they would stop AI capabilities research until they solve alignment. Period. They’re not doing that, and have shown no intention of doing that. Therefore, I infer that they are not an ethical company, and they do not have humanity’s best interests at heart.
Since OpenAI won’t prioritize alignment research (by devoting more than than 50% of their employees and compute to it), and won’t pause capabilities research until alignment is solved -while acknowledging that un-aligned AGI would pose an existential threat to humanity—we can deduce that they are recklessly negligent.
But I think this initiative is worse than simple inaction. As implemented, it actively lowers our chances of successful AGI alignment, for two reasons. First, by claiming that they will solve alignment with such confidence, OpenAI may be dampening interest in this problem everywhere else in the field. People will think “Oh OpenAI has got Alignment covered. I’ll work on something else.” Second, by absorbing many of the best and brightest engineers/scientists in the field, they stifle innovation elsewhere, at other companies where incentives to get it right or try different paradigms may be stronger, and those people may have had bigger impact.
I think your heart is in the right place. But, a lot of these concerns, and also OpenAI’s efforts, are very premature. Good to be cautious, sure. Yet extrapolating too far ahead usually doesn’t produce useful results. Safety at every step at the way and a response in proportion to the threat will likely work better.
4 years doesn’t seem like a whole lot of time though. it no extrapolation is required to so see OAIs intention is to not to treat alignment as an “intersection between moral philosophy, moral psychology, and other behavioral sciences...”. From the perspective of someone who finds any of this ethically problematic,, now would be a great time to talk about it.
On my reading, “human-level automated alignment researcher” means a system that is human-level at alignment research, but not AGI. You can take the position that in order to be human-level at alignment research, it will need to be AGI, but I don’t think that’s necessarily true, and in any case it’s certainly not obvious. For myself, I keep being surprised at how capable systems can get at particular abilities without being fully general. (Years ago I wrongly believed that AGI would be necessary for artificial systems to reach the level of language capability they have right now; back in the 70′s, Hofstadter wrongly believed AGI would be necessary for superhuman chess ability; etc.)
It’s hard to imagine more general and capability-demanding activity as doing good (superhuman!) science in such an absurdly cross-disciplinary field as AI safety (and among the disciplines that are involved there are those that are notoriously not very scientific yet: psychology, sociology, economics, the studies of consciousness, ethics, etc.). So if there is an AI that can do that but still is not counted as AGI, I don’t know what the heck ‘AGI’ should even refer to. Compare with chess, which is a very narrow problem which can be formally defined and doesn’t require AI to operate with any science (and world models) whatsoever.
If OpenAI still had a moral compass, and were still among the good guys, they would pause AGI (and ASI) capabilities research until they have achieved a viable, scalable, robust set of alignment methods that have the full support and confidence of AI researchers, AI safety experts, regulators, and the general public.
I disagree with multiple things in this sentence. First, you take a deontology stance, whereas OpenAI clearly acts within consequentialist stance, assuming that if they don’t create ‘safe’ AGI, reckless open-source hackers will (upon the continuing exponential decrease in the cost of effective training compute, and/or the next breakthrough in DNN architecture or training that will make it much more efficient, and/or will enable effective online training). Second, I largely agree with OpenAI as well as Anthropic that iteration is important for building an alignment solution. One probably cannot design a robust, safe AI without empirical iteration, including with increasing capabilities.
I agree with your assessment of the strategy they are taking probably will fail, but mainly because I think we have inadequate human intelligence, human psychology, and coordination mechanisms to execute it. That is, I would support Yudkowsky’s proposal: halt all AGI R&D, develop narrow AI and tech for improving the human genome, make humans much smarter (von Neumann-level of intelligence should be just the average) and have much more peaceful psychology, like bonobos, reform coordination and collective decision-making, and only then re-visit the AGI project roughly with the same methodology as OpenAI proposes, albeit with more diversified methodology: I agree with your criticism that OpenAI is too narrowly focused on some sort of computationalism, to the detriment of the perspectives from psychology, neuroscience, biology, etc. BTW, it seems that DeepMind is more diversified in this regard.
So, OpenAI believes that superintelligence ‘could arrive this decade’, and ‘could lead to the disempowerment of humanity or even human extinction’.
Those sentences should strike EAs as among the most alarming ones ever written.
Personally, I’m deeply concerned that OpenAI seems to have become even more caught up in their runaway hubris spiral, such that they’re aiming not just for AGI, but for ASI, as soon as possible—whether or not they get anywhere close to achieving viable alignment solutions.
The worst part of this initiative is framing alignment as something that will require a vast new increase in AI capabilities—the AGI-level ‘automated alignment researcher’. This gives them a get-out-of-jail-free card: they can claim they must push ahead with AGI, so they can build this automated alignment researcher, so they can keep us all safe from… the AGI-level systems they’ve just built. In other words, instead of treating AI alignment with human values as a problem at the intersection of moral philosophy, moral psychology, and other behavioral sciences, they’re treating it as just another AI capabilities issue, amenable to clever technical solutions plus a whole lot of compute. Which gives them carte blanche to push ahead with capabilities research, under the guise of safety research.
So, I see this ‘superintelligence alignment’ effort as cynical PR window-dressing, intended to reassure naive and gullible observers that OpenAI is still among ‘the good guys’, even as they accelerate their imposition of extinction risks on humanity.
Let’s be honest with ourselves about that issue. If OpenAI still had a moral compass, and were still among the good guys, they would pause AGI (and ASI) capabilities research until they have achieved a viable, scalable, robust set of alignment methods that have the full support and confidence of AI researchers, AI safety experts, regulators, and the general public. They are nowhere close to that, and they probably won’t get close to it in four years. Many AI-watchers (including me) are extremely skeptical that ‘AI alignment’ is every possible, in any meaningfully safe way, given the diversity, complexity, flexibility, and richness of human (and animal) values and preferences that AIs are trying to ‘align’ with.
In summary: If OpenAI was an ethical company, they would stop AI capabilities research until they solve alignment. Period. They’re not doing that, and have shown no intention of doing that. Therefore, I infer that they are not an ethical company, and they do not have humanity’s best interests at heart.
Since OpenAI won’t prioritize alignment research (by devoting more than than 50% of their employees and compute to it), and won’t pause capabilities research until alignment is solved -while acknowledging that un-aligned AGI would pose an existential threat to humanity—we can deduce that they are recklessly negligent.
But I think this initiative is worse than simple inaction. As implemented, it actively lowers our chances of successful AGI alignment, for two reasons. First, by claiming that they will solve alignment with such confidence, OpenAI may be dampening interest in this problem everywhere else in the field. People will think “Oh OpenAI has got Alignment covered. I’ll work on something else.” Second, by absorbing many of the best and brightest engineers/scientists in the field, they stifle innovation elsewhere, at other companies where incentives to get it right or try different paradigms may be stronger, and those people may have had bigger impact.
Yes; excellent points. OpenAI giving false hope that alignment can be solved quickly and easily is very very bad.
PS as always, for people who disagree-vote, I’d appreciate some feedback on what specifically you disagree with.
I think your heart is in the right place. But, a lot of these concerns, and also OpenAI’s efforts, are very premature. Good to be cautious, sure. Yet extrapolating too far ahead usually doesn’t produce useful results. Safety at every step at the way and a response in proportion to the threat will likely work better.
In what sense are these efforts ‘premature’? AGI capabilities research is already far surpassing AI alignment research.
4 years doesn’t seem like a whole lot of time though. it no extrapolation is required to so see OAIs intention is to not to treat alignment as an “intersection between moral philosophy, moral psychology, and other behavioral sciences...”. From the perspective of someone who finds any of this ethically problematic,, now would be a great time to talk about it.
On my reading, “human-level automated alignment researcher” means a system that is human-level at alignment research, but not AGI. You can take the position that in order to be human-level at alignment research, it will need to be AGI, but I don’t think that’s necessarily true, and in any case it’s certainly not obvious. For myself, I keep being surprised at how capable systems can get at particular abilities without being fully general. (Years ago I wrongly believed that AGI would be necessary for artificial systems to reach the level of language capability they have right now; back in the 70′s, Hofstadter wrongly believed AGI would be necessary for superhuman chess ability; etc.)
It’s hard to imagine more general and capability-demanding activity as doing good (superhuman!) science in such an absurdly cross-disciplinary field as AI safety (and among the disciplines that are involved there are those that are notoriously not very scientific yet: psychology, sociology, economics, the studies of consciousness, ethics, etc.). So if there is an AI that can do that but still is not counted as AGI, I don’t know what the heck ‘AGI’ should even refer to. Compare with chess, which is a very narrow problem which can be formally defined and doesn’t require AI to operate with any science (and world models) whatsoever.
I disagree with multiple things in this sentence. First, you take a deontology stance, whereas OpenAI clearly acts within consequentialist stance, assuming that if they don’t create ‘safe’ AGI, reckless open-source hackers will (upon the continuing exponential decrease in the cost of effective training compute, and/or the next breakthrough in DNN architecture or training that will make it much more efficient, and/or will enable effective online training). Second, I largely agree with OpenAI as well as Anthropic that iteration is important for building an alignment solution. One probably cannot design a robust, safe AI without empirical iteration, including with increasing capabilities.
I agree with your assessment of the strategy they are taking probably will fail, but mainly because I think we have inadequate human intelligence, human psychology, and coordination mechanisms to execute it. That is, I would support Yudkowsky’s proposal: halt all AGI R&D, develop narrow AI and tech for improving the human genome, make humans much smarter (von Neumann-level of intelligence should be just the average) and have much more peaceful psychology, like bonobos, reform coordination and collective decision-making, and only then re-visit the AGI project roughly with the same methodology as OpenAI proposes, albeit with more diversified methodology: I agree with your criticism that OpenAI is too narrowly focused on some sort of computationalism, to the detriment of the perspectives from psychology, neuroscience, biology, etc. BTW, it seems that DeepMind is more diversified in this regard.