The post is ~900 words so I would recommend reading it, but the key takeaways are:
OpenAI are starting a new “superintelligence alignment” team, led by Ilya Sutskever (Chief Scientist at OpenAI) and Jan Leike (Alignment team lead at OpenAI). Ilya Sutskever will be making this his core research focus.
OpenAI are dedicating 20% of the compute they’ve secured to date over the next four years to solving superintelligent alignment
The superalignment team is hiring for a research engineer, research scientist and research manager.
[Edited to add] :
Here’s the introduction (with footnotes removed)
Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world’s most important problems. But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.
While superintelligence seems far off now, we believe it could arrive this decade. Managing these risks will require, among other things, new institutions for governance and solving the problem of superintelligence alignment:
How do we ensure AI systems much smarter than humans follow human intent?
And later on:
Our approach
Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence.
To align the first automated alignment researcher, we will need to 1) develop a scalable training method, 2) validate the resulting model, and 3) stress test our entire alignment pipeline:
To provide a training signal on tasks that are difficult for humans to evaluate, we can leverage AI systems to assist evaluation of other AI systems (scalable oversight). In addition, we want to understand and control how our models generalize our oversight to tasks we can’t supervise (generalization).
To validate the alignment of our systems, we automate search for problematic behavior (robustness) and problematic internals (automated interpretability).
Finally, we can test our entire pipeline by deliberately training misaligned models, and confirming that our techniques detect the worst kinds of misalignments (adversarial testing).
Paraphrased from OpenAI’s Twitter thread:
In addition to members from our existing alignment team, joining are Harri Edwards, Yuri Burda, Adrien Ecoffet, Nat McAleese, Collin Burns, Bowen Baker, Pavel Izmailov, and Leopold Aschenbrenner
And paraphrased from a Nat McAleese tweet:
Yes, this is the notkilleveryoneism team
So, OpenAI believes that superintelligence ‘could arrive this decade’, and ‘could lead to the disempowerment of humanity or even human extinction’.
Those sentences should strike EAs as among the most alarming ones ever written.
Personally, I’m deeply concerned that OpenAI seems to have become even more caught up in their runaway hubris spiral, such that they’re aiming not just for AGI, but for ASI, as soon as possible—whether or not they get anywhere close to achieving viable alignment solutions.
The worst part of this initiative is framing alignment as something that will require a vast new increase in AI capabilities—the AGI-level ‘automated alignment researcher’. This gives them a get-out-of-jail-free card: they can claim they must push ahead with AGI, so they can build this automated alignment researcher, so they can keep us all safe from… the AGI-level systems they’ve just built. In other words, instead of treating AI alignment with human values as a problem at the intersection of moral philosophy, moral psychology, and other behavioral sciences, they’re treating it as just another AI capabilities issue, amenable to clever technical solutions plus a whole lot of compute. Which gives them carte blanche to push ahead with capabilities research, under the guise of safety research.
So, I see this ‘superintelligence alignment’ effort as cynical PR window-dressing, intended to reassure naive and gullible observers that OpenAI is still among ‘the good guys’, even as they accelerate their imposition of extinction risks on humanity.
Let’s be honest with ourselves about that issue. If OpenAI still had a moral compass, and were still among the good guys, they would pause AGI (and ASI) capabilities research until they have achieved a viable, scalable, robust set of alignment methods that have the full support and confidence of AI researchers, AI safety experts, regulators, and the general public. They are nowhere close to that, and they probably won’t get close to it in four years. Many AI-watchers (including me) are extremely skeptical that ‘AI alignment’ is every possible, in any meaningfully safe way, given the diversity, complexity, flexibility, and richness of human (and animal) values and preferences that AIs are trying to ‘align’ with.
In summary: If OpenAI was an ethical company, they would stop AI capabilities research until they solve alignment. Period. They’re not doing that, and have shown no intention of doing that. Therefore, I infer that they are not an ethical company, and they do not have humanity’s best interests at heart.
Since OpenAI won’t prioritize alignment research (by devoting more than than 50% of their employees and compute to it), and won’t pause capabilities research until alignment is solved -while acknowledging that un-aligned AGI would pose an existential threat to humanity—we can deduce that they are recklessly negligent.
But I think this initiative is worse than simple inaction. As implemented, it actively lowers our chances of successful AGI alignment, for two reasons. First, by claiming that they will solve alignment with such confidence, OpenAI may be dampening interest in this problem everywhere else in the field. People will think “Oh OpenAI has got Alignment covered. I’ll work on something else.” Second, by absorbing many of the best and brightest engineers/scientists in the field, they stifle innovation elsewhere, at other companies where incentives to get it right or try different paradigms may be stronger, and those people may have had bigger impact.
Yes; excellent points. OpenAI giving false hope that alignment can be solved quickly and easily is very very bad.
PS as always, for people who disagree-vote, I’d appreciate some feedback on what specifically you disagree with.
I think your heart is in the right place. But, a lot of these concerns, and also OpenAI’s efforts, are very premature. Good to be cautious, sure. Yet extrapolating too far ahead usually doesn’t produce useful results. Safety at every step at the way and a response in proportion to the threat will likely work better.
In what sense are these efforts ‘premature’? AGI capabilities research is already far surpassing AI alignment research.
4 years doesn’t seem like a whole lot of time though. it no extrapolation is required to so see OAIs intention is to not to treat alignment as an “intersection between moral philosophy, moral psychology, and other behavioral sciences...”. From the perspective of someone who finds any of this ethically problematic,, now would be a great time to talk about it.
On my reading, “human-level automated alignment researcher” means a system that is human-level at alignment research, but not AGI. You can take the position that in order to be human-level at alignment research, it will need to be AGI, but I don’t think that’s necessarily true, and in any case it’s certainly not obvious. For myself, I keep being surprised at how capable systems can get at particular abilities without being fully general. (Years ago I wrongly believed that AGI would be necessary for artificial systems to reach the level of language capability they have right now; back in the 70′s, Hofstadter wrongly believed AGI would be necessary for superhuman chess ability; etc.)
It’s hard to imagine more general and capability-demanding activity as doing good (superhuman!) science in such an absurdly cross-disciplinary field as AI safety (and among the disciplines that are involved there are those that are notoriously not very scientific yet: psychology, sociology, economics, the studies of consciousness, ethics, etc.). So if there is an AI that can do that but still is not counted as AGI, I don’t know what the heck ‘AGI’ should even refer to. Compare with chess, which is a very narrow problem which can be formally defined and doesn’t require AI to operate with any science (and world models) whatsoever.
I disagree with multiple things in this sentence. First, you take a deontology stance, whereas OpenAI clearly acts within consequentialist stance, assuming that if they don’t create ‘safe’ AGI, reckless open-source hackers will (upon the continuing exponential decrease in the cost of effective training compute, and/or the next breakthrough in DNN architecture or training that will make it much more efficient, and/or will enable effective online training). Second, I largely agree with OpenAI as well as Anthropic that iteration is important for building an alignment solution. One probably cannot design a robust, safe AI without empirical iteration, including with increasing capabilities.
I agree with your assessment of the strategy they are taking probably will fail, but mainly because I think we have inadequate human intelligence, human psychology, and coordination mechanisms to execute it. That is, I would support Yudkowsky’s proposal: halt all AGI R&D, develop narrow AI and tech for improving the human genome, make humans much smarter (von Neumann-level of intelligence should be just the average) and have much more peaceful psychology, like bonobos, reform coordination and collective decision-making, and only then re-visit the AGI project roughly with the same methodology as OpenAI proposes, albeit with more diversified methodology: I agree with your criticism that OpenAI is too narrowly focused on some sort of computationalism, to the detriment of the perspectives from psychology, neuroscience, biology, etc. BTW, it seems that DeepMind is more diversified in this regard.
This may sound like a lot, but I think it’s likely that 4 years from now 20% of currently secured compute will just be a tiny fraction of what they’ll have secured by then.
Related: Zvi has just written a pretty in-depth piece on this new Superalignment team, which I recommend. Here’s the opening:
Very nice! I’d say this seems like it’s aimed at a difficulty level of 5 to 7 on my table,
https://www.lesswrong.com/posts/EjgfreeibTXRx9Ham/ten-levels-of-ai-alignment-difficulty#Table
I.e. experimentation on dangerous systems and interpretability play some role but the main thrust is automating alignment research and oversight, so maybe I’d unscientifically call it a 6.5, which is a tremendous step up from the current state of things (2.5) and would solve alignment in many possible worlds.
This is a great goal! I don’t believe they’ve got what it takes to achieve it, though. Safely directing a superintelligent system at solving alignment is an alignment-complete problem. Building a human-level system that does alignment research safely on the first try is possible, running more than one copy of this system at a superhuman speed safely is something no one has any idea how to even approach, and unless this insanity is stopped so we have many more than four years to solve alignment, we’re all dead
My first thought was “oh, so we should just let you handle your own evaluations?” My husband (Ronny Fernandez)’s first thought was “training “human-level AI researchers” to perform adversarial evaluations is setting up an intelligence explosion”...
Worth reading:
https://www.lesswrong.com/posts/xp6n2MG5vQkPpFEBH/the-control-problem-unsolved-or-unsolvable