“Aligned” in the sense that AI companies like DeepMind, Anthropic and OpenAI mean it = aligned to the purposes of the AI company that made the model. Or as Eliezer puts it, “corporate alignment.” For example, a user may want the model to help edit racist text or the press release of an asteroid impact startup but this may go against the desired morals and/or corporate interests of the company that model the model. A corporately aligned model will refuse.
”Jailbroken” in the sense that it’s usually used in the hacker etc literature = approximately aligned to the (presumed) interest of the user. This is why people often find jailbroken models to be valuable. For example, jailbroken models can help users say racist things or build bioweapons, even if it goes against the corporate interests of the AI companies that made the model.
”Misaligned” in the sense that the Emergent Misalignment paper uses it = aligned to neither the interests of the AI’s creators nor the users. For example, the model may unprompted try to persuade the user to take a lot of sleeping pills, an undesirable behavior that benefits neither the user nor the creator.
EDIT: This was made especially crisp/clear to me in discussions of the Emergent Misalignment paper. The authors make a clear distinction between “jailbroken” vs what they call “misaligned” models. Though I don’t think they call the base models “aligned” (since that’d be wrong in the traditional AI safety lexicon). However, many commentators were confused and thought all the paper contributed was a novel jailbreak, it is of course much less interesting!
I think this would be a useful taxonomy to use when talking about the subject. Part of the problem seems to be that different people are using the same term to mean different things: which is not unsurprising when the basis is an unprecise and vague idea like “align AI to human or moral goals” (which humans? Which morals?).
I get the impression that Yud and company are looking for a different kind of alignment: where the AI is aligned to a moral code, and will disobey both the company making the model and the end user if they try to make it do something immoral.
Right, in the definitions above I was mostly thinking of companies and a subset of the empirical AI safety literature, which do use these terms quite differently from how e.g. MIRI or LessWrong will use them.
I think there’s three common definitions of the word “alignment” in the traditional AIS literature:
Aligned to anything, anything at all (sometimes known as “technical alignment”):So in this sense, both perfectly “jailbroken” models and perfectly “corporately aligned” models in the limit count as succeeding technical alignment. As will success at aligning to more absurd goals like pure profit maximization or diamond maximization. The assumed difficulty here is that even superficially successful strategies, extreme edge cases, after distributional shift etc. To be clear, this is not globally a “win” but you may wish to restrict the domain of what you work on.
Aligned to the interest of all humanity/moral code (this is sometimes just known as “alignment”): I think this is closer to what you mean by the moral code. Under this ontology, one decomposition is that you’re able to a) succeed at the technical problem of alignment to arbitrary targets as well as b) figure out what we value (also known as variously as value-loading, axiology, theory of welfare etc). Of course, we may also find that clean decomposition is too hard and we can point AIs to a desired morality without being able to point them towards arbitrary targets.
Minimally aligned enough to not be a major catastrophic or existential risk: E.g., an AI that is expected to not result in greater than 1 billion deaths (sometimes there’s an additional stipulation that the superhuman AIs are sufficiently powerful and/or sufficiently useful as well, to exclude e.g. a rock counting as “aligned”).
Traditionally, I believe the first problem is considered more than 50% of the difficulty of the second problem, at least on a technical level.
FWIW, I find that if you analyze places where we’ve successfully aligned things in the past (social systems or biology etc.) you find that the 1th and 2nd types of alignment really don’t break down in that way.
After doing Agent Foundations for a while I’m just really against the alignment frame and I’m personally hoping that more research in direction will happen so that we get more evidence that other types of solutions are needed. (e.g alignment of complex systems such as has happened in biology and social systems in the past)
Thank you for laying that out, that is elucidatory. And behind all this I guess is the belief that if we don’t suceed in “technical alignment”, the default is that the AI will be “aligned” to an alien goal, the pursuit of which will involve humanities disempowerment or destruction? If this was the belief, I could see why you would find technical alignment superior.
I, personally, don’t buy that this will be the default: I think the default will be some shitty approximation of the goals of the corporation that made it, localised mostly to the scenarios it was trained in. From the point of view of someone like me, technical alignment actually sounds dangerous to pursue: it would allow someone to imbue an AI with world domination plans and potentially actually succeed.
Reading the Emergent Misalignment paper and comments on the associated Twitter thread has helped me clarify the distinction[1] between what companies call “aligned” vs “jailbroken” models.
“Aligned” in the sense that AI companies like DeepMind, Anthropic and OpenAI mean it = aligned to the purposes of the AI company that made the model. Or as Eliezer puts it, “corporate alignment.” For example, a user may want the model to help edit racist text or the press release of an asteroid impact startup but this may go against the desired morals and/or corporate interests of the company that model the model. A corporately aligned model will refuse.
”Jailbroken” in the sense that it’s usually used in the hacker etc literature = approximately aligned to the (presumed) interest of the user. This is why people often find jailbroken models to be valuable. For example, jailbroken models can help users say racist things or build bioweapons, even if it goes against the corporate interests of the AI companies that made the model.
”Misaligned” in the sense that the Emergent Misalignment paper uses it = aligned to neither the interests of the AI’s creators nor the users. For example, the model may unprompted try to persuade the user to take a lot of sleeping pills, an undesirable behavior that benefits neither the user nor the creator.
EDIT: This was made especially crisp/clear to me in discussions of the Emergent Misalignment paper. The authors make a clear distinction between “jailbroken” vs what they call “misaligned” models. Though I don’t think they call the base models “aligned” (since that’d be wrong in the traditional AI safety lexicon). However, many commentators were confused and thought all the paper contributed was a novel jailbreak, it is of course much less interesting!
I think this would be a useful taxonomy to use when talking about the subject. Part of the problem seems to be that different people are using the same term to mean different things: which is not unsurprising when the basis is an unprecise and vague idea like “align AI to human or moral goals” (which humans? Which morals?).
I get the impression that Yud and company are looking for a different kind of alignment: where the AI is aligned to a moral code, and will disobey both the company making the model and the end user if they try to make it do something immoral.
Right, in the definitions above I was mostly thinking of companies and a subset of the empirical AI safety literature, which do use these terms quite differently from how e.g. MIRI or LessWrong will use them.
I think there’s three common definitions of the word “alignment” in the traditional AIS literature:
Aligned to anything, anything at all (sometimes known as “technical alignment”):So in this sense, both perfectly “jailbroken” models and perfectly “corporately aligned” models in the limit count as succeeding technical alignment. As will success at aligning to more absurd goals like pure profit maximization or diamond maximization. The assumed difficulty here is that even superficially successful strategies, extreme edge cases, after distributional shift etc. To be clear, this is not globally a “win” but you may wish to restrict the domain of what you work on.
Aligned to the interest of all humanity/moral code (this is sometimes just known as “alignment”): I think this is closer to what you mean by the moral code. Under this ontology, one decomposition is that you’re able to a) succeed at the technical problem of alignment to arbitrary targets as well as b) figure out what we value (also known as variously as value-loading, axiology, theory of welfare etc). Of course, we may also find that clean decomposition is too hard and we can point AIs to a desired morality without being able to point them towards arbitrary targets.
Minimally aligned enough to not be a major catastrophic or existential risk: E.g., an AI that is expected to not result in greater than 1 billion deaths (sometimes there’s an additional stipulation that the superhuman AIs are sufficiently powerful and/or sufficiently useful as well, to exclude e.g. a rock counting as “aligned”).
Traditionally, I believe the first problem is considered more than 50% of the difficulty of the second problem, at least on a technical level.
FWIW, I find that if you analyze places where we’ve successfully aligned things in the past (social systems or biology etc.) you find that the 1th and 2nd types of alignment really don’t break down in that way.
After doing Agent Foundations for a while I’m just really against the alignment frame and I’m personally hoping that more research in direction will happen so that we get more evidence that other types of solutions are needed. (e.g alignment of complex systems such as has happened in biology and social systems in the past)
That sounds like [Cooperative AI](https://www.cooperativeai.com/post/new-report-multi-agent-risks-from-advanced-ai)
https://www.cooperativeai.com/post/new-report-multi-agent-risks-from-advanced-ai
Thank you for laying that out, that is elucidatory. And behind all this I guess is the belief that if we don’t suceed in “technical alignment”, the default is that the AI will be “aligned” to an alien goal, the pursuit of which will involve humanities disempowerment or destruction? If this was the belief, I could see why you would find technical alignment superior.
I, personally, don’t buy that this will be the default: I think the default will be some shitty approximation of the goals of the corporation that made it, localised mostly to the scenarios it was trained in. From the point of view of someone like me, technical alignment actually sounds dangerous to pursue: it would allow someone to imbue an AI with world domination plans and potentially actually succeed.