Many new researchers are going into AI alignment. For a variety of reasons, they may choose to work for organizations such as Anthropic or OpenAI. Chances are good that a new researcher will be interested in “interpretability”.
A creeping concern for many: “Is my research going to cause AGI ruin? Am I making the most powerful AI systems more powerful, even though I’m trying to make them safer?” Maybe they’ve even heard someone say that “mechanistic interpretability is capabilities research”. This essay dissects the specific case of interpretability research, to figure out when it does more harm than good.
What is Interpretability?
How do neural networks “think”? When you input some tokens into ChatGPT, how exactly does it decide what the best next-token is? What attributes of human language does it keep track of, and how does it do so? Interpretability is the sub-area of AI research that tackles these sorts of question.
Interpretability can be likened to investigating human psychology from a “bottom-up” approach of observing neurons, cortex structures, neurotransmitters, and similar low-level entities. This focus on the mechanics of a mind’s “substrate” (whether in biological neurons or in artificial neural networks) has obvious strengths, yet subtler weaknesses we’ll explore later.
One example of interpretability work is the recent “neurons” work by OpenAI. In their paper, “Language models can explain neurons in language models”, they tell GPT-4 to write explanations of the individual neurons within the smaller GPT-2 model. The idea is to gain human-readable understanding of what a large language model (LLM) is thinking by seeing which neurons correspond to which output-components. So one neuron’s activation-state may correspond to the presence of fractions, while another codes for times of day.
We also have research such as “Progress measures for grokking via mechanistic interpretability”. In this paper, the authors first train a neural network to perform a math operation (whose answer is easily checkable). Then, they analyze the resulting network’s structure to reverse-engineer the algorithm it “learned” to use. While human mathematicians and engineers have developed their own ways to solve the math problem (addition modulo a prime number), the neural network eventually hit on its own nonstandard method for doing so.
When is This Dangerous?
I posit that interpretability work is “dangerous” when it enhances the overall capabilities of an AI system, without making that system more aligned with human goals. This tracks well with the increasingly-popular notion of “speeding up capabilities research VS speeding up alignment research”. We prefer when our work counterfactually increases AI alignment, while not otherwise speeding up the development of AGI capabilities.
The key fact about interpretability research, which determines its safety/usefulness under the above criteria, is whether it enhances human control over an AI system. This suggests a few concrete rules-of-thumb, which a researcher can apply to their interpretability project P:
If P gives us a higher-resolution picture of an AI’s thought patterns, without giving us a way to reliably change them, then P is dangerous interpretability research.
If P is used to make a relied-on AI system less-powerful or less-general, yet safer for humans to use, then P is less dangerous.
If P makes it easier/more efficient to train powerful AI models, then P is dangerous. (This would be similar to making every GPU on Earth 10x as energy-efficient or 10x as fast at its computations: clearly speeding up the development of dangerous capabilities.)
If P is used in conjunction with, or as, a “steering” mechanism to control an AI’s behavior, then P is less dangerous.
The Sealed Interpretability Lab
One thought experiment can show us the potential dangers of interpretability research in greater detail. This is based on a question I was asked by “woog” on Discord.
Imagine a lab whose output is sealed off from the rest of the world. Its researchers can look at other public research, but they can’t release anything learned at the lab. This lab’s sole focus is AI interpretability, revealing ML systems’ “thoughts” to human observers.
The guiding question: If somebody works at this lab, are they speeding up capabilities research?
One detail that helps answer this question, is what kinds of AI systems the lab is working with.
If the lab can only work with existing ML models, such as ChatGPT, then we presume it cannot train its own models. This reduces the computing-power requirements of the lab, which already makes it unlikely to advance capabilities through “just scaling it up”. However, its actions create knowledge that in general would speed up capabilities development, either internally or when sharing research with trusted partners.
If the lab can create its own ML models, especially of a size comparable to the state-of-the-art LLMs, then it’s likely to advance capabilities research.
What happened here? If the interpretability-only lab can build large models, it can cause doom… but the same holds for merely working with existing large models? How can that be?
If the “rules of thumb” noted above apply to most interpretability research, then interpretability research can easily end up making it easier to develop AI capabilities. This could make weaker models stronger, and strong models even more strong. So to make things truly safe, the interpretability-only lab can’t work with the newest models… the ones being used in real life, and which are the most likely to be dangerous and deceptive. Toy-model research could be useless (since it’s “easy to interpret” at a glance), large-model research could increase the dangerous capabilities of existing AI systems, and cutting-edge-model research itself speeds up capabilities progress.
It gets even worse from here: If the interpretability lab, as stated, never releases research, then it can’t provide useful interpretability techniques to the top capabilities-increasing labs. On the other hand, if those labs are doing things besides alignment (which they currently are), they are likely to use the interpretability techniques to use their models more efficiently:
If the interpretability work reveals problems with an ML model’s thought patterns, we may or may not have easy ways to correct those thought patterns directly, rather than the outputs. If we do find ways to correct an AI’s thought patterns, that would be progress on alignment (see “Just Retarget The Search” below). This could be verified (but not necessarily aided) by interpretability.
If the interpretability work only reveals surface-level problems, it can leave a model’s deeper malignant thought patterns untouched, while increasing the confidence placed in it by human operators. John Wentworth pointed out something sort-of-similar for the technique of Reinforcement Learning from Human Feedback (RLHF); the shallow easy-to-fix problems get fixed faster, while the deeper problems are hidden from view.
Basically, interpretability research can get more capabilities out of current state-of-the-art (SOTA) models, and can guide the capabilities-training of future models.
Another detail: How “sealed” is this interpretability lab?
If the lab never releases any of its interpretability research to anybody, then no other AI developers can benefit from alignment-enhancing interpretability work.
If the lab only releases its interpretability work to a few trusted top-level AI labs, those labs are likely to use the work to increase the capabilities of their models, without improving the “steer-ability” (see below) of them. As elaborated before, this can happen despite the intentions of the top-level labs.
If the lab publicly releases its interpretability results for all to see, then all the above problems can spread to every other lab.
We end up with a “damned if you do, damned if you don’t” decision-tree. Each leaf can speed up capabilities through sharing, speed up capabilities through independent model-building, or waste the resources of alignment donors.
Any interpretability-only research, can enhance the capabilities of existing models.
Interpretability research, when mixed with capabilities research, advances capabilities overall.
Progress on interpretability can easily be repurposed to use unaligned models more efficiently. This can be thought of as “increasing capabilities”.
With no progress on interpretability, the interpretability-only lab has no purpose.
Now, given how detail-contingent many of these scenarios are, it’s plausible that an organization could fix or avoid all of them. However, unless more of the top capabilities labs have info/exfohazard policies I’m not aware of, there’s little evidence that these groups are optimizing against the breadth of failure modes described here.
What Would the World Look Like, Otherwise?
To get a better sense of whether interpretability work is dangerous, we can imagine conditions that would be true if they weren’t dangerous. That is, in a world where interpretability work was accelerating alignment faster than capabilities (or was accelerating neither), what would we expect to see?
There should be multiple competing schools of thought, giving different answers to “how do neural networks think?”. When new interpretability research is released by a top lab, the results are held up as evidence for/against such answers.
As interpretability research progresses, its techniques are adopted for use in the largest/most-important ML models. If OpenAI comes up with an interpretability method, that quickly gets used to make ChatGPT and Bing AI safer for users, even if it makes them less generally-capable.
Interpretability tactics slow down, or don’t impact the speed of, new advances in capabilities.
Do we actually see these things in real life?
While there are different research agendas for AI alignment, and multiple schools of thought for “how a mind works”, they don’t seem to be impacted much by new interpretability research.
Some interpretability techniques are used in ML training. However, I am not aware of any time when information uncovered by an interpretability tool has led to a change-of-course or a deeper-alignment solution in a mainstream model.
Capabilities continues to advance quickly, despite the growing work in interpretability. Either state-of-the-art models aren’t built using new interpretability techniques, or they are (yet keep making mistakes and being hard-to-control), or they’re helping in a way that’s hard for outsiders to observe and verify. This is more of a point for “little/no impact”, which isn’t so bad.
Overall, it looks like interpretability work is often ignored or not-very-useful in practice. This is a far cry from it being fully-dangerous, at least at present. Maybe it is helping alignment, but work on it is slow. (Interpretability has been around since at least 2018, but that may not be enough time for its work to bear fruit.)
When Interpretability is Still Important
I generally break down the problem of AI alignment into two subproblems:
Steering cognition: Can we control the thought and behavior patterns of a powerful mind at all? This is the question behind the rocket alignment problem analogy. Currently, we have large, inscrutable neural networks that output increasingly-smart answers to given questions. We can’t easily or reliably guide a neural network to avoid unwanted behaviors or thought patterns.
Deciding/implanting values: Even if we can steer a powerful AI system to think and behave in safe/friendly ways, how do we then point it towards the best values for the future? This vein of research includes the concept of Coherent Extrapolated Volition, the value-loading part of the QACI alignment setup, and (in my view) the idea of moral uncertainty. If the Qualia Research Institute focused consistently on their mission to understand “what makes a being sentient at all?” and “what experiences will be positive or negative for sentient beings”, their work would generally be on the values-side as well.
It seems that interpretability work would be, not only helpful, but essential for the “steering cognition” subproblem. After all, if you cannot discern a boat’s location, you would be hard-pressed to get “better” at steering it. The same is true for the internal mechanisms of artificial minds. If we can’t tell what an AI system is “thinking”, how do we know if we’re really “in control”?
However, you’ll note that interpretability on its own does not solve either of the two difficult subproblems listed. If you’re stuck in a self-driving car that’s going to ram into a wall, having a more-accurate prediction of the impact-angle is not going to stop or steer the car out of harm’s way.
Nevertheless, “knowing when we’re steering” could still be centrally-important for “solving steering”.
Wentworth’s “Just Retarget The Search” essay shows us a potential instantiation of this idea. Imagine a day when interpretability tools are good enough to identify higher-level “modules” for general reasoning, search, and goal-directedness in AI systems. If these higher-level modules can be picked out, their data can then be rewritten so “target” what human want. This mostly or entirely solves the “steering cognition” subproblem. Under certain assumptions, such as the “natural abstraction hypothesis” being true (i.e. the aforementioned “modules” existing), this use of alignment would be quite safe and alignment-oriented. But this exception itself demonstrates why interpretability is not enough; some theoretical backing is likely still needed, so we can tell if we’re “binding” the AI’s behavior in full, or just one part of its cognition. Even if interpretability is essential, that does not preclude it from being dangerous in the ways described earlier.
The Implications for P(doom)
If AGI is developed by 2070, will it become uncontrollable by humans, in a way that causes an existential catastrophe?
On its face, interpretability work is supposed to lower the odds of that occurring. As noted above, interpretability work can help us confirm the viability of alignment solutions for steering cognition. But it doesn’t really give us those steering solutions, and it’s unlikely to do so before a dangerous AGI system is developed.
Reasonably, most interpretability work is at risk of increasing humanity’s P(doom). In particular, the following criteria modulate the resulting risk-change:
If interpretability research isn’t tightly coupled to cognition-steering research, it could increase P(doom).
If interpretability research is released to the public and/or top capabilities labs, it could increase P(doom).
If interpretability research is either too low-level to help humans steer cognition (due to remaining inscrutability), or too surface-level/outputs-based to detect deeper misalignment with human objectives, it wouldn’t decrease P(doom).
If interpretability research continues to get more resources and researcher-manpower (or be a more-parallelizable use of those resources) than more-direct alignment research paths, it could increase P(doom) by competing with those paths.
In closing, if alignment-conscious researchers continue going into the interpretability subfield, the probability of AGI ruin will tend to increase.
Why and When Interpretability Work is Dangerous
This essay was partly based on discussions with “woog” on Discord. Further thanks to the gears to ascension, for inspiring this post with an offhand comment. This is also an entry for the Open Philanthropy AI Worldviews Contest.
Many new researchers are going into AI alignment. For a variety of reasons, they may choose to work for organizations such as Anthropic or OpenAI. Chances are good that a new researcher will be interested in “interpretability”.
A creeping concern for many: “Is my research going to cause AGI ruin? Am I making the most powerful AI systems more powerful, even though I’m trying to make them safer?” Maybe they’ve even heard someone say that “mechanistic interpretability is capabilities research”. This essay dissects the specific case of interpretability research, to figure out when it does more harm than good.
What is Interpretability?
How do neural networks “think”? When you input some tokens into ChatGPT, how exactly does it decide what the best next-token is? What attributes of human language does it keep track of, and how does it do so? Interpretability is the sub-area of AI research that tackles these sorts of question.
Interpretability can be likened to investigating human psychology from a “bottom-up” approach of observing neurons, cortex structures, neurotransmitters, and similar low-level entities. This focus on the mechanics of a mind’s “substrate” (whether in biological neurons or in artificial neural networks) has obvious strengths, yet subtler weaknesses we’ll explore later.
One example of interpretability work is the recent “neurons” work by OpenAI. In their paper, “Language models can explain neurons in language models”, they tell GPT-4 to write explanations of the individual neurons within the smaller GPT-2 model. The idea is to gain human-readable understanding of what a large language model (LLM) is thinking by seeing which neurons correspond to which output-components. So one neuron’s activation-state may correspond to the presence of fractions, while another codes for times of day.
We also have research such as “Progress measures for grokking via mechanistic interpretability”. In this paper, the authors first train a neural network to perform a math operation (whose answer is easily checkable). Then, they analyze the resulting network’s structure to reverse-engineer the algorithm it “learned” to use. While human mathematicians and engineers have developed their own ways to solve the math problem (addition modulo a prime number), the neural network eventually hit on its own nonstandard method for doing so.
When is This Dangerous?
I posit that interpretability work is “dangerous” when it enhances the overall capabilities of an AI system, without making that system more aligned with human goals. This tracks well with the increasingly-popular notion of “speeding up capabilities research VS speeding up alignment research”. We prefer when our work counterfactually increases AI alignment, while not otherwise speeding up the development of AGI capabilities.
The key fact about interpretability research, which determines its safety/usefulness under the above criteria, is whether it enhances human control over an AI system. This suggests a few concrete rules-of-thumb, which a researcher can apply to their interpretability project P:
If P gives us a higher-resolution picture of an AI’s thought patterns, without giving us a way to reliably change them, then P is dangerous interpretability research.
If P is used to make a relied-on AI system less-powerful or less-general, yet safer for humans to use, then P is less dangerous.
If P makes it easier/more efficient to train powerful AI models, then P is dangerous. (This would be similar to making every GPU on Earth 10x as energy-efficient or 10x as fast at its computations: clearly speeding up the development of dangerous capabilities.)
If P is used in conjunction with, or as, a “steering” mechanism to control an AI’s behavior, then P is less dangerous.
The Sealed Interpretability Lab
One thought experiment can show us the potential dangers of interpretability research in greater detail. This is based on a question I was asked by “woog” on Discord.
Imagine a lab whose output is sealed off from the rest of the world. Its researchers can look at other public research, but they can’t release anything learned at the lab. This lab’s sole focus is AI interpretability, revealing ML systems’ “thoughts” to human observers.
The guiding question: If somebody works at this lab, are they speeding up capabilities research?
One detail that helps answer this question, is what kinds of AI systems the lab is working with.
If the lab can only work with existing ML models, such as ChatGPT, then we presume it cannot train its own models. This reduces the computing-power requirements of the lab, which already makes it unlikely to advance capabilities through “just scaling it up”. However, its actions create knowledge that in general would speed up capabilities development, either internally or when sharing research with trusted partners.
If the lab can create its own ML models, especially of a size comparable to the state-of-the-art LLMs, then it’s likely to advance capabilities research.
What happened here? If the interpretability-only lab can build large models, it can cause doom… but the same holds for merely working with existing large models? How can that be?
If the “rules of thumb” noted above apply to most interpretability research, then interpretability research can easily end up making it easier to develop AI capabilities. This could make weaker models stronger, and strong models even more strong. So to make things truly safe, the interpretability-only lab can’t work with the newest models… the ones being used in real life, and which are the most likely to be dangerous and deceptive. Toy-model research could be useless (since it’s “easy to interpret” at a glance), large-model research could increase the dangerous capabilities of existing AI systems, and cutting-edge-model research itself speeds up capabilities progress.
(As usual, the more powerful an AI system gets, the harder it is to align properly. Interpretability, without the steering mechanisms that are likely the core of AI alignment, doesn’t help this.)
It gets even worse from here: If the interpretability lab, as stated, never releases research, then it can’t provide useful interpretability techniques to the top capabilities-increasing labs. On the other hand, if those labs are doing things besides alignment (which they currently are), they are likely to use the interpretability techniques to use their models more efficiently:
If the interpretability work reveals problems with an ML model’s thought patterns, we may or may not have easy ways to correct those thought patterns directly, rather than the outputs. If we do find ways to correct an AI’s thought patterns, that would be progress on alignment (see “Just Retarget The Search” below). This could be verified (but not necessarily aided) by interpretability.
If the interpretability work only reveals surface-level problems, it can leave a model’s deeper malignant thought patterns untouched, while increasing the confidence placed in it by human operators. John Wentworth pointed out something sort-of-similar for the technique of Reinforcement Learning from Human Feedback (RLHF); the shallow easy-to-fix problems get fixed faster, while the deeper problems are hidden from view.
Basically, interpretability research can get more capabilities out of current state-of-the-art (SOTA) models, and can guide the capabilities-training of future models.
Another detail: How “sealed” is this interpretability lab?
If the lab never releases any of its interpretability research to anybody, then no other AI developers can benefit from alignment-enhancing interpretability work.
If the lab only releases its interpretability work to a few trusted top-level AI labs, those labs are likely to use the work to increase the capabilities of their models, without improving the “steer-ability” (see below) of them. As elaborated before, this can happen despite the intentions of the top-level labs.
If the lab publicly releases its interpretability results for all to see, then all the above problems can spread to every other lab.
We end up with a “damned if you do, damned if you don’t” decision-tree. Each leaf can speed up capabilities through sharing, speed up capabilities through independent model-building, or waste the resources of alignment donors.
Any interpretability-only research, can enhance the capabilities of existing models.
Interpretability research, when mixed with capabilities research, advances capabilities overall.
Progress on interpretability can easily be repurposed to use unaligned models more efficiently. This can be thought of as “increasing capabilities”.
With no progress on interpretability, the interpretability-only lab has no purpose.
Now, given how detail-contingent many of these scenarios are, it’s plausible that an organization could fix or avoid all of them. However, unless more of the top capabilities labs have info/exfohazard policies I’m not aware of, there’s little evidence that these groups are optimizing against the breadth of failure modes described here.
What Would the World Look Like, Otherwise?
To get a better sense of whether interpretability work is dangerous, we can imagine conditions that would be true if they weren’t dangerous. That is, in a world where interpretability work was accelerating alignment faster than capabilities (or was accelerating neither), what would we expect to see?
There should be multiple competing schools of thought, giving different answers to “how do neural networks think?”. When new interpretability research is released by a top lab, the results are held up as evidence for/against such answers.
As interpretability research progresses, its techniques are adopted for use in the largest/most-important ML models. If OpenAI comes up with an interpretability method, that quickly gets used to make ChatGPT and Bing AI safer for users, even if it makes them less generally-capable.
Interpretability tactics slow down, or don’t impact the speed of, new advances in capabilities.
Do we actually see these things in real life?
While there are different research agendas for AI alignment, and multiple schools of thought for “how a mind works”, they don’t seem to be impacted much by new interpretability research.
Some interpretability techniques are used in ML training. However, I am not aware of any time when information uncovered by an interpretability tool has led to a change-of-course or a deeper-alignment solution in a mainstream model.
Capabilities continues to advance quickly, despite the growing work in interpretability. Either state-of-the-art models aren’t built using new interpretability techniques, or they are (yet keep making mistakes and being hard-to-control), or they’re helping in a way that’s hard for outsiders to observe and verify. This is more of a point for “little/no impact”, which isn’t so bad.
Overall, it looks like interpretability work is often ignored or not-very-useful in practice. This is a far cry from it being fully-dangerous, at least at present. Maybe it is helping alignment, but work on it is slow. (Interpretability has been around since at least 2018, but that may not be enough time for its work to bear fruit.)
When Interpretability is Still Important
I generally break down the problem of AI alignment into two subproblems:
Steering cognition: Can we control the thought and behavior patterns of a powerful mind at all? This is the question behind the rocket alignment problem analogy. Currently, we have large, inscrutable neural networks that output increasingly-smart answers to given questions. We can’t easily or reliably guide a neural network to avoid unwanted behaviors or thought patterns.
Deciding/implanting values: Even if we can steer a powerful AI system to think and behave in safe/friendly ways, how do we then point it towards the best values for the future? This vein of research includes the concept of Coherent Extrapolated Volition, the value-loading part of the QACI alignment setup, and (in my view) the idea of moral uncertainty. If the Qualia Research Institute focused consistently on their mission to understand “what makes a being sentient at all?” and “what experiences will be positive or negative for sentient beings”, their work would generally be on the values-side as well.
It seems that interpretability work would be, not only helpful, but essential for the “steering cognition” subproblem. After all, if you cannot discern a boat’s location, you would be hard-pressed to get “better” at steering it. The same is true for the internal mechanisms of artificial minds. If we can’t tell what an AI system is “thinking”, how do we know if we’re really “in control”?
However, you’ll note that interpretability on its own does not solve either of the two difficult subproblems listed. If you’re stuck in a self-driving car that’s going to ram into a wall, having a more-accurate prediction of the impact-angle is not going to stop or steer the car out of harm’s way.
Nevertheless, “knowing when we’re steering” could still be centrally-important for “solving steering”.
Wentworth’s “Just Retarget The Search” essay shows us a potential instantiation of this idea. Imagine a day when interpretability tools are good enough to identify higher-level “modules” for general reasoning, search, and goal-directedness in AI systems. If these higher-level modules can be picked out, their data can then be rewritten so “target” what human want. This mostly or entirely solves the “steering cognition” subproblem. Under certain assumptions, such as the “natural abstraction hypothesis” being true (i.e. the aforementioned “modules” existing), this use of alignment would be quite safe and alignment-oriented. But this exception itself demonstrates why interpretability is not enough; some theoretical backing is likely still needed, so we can tell if we’re “binding” the AI’s behavior in full, or just one part of its cognition. Even if interpretability is essential, that does not preclude it from being dangerous in the ways described earlier.
The Implications for P(doom)
If AGI is developed by 2070, will it become uncontrollable by humans, in a way that causes an existential catastrophe?
On its face, interpretability work is supposed to lower the odds of that occurring. As noted above, interpretability work can help us confirm the viability of alignment solutions for steering cognition. But it doesn’t really give us those steering solutions, and it’s unlikely to do so before a dangerous AGI system is developed.
Reasonably, most interpretability work is at risk of increasing humanity’s P(doom). In particular, the following criteria modulate the resulting risk-change:
If interpretability research isn’t tightly coupled to cognition-steering research, it could increase P(doom).
If interpretability research is released to the public and/or top capabilities labs, it could increase P(doom).
If interpretability research is either too low-level to help humans steer cognition (due to remaining inscrutability), or too surface-level/outputs-based to detect deeper misalignment with human objectives, it wouldn’t decrease P(doom).
If interpretability research continues to get more resources and researcher-manpower (or be a more-parallelizable use of those resources) than more-direct alignment research paths, it could increase P(doom) by competing with those paths.
In closing, if alignment-conscious researchers continue going into the interpretability subfield, the probability of AGI ruin will tend to increase.
Further Reading
Should we publish mechanistic interpretability research?
AGI-Automated Interpretability is Suicide
How Interpretability can be Impactful
How many e-prints on AI Safety, Interpretability or Explainability will be published on arXiv from 2021 through 2026?