In âUnpacking Classic Arguments for AI Riskâ, you defined The Process Orthogonality Thesis as: The process of imbuing a system with capabilities and the process of imbuing a system with goals are orthogonal.
Then, gave several examples of cases where this does not hold: thermostat, Deep Blue, OpenAI Five, the Human brain. Could you elaborate a bit on these examples?
I am a bit confused about it. In Deep Blue, I think that most of the progress has been general computational advances, and the application of an evaluation system given later. The human brain value system can be changed quite a lot without apparent changes in the capacity to achieve oneâs goals (consider psychopaths for extreme example here).
Also, general RL systems have had successes in applying themselves to many different circumstances. Say, the work of DeepMind on Atari. Doesnât that point in favor of the Process Orthogonality Thesis?
I think that my description of the thesis (and, actually, my own thinking on it) is a bit fuzzy. Nevertheless, hereâs roughly how Iâm thinking about it:
First, letâs say that an agent has the âgoalâ of doing X if itâs sometimes useful to think of the system as âtrying to do X.â For example, itâs sometimes useful to think of a person as âtryingâ to avoid pain, be well-liked, support their family, etc. Itâs sometimes useful to think of a chess program as âtryingâ to win games of chess.
Agents are developed through a series of changes. In the case of a âhand-codedâ AI system, the changes would involve developers adding, editing, or removing lines of code. In the case of an RL agent, the changes would typically involve a learning algorithm updating the agentâs policy. In the case of human evolution, the changes would involve genetic mutations.
If the âprocess orthogonality thesisâ were true, then this would mean that we can draw a pretty clean line between between âchanges that affect an agentâs capabilitiesâ and âchanges that affect an agentâs goals.â Instead, I want to say that itâs really common for changes to affect both capabilities and goals. In practice, we canât draw a clean line between âcapability genesâ and âgoal genesâ or between âRL policy updates that change goalsâ and âRL policy updates that change capabilities.â Both goals and capabilities tend to take shape together.
That being said, it is true that some changes do, intuitively, mostly just affect either capabilities or goals. I wouldnât be surprised, for example, if itâs possible to introduce a minus sign somewhere into Deep Blueâs code and transform it into a system that looks like itâs trying to lose at chess; although the system will probably be less good at losing than it was a winning, it may still be pretty capable. So the processes of changing a systemâs capabilities and changing its goals can still come apart to some degree.
Itâs also possible to do fundamental research and engineering work that is useful for developing a wide variety of systems. For example, hardware progress has, in general, made it easier to develop highly competent RL agents in all sorts of domains. But, when it comes time to train a new RL agent, its goals and capabilities will still take shape together.
Thanks! This does clarify things for me, and I think that the definition of a âgoalâ is very helpful here. I do still have some uncertainty here about the claim of process orthogonality which I can better understand:
Letâs define an âinstrumental goalâ as a goal X for which there is a goal Y such that whenever it is useful to think of the agent as âtrying to do Xâ it is in fact also useful to think of it as âtrying to do Yâ; In this case we can think that X is instrumental to Y. Instrumental goals can be generated at the development phase or by the agent itself (implicitly or explicitly).
I think that the (non-process) orthogonality thesis does not hold with respect to instrumental goals. A better selection of instrumental goals will enable better capabilities, and with greater capabilities comes greater planning capacity.
Therefore, the process orthogonality thesis does not hold as well for instrumental goals. This means that instrumental goals are usually not the goals of interest when trying to discern between process and non-process orthogonality theses, and we should focus on terminal goal (those which arenât instrumental).
In the case of an RL agent or Deep Blue, I can only see one terminal goalâmaximize defined score or win chess. These wonât really be change together with capabilities.
I thought a bit about humans, but I feel that this is much more complicated and needs more nuanced definitions of goals. (is avoiding suffering a terminal goal? It seems that way, but who is doing the thinking in which it is useful to think of one thing or another as a goal? Perhaps the goal is to reduce specific neuronal activity for which avoiding suffering is merely instrumental?)
I thought a bit about humans, but I feel that this is much more complicated and needs more nuanced definitions of goals. (is avoiding suffering a terminal goal? It seems that way, but who is doing the thinking in which it is useful to think of one thing or another as a goal? Perhaps the goal is to reduce specific neuronal activity for which avoiding suffering is merely instrumental?)
Iâm actually not very optimistic about a more complex or formal definition of goals. In my mind, the concept of a âgoalâ is often useful, but itâs sort of an intrinisically fuzzy or fundamentally pragmatic concept. I also think that, in practice, the distinction between an âintrinsicâ and âinstrumentalâ goal is pretty fuzzy in the same way (although I think your definition is a good one).
Ultimately, agents exhibit behaviors. Itâs often useful to try to summarize these behaviors in terms of what sorts of things the agent is fundamentally âtryingâ to do and in terms of the âcapabilitiesâ that the agent brings to bear. But I think this is just sort of a loose way of speaking. I donât really think, for example, that there are principled/âdefinitive answers to the questions âWhat are all of my catâs goals?â, âWhich of my catâs goals are intrinsic?â, or âWhatâs my catâs utility function?â Even if we want to move beyond behavioral definitions of goals, to ones that focus on cognitive processes, I think these sorts of questions will probably still remain pretty fuzzy.
(I think that this way of thinkingâin which evolutionary or engineering selection processes ultimately act on âbehaviors,â which can only somewhat informally or imprecisely be described in terms of âcapabilitiesâ and âgoalsââalso probably has an influence on my relative optimism about AI alignment. )
I was thinking it over, and I think that I was implicitly assuming that process orthogonality follows from orthogonality in some form or something like that.
The Deep Blue question still holds, I think.
The human brain should be thought of as designed by evolution. What I wrote is strictly (non-process) orthogonality. An example could be given that the cognitive breakthrough might have been enlargement of the neocortex, while civilization was responsible for the values.
I guess that the point is that there are example of non-orthogonality? (Say, the evaluation function of DeepBlue being critical for itâs success)
In âUnpacking Classic Arguments for AI Riskâ, you defined The Process Orthogonality Thesis as: The process of imbuing a system with capabilities and the process of imbuing a system with goals are orthogonal.
Then, gave several examples of cases where this does not hold: thermostat, Deep Blue, OpenAI Five, the Human brain. Could you elaborate a bit on these examples?
I am a bit confused about it. In Deep Blue, I think that most of the progress has been general computational advances, and the application of an evaluation system given later. The human brain value system can be changed quite a lot without apparent changes in the capacity to achieve oneâs goals (consider psychopaths for extreme example here).
Also, general RL systems have had successes in applying themselves to many different circumstances. Say, the work of DeepMind on Atari. Doesnât that point in favor of the Process Orthogonality Thesis?
I think that my description of the thesis (and, actually, my own thinking on it) is a bit fuzzy. Nevertheless, hereâs roughly how Iâm thinking about it:
First, letâs say that an agent has the âgoalâ of doing X if itâs sometimes useful to think of the system as âtrying to do X.â For example, itâs sometimes useful to think of a person as âtryingâ to avoid pain, be well-liked, support their family, etc. Itâs sometimes useful to think of a chess program as âtryingâ to win games of chess.
Agents are developed through a series of changes. In the case of a âhand-codedâ AI system, the changes would involve developers adding, editing, or removing lines of code. In the case of an RL agent, the changes would typically involve a learning algorithm updating the agentâs policy. In the case of human evolution, the changes would involve genetic mutations.
If the âprocess orthogonality thesisâ were true, then this would mean that we can draw a pretty clean line between between âchanges that affect an agentâs capabilitiesâ and âchanges that affect an agentâs goals.â Instead, I want to say that itâs really common for changes to affect both capabilities and goals. In practice, we canât draw a clean line between âcapability genesâ and âgoal genesâ or between âRL policy updates that change goalsâ and âRL policy updates that change capabilities.â Both goals and capabilities tend to take shape together.
That being said, it is true that some changes do, intuitively, mostly just affect either capabilities or goals. I wouldnât be surprised, for example, if itâs possible to introduce a minus sign somewhere into Deep Blueâs code and transform it into a system that looks like itâs trying to lose at chess; although the system will probably be less good at losing than it was a winning, it may still be pretty capable. So the processes of changing a systemâs capabilities and changing its goals can still come apart to some degree.
Itâs also possible to do fundamental research and engineering work that is useful for developing a wide variety of systems. For example, hardware progress has, in general, made it easier to develop highly competent RL agents in all sorts of domains. But, when it comes time to train a new RL agent, its goals and capabilities will still take shape together.
(Hope that clarifies things at least a bit!)
Thanks! This does clarify things for me, and I think that the definition of a âgoalâ is very helpful here. I do still have some uncertainty here about the claim of process orthogonality which I can better understand:
Letâs define an âinstrumental goalâ as a goal X for which there is a goal Y such that whenever it is useful to think of the agent as âtrying to do Xâ it is in fact also useful to think of it as âtrying to do Yâ; In this case we can think that X is instrumental to Y. Instrumental goals can be generated at the development phase or by the agent itself (implicitly or explicitly).
I think that the (non-process) orthogonality thesis does not hold with respect to instrumental goals. A better selection of instrumental goals will enable better capabilities, and with greater capabilities comes greater planning capacity.
Therefore, the process orthogonality thesis does not hold as well for instrumental goals. This means that instrumental goals are usually not the goals of interest when trying to discern between process and non-process orthogonality theses, and we should focus on terminal goal (those which arenât instrumental).
In the case of an RL agent or Deep Blue, I can only see one terminal goalâmaximize defined score or win chess. These wonât really be change together with capabilities.
I thought a bit about humans, but I feel that this is much more complicated and needs more nuanced definitions of goals. (is avoiding suffering a terminal goal? It seems that way, but who is doing the thinking in which it is useful to think of one thing or another as a goal? Perhaps the goal is to reduce specific neuronal activity for which avoiding suffering is merely instrumental?)
Iâm actually not very optimistic about a more complex or formal definition of goals. In my mind, the concept of a âgoalâ is often useful, but itâs sort of an intrinisically fuzzy or fundamentally pragmatic concept. I also think that, in practice, the distinction between an âintrinsicâ and âinstrumentalâ goal is pretty fuzzy in the same way (although I think your definition is a good one).
Ultimately, agents exhibit behaviors. Itâs often useful to try to summarize these behaviors in terms of what sorts of things the agent is fundamentally âtryingâ to do and in terms of the âcapabilitiesâ that the agent brings to bear. But I think this is just sort of a loose way of speaking. I donât really think, for example, that there are principled/âdefinitive answers to the questions âWhat are all of my catâs goals?â, âWhich of my catâs goals are intrinsic?â, or âWhatâs my catâs utility function?â Even if we want to move beyond behavioral definitions of goals, to ones that focus on cognitive processes, I think these sorts of questions will probably still remain pretty fuzzy.
(I think that this way of thinkingâin which evolutionary or engineering selection processes ultimately act on âbehaviors,â which can only somewhat informally or imprecisely be described in terms of âcapabilitiesâ and âgoalsââalso probably has an influence on my relative optimism about AI alignment. )
I was thinking it over, and I think that I was implicitly assuming that process orthogonality follows from orthogonality in some form or something like that.
The Deep Blue question still holds, I think.
The human brain should be thought of as designed by evolution. What I wrote is strictly (non-process) orthogonality. An example could be given that the cognitive breakthrough might have been enlargement of the neocortex, while civilization was responsible for the values.
I guess that the point is that there are example of non-orthogonality? (Say, the evaluation function of DeepBlue being critical for itâs success)