Reinforcement learning does not select for reward maximising agents in general
Reward “upweight certain kinds of actions in certain kinds of situations, and therefore reward chisels cognitive grooves into agents”
I’m thus very sceptical that systems optimised via reinforcement learning to be capable in a wide variety of domains/tasks converge towards maximising a simple expected utility function
I am not aware that humanity actually knows training paradigms that select for expected utility maximisers
Our most capable/economically transformative AI systems are not agents and are definitely not expected utility maximisers
The do not become agents in the limit and expected utility maximisation is a particular kind of agency
I am seriously entertaining the hypothesis that expected utility maximisation is anti-natural to selection for general intelligence
I’m not under the impression that systems optimised by stochastic gradient descent to be generally capable optimisers converge towards expected utility maximisers
The generally capable optimisers produced by evolution aren’t expected utility maximisers
I’m starting to suspect that “search like” optimisation processes for general intelligence do not in general converge towards expected utility maximisers
I.e. it may end up being the case that the only way to create a generally capable expected utility maximiser is to explicitly design one
And we do not know how to design capable optimisers for rich environments
While it may be in theory feasible to design systems at any intelligence level with any final goal
In practice, we cannot design capable optimisers.
For intelligent systems created by “search like” optimisation, final goals are not orthogonal to cognitive ability
Sufficiently hard optimisation for most cognitive tasks would not converge towards selecting for generally capable systems
In the limit, what do systems selected for playing Go converge towards?
I posit that said limit is not “general intelligence”
The cognitive tasks/domain on which a system was optimised for performance on may instantiate an upper bound on the general capabilities of the system
You do not need much optimisation power to attain optimal performance in logical tic tac toe
Systems selected for performance at logical tic tac toe should be pretty weak narrow optimisers because that’s all that’s required for optimality in that domain
I don’t expect the systems that matter (in the par human or strongly superhuman regime) to be expected utility maximisers. I think arguments for AI x-risk that rest on expected utility maximisers are mostly disconnected from reality. I suspect that discussing the perils of expected utility maximisation in particular — as opposed to e.g. dangers from powerful (consequentialist?) optimisation processes — is somewhere between being a distraction and being actively harmful[3].
I do not think expected utility maximisation is the limit of what generally capable optimisers look like[4].
Arguments for Expected Utility Maximisation Are Unnecessary
I don’t think the case for existential risks from AI safety rest on expected utility maximisation. I kind of stopped alieving expected utility maximisers a while back (only recently have I synthesised explicit beliefs that reject it), but I still plan on working on AI existential safety, because I don’t see the core threat as resulting from expected utility maximisation.
The reasons I consider AI an existential threat mostly rely on:
Instrumental convergence for consequentialist/goal directed systems
A system doesn’t need to be a utility maximiser for a simple utility function to be goal directed (again, see humans)
Selection pressures for power seeking systems
Reasons
More economically productive/useful
Some humans are power seeking
Power seeking systems promote themselves/have better reproductive fitness
Human disempowerment is the immediate existential catastrophe scenario I foresee from power seeking
Bad game theoretic equilibria
This could lead towards dystopian scenarios in multipolar outcomes
Humans getting outcompeted by AI systems
Could slowly lead to an extinction
I do not actually expect extinction near term, but it’s not the only “existential catastrophe”:
I optimised for writing this quickly. So my language may be stronger/more confident that I actually feel. I may not have spent as much time accurately communicating my uncertainty as may have been warranted.
There may be significant opportunity cost via diverting attention from other more plausible pathways to doom.
In general, I think exposing people to bad arguments for a position is a poor persuasive strategy as people who dismiss said bad arguments may (rationally) update downwards on the credibility of the position.
I don’t necessarily think agents are that limit either. But as “Why Subagents?” shows, expected utility maximisers aren’t the limit of idealised agency.
Why The Focus on Expected Utility Maximisers?
Epistemic Status
Unsure[1], partially noticing my own confusion. Hoping Cunningham’s Law can help resolve it.
Related Answer
Confusions About Arguments From Expected Utility Maximisation
Some MIRI people (e.g. Rob Bensinger) still highlight EU maximisers as the paradigm case for existentially dangerous AI systems. I’m confused by this for a few reasons:
Not all consequentialist/goal directed systems are expected utility maximisers
E.g. humans
Some recent developments make me sceptical that VNM expected utility are a natural form of generally intelligent systems
Wentworth’s subagents provide a model for inexploitable agents that don’t maximise a simple unitary utility function
The main requirement for subagents to be a better model than unitary agents is path dependent preferences or hidden state variables
Alternatively, subagents natively admit partial orders over preferences
If I’m not mistaken, utility functions seem to require a (static) total order over preferences
This might be a very unreasonable ask; it does not seem to describe humans, animals, or even existing sophisticated AI systems
I think the strongest implication of Wentworth’s subagents is that expected utility maximisation is not the limit or idealised form of agency
Shard Theory suggests that trained agents (via reinforcement learning[2]) form value “shards”
Values are inherently “contextual influences on decision making”
Hence agents do not have a static total order over preferences (what a utility function implies) as what preferences are active depends on the context
Preferences are dynamic (change over time), and the ordering of them is not necessarily total
This explains many of the observed inconsistencies in human decision making
A multitude of value shards do not admit analysis as a simple unitary utility function
Reward is not the optimisation target
Reinforcement learning does not select for reward maximising agents in general
Reward “upweight certain kinds of actions in certain kinds of situations, and therefore reward chisels cognitive grooves into agents”
I’m thus very sceptical that systems optimised via reinforcement learning to be capable in a wide variety of domains/tasks converge towards maximising a simple expected utility function
I am not aware that humanity actually knows training paradigms that select for expected utility maximisers
Our most capable/economically transformative AI systems are not agents and are definitely not expected utility maximisers
Such systems might converge towards general intelligence under sufficiently strong selection pressure but do not become expected utility maximisers in the limit
The do not become agents in the limit and expected utility maximisation is a particular kind of agency
I am seriously entertaining the hypothesis that expected utility maximisation is anti-natural to selection for general intelligence
I’m not under the impression that systems optimised by stochastic gradient descent to be generally capable optimisers converge towards expected utility maximisers
The generally capable optimisers produced by evolution aren’t expected utility maximisers
I’m starting to suspect that “search like” optimisation processes for general intelligence do not in general converge towards expected utility maximisers
I.e. it may end up being the case that the only way to create a generally capable expected utility maximiser is to explicitly design one
And we do not know how to design capable optimisers for rich environments
We can’t even design an image classifier
I currently disbelieve the strong orthogonality thesis translated to practice
While it may be in theory feasible to design systems at any intelligence level with any final goal
In practice, we cannot design capable optimisers.
For intelligent systems created by “search like” optimisation, final goals are not orthogonal to cognitive ability
Sufficiently hard optimisation for most cognitive tasks would not converge towards selecting for generally capable systems
In the limit, what do systems selected for playing Go converge towards?
I posit that said limit is not “general intelligence”
The cognitive tasks/domain on which a system was optimised for performance on may instantiate an upper bound on the general capabilities of the system
You do not need much optimisation power to attain optimal performance in logical tic tac toe
Systems selected for performance at logical tic tac toe should be pretty weak narrow optimisers because that’s all that’s required for optimality in that domain
I don’t expect the systems that matter (in the par human or strongly superhuman regime) to be expected utility maximisers. I think arguments for AI x-risk that rest on expected utility maximisers are mostly disconnected from reality. I suspect that discussing the perils of expected utility maximisation in particular — as opposed to e.g. dangers from powerful (consequentialist?) optimisation processes — is somewhere between being a distraction and being actively harmful[3].
I do not think expected utility maximisation is the limit of what generally capable optimisers look like[4].
Arguments for Expected Utility Maximisation Are Unnecessary
I don’t think the case for existential risks from AI safety rest on expected utility maximisation. I kind of stopped alieving expected utility maximisers a while back (only recently have I synthesised explicit beliefs that reject it), but I still plan on working on AI existential safety, because I don’t see the core threat as resulting from expected utility maximisation.
The reasons I consider AI an existential threat mostly rely on:
Instrumental convergence for consequentialist/goal directed systems
A system doesn’t need to be a utility maximiser for a simple utility function to be goal directed (again, see humans)
Selection pressures for power seeking systems
Reasons
More economically productive/useful
Some humans are power seeking
Power seeking systems promote themselves/have better reproductive fitness
Human disempowerment is the immediate existential catastrophe scenario I foresee from power seeking
Bad game theoretic equilibria
This could lead towards dystopian scenarios in multipolar outcomes
Humans getting outcompeted by AI systems
Could slowly lead to an extinction
I do not actually expect extinction near term, but it’s not the only “existential catastrophe”:
Human disempowerment
Various forms of dystopia
I optimised for writing this quickly. So my language may be stronger/more confident that I actually feel. I may not have spent as much time accurately communicating my uncertainty as may have been warranted.
Correct me if I’m mistaken, but I’m under the impression that RL is the main training paradigm we have that selects for agents.
I don’t necessarily expect that our most capable systems would be trained via reinforcement learning, but I think our most agentic systems would be.
There may be significant opportunity cost via diverting attention from other more plausible pathways to doom.
In general, I think exposing people to bad arguments for a position is a poor persuasive strategy as people who dismiss said bad arguments may (rationally) update downwards on the credibility of the position.
I don’t necessarily think agents are that limit either. But as “Why Subagents?” shows, expected utility maximisers aren’t the limit of idealised agency.