I’m a Postdoctoral Research Fellow at Oxford University’s Global Priorities Institute.
Previously, I was a Philosophy Fellow at the Center for AI Safety.
So far, my work has mostly been about the moral importance of future generations. Going forward, it will mostly be about AI.
You can email me at elliott.thornley@philosophy.ox.ac.uk
Great post!
For example, maybe, according to you, you’re an “all men are created equal” type. That is, you treat all men equally. Maybe you even write a fancy document about this, and this document gets involved in the founding of a country, or something.
There’s a thing philosophy can do, here, which is to notice that you still own slaves. Including: male slaves. And it can do that whole “implication” thing, about how, Socrates is a man, you treat all men equally, therefore you treat Socrates equally, except oh wait, you don’t, he’s your slave.
Charles Mills points out another tension in the Declaration of Independence: about 900 words after ‘all men are created equal’, there’s a sentence about ‘merciless Indian Savages’.
I think all of these objections would be excellent if I were arguing against this claim:
Agents are rationally required to satisfy the VNM axioms.
But I’m arguing against this claim:
Sufficiently-advanced artificial agents will satisfy the VNM axioms.
And given that, I think your objections miss the mark.
On your first point, I’m prepared to grant that agents have no reason to rule out option A- at node 2. All I need to claim is that advanced artificial agents might rule out option A- at node 2. And I think my argument makes that claim plausible.
On your second point, Decision-Tree Separability doesn’t rule out my proposal. What would rule it out is Decision-Tree Separability*:
sufficiently-advanced artificial agents’ dispositions to choose options at a choice node will not depend on other parts of the decision tree than those that can be reached from that node.
And whatever the merits of Decision-Tree Separability, Decision-Tree Separability* seems to me not very plausible.
On your third point, (whether or not most humans are irrational) most humans are non-consequentialists. So even if it is no more plausible that artificial agents will be non-consequentialists than humans, it can be plausible that artificial agents will be non-consequentialists. And it is relevant that advanced artificial agents could be better at remembering their past decisions than humans. That would make them better able to act in accordance with the policy that I suggest.
There’s a complication here related to a point that Rohin makes : if we can only see an agent’s decisions and we know nothing about its preferences, all behavior can be rationalized as EU maximization.
But suppose we set that complication aside. Suppose we know this about an agent’s preferences:
There is some option A such that the agent strictly prefers A+$1
Then we can observe violations of Completeness. Suppose that we first offer our agent a choice between A and some other option B, and that the agent chooses A. Then we give the agent the chance to trade in A for B, and the agent takes the trade. That indicates that the agent does not strictly prefer A to B and does not strictly prefer B to A. Two possibilities remain: either the agent is indifferent between A and B, or the agent has a preferential gap between A and B.
Now we offer our agent another choice: stick with B, or trade it in for A+$1. If the agent is indifferent between A and B, they will strictly prefer A+$1 to B (because indifference is sensitive to all sweetenings and sourings), and so we will observe the agent taking the trade. If we observe that the agent doesn’t take the trade, then they must have a preferential gap between A and B, and so their preferences must be incomplete.
Ah I see! Yep, agree with that.
Nice point. The rough answer is ‘Yes, but only once the agent has turned down a sufficiently wide array of options.’ Depending on the details, that might never happen or only happen after a very long time.
I’ve had a quick think about the more precise answer, and I think it is:
The agent’s preferences will be functionally complete once and only once it is the case that, for all pairs of options between which the agent has a preferential gap, the agent has turned down an option that is strictly preferred to one of the options in the pair.
I didn’t mean to suggest it was new! I remember that part of your book.
Your second point seems to me to get the dialectic wrong. We can read coherence arguments as saying:
Sufficiently-advanced artificial agents won’t pursue dominated strategies, so they’ll have complete preferences.
I’m pointing out that that inference is poor. Advanced artificial agents might instead avoid dominated strategies by acting in accordance with the policy that I suggest.
I’m still thinking about your last point. Two quick thoughts:
It seems like most humans aren’t consequentialists.
Advanced artificial agents could have better memories of their past decisions than humans.
it seems like we agree on the object-level facts
I think that’s right.
Often a lot of the important “assumptions” in a theorem are baked into things like the type signature of a particular variable or the definitions of some key terms; in my toy theorem above I give two examples (completeness and lack of time-dependence). You are going to lose some information about what the theorem says when you convert it from math to English; an author’s job is to communicate the “important” parts of the theorem (e.g. the conclusion, any antecedents that the reader may not agree with, implications of the type signature that limit the applicability of the conclusion), which will depend on the audience.
Yep, I agree with all of this.
Converting math into English is a tricky business.
Often, but not in this case. If authors understood the above points and meant to refer to the Complete Class Theorem, they need only have said:
If an agent has complete, transitive preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.
(And they probably wouldn’t have mentioned Cox, Savage, etc.)
Yup, good point, I think it doesn’t change the conclusion.
I think it does. If the money-pump for transitivity needs Completeness, and Completeness is doubtful, then the money-pump for transitivity is doubtful too.
Thanks for the comment! In this context, where we’re arguing about whether sufficiently-advanced artificial agents will satisfy the VNM axioms, I only have to give up Decision-Tree Separability*:
Sufficiently-advanced artificial agents’ dispositions to choose options at a choice node will not depend on other parts of the decision tree than those that can be reached from that node.
And Decision-Tree Separability* isn’t particularly plausible. It’s false if any sufficiently-advanced artificial agent acts in accordance with the following policy: ‘if I previously turned down some option X, I will not choose any option that I strictly disprefer to X.’ And it’s easy to see why agents might act in accordance with that policy: it makes them immune to money-pumps for Completeness.
Also, it seems as if one of the major downsides of resolute choice is that agents sometimes have to act against their preferences. But, as I argue in the post, artificial agents with incomplete preferences who act in accordance with the policy above will never have to act against their preferences.
So, you would agree that the following is an English description of a theorem:
If an agent has complete, transitive preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.
Yep, I agree with that.
I feel pretty fine with justifying the transitive part via theorems basically like the one I gave above.
Note that your money-pump justifies acyclicity (The agent does not strictly prefer A to B, B to C, and C to A) rather than the version of transitivity necessary for the VNM and Complete Class theorems (If the agent weakly prefers A to B, and B to C, then the agent weakly prefers A to C). Gustafsson thinks you need Completeness to get a money-pump for this version of transitivity working (see footnote 8 on page 3), and I’m inclined to agree.
when you have intransitive preferences it’s not even clear what a “dominated strategy” would be.
A dominated strategy would be a strategy which leads you to choose an option that is worse in some respect than another available option and not better than that other available option in any respect. For example, making all the trades and getting A- in the decision-situation below would be a dominated strategy, since you could have made no trades and got A:
So I still think that it is basically incorrect to say:
And yet the error seems to have gone uncorrected for more than a decade.
The error is claiming that
There exist theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
I haven’t seen anyone point out that that claim is false.
That said, one could reason as follows:
Rohin, John, and others have argued that agents with incomplete preferences can act in accordance with policies that make them immune to pursuing dominated strategies.
Agents with incomplete preferences cannot be represented as maximizing expected utility.
So, if Rohin’s, John’s, and others’ arguments are sound, there cannot exist theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
Then one would have corrected the error. But since the availability of this kind of reasoning is easily missed, it seems worth correcting the error directly.
Theorems are typically of the form “Suppose X, then Y”; what is X if not an assumption?
X is an antecedent.
Consider an example. Imagine I claim:
Suppose James is a bachelor. Then James is unmarried.
In making this claim, I am not assuming that James is a bachelor. My claim is true whether or not James is a bachelor.
I might temporarily assume that James is a bachelor, and then use that assumption to prove that James is unmarried. But when I conclude ‘Suppose James is a bachelor. Then James is unmarried’, I discharge that initial assumption. My conclusion no longer depends on it. Any conclusion which can be proved with no undischarged assumptions is a theorem.
Theorem. Suppose there is a set of possible worlds , and an agent that given a current world and a proposed new world specifies how much money it would pay to switch to from . Suppose further than cannot be money pumped, that is, there is no sequence of worlds such that (1) and (2) . Then must be transitive in the following sense: for any , if and , then .
Proof. Suppose is not transitive, so there exists some where , , and . But then then the sequence is a money pump, leading to a contradiction.
I agree that this is a theorem. But it’s not a ‘coherence theorem’ (at least not in the way that I’ve used the term in this post, and not in the way that previous authors seem to have used the term [see the Appendix]): it doesn’t state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. It states only that, unless an agent’s preferences are acyclic, that agent is liable to pursue strategies that are dominated by some other available strategy.
You can call it a ‘coherence theorem’. Then it would be true that coherence theorems exist. But the important point remains: Premise 1 of the coherence argument is false. There are no theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. VNM doesn’t say that, Savage doesn’t say that, Bolker-Jeffrey doesn’t say that, Dutch Books don’t say that, Cox doesn’t say that, Complete Class doesn’t say that.
For the second argument (that the completeness axiom is suspicious): I think this is basically expressing the same sort of objection that I express here, particularly the section “There are no coherence arguments that say you must have preferences”. I didn’t tie it to the Completeness axiom because I think it’s a mistake to get bogged down in the details of the specific assumptions present in theorems when you can make the same point in English, but it is the same conceptual point, as far as I can tell.
I agree. I think the points that you make in that post are good.
For what it’s worth my position here is “you can’t argue for AI risk solely via coherence theorems; you also have to argue for why the AI will be goal-directed in the first place, but there are plausible arguments for that conclusion (which are not based on coherence arguments)”.
I agree with this too.
Two points, made in order of importance:
(1) How we define the term ‘coherence theorems’ doesn’t matter. What matters is that Premise 1 (striking out the word ‘coherence’, if you like) is false.
(2) The way I define the term ‘coherence theorems’ seems standard.
Now making point (1) in more detail:
Reserve the term ‘coherence theorems’ for whatever you like. Premise 1 is false: there are no theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. The VNM Theorem doesn’t say that, nor does Savage’s Theorem, nor does Bolker-Jeffrey, nor do Dutch Books, nor does Cox’s Theorem, nor does the Complete Class Theorem. That is the error in coherence arguments. Premise 1 is false.
Now for point (2):
I take the Appendix to make plausible enough that my use of the term ‘coherence theorems’ is standard, at least in online discussions. Here are some quotations.
1.
Now, by the general idea behind coherence theorems, since we can’t view this behavior as corresponding to expected utilities, we ought to be able to show that it corresponds to a dominated strategy somehow
2.
Roughly, the general claim these theorems make is that any system either (a) acts like an expected utility maximizer under some probabilistic model, or (b) throws away resources in a pareto-suboptimal manner.
3.
Summary: Violations of coherence constraints in probability theory and decision theory correspond to qualitatively destructive or dominated behaviors.
Again, we see a manifestation of a powerful family of theorems showing that agents which cannot be seen as corresponding to any coherent probabilities and consistent utility function will exhibit qualitatively destructive behavior
4.
One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy.
5.
‘Coherence arguments’ mean that if you don’t maximize ‘expected utility’ (EU)—that is, if you don’t make every choice in accordance with what gets the highest average score, given consistent preferability scores that you assign to all outcomes—then you will make strictly worse choices by your own lights than if you followed some alternate EU-maximizing strategy (at least in some situations, though they may not arise). For instance, you’ll be vulnerable to ‘money-pumping’—being predictably parted from your money for nothing.
6.
The overall message here is that there is a set of qualitative behaviors and as long you do not engage in these qualitatively destructive behaviors, you will be behaving as if you have a utility function.
7.
I think that to contain the concept of Utility as it exists in me, you would have to do homework exercises I don’t know how to prescribe. Maybe one set of homework exercises like that would be showing you an agent, including a human, making some set of choices that allegedly couldn’t obey expected utility, and having you figure out how to pump money from that agent (or present it with money that it would pass up).
8.
The view that utility maximizers are inevitable is supported by a number of coherence theories developed early on in game theory which show that any agent without a consistent utility function is exploitable in some sense.
Maybe the term ‘coherence theorems’ gets used differently elsewhere. That is okay. See point (1).
Using ‘coherence theorems’ with a meaning that is as standard as any, and explaining that meaning within two sentences, seems fine to me.
I would have hoped you reached the second sentence before skimming! I define what I mean (and what I take previous authors to mean) by ‘coherence theorems’ there.
I cite John in the post!
I’m following previous authors in defining ‘coherence theorems’ as
theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
On that definition, there are no coherence theorems. VNM is not a coherence theorem, nor is Savage’s Theorem, nor is Bolker-Jeffrey, nor are Dutch Book Arguments, nor is Cox’s Theorem, nor is the Complete Class Theorem.
there are theorems that are relevant to the question of agent coherence
I’d have no problem with authors making that claim.
I would urge the author to create an actual concrete situation that doesn’t seem very dumb in which a highly intelligence, powerful and economically useful system has non-complete preferences
Working on it.
Sounds good!
I haven’t read this post yet, but it sounds like you might be interested in this paper on existential risks from a Thomist Christian perspective if you haven’t seen it already.
Thanks for the comment!
What we mean with this sentence is that there are clear moral objections against governments pursuing [perhaps we should have said ‘instituting’] democratically unacceptable policies. We don’t mean to suggest that there’s anything wrong with citizens advocating for policies that are currently democratically unacceptable with the aim of making them democratically acceptable.