But can you produce a finite upper bound on our lightcone that you’re 100% confident nothing can pass? (It doesn’t have to be tight.)
I think Vasco already made this point elsewhere, but I don’t see why you need certainty about any specific line to have finite expectation. If for the counterfactual payoff x, you think (perhaps after a certain point) xP(x) approaches 0 as x tends to infinity, it seems like you get finite expectation without ever having absolute confidence in any boundary (this applies to life expectancy, too).
Section II from Carlsmith, 2021 is one of the best arguments for acausal influence I’m aware of, in case you’re interested in something more convincing. (FWIW, I also thought acausal influence was crazy for a long time, and I didn’t find Newcomb’s problem to be a compelling reason to reject causal decision theory.)
Thanks! I had a look, and it still doesn’t persuade me, for much the reasons Newcomb’s problem didn’t. In roughly ascending importance
Maybe this just a technicality, but the claim ‘you are exposed to exactly identical inputs’ seems impossible to realise with perfect precision. The simulator itself must differ in the two cases. So in the same way that outputs of two instances of a software program being run, even on the same computer in the same environment can theoretically differ for various reasons (looking at a high enough zoom level they will differ), the two simulations can’t be guaranteed identical (Carlsmith even admits this with ‘absent some kind of computer malfunction’, but just glosses over it). On the one hand, this might be too fine a distinction to matter in practice; on the other, if I’m supposed to believe a wildly counterintuitive proposition instead of a commonsense one that seems to work fine in the real world, based on supposed logical necessity that it turns out isn’t logically necessary, I’m going to be very sceptical of the proposition even if I can’t find a stronger reason to reject it.
The thought experiment gives no reason why the AI system should actually believe it’s in the scenario described, and that seems like a crucial element in its decision process. If in the real world, someone put me in a room with a chalkboard and told me this is what was happening, no matter what evidence they showed, I would have some element of doubt, both of their ability (cf point 1) but more importantly their motivations. If I discovered that the world was so bizarre as in this scenario, it would be at best a coinflip for me that I should take them at face value.
It seems contradictory to frame decision theory as applying to ‘a deterministic AI system’ whose clones ‘will make the same choice, as a matter of logical necessity’. There’s a whole free will debate lurking underneath any decision theoretic discussion involving recognisable agents that I don’t particularly want to get into—but if you’re taking away all agency from the ‘agent’, it’s hard to see what it means to advocate it adopting a particular decision theory. At that point the AI might as well be a rock, and I don’t feel like anyone is concerned about which decision theory rocks ‘should’ adopt.
This follows from the theorems I cited, but I didn’t include proofs of the theorems here. The proofs are technical and tricky,[1] and I didn’t want to make my post much longer or spend so much more time on it. Explaining each proof in an intuitive way could probably be a post on its own.
I would be less interested to see a reconstruction of a proof of the theorems and more interested to see them stated formally and a proof of the claim that it follows from them.
On Carlsmith’s example, we can just make it a logical necessity by assuming more. And, as you acknowledge the possibility, some distinctions can be too fine. Maybe you’re only 5% sure your copy exists at all and the conditions are right for you to get $1 million from your copy sending it.
5%*$1 million = $50,000 > $1,000, so you still make more in expectation from sending a million dollars. You break even in expected money if your decision to send $1 million increases your copy’s probability of sending $1 million by 1⁄1,000.
I do find it confusing to think about decision-making under determinism, but I think 3 proves too much. I don’t think quantum indeterminacy or randomness saves free will or agency if it weren’t already saved, and we don’t seem to have any other options, assuming physicalism and our current understanding of physics.
I think Vasco already made this point elsewhere, but I don’t see why you need certainty about any specific line to have finite expectation. If for the counterfactual payoff x, you think (perhaps after a certain point) xP(x) approaches 0 as x tends to infinity, it seems like you get finite expectation without ever having absolute confidence in any boundary (this applies to life expectancy, too).
Ya, I agree you don’t need certainty about the bound, but now you need certainty about the distribution not being heavy-tailed at all. Suppose your best guess is that it looks like some distribution X, with finite expected value. Now, I suggest that it might actually be Y, which is heavy-tailed (has infinite expected value). If you assign any nonzero probability to that being right, e.g. switch to pY+(1−p)X for some p>0, then your new distribution is heavy-tailed, too. In general, if you think there’s some chance you’d come to believe it’s heavy-tailed, then you should believe now that it’s heavy-tailed, because a probabilistic mixture with a heavy-tailed distribution is heavy-tailed. Or, if you think there’s some chance you’d come to believe there’s some chance it’s heavy-tailed, then you should believe now that it’s heavy-tailed.
(Vasco’s claim was stronger: the difference is exactly 0 past some point.)
I would be less interested to see a reconstruction of a proof of the theorems and more interested to see them stated formally and a proof of the claim that it follows from them.
Hmm, I might be misunderstanding.
I already have formal statements of the theorems in the post:
Stochastic Dominance, Anteriority and Impartiality are jointly inconsistent.
Stochastic Dominance, Separability and Impartiality are jointly inconsistent.
All of those terms are defined in the section Anti-utilitarian theorems. I guess I defined Impartiality a bit informally and might have hidden some background assumptions (preorder, so reflexivity + transitivity, and the set of prospects is every probability distribution over outcomes in the set of outcomes), but the rest were formally defined.
Then, from 1, assuming Stochastic Dominance and Impartiality, Anteriority must be false. From 2, assuming Stochastic Dominance and Impartiality, Separability must be false. Therefore assuming Stochastic Dominance and Impartiality, Anteriority and Separability must both be false.
I think Vasco already made this point elsewhere, but I don’t see why you need certainty about any specific line to have finite expectation. If for the counterfactual payoff x, you think (perhaps after a certain point) xP(x) approaches 0 as x tends to infinity, it seems like you get finite expectation without ever having absolute confidence in any boundary (this applies to life expectancy, too).
Thanks! I had a look, and it still doesn’t persuade me, for much the reasons Newcomb’s problem didn’t. In roughly ascending importance
Maybe this just a technicality, but the claim ‘you are exposed to exactly identical inputs’ seems impossible to realise with perfect precision. The simulator itself must differ in the two cases. So in the same way that outputs of two instances of a software program being run, even on the same computer in the same environment can theoretically differ for various reasons (looking at a high enough zoom level they will differ), the two simulations can’t be guaranteed identical (Carlsmith even admits this with ‘absent some kind of computer malfunction’, but just glosses over it). On the one hand, this might be too fine a distinction to matter in practice; on the other, if I’m supposed to believe a wildly counterintuitive proposition instead of a commonsense one that seems to work fine in the real world, based on supposed logical necessity that it turns out isn’t logically necessary, I’m going to be very sceptical of the proposition even if I can’t find a stronger reason to reject it.
The thought experiment gives no reason why the AI system should actually believe it’s in the scenario described, and that seems like a crucial element in its decision process. If in the real world, someone put me in a room with a chalkboard and told me this is what was happening, no matter what evidence they showed, I would have some element of doubt, both of their ability (cf point 1) but more importantly their motivations. If I discovered that the world was so bizarre as in this scenario, it would be at best a coinflip for me that I should take them at face value.
It seems contradictory to frame decision theory as applying to ‘a deterministic AI system’ whose clones ‘will make the same choice, as a matter of logical necessity’. There’s a whole free will debate lurking underneath any decision theoretic discussion involving recognisable agents that I don’t particularly want to get into—but if you’re taking away all agency from the ‘agent’, it’s hard to see what it means to advocate it adopting a particular decision theory. At that point the AI might as well be a rock, and I don’t feel like anyone is concerned about which decision theory rocks ‘should’ adopt.
I would be less interested to see a reconstruction of a proof of the theorems and more interested to see them stated formally and a proof of the claim that it follows from them.
On Carlsmith’s example, we can just make it a logical necessity by assuming more. And, as you acknowledge the possibility, some distinctions can be too fine. Maybe you’re only 5% sure your copy exists at all and the conditions are right for you to get $1 million from your copy sending it.
5%*$1 million = $50,000 > $1,000, so you still make more in expectation from sending a million dollars. You break even in expected money if your decision to send $1 million increases your copy’s probability of sending $1 million by 1⁄1,000.
I do find it confusing to think about decision-making under determinism, but I think 3 proves too much. I don’t think quantum indeterminacy or randomness saves free will or agency if it weren’t already saved, and we don’t seem to have any other options, assuming physicalism and our current understanding of physics.
Ya, I agree you don’t need certainty about the bound, but now you need certainty about the distribution not being heavy-tailed at all. Suppose your best guess is that it looks like some distribution X, with finite expected value. Now, I suggest that it might actually be Y, which is heavy-tailed (has infinite expected value). If you assign any nonzero probability to that being right, e.g. switch to pY+(1−p)X for some p>0, then your new distribution is heavy-tailed, too. In general, if you think there’s some chance you’d come to believe it’s heavy-tailed, then you should believe now that it’s heavy-tailed, because a probabilistic mixture with a heavy-tailed distribution is heavy-tailed. Or, if you think there’s some chance you’d come to believe there’s some chance it’s heavy-tailed, then you should believe now that it’s heavy-tailed.
(Vasco’s claim was stronger: the difference is exactly 0 past some point.)
Hmm, I might be misunderstanding.
I already have formal statements of the theorems in the post:
Stochastic Dominance, Anteriority and Impartiality are jointly inconsistent.
Stochastic Dominance, Separability and Impartiality are jointly inconsistent.
All of those terms are defined in the section Anti-utilitarian theorems. I guess I defined Impartiality a bit informally and might have hidden some background assumptions (preorder, so reflexivity + transitivity, and the set of prospects is every probability distribution over outcomes in the set of outcomes), but the rest were formally defined.
Then, from 1, assuming Stochastic Dominance and Impartiality, Anteriority must be false. From 2, assuming Stochastic Dominance and Impartiality, Separability must be false. Therefore assuming Stochastic Dominance and Impartiality, Anteriority and Separability must both be false.