Sure there are costs to giving the wallet away, but if you’re an EV-maximiser you probably still should.
You could tell the mugger that you don’t want to give him the wallet because you’d feel like an idiot if you did, and that you’ve got an amazing alternative way to do good with the money. He could then just promise you that he’ll create way more utility than originally promised until it makes up for any such costs or missed opportunities (in EV terms).
First, just to clarify, are you still pushing the realistic-mugger objection as a demonstration of your argument? That (combined with the part about AI safety, which I address later) seemed like the bulk of your original argument.
Second, you seem to bring up a separate idea:
He could then just promise you that he’ll create way more utility than originally promised until it makes up for any such costs or missed opportunities (in EV terms).
Okay, so I immediately further discount the likelihood of that being true (e.g., “if that’s true, why didn’t you just say so to begin with...”) which offsets a large portion of the claimed increase, and I just repeat the original reasoning process: “insofar as there is any credence to assign to what this person is saying, there are better reasons to think that sacrificing my resources, reputation, psychological wellbeing, time, etc. will have worse expected value: if it’s actually possible to break the laws of physics and produce a googolplex utilons, it’s unlikely that giving over the wallet is the way to achieve that… etc.”
Ultimately, the intuition/answer is fairly obvious for me, and intuition is a valid input in a decision framework. (If someone wants to object here and say that decision theories have to eschew such forms of intuition and must strictly rely on fully-fleshed out arguments and/or other formal processes, that just seems to be universally impossible regardless of whether the situation is about Pascal’s mugging, a simple prisoner’s dilemma, or anything else.)
Also:
[…] if people work towards increasing a particular variable they are more likely to increase it than decrease it (all other things equal).
Why can’t I just copy that reasoning and think “by choosing to not hand over my wallet to Trass, the Tin-and-Brass God of Mugging, I am working towards reducing the likelihood of existential catastrophe, which means I am more likely to be reducing it. QED”
(To be clear, I’m not disputing that it’s reasonable to think that working on AI safety generally has positive expected value in terms of increasing AI safety vs. decreasing AI safety, but that’s totally detached from Pascalian mugging.)
First, just to clarify, are you still pushing the realistic-mugger objection as a demonstration of your argument?
I used the realistic mugger scenario to illustrate a point I think also applies to the original Pascal’s mugging. I’m now discussing the original Pascal’s mugging.
In the original Pascal’s mugging the key idea is that the mugger can promise more and more utility to overcome any barriers you might have to handing over the wallet. Yes the probability you think the mugger is telling the truth might decrease too, but at some point the probability will stop decreasing or at least won’t decrease as fast as the mugger can promise more. Remember the mugger can promise ANY amount of utility and numbers can get unimaginely big. Of course he can easily promise enough to overcome any reasonable decrease in probability.
Why can’t I just copy that reasoning and think “by choosing to not hand over my wallet to Trass, the Tin-and-Brass God of Mugging, I am working towards reducing the likelihood of existential catastrophe, which means I am more likely to be reducing it. QED”
Not sure what you’re getting at. I’m saying that, on average, people trying to achieve a goal will mean they are more likely to achieve it than if they weren’t trying to achieve the goal. This was simply to counter Sanjay’s point about AI and is indeed detached to the discussion of Pascal’s mugging.
In the original Pascal’s mugging the key idea is that the mugger can promise more and more utility to overcome any barriers you might have to handing over the wallet. Yes the probability you think the mugger is telling the truth might decrease too, but at some point the probability will stop decreasing or at least won’t decrease as fast as the mugger can promise more.
You’re only responding to the first (and admittedly less important) half of my response. The key issue is that you can just use that same logic against the new claims and still defeat the : for any given increase in the mugger’s claims, you can come up with an equally large (but still more probable) increase in opportunity cost or other objections. If the mugger says “no wait, actually I’ll give you ten googolplex of utilons instead of just one googolplex, and I’ll bet you don’t think that less than 10% as likely,” I could just say “Hmm, yeah maybe it’s only 20% as likely so that would double the expected value, except actually I now think the opportunity costs and other downsides are two to three times greater than what I thought before, so actually the costs still outweigh the gains.”
If we want to just end this back and forth, we’ll just jump straight to claims of infinity: a 20% chance of achieving an infinitely good outcome has higher expected value (or, is a better option) than a 10% chance of achieving an infinitely good outcome, ceteris paribus.
Why can’t I just copy that reasoning and think “by choosing to not hand over my wallet to Trass, the Tin-and-Brass God of Mugging, I am working towards reducing the likelihood of existential catastrophe, which means I am more likely to be reducing it. QED”
Not sure what you’re getting at. I’m saying that, on average, people trying to achieve a goal will mean they are more likely to achieve it than if they weren’t trying to achieve the goal. This was simply to counter Sanjay’s point about AI and is indeed detached to the discussion of Pascal’s mugging.
I don’t think I originally saw that part of Sanjay’s comment/post, and I agree that there is probably something wrong with his arguments there. I partially thought you were making that point as an additional argument in favor of giving your wallet to the mugger, since you would be doing so in order to try to reduce x-risk.
“Hmm, yeah maybe it’s only 20% as likely so that would double the expected value, except actually I now think the opportunity costs and other downsides are two to three times greater than what I thought before, so actually the costs still outweigh the gains.”
But do you actually have a good reason to believe the costs and opportunity costs will rapidly increase to unimaginably high levels? Why would this be the case?
The costs and opportunity costs shouldn’t vary much (if at all) all depending on the amount the mugger is promising. They are quite independent.
But do you actually have a good reason to believe the costs and opportunity costs will rapidly increase to unimaginably high levels? Why would this be the case?
A few responses:
1. One which I already noted:
“insofar as there is any credence to assign to what this person is saying, there are better reasons to think that sacrificing my resources, reputation, psychological wellbeing, time, etc. will have worse expected value: if it’s actually possible to break the laws of physics and produce a googolplex utilons, it’s unlikely that giving over the wallet is the [best] way to achieve that [in expectation]… etc.”
In other words, there are reasons to believe:
1a. The opportunity costs[1] are correlated with outcome magnitudes that this person is claiming;
1b. The risk-costs[2] are correlated (as above...)
2. No, I don’t have that many “good” reasons, but that’s been a key point I’ve tried emphasizing: you don’t need good, explicit reasoning to fight completely absurd reasoning, you just need better/less-bad reasoning, which can still be fairly dubious.
Such as “if I give my wallet over now, I won’t have money to give to a REAL Trass or Smiggledorfian fart goblin later on,” or “if I waste my time and money here, I have less time to ponder the design of the universe and find a flaw in the matrix that allows me to escape and live forever in paradise (or to figure out which supernatural/religious belief is valid and act on that to get infinite utilons).”
Such as “if I give my wallet over now, I won’t have money to give to a REAL Trass or Smiggledorfian fart goblin later on,” or “if I waste my time and money here, I have less time to ponder the design of the universe and find a flaw in the matrix that allows me to escape and live forever in paradise (or to figure out which supernatural/religious belief is valid and act on that to get infinite utilons).
Again I just think the mugger being able to say any number means they can overcome any alternative (in EV terms).
For example, you can calculate the EV of the possibility of meeting a real wizard in a year and them generating an insane amount of utility. Your EV calculation will spit out some sort of number right? Well the mugger in front of you can beat whatever number that is by saying whatever number they need to to beat it.
Or maybe the EV is undefined because you start talking about infinite utility (as you allude to). In this case EV reasoning pretty much breaks down.
Which brings me to the point that rejecting EV reasoning in the first place can get you out of handing over the wallet. Maybe there’s a problem with EV reasoning, for example when dealing with infinite utilities, or a problem with EV reasoning when probabilities get really small.
Indeed I’m not sure we should actually hand over the wallet because I’m open to the possibility EV reasoning is flawed, at least in this case. See the St. Petersburg Paradox and the Pasadena Game for potential issues with EV reasoning. I still think EV reasoning is pretty useful in more regular circumstances though.
Such as, “Actually, this person is testing me, and will do the exact opposite of what they claim if and only if I do what they are telling me to do.”
My response to this is the same as the one I gave originally to Sanjay. I don’t think this is a compelling argument.
EDIT: Also see Derek Shiller’s comment which I endorse.
For example, you can calculate the EV of the possibility of meeting a real wizard in a year and them generating an insane amount of utility. Your EV calculation will spit out some sort of number right? Well the mugger in front of you can beat whatever number that is by saying whatever number they need to to beat it.
Hence why I wrote out “infinite utilons” with such emphasis and in a previous comment also wrote “If we want to just end this back and forth, we’ll just jump straight to claims of infinity [...]”. But you do continue:
Or maybe the EV is undefined because you start talking about infinite utility (as you allude to). In this case EV reasoning pretty much breaks down.
I disagree (at least insofar as I conceptualize EV and if you are just saying “you can’t compare fractional infinities”), as I already asserted:
a 20% chance of achieving an infinitely good outcome has higher expected value (or, is a better option) than a 10% chance of achieving an infinitely good outcome, ceteris paribus.
Summarizing your thoughts, you offer the following conclusion:
Which brings me to the point that rejecting EV reasoning in the first place can get you out of handing over the wallet. Maybe there’s a problem with EV reasoning, for example when dealing with infinite utilities, or a problem with EV reasoning when probabilities get really small.
Indeed I’m not sure we should actually hand over the wallet because I’m open to the possibility EV reasoning is flawed, at least in this case. See the St. Petersburg Paradox and the Pasadena Game for potential issues with EV reasoning.
I’m also somewhat open to the idea that EV reasoning is flawed, especially my interpretation of the concept. However:
I am very skeptical of rejecting EV reasoning without any kind of replacement, which I don’t think I’ve seen you offer.
This conversation has not moved me closer to believing that I should reject EV reasoning due to Pascalian mugging objections, of which I don’t see any standing/compelling examples. In fact, it is through EV reasoning that I find these objections uncompelling.
The St. Petersburg Paradox and Pasadena Game are interesting objections with which I’m decently familiar (especially the former); I’ve written one or two somewhat long comments about the former some time ago. However, these are also not persuasive, for some of the same reasons I’m articulating here (e.g., it might be equally likely that shoving your $100 bill in some random tree hollow is going to reveal a Korok that breaks the Matrix and gives you infinite utilons than that you are going to flip heads for an infinitely long time).
You know, I actually think I’ve gone completely down the wrong lines with my argument and if I could erase my previous comments I would.
The point is that Pascal’s mugging is a thought experiment to illustrate a point, and I can alter it as required. So I can remove the possibility of amazing alternatives for the money in the wallet if I have to.
E.g. let’s say you have a large amount of money in your wallet and, for whatever reason, you can only use that money to buy yourself stuff you really like. E.g. let’s say it’s a currency that you know can only be taken by a store that sells fancy cars, clothes and houses. Assume you really do know you can’t use the money for anything else (it’s a thought experiment so I can make this restriction).
Now imagine that you run into a Pascal’s mugging. Your “opportunity costs are correlated with outcome magnitudes that the person is claiming” argument no longer applies in this thought experiment. Do you now hand over the wallet and miss out on all of that amazing stuff you could have bought from the store?
I know this all sounds weird, but thought experiments are weird...
Well, with those extra assumptions I would no longer consider it a Pascalian Mugging, I would probably just consider it a prone-to-mislead thought experiment.
Would I take a 1*10−10 chance of getting 1015 utilons over the option of a 100% chance of 1 utilon? Well, if we assume that:
Utilons are the only thing that matter for morality/wellbeing (ignore all moral uncertainty),
I don’t have any additional time to reduce my uncertainty (or I have maximal certainty of the 1*10−10 chance estimate),
The 1*10−10 chance estimate already takes into account all of the outside views, intuitions, and any other arguments that says “this is extremely improbable”,
There’s no such thing as opportunity cost,
Diminishing marginal returns don’t apply,
Psychological and social factors are non-existent,
Your answer to this hypothetical question will not be used to judge your intelligence/character in the real world,
(And how about for good measure, “Assume away any other argument which could dispute the intended conclusion”),
Then the answer is probably yes?
I have a hard-to-articulate set of thoughts on what makes some thought experiments valuable vs. misleading, with one of them being something along the lines of “It’s unhelpful to generate an unrealistic thought experiment and then use the answer/response produced by a framework/system as an argument against using that framework/system in the real world if the argument is (at least implicitly) ‘Wow, look at what an unreasonable answer this could produce for real world problems,’ especially given how people may be prone to misinterpret such evidence (especially due to framing effects) for frameworks/systems that they are unfamiliar with or are already biased against.”
But I don’t have time to get into that can of worms, unfortunately.
Pascal’s mugging is only there to uncover ifthere might be a potential problem with using EV maximisation when probabilities get very small. It’s a thought experiment in decision theory. For that reason I actually think my altered thought experiment is useful, as I think you were introducing complications that distract from this central message of the thought experiment. Pascal’s mugging doesn’t in itself say anything about the relevance of these issues to real life. It may be all a moot point at the end of the day.
It sounds to me as if you don’t see any issues with EV maximisation when probabilities get very small. So in my altered thought experiment you would indeed give away your wallet to some random dude claiming to be a wizard, thereby giving up all those awesome things from the store. It’s worth at least noting that many people wouldn’t do the same, and who is right or wrong is where the interesting conundrum lies.
It’s worth at least noting that many people wouldn’t do the same
I don’t think many people are capable of actually internalizing all of the relevant assumptions that in real life would be totally unreasonable, nor do most people have a really good sense of why they have certain intuitions in the first place. So, it’s not particularly surprising/interesting that people would have very different views on this question.
Sure there are costs to giving the wallet away, but if you’re an EV-maximiser you probably still should.
You could tell the mugger that you don’t want to give him the wallet because you’d feel like an idiot if you did, and that you’ve got an amazing alternative way to do good with the money. He could then just promise you that he’ll create way more utility than originally promised until it makes up for any such costs or missed opportunities (in EV terms).
First, just to clarify, are you still pushing the realistic-mugger objection as a demonstration of your argument? That (combined with the part about AI safety, which I address later) seemed like the bulk of your original argument.
Second, you seem to bring up a separate idea:
Okay, so I immediately further discount the likelihood of that being true (e.g., “if that’s true, why didn’t you just say so to begin with...”) which offsets a large portion of the claimed increase, and I just repeat the original reasoning process: “insofar as there is any credence to assign to what this person is saying, there are better reasons to think that sacrificing my resources, reputation, psychological wellbeing, time, etc. will have worse expected value: if it’s actually possible to break the laws of physics and produce a googolplex utilons, it’s unlikely that giving over the wallet is the way to achieve that… etc.”
Ultimately, the intuition/answer is fairly obvious for me, and intuition is a valid input in a decision framework. (If someone wants to object here and say that decision theories have to eschew such forms of intuition and must strictly rely on fully-fleshed out arguments and/or other formal processes, that just seems to be universally impossible regardless of whether the situation is about Pascal’s mugging, a simple prisoner’s dilemma, or anything else.)
Also:
Why can’t I just copy that reasoning and think “by choosing to not hand over my wallet to Trass, the Tin-and-Brass God of Mugging, I am working towards reducing the likelihood of existential catastrophe, which means I am more likely to be reducing it. QED”
(To be clear, I’m not disputing that it’s reasonable to think that working on AI safety generally has positive expected value in terms of increasing AI safety vs. decreasing AI safety, but that’s totally detached from Pascalian mugging.)
I used the realistic mugger scenario to illustrate a point I think also applies to the original Pascal’s mugging. I’m now discussing the original Pascal’s mugging.
In the original Pascal’s mugging the key idea is that the mugger can promise more and more utility to overcome any barriers you might have to handing over the wallet. Yes the probability you think the mugger is telling the truth might decrease too, but at some point the probability will stop decreasing or at least won’t decrease as fast as the mugger can promise more. Remember the mugger can promise ANY amount of utility and numbers can get unimaginely big. Of course he can easily promise enough to overcome any reasonable decrease in probability.
Not sure what you’re getting at. I’m saying that, on average, people trying to achieve a goal will mean they are more likely to achieve it than if they weren’t trying to achieve the goal. This was simply to counter Sanjay’s point about AI and is indeed detached to the discussion of Pascal’s mugging.
You’re only responding to the first (and admittedly less important) half of my response. The key issue is that you can just use that same logic against the new claims and still defeat the : for any given increase in the mugger’s claims, you can come up with an equally large (but still more probable) increase in opportunity cost or other objections. If the mugger says “no wait, actually I’ll give you ten googolplex of utilons instead of just one googolplex, and I’ll bet you don’t think that less than 10% as likely,” I could just say “Hmm, yeah maybe it’s only 20% as likely so that would double the expected value, except actually I now think the opportunity costs and other downsides are two to three times greater than what I thought before, so actually the costs still outweigh the gains.”
If we want to just end this back and forth, we’ll just jump straight to claims of infinity: a 20% chance of achieving an infinitely good outcome has higher expected value (or, is a better option) than a 10% chance of achieving an infinitely good outcome, ceteris paribus.
I don’t think I originally saw that part of Sanjay’s comment/post, and I agree that there is probably something wrong with his arguments there. I partially thought you were making that point as an additional argument in favor of giving your wallet to the mugger, since you would be doing so in order to try to reduce x-risk.
But do you actually have a good reason to believe the costs and opportunity costs will rapidly increase to unimaginably high levels? Why would this be the case?
The costs and opportunity costs shouldn’t vary much (if at all) all depending on the amount the mugger is promising. They are quite independent.
A few responses:
1. One which I already noted:
In other words, there are reasons to believe:
1a. The opportunity costs[1] are correlated with outcome magnitudes that this person is claiming;
1b. The risk-costs[2] are correlated (as above...)
2. No, I don’t have that many “good” reasons, but that’s been a key point I’ve tried emphasizing: you don’t need good, explicit reasoning to fight completely absurd reasoning, you just need better/less-bad reasoning, which can still be fairly dubious.
Such as “if I give my wallet over now, I won’t have money to give to a REAL Trass or Smiggledorfian fart goblin later on,” or “if I waste my time and money here, I have less time to ponder the design of the universe and find a flaw in the matrix that allows me to escape and live forever in paradise (or to figure out which supernatural/religious belief is valid and act on that to get infinite utilons).”
Such as, “Actually, this person is testing me, and will do the exact opposite of what they claim if and only if I do what they are telling me to do.”
Again I just think the mugger being able to say any number means they can overcome any alternative (in EV terms).
For example, you can calculate the EV of the possibility of meeting a real wizard in a year and them generating an insane amount of utility. Your EV calculation will spit out some sort of number right? Well the mugger in front of you can beat whatever number that is by saying whatever number they need to to beat it.
Or maybe the EV is undefined because you start talking about infinite utility (as you allude to). In this case EV reasoning pretty much breaks down.
Which brings me to the point that rejecting EV reasoning in the first place can get you out of handing over the wallet. Maybe there’s a problem with EV reasoning, for example when dealing with infinite utilities, or a problem with EV reasoning when probabilities get really small.
Indeed I’m not sure we should actually hand over the wallet because I’m open to the possibility EV reasoning is flawed, at least in this case. See the St. Petersburg Paradox and the Pasadena Game for potential issues with EV reasoning. I still think EV reasoning is pretty useful in more regular circumstances though.
My response to this is the same as the one I gave originally to Sanjay. I don’t think this is a compelling argument.
EDIT: Also see Derek Shiller’s comment which I endorse.
Hence why I wrote out “infinite utilons” with such emphasis and in a previous comment also wrote “If we want to just end this back and forth, we’ll just jump straight to claims of infinity [...]”. But you do continue:
I disagree (at least insofar as I conceptualize EV and if you are just saying “you can’t compare fractional infinities”), as I already asserted:
Summarizing your thoughts, you offer the following conclusion:
I’m also somewhat open to the idea that EV reasoning is flawed, especially my interpretation of the concept. However:
I am very skeptical of rejecting EV reasoning without any kind of replacement, which I don’t think I’ve seen you offer.
This conversation has not moved me closer to believing that I should reject EV reasoning due to Pascalian mugging objections, of which I don’t see any standing/compelling examples. In fact, it is through EV reasoning that I find these objections uncompelling.
The St. Petersburg Paradox and Pasadena Game are interesting objections with which I’m decently familiar (especially the former); I’ve written one or two somewhat long comments about the former some time ago. However, these are also not persuasive, for some of the same reasons I’m articulating here (e.g., it might be equally likely that shoving your $100 bill in some random tree hollow is going to reveal a Korok that breaks the Matrix and gives you infinite utilons than that you are going to flip heads for an infinitely long time).
You know, I actually think I’ve gone completely down the wrong lines with my argument and if I could erase my previous comments I would.
The point is that Pascal’s mugging is a thought experiment to illustrate a point, and I can alter it as required. So I can remove the possibility of amazing alternatives for the money in the wallet if I have to.
E.g. let’s say you have a large amount of money in your wallet and, for whatever reason, you can only use that money to buy yourself stuff you really like. E.g. let’s say it’s a currency that you know can only be taken by a store that sells fancy cars, clothes and houses. Assume you really do know you can’t use the money for anything else (it’s a thought experiment so I can make this restriction).
Now imagine that you run into a Pascal’s mugging. Your “opportunity costs are correlated with outcome magnitudes that the person is claiming” argument no longer applies in this thought experiment. Do you now hand over the wallet and miss out on all of that amazing stuff you could have bought from the store?
I know this all sounds weird, but thought experiments are weird...
Well, with those extra assumptions I would no longer consider it a Pascalian Mugging, I would probably just consider it a prone-to-mislead thought experiment.
Would I take a 1*10−10 chance of getting 1015 utilons over the option of a 100% chance of 1 utilon? Well, if we assume that:
Utilons are the only thing that matter for morality/wellbeing (ignore all moral uncertainty),
I don’t have any additional time to reduce my uncertainty (or I have maximal certainty of the 1*10−10 chance estimate),
The 1*10−10 chance estimate already takes into account all of the outside views, intuitions, and any other arguments that says “this is extremely improbable”,
There’s no such thing as opportunity cost,
Diminishing marginal returns don’t apply,
Psychological and social factors are non-existent,
Your answer to this hypothetical question will not be used to judge your intelligence/character in the real world,
(And how about for good measure, “Assume away any other argument which could dispute the intended conclusion”),
Then the answer is probably yes?
I have a hard-to-articulate set of thoughts on what makes some thought experiments valuable vs. misleading, with one of them being something along the lines of “It’s unhelpful to generate an unrealistic thought experiment and then use the answer/response produced by a framework/system as an argument against using that framework/system in the real world if the argument is (at least implicitly) ‘Wow, look at what an unreasonable answer this could produce for real world problems,’ especially given how people may be prone to misinterpret such evidence (especially due to framing effects) for frameworks/systems that they are unfamiliar with or are already biased against.”
But I don’t have time to get into that can of worms, unfortunately.
Pascal’s mugging is only there to uncover if there might be a potential problem with using EV maximisation when probabilities get very small. It’s a thought experiment in decision theory. For that reason I actually think my altered thought experiment is useful, as I think you were introducing complications that distract from this central message of the thought experiment. Pascal’s mugging doesn’t in itself say anything about the relevance of these issues to real life. It may be all a moot point at the end of the day.
It sounds to me as if you don’t see any issues with EV maximisation when probabilities get very small. So in my altered thought experiment you would indeed give away your wallet to some random dude claiming to be a wizard, thereby giving up all those awesome things from the store. It’s worth at least noting that many people wouldn’t do the same, and who is right or wrong is where the interesting conundrum lies.
I don’t think many people are capable of actually internalizing all of the relevant assumptions that in real life would be totally unreasonable, nor do most people have a really good sense of why they have certain intuitions in the first place. So, it’s not particularly surprising/interesting that people would have very different views on this question.