I’m curious what people who’re more familiar with infinite ethics think of Manheim & Sandberg’s What is the upper limit of value?, in particular where they discuss infinite ethics (emphasis mine):
Bostrom’s discussion of infinite ethics is premised on the moral relevance of physically inaccessible value. That is, it assumes that aggregative utilitarianism is over the full universe, rather than the accessible universe. This requires certain assumptions about the universe, as well as being premised on a variant of the incomparability argument that we dismissed above, but has an additional response which is possible, presaged earlier. Namely, we can argue that this does not pose a problem for ethical decision-making even using aggregative ethics, because the consequences of any ethical decision can have only a finite (difference in) value. This is because the value of a moral decision relates only to the impact of that decision. Anything outside of the influenced universe is not affected, and the arguments above show that the difference any decision makes is finite.
I first read their paper a few years ago and found their arguments for the finiteness of value persuasive, as well as their collectively-exhaustive responses in section 4 to possible objections. So ever since then I’ve been admittedly confused by claims that the problems of infinite ethics still warrant concern w.r.t. ethical decision-making (e.g. I don’t really buy Joe Carlsmith’s arguments for acknowledging that infinities matter in this context, same for Toby Ord’s discussion in a recent 80K podcast). What am I missing?
I haven’t read the paper, but a simple objection is that you’re never going to be certain your actions only have finite effects, because you should only assign credence 0 to contradictions. (I don’t actually know the argument for the latter, but some philosophers believe it.) So you have to deal with the very, very small but not literally 0 chance that your actions will have an infinitely good/bad outcome because your current theories of how the universe works are wrong. However, anything with a chance of bringing about an infinitely good or bad outcome has an infinite expected value or an undefined one. So unless all expected values are undefined (which brings it own problems) you have to deal with infinite expected values, which is enough to cause trouble.
Manheim and Sandberg address your objection in the paper persuasively (to me personally), so let me quote them, since directly addressing these arguments might change my mind. @MichaelStJules I’d be keen to get your take on this as well. (I’m not quoting the footnotes, even though they were key to persuading me too.)
Section 4.1, “Rejecting Physics”:
4.1.1 Pessimistic Meta-induction and expectations of falsification
The pessimistic meta-induction warns that since many past successful scientific theories were found to be false, we have no reason expect that our currently successful theories are approximately true. Hence, for example, the above constraints on information processing are not guaranteed to imply finitude. Indeed, many of them are based on information physics that is weakly understood and liable to be updated in new directions. If physics in our universe does, in fact, allow for access to infinite matter, energy, time, or computation through some as-yet-undiscovered loophole, it would undermine the central claim to finitude.
This criticism cannot be refuted, but there are two reasons to be at least somewhat skeptical. First, scientific progress is not typically revisionist, but rather aggregative. Even the scientific revolutions of Newton, then Einstein, did not eliminate gravity, but rather explained it further. While we should regard the scientific input to our argument as tentative, the fallibility argument merely shows that science will likely change. It does not show that it will change in the direction of allowing infinite storage. Second, past results in physics have increasingly found strict bounds on the range of physical phenomena rather than unbounding them. Classical mechanics allow for far more forms of dynamics than relativistic mechanics, and quantum mechanics strongly constrain what can be known and manipulated on small scales.
While all of these arguments in defense of physics are strong evidence that it is correct, it is reasonable to assign a very small but non-zero value to the possibility that the laws of physics allow for infinities. In that case, any claimed infinities based on a claim of incorrect physics can only provide conditional infinities. And those conditional infinities may be irrelevant to our decisionmaking, for various reasons.
4.1.2 Boltzmann Brains, Decisions, and the indefinite long-term
One specific possible consideration for an infinity is that after the heat-death of the universe there will be an indefinitely long period where Boltzmann brains can be created from random fluctuations. Such brains are isomorphic to thinking human brains, and in the infinite long-term, an infinite number of such brains might exist [ 34]. If such brains are morally relevant, this seems to provide a value infinity.
We argue that even if these brains have moral value, it is by construction impossible to affect their state, or the distribution of their states. This makes their value largely irrelevant to decision-making, with one caveat. That is, if a decision-maker believes that these brains have positive or negative moral value, it could influence decisions about whether decisions that could (or would intentionally) destroy space-time, for instance, by causing a false-vacuum collapse. Such an action would be a positive or negative decision, depending on whether the future value of a non-collapsed universe is otherwise positive or negative. Similar and related implications exist depending on whether a post-collapse universe itself has a positive or negative moral value.
Despite the caveat, however, a corresponding (and less limited) argument can be made about decisionmaking for other proposed infinities that cannot be affected. For example, inaccessible portions of the universe, beyond the reachable light-cone, cannot be causally influenced. As long as we maintain that we care about the causal impacts of decisions, they are irrelevant to decisionmaking.
Section 4.2.4 more directly addresses the objection I think. (Unfortunately the copy-pasting doesn’t preserve the mathematical formatting, so perhaps it’d be clearer to just look at page 12 of their paper; in particular I’ve simplified their notation for $1 in 2020 to just $1):
4.2.4 Bounding Probabilities
As noted above, any act considered by a rational decision maker, whether consequentialist or otherwise, is about preferences over a necessarily finite number of possible decisions. This means that if we restrict a decision-maker or ethical system to finite, non-zero probabilities relating to finite value assigned to each end state, we end up with only finite achievable value. The question is whether probabilities can in fact be bounded in this way.
We imagine Robert, faced with a choice between getting $1 with certainty, and getting $100 billion with some probability. Given that there are two choices, Robert assigns utility in proportion to the value of the outcome weighted by the probability. If the probability is low enough, yet he chooses the option, it implies that the value must be correspondingly high.
As a first argument, imagine Robert rationally believes there is a probability of 10^−100 of receiving the second option, and despite the lower expected dollar value, chooses it. This implies that he values receiving $100 billion at approximately 10^100x the value of receiving $1. While this preference is strange, it is valid, and can be used to illustrate why Bayesians should not consider infinitesimal probabilities valid.
To show this, we ask what would be needed for Robert to be convinced this unlikely event occurred. Clearly, Robert would need evidence, and given the incredibly low prior probability, the evidence would need to be stupendously strong. If someone showed Robert that his bank balance was now $100 billion higher, that would provide some evidence for the claim—but on its own, a bank statement can be fabricated, or in error. This means the provided evidence is not nearly enough to convince him that the event occurred. In fact, with such a low prior probability, it seems plausible that Robert could have everyone he knows agree that it occurred, see newspaper articles about the fact, and so on, and given the low prior odds assigned, still not be convinced. Of course, in the case that the event happened, the likelihood of getting all of that evidence will be much higher, causing him to update towards thinking it occurred.
A repeatable experiment which generates uncorrelated evidence could provide far more evidence over time, but complete lack of correlation seems implausible; checking the bank account balance twice gives almost no more evidence than checking it once. And as discussed in the appendix, even granting the possibility of such evidence generation, the amount possible is still bounded by available time, and therefore finite.
Practically, perhaps the combination of evidence reaches odds of 10^50:1 that the new money exists versus that it does not. Despite this, if he truly assigned the initially implausibly low probability, any feasible update would not be enough to make the event, receiving the larger sum, be a feasible contender for what Robert should conclude. Not only that, but we posit that a rational decision maker should know, beforehand, that he cannot ever conclude that the second case occurs.
If he is, in fact, a rational decision maker, it seems strange to the point of absurdity for him to to choose something he can never believe occurred, over the alternative of a certain small gain.
Generally, then, if an outcome is possible, at some point a rational observer must be able to be convinced, by aggregating evidence, that it occurred. Because evidence is a function of physical reality, the possible evidence is bounded, just as value itself is limited by physical constraints. We suggest (generously) that the strength of this evidence is limited to odds of the number of possible quantum states of the visible universe — a huge but finite value — to 1. If the prior probability assigned to an outcome is too low to allow for a decision maker to conclude it has occurred given any possible universe, no matter what improbable observations occur, we claim the assigned probability is not meaningful for decision making. As with the bound on lexicographic preferences, this bound allows for an immensely large assignment of value, even inconceivably so, but it is again still finite.
Just noting for my own future edification this LW exchange between David Manheim (who argues that infinite ethics is irrelevant to actual decisions, per paper above) and Joe Carlsmith (who argues the opposite, per essay above), which only increased my conviction that Manheim and Anders Sandberg were right.
FWIW here’s Claude Sonnet 3.5 attempting to first steelman Carlsmith’s essay and then being neutral about which stance is more right:
Steelman of Carlsmith’s key arguments:
Even if infinite value/disvalue isn’t currently accessible, we should have non-zero credence in possibilities that would make it accessible (e.g. discovering our physics is wrong in specific ways). Given expected value theory, a non-zero credence in infinite value should dominate our calculations.
We face decisions with potentially infinite consequences even under current physics through acausal influence—if we accept certain decision theories and cosmological models suggesting infinite copies of us exist, our choices could influence an infinite number of copies.
Even if we can’t access infinities in practice, ethics shouldn’t fall silent when considering infinite cases. We have strong intuitions about choices like “infinite heaven vs infinite hell” that a complete ethical theory needs to address.
The mathematical/philosophical challenges infinite ethics raises reveal important limitations in our ethical frameworks that we should take seriously, even if just to better understand finite ethics.
Manheim/Sandberg’s key counterarguments:
Under our best physics, the accessible universe is fundamentally finite in space, time, and computational capacity.
Any decision theory allowing infinite value requires rejecting either current physics, rational preferences, or established decision theory in problematic ways.
We can represent supposedly “infinite” values (like lexicographic preferences) with very large but finite numbers while preserving all decision-relevant distinctions.
Ethics should focus on actionable choices that make a difference to actual moral patients in our universe.
My analysis: While Carlsmith makes interesting theoretical points, Manheim/Sandberg seem more right for practical purposes. The key considerations:
Their physics arguments about fundamental finitude seem solid and would need extraordinary evidence to overturn.
Even if we assign some credence to infinite possibilities, we can’t operationalize that credence in a way that helps real decision-making.
The practical ethical questions we face (e.g. existential risk, animal welfare, global poverty) don’t require solving infinite ethics.
Time spent on infinite ethics likely trades off against work on pressing finite problems where we can make concrete progress.
So while infinite ethics raises fascinating theoretical questions, I lean towards Manheim/Sandberg’s view that it’s not practically relevant for “doing good better.” That said, I appreciate Carlsmith’s rigorous exploration of the theoretical challenges, which may yield insights for finite ethics.
The prudent path seems to be: Note the theoretical challenges infinite ethics raises, remain humble about our ethical frameworks, but focus our efforts on tractable finite problems where we can clearly help actual moral patients.
If you maximize expcted value, you should be taking expected values through small probabilities, including that we have the physics wrong or that things could go on forever (or without hard upper bound) temporally. Unless you can be 100% in no infinities, then your expected values will be infinite or undefined. And there are, I think, hypotheses that can’t be ruled out and that
could involve infinite affectable value.
In response to Carl Shulman on acausal influence, David Manheim said to renormalize. I’m sympathetic and would probably agree with doing something similar, but the devil is in the details. There may be no very uniquely principled way to do this, and some things can still break down, e.g. you get actions that are morally incomparable.
And there are, I think, hypotheses that can’t be ruled out and that could involve infinite affectable value.
This is my crux, I think. I have yet to find a single persuasive example of an ethical decision I might face for which incorporating infinite ethics considerations suggests a different course of action. I don’t remember if Carlsmith’s essay provided any such examples; if it did I likely did not find them persuasive, since I skimmed it with this focus in mind. I interpreted Manheim & Sandberg’s paper to say that I likely wouldn’t find any such examples if I kept looking.
You could want to do acausal trades and cooperate with agents causally disconnected from you. You’ll expect that those who reason (sufficiently) similarly would do the same in return, and that you would cooperate would be evidence for them cooperating and make it more likely.
If you were difference-making risk averse locally, e.g. you don’t care about making a huge difference with very very tiny probability, by taking acausal influence into account, you should be (possibly much) less difference-making risk averse, according to Wilkinson.
I don’t see why acausal trade makes infinite ethics decision-relevant for essentially the reasons Manheim & Sandberg discuss in Section 4.5 – acausal trade alone doesn’t imply infinite value; footnote 41′s “In mainstream cosmological theories, there is a single universe, and the extent can be large but finite even when considering the unreachable portion (e.g. in closed topologies). In that case, these alternative decision theories are useful for interaction with unreachable beings, or as ways to interact with powerful predictors, but still do not lead to infinities”; physical limits on information storage and computation would still apply to any acausal coordination.
They aren’t asserting that the whole universe, including the unreachable portion, is finite in extent with certainty. They’re just saying that it’s possible, and they also note infinite is possible too in the sentence after which that footnote follows.
Even if you think a universe with infinite spatial extent is very unlikely, you should still be entertaining the possibility. If there’s a chance it’s infinite and you can have infinite impact (before renormalizing), a risk neutral expected value reasoner should wager on that.
FWIW, I’m sympathetic to their arguments in that section against expected value maximization, or that at least undermine the arguments for it. I’m not totally convinced of expected value maximization myself.
However, that doesn’t give a positive case for ignoring these infinities. I find infinite acausal impacts not too unlikely, personally, because both that acausal influence is possible seems more likely than not and that the universe is infinite in spatial extent (and in the right way to be influenced infinitely acausally) seems not too unlikely.
Rob Wiblin: OK, so the argument is something like valuing is a process that requires information to be encoded, and information to be processed — and there are just maximum limits on how much information can be encoded and processed given a particular amount of mass and given a finite amount of mass and energy. So that ultimately is going to set the limit on how much valuing can be done physically in our universe. No matter what things we create, no matter what minds we generate, there’s going to be some finite limit there. That’s basically it?
Anders Sandberg: That’s it. In some sense, this is kind of trivial. I think some readers would no doubt feel almost cheated, because they wanted to know that metaphysical limit for value, and we can’t say anything about that. But it seems very likely that if value has to have to do with some entity that is doing the valuing, then there is always going to be this limit — especially since the universe is inconveniently organised in such a way that we can’t get hold of infinite computational power, as far as we know.
I believe the effects of one’s actions decay to practically 0 after at most around 100 years, so I do not think it matters whether the theoretically affectable universe is infinite or not. Even if it was, one could simply use limits to figure out which action is best.
Thanks Vasco. While I agree with what I interpret to be your actionable takeaway (to ethically act as if our actions’ consequences are finitely circumscribed in time and space), I don’t see where your confidence comes from that the effects of one’s actions decay to practically 0 after at most around 100 years, especially given that longtermists explicitly seek and focus on such actions. I’m guessing you have a writeup on the forum elaborating on your reasoning, in which case would you mind linking to it?
I’m curious what people who’re more familiar with infinite ethics think of Manheim & Sandberg’s What is the upper limit of value?, in particular where they discuss infinite ethics (emphasis mine):
I first read their paper a few years ago and found their arguments for the finiteness of value persuasive, as well as their collectively-exhaustive responses in section 4 to possible objections. So ever since then I’ve been admittedly confused by claims that the problems of infinite ethics still warrant concern w.r.t. ethical decision-making (e.g. I don’t really buy Joe Carlsmith’s arguments for acknowledging that infinities matter in this context, same for Toby Ord’s discussion in a recent 80K podcast). What am I missing?
I haven’t read the paper, but a simple objection is that you’re never going to be certain your actions only have finite effects, because you should only assign credence 0 to contradictions. (I don’t actually know the argument for the latter, but some philosophers believe it.) So you have to deal with the very, very small but not literally 0 chance that your actions will have an infinitely good/bad outcome because your current theories of how the universe works are wrong. However, anything with a chance of bringing about an infinitely good or bad outcome has an infinite expected value or an undefined one. So unless all expected values are undefined (which brings it own problems) you have to deal with infinite expected values, which is enough to cause trouble.
Manheim and Sandberg address your objection in the paper persuasively (to me personally), so let me quote them, since directly addressing these arguments might change my mind. @MichaelStJules I’d be keen to get your take on this as well. (I’m not quoting the footnotes, even though they were key to persuading me too.)
Section 4.1, “Rejecting Physics”:
Section 4.2.4 more directly addresses the objection I think. (Unfortunately the copy-pasting doesn’t preserve the mathematical formatting, so perhaps it’d be clearer to just look at page 12 of their paper; in particular I’ve simplified their notation for $1 in 2020 to just $1):
Just noting for my own future edification this LW exchange between David Manheim (who argues that infinite ethics is irrelevant to actual decisions, per paper above) and Joe Carlsmith (who argues the opposite, per essay above), which only increased my conviction that Manheim and Anders Sandberg were right.
FWIW here’s Claude Sonnet 3.5 attempting to first steelman Carlsmith’s essay and then being neutral about which stance is more right:
If you maximize expcted value, you should be taking expected values through small probabilities, including that we have the physics wrong or that things could go on forever (or without hard upper bound) temporally. Unless you can be 100% in no infinities, then your expected values will be infinite or undefined. And there are, I think, hypotheses that can’t be ruled out and that could involve infinite affectable value.
In response to Carl Shulman on acausal influence, David Manheim said to renormalize. I’m sympathetic and would probably agree with doing something similar, but the devil is in the details. There may be no very uniquely principled way to do this, and some things can still break down, e.g. you get actions that are morally incomparable.
This is my crux, I think. I have yet to find a single persuasive example of an ethical decision I might face for which incorporating infinite ethics considerations suggests a different course of action. I don’t remember if Carlsmith’s essay provided any such examples; if it did I likely did not find them persuasive, since I skimmed it with this focus in mind. I interpreted Manheim & Sandberg’s paper to say that I likely wouldn’t find any such examples if I kept looking.
You could want to do acausal trades and cooperate with agents causally disconnected from you. You’ll expect that those who reason (sufficiently) similarly would do the same in return, and that you would cooperate would be evidence for them cooperating and make it more likely.
If you were difference-making risk averse locally, e.g. you don’t care about making a huge difference with very very tiny probability, by taking acausal influence into account, you should be (possibly much) less difference-making risk averse, according to Wilkinson.
I don’t see why acausal trade makes infinite ethics decision-relevant for essentially the reasons Manheim & Sandberg discuss in Section 4.5 – acausal trade alone doesn’t imply infinite value; footnote 41′s “In mainstream cosmological theories, there is a single universe, and the extent can be large but finite even when considering the unreachable portion (e.g. in closed topologies). In that case, these alternative decision theories are useful for interaction with unreachable beings, or as ways to interact with powerful predictors, but still do not lead to infinities”; physical limits on information storage and computation would still apply to any acausal coordination.
I’ll look into Wilkinson’s paper, thanks.
They aren’t asserting that the whole universe, including the unreachable portion, is finite in extent with certainty. They’re just saying that it’s possible, and they also note infinite is possible too in the sentence after which that footnote follows.
Even if you think a universe with infinite spatial extent is very unlikely, you should still be entertaining the possibility. If there’s a chance it’s infinite and you can have infinite impact (before renormalizing), a risk neutral expected value reasoner should wager on that.
FWIW, I’m sympathetic to their arguments in that section against expected value maximization, or that at least undermine the arguments for it. I’m not totally convinced of expected value maximization myself.
However, that doesn’t give a positive case for ignoring these infinities. I find infinite acausal impacts not too unlikely, personally, because both that acausal influence is possible seems more likely than not and that the universe is infinite in spatial extent (and in the right way to be influenced infinitely acausally) seems not too unlikely.
But I am optimistic about renormalization.
Sandberg’s recent 80K podcast interview transcript has this quote:
Hi Mo,
I believe the effects of one’s actions decay to practically 0 after at most around 100 years, so I do not think it matters whether the theoretically affectable universe is infinite or not. Even if it was, one could simply use limits to figure out which action is best.
Thanks Vasco. While I agree with what I interpret to be your actionable takeaway (to ethically act as if our actions’ consequences are finitely circumscribed in time and space), I don’t see where your confidence comes from that the effects of one’s actions decay to practically 0 after at most around 100 years, especially given that longtermists explicitly seek and focus on such actions. I’m guessing you have a writeup on the forum elaborating on your reasoning, in which case would you mind linking to it?
My post Reducing the nearterm risk of human extinction is not astronomically cost-effective? is somewhat related, but it does not empirically analyse how fast effects decay over time. Uncertainty over time and Bayesian updating is the best analysis on this I am aware of. I have just updated the comment I had left there to explain my claim that effects decay to practically 0 after at most 100 years.
Much appreciated, thanks again Vasco.