Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
As far as I can tell, we don’t know of any principles that satisfy both (1) they guide our actions in all or at least most situations and (2) when taken seriously, they don’t lead to crazy town. So our options seem to be (a) don’t use principles to guide actions, (b) go to crazy town, or (c) use principles to guide actions but be willing to abandon them when the implications get too crazy.
I wish this post had less of a focus on utilitarianism and more on whether we should be doing (a), (b) or (c).
(I am probably not going to respond to comments about how specific principles satisfy (1) and (2) unless it actually seems plausible to me that they might satisfy (1) and (2).)
Thanks for your comment.
It’s not clear to me that (a), (b), and (c) are the only options—or rather, there are a bunch of different variants (c’), (c″), (c‴). Sure, you can say “use principles to guide action until they get too crazy’, but you can also say ‘use a multiplicity of principles to guide action until they conflict’, or ‘use a single principle to guide action until you run into cases where it is difficult to judge how the principle applies’, or so on. There are lots of different rules of thumb to tell you when and how principles run out, none of them entirely satisfactory.
And, importantly, there is no meta-principle that tells you when and how to apply object-level principles. Intuitively there’s an infinite regress argument for this claim, but since infinite regress arguments are notoriously unreliable I also tried to explain in more detail why it’s true in this post: any meta-principle would be vulnerable in the same way to the train to crazy town. And so, if you have the worries about the train to crazy town that are expressed in this post, you have to move away from the realm of pure rationalist moral philosophy and begin to use principles as context-specific guides, exercising particular judgments to figure out when they are relevant.
And so, in order to figure out when to use principles, you have to examine some specific principles in specific context, rather than trying very abstract philosophising to get the meta-principle. That was why I focussed so much of utilitarianism, and the specific context within which it is a useful way to think about ethics: I think this kind of specific analysis is the only way to figure out the limits of our principles. Your comment seems to assume that I could do some abstract philosophising to decide between (a), (b), (c), (c’), (c″), etc.; but for the very reasons discussed in this post, I don’t think that’s an option.
Why is criterion (1) something we want? Isn’t it enough to find principles that would guide us in some or many situations? Or even just in “many of the currently relevant-to-EA situations”?
I think there plausibly are principles that achieve (1) and (2), but they’ll give up either transitivity or the independence of irrelevant alternatives, and if used to guide actions locally without anticipating your own future decisions and without the ability to make precommitments, lead to plausibly irrational behaviour (and more than usual than just with known hard problems like Newcomb’s and Parfit’s hitchhiker). I don’t think those count as “crazy towns”, but they’re things people find undesirable or see as “inconsistent”. Also, they might require more arbitrariness than usual, e.g. picking thresholds, or nonlinear monotonically increasing functions.
Principles I have in mind (although they need to be extended or combined with others to achieve 1):
Partial/limited aggregation, although I don’t know if they’re very well-developed, especially to handle uncertainty (and some extensions may have horrific crazy town counterexamples, like https://forum.effectivealtruism.org/posts/smgFKszHPLfoBEqmf/partial-aggregation-s-utility-monster). The Repugnant Conclusion and extensions (https://link.springer.com/article/10.1007/s00355-021-01321-2 ) can be avoided this way, and I think limiting and totally forbidding aggregation are basically the only ways to do so, but totally forbidding aggregation probably leads to crazy towns.
Difference-making risk aversion, to prevent fanaticism for an otherwise unbounded theory (objections of stochastic dominance and, if universalized, collective defeat here https://globalprioritiesinstitute.org/on-the-desire-to-make-a-difference-hilary-greaves-william-macaskill-andreas-mogensen-and-teruji-thomas-global-priorities-institute-university-of-oxford/ , https://www.youtube.com/watch?v=HT2w5jGCWG4 and https://forum.effectivealtruism.org/posts/QZujaLgPateuiHXDT/concerns-with-difference-making-risk-aversion , and some other objections and responses here https://forum.effectivealtruism.org/posts/sEnkD8sHP6pZztFc2/fanatical-eas-should-support-very-weird-projects?commentId=YqNWwzdpvmFbbXyoe#YqNWwzdpvmFbbXyoe ). There’s also a modification that never locally violates stochastic dominance, by replacing the outcome distributions with their quantile functions (i.e. sorting each outcome distribution statewise), but it’s also not implausible to me that we should just give up stochastic dominance and focus on statewise dominance, because stochastic dominance doesn’t guarantee conditional worseness, difference-making may be fundamentally a statewise concern, and stochastic dominance is fanatical with the right background noise when (deterministic) value is additive (https://www.journals.uchicago.edu/doi/10.1086/705555).
A non-fanatical approach to normative uncertainty or similar, but (possibly) instead of weighing different theories by subjective credences, you think of it as weighing principles or reasons by all-things-considered intuitive weights within a single theory. This interpretation makes most sense if you’re either a moral realist who treats ethics as fundamentally pluralistic or vague, or you’re a moral antirealist. However, again, this will probably lead to violations of transitivity or the independence of irrelevant alternatives to avoid fanaticism, unless you assume reasons are always bounded in strength, which rules out risk-neutral EV-maximizing total utilitarianism.
Also, I normally think of crazy town as requiring an act, rather than an omission, so distinguishing the two could help, although that might be unfair, and I suppose you can imagine forced choices.
Thank you for the post which I really liked! Just two short comments:
It is not clear to me why the problems of utilitarianism should inevitably lead to a form of fanaticism, under promising frameworks for moral uncertainty. At least this seems not to follow on the account of moral uncertainty of MacAskill, Ord and Bykvist (2020) which is arguably the most popular one for at least two reasons: a) Once the relevant credence distribution includes ethical theories which are not intertheoretically comparable or are merely ordinal-scale, then theories in which one has small credences (including totalist utilitariansm) won’t always dictate how to act. b) Some other ethical theories, e.g. Kantian theories which unconditionally forbid killing, seem (similar to totalist utilitarianism) to place extremely high (dis)value on certain actions.
It would be interesting to think about how distinctions between different version of utilitarianism would factor into your argument. In particular, you could be an objective utilitarian (who thinks that the de facto moral worth of an action is completely determined by its de facto consequences for total wellbeing) without believing (a) that expected value theory is the correct account of decision making under uncertainty or (b) that the best method in practice for maximizing wellbeing is to frequently explicitly calculate expected value. Version (b) would be so-called multi-level utilitarianism.
Thanks for your comment! I was considering writing much more about moral uncertainty, since I think it’s an important topic here, but the post was long enough as it is. But you and other commenters below have all pulled me up on this one, so it’s worth being more explicit. I thus hope it’s OK for this reply to you to serve as a general reply to a lot of themes related to moral uncertainty in the comment section, to avoid repeating myself too much!
Starting with 1(b), the question of unconditional ‘deontological constraints’: this works in theory, but I don’t think it applies in practice. The (dis)value placed on specific actions can’t just be ‘extremely high’, because then it can still be swamped by utilitarianism over unbounded choice sets; it has to be infinite, such that (e.g.) intentional killing is infinitely disvaluable and no finite source of value, no matter how arbitrarily large, could outweigh it. This gets you around the impossibility proof, which as mentioned relies on order properties of the reals that don’t hold for the extended reals—roughly, the value of utility is always already infinitesimal relative to the infinite sources of (dis)value, so the marginal value of utility doesn’t need to decline asymptotically to avoid swamping them.
But in practice, I just don’t see what marginally plausible deontological constraints could help a mostly-consequentialist theory avoid the train to crazy town. These constraints work to avoid counterexamples like e.g. the transplant case, where intuitively there is another principle at play that overrides utility considerations. In these cases, deontological constraints are simple, intuitive, and well-motivated. But in the cases I’m concerned with in this post, like Hurka’s St Petersburg Paradox, it’s not clear that Kantian-style constraints on murder or lying really help the theory—especially because of the role of risk in the example. To get around this example with deontological constraints, you either have to propose wildly implausible constraints like ‘never accept any choices with downside risk to human life’, or have an ad hoc restriction specifically designed to get around this case in particular—the latter of which seems a) epistemically dodgy and b) liable to be met with a slightly adjusted counterexample. I just don’t see how you could avoid all such cases with any even mildly plausible deontological constraint.
Beyond these kinds of ‘lexical’ approaches, there are various other attempts to avoid fanaticism while respecting considerations of utility at scale—your 1(a). But by Cowen’s proof, if these are indeed to work, they must deny the universal domain condition—as indeed, the theories mentioned tend to! I mentioned the moral parliament explicitly, but note also that (e.g.) if you accept that certain intertheoretical comparisons cannot be made, then you have ipso facto denied universal domain and accepted a certain level of incomparability and pluralism.
The difference between me and you is just that you’ve only accepted incomparability at the meta-level (it applies comparing different moral theories), whereas I’m encouraging you to adopt it at the object level (it should apply to the act of thinking about ethics in the first instance). But I see no coherent way to hold ‘we can have incomparability in our meta-level theorising, but it must be completely banned from our object-level theorising’! There are many potential rationalistic reasons you might offer for why incomparability and incommensurability should be banished from moral philosophy; but none of these are available to you if you take on a framework for moral uncertainty that avoids fanaticism by denying universal domain. So accepting these kinds of positions about moral uncertainty just seems to me like an unstable halfway house between true rationalistic moral philosophy (on the one hand) and pluralism (on the other).
Thank you for your replies! In essence, I don’t think I disagree much with any of your points. I will mainly add different points of emphasis:
I think one argument I was gesturing at is a kind of divide-and-conquer strategy where some standard moves of utilitarians or moral uncertainty adherents can counter some of the counterintuitive implications (walks to crazy town) you point to. For instance, the St. Petersburg Paradox seems to be a objection to expected value utilitarianism, not for every form of the view. Similarly, some of the classical counterexamples to utilitarianism (e.g., some variants of trolley cases) involve violations of plausible deontological constraints. Thus, if you have a non-negligible credence in a moral view which posits unconditional prohibitions of such behavior, you don’t need to buy the implausible implication (under moral uncertainty). But you are completely correct that there will remain some, maybe many, implications that many find counterintuitive or crazy, e.g., the (very) repugnant conclusion (if you are totalist utilitarian). Personally, I tend to be less troubled by these cases and suspect that we perhaps should bite some of these bullets, but to justify this would of course require a longer argument (which someone with different intuitions won’t likely be tempted by, in any case).
The passage of your text which seemed most relevant to multi-level utilitarianism is the following: “In practice, Effective Altruists are not willing to purchase theoretical coherence at the price of absurdity; they place utilitarian reasoning in a pluralist context. They may do this unreflectively, and I think they do it imperfectly; but it is an existence proof of a version of Effective Altruism that accepts that utility considerations are embedded in a wider context, and tempers them with judgment.“ One possible explanation of this observation is that the EA’s which are utilitarians are often multi-level utilitarians who consciously and intentionally use considerations beyond maximizing utility in practical decision-situation. If that were true, it would raise the interesting question what difference adopting a pluralist normative ethics, as opposed to a universal-domain utilitarianism, would make for effective altruist practice (I do not mean to imply that there aren’t difference).
With respect to moral uncertainty, I interpret you as agreeing that the most common effective altruist views actually avoid fanaticism. This then raises the question whether accepting incomparability at the meta-level (between normative theories) gives you reasons to also (or instead) accept incomparability at the object-level (between first-order moral reasons for or against actions). I am not sure about that. I am sympathetic to your point that it might be strange to hold that ‘we can have incomparability in our meta-level theorising, but it must be completely banned from our object-level theorising‘. At the same time, at least some of the reasons for believing in meta-level incomparability are quite independent from the relevant object-level arguments, so you might have good reasons to believe in it only on the meta-level. Also, the sort of incomparability seems different. As I understand your view, it says that different kinds of moral reasons can favor or oppose a course of action such that we sometimes have to use our faculties for particular, context-sensitive moral judgements, without being able to resort to a universal meta-principle that tells us how to weigh the relevant moral reasons. By contrast, the moral uncertainty view posits precisely such a meta-principle, e.g. variance voting. So I can see how one might think that the second-order incomparability is acceptable but yours is unacceptable (although this is not my view).
On 2: I think the point is simply that, as noted in footnote 8, the ‘train to crazy town’ reasoning can apply quite directly to comparisons between states of affairs with no lingering uncertainty (Savagean consequences). When we apply the reasoning in this way, two features arise:
(a) Uncertainty, and frameworks for dealing with uncertainty, no longer have a role to play as we are certain about outcomes. This is the case with e.g. the Very Repugnant Conclusion.
(b) The absurdities that are generated apply directly at the level of axiology, rather than ‘infecting’ axiology via normative ethics. If we read multi-level utilitarianism as an attempt to insulate axiology from ethics, then it can’t help in this case. Of course, multi-level utilitarians are often more willing to be bullet-biters! But the point is just that they do have to bite the bullet.
This comment might seem somewhat tangential but my main point is that the problem you are trying to solve is unsolvable and we might be better off reframing the question/solution.
My views
(1) anti realism is true
Every train/stop is equally crazy/arbitrary.
(2) The EA community has a very nebulous relationship with meta-ethics
obligatory the community is not a monolith
I see lots of people who aren’t anti-realists, lots who are
Community has no political system, so logic/persuasion is often the only way to push many things forward (unless you control resources)
if anti-realism is true there is no logical way to pick a worldview
(if anti-realism is true) most of this discussion is just looping and/or trying to persuade other EAs. I say that as someone who likes to loop about this stuff.
most of the power is in the hands of the high status early joiners (we constantly reference Karnofsky or Macaskill as if they have some special insight on meta-ethics) and rich people who join the community (give money to whatever supports their worldview).
(3) I think the community should explicitly renounce its relationship with utilitarianism or any other ethical worldview.
Let subgroups pop up that explicitly state their worldview, and the political system they will use to try to get there. e.g. utilitarianism + democracy, welfarism + dictatorship, etc.
You can reflect upon your moral views and divy up your funds/time to the groups accordingly.
These groups will become the primary retro funders for impact markets, since their goals are clear, and they will have the institutions to decide how they measure impact.
and also just the main funders but I wanted to emphasize that this will have good synergy with higher tech philanthropy.
I feel that this is much more transparent, honest, and clear. This is personally important to me.
We can stop arguing about this sort of stuff and let the money talk.
EA as a “public forum”, not an agent with power
‘if anti-realism is true there is no logical way to pick a worldview’
This seems like an obvious ‘crux’ here—I don’t think this is true? e.g., Simon Blackburn would insist that there are ‘logical’ ways to pick a worldview despite being committed to the view that value is just something we project onto the world, not something that is a property of the world in itself.
If it wasn’t clear I meant logical from first principles.
If you still disagree with the above statement I would definitely want to hear what your line of thinking is. I don’t know much about Blackburn or quasi-realism but I’ll check it out.
With the clarification that you specifically meant ‘from first principles’, I’m not sure how your point is supposed to be relevant. I agree that ‘there is no logical way to pick a worldview from first principles’, but just because you can’t rationally conclude something from first principles doesn’t mean you can’t rationally conclude it at all. There’s no way to get to science from first principles, but rational scientific argumentation still exists. Likewise, there might be no logical first-principles derivation of morality, but I can still apply logical reasoning to ethical questions to try to figure things out. So the idea that there’s no first-principles derivation of morality—which the anti-realist and the realist can both accept, by the way, there’s no connection here with anti-realism—is just irrelevant to your other points.
The implication here is you are rationally concluding it using your own values as a starting point. I don’t think there is an easy way to adjucate disagreements when this is how we arrive at views. You are trying to talk about meta-ethics but this will turn into a political philosophy question.
I have heard this argument multiple times and I still don’t get it. Science is a process that we can check the results of in real life, because it deals with real quantities. The fact that science doesn’t exist from first principles doesn’t mean we can use empirics to back up the validity of science in pushing forth our understanding of the world. We have seen time and time again that science does work for the purposes we want it to work for. Morality isn’t a physical quantity. We can’t use empiricism w/respect to morals. Hence we only have logic to lean on.
Sort of agree but not really. Here are the options as I see them.
Objective Morality doesn’t exist (anti-realism). This directly implies there is no first-principles derivation because if there was it would exist.
Objective Morality exists and we can prove it (maybe we can call this secular realism)
Objective Morality exists and we can’t prove it (potentially offensive but this basically sounds like religion or faith)
If you are saying you buy into bullet point 3 that’s fine but I would be upset if EA accepted this explicitly or implicitly.
I think you’ve run together several different positions about moral epistemology and meta-ethics. Your three bullet points definitely do not describe the whole range of positions here. For example: RM Hare was an anti-realist (the anti-realist par excellence, even) but believed in a first principles derivation of morality; you may have come across his position in the earlier works of his most famous student, Peter Singer. (Singer has since become a realist, under the influence of Derek Parfit). Likewise, you can have those who are as realist as realists can be, and who accept that we can know moral truths, but not that we can prove them. This seems to be what you’re denying in your comment—you think the only hope for moral epistemology is first-principles logic—but that’s a strong claim, and pretty much all meta-ethical naturalists have accounts of how we can know morality through some kind of natural understanding.
For myself, I’m a pretty strong anti-realist, but for reasons that have very little to do with traditional questions of moral epistemology; so I actually have a lot of sympathy with the accounts of moral epistemology given by many different metaethical naturalists, as well as by those who have straddled the realist/anti-realist line (e.g. constructivists, or Crispin Wright whatever name you want to give to his position), if their positions are suitably modified.
I’m confused. He thinks you can derive that morality doesn’t exist or he thinks you can derive something that doesn’t exist?
I mean it depends on what you mean by moral epistemology. If you just mean a decision tree that I might like to use for deciding my morals I think it exists. If you mean a decision tree that I Should follow then I disagree.
Thanks for this fantastic post and your comments, I both enjoyed and learned from them.
Is the following a reasonable summary of your position? Is there anything you would change or add?
Thank you very much for writing this—I have found it incredibly useful and wish all complex philosophical texts would be followed by something like this. Just wanted to give you extra feel-good for doing this because it is well deserved as it helped me immensely to fit all that I read into place.
Thanks so much! I’m really glad this was helpful for you.
Thank you for your fantastic summary! Yes, I think that’s a great account of what I’m saying in this post.
“Various theories of moral uncertainty exist, outlining how this aggregation works; but none of them actually escape the issue. The theories of moral uncertainty that Effective Altruists rely on are themselves frameworks for commensurating values and systematically ranking options, and (as such) they are also vulnerable to ‘value dictatorship’, where after some point the choices recommended by utilitarianism come to swamp the recommendations of other theories. In the literature, this phenomenon is well-known as ‘fanaticism’.[10]”
This seems too strong. Doesn’t this only apply to maximizing expected choiceworthiness with intertheoretic comparisons, among popular approaches? My impression is that none of the other popular approaches are fanatical. You mention moral parliament in the footnote as a non-fanatical exception, but there’s also MEC variance voting, formal bargain-theoretic approaches and my favourite theory.
Also, theories with value lexicality can swamp utilitarianism, although this also seems fanatical.
Hi Michael, thanks for your comment—since it overlapped quite significantly with Leonard’s comment, I have tried to address both of your concerns together in one of my replies to Leonard. Let me know if I failed to reply sufficiently to some of your points!
I think the quote literally mischaracterizes approaches to normative uncertainty that EAs rely on as all leading to fanaticism, when almost none do, only really MEC with intertheoretical comparisons. Whether the ones that deny intertheoretical comparisons also reject universal domain is a separate issue.
Or maybe I misunderstood, and you’re only referring to versions of MEC with intertheoretical comparisons?
I think that approaches other than MEC + ITT aren’t typically clear enough about universal domain, and can be interpreted in several different ways because they’re not completely formally specified. (Not itself a criticism!) But (a corollary of the argument in this post) these frameworks only actually avoid fanaticism if they deny universal domain. So it isn’t clear whether they fall victim too fanaticism, pace your claim that ‘almost none do’.
I think—purely personal impression—that most people who think very hard about moral uncertainty are committed to quite a rationalistic view of ethics, and that this leads quite naturally to universal domain; so I said that, interpreting this into their frameworks, they will fall victim to fanaticism. But I mentioned the moral parliament as an example of an approach where the proponents have explicitly denied universal domain, as an example of the other possibility. However you interpret it, though, it’s an either/or situation.
(Lexical orders / deontological constraints are a separate issue, as you mention.)
Assuming all your moral theories are at least interval-scale, I think variance voting and similar normalization-based methods
are basically completely formally specified (at least as much as any other approach I’ve seen, like MEC+intertheoretical comparisons).
are never fanatical with respect to your credences in theories iff you normalize theories based on the options you actually have available and so violate the independence of irrelevant alternatives (IIA).
Maybe violating IIA conflicts with how you think about “universal domain”, and IIA is usually assumed without much comment. If you think the quality of an outcome is based entirely on the features of that outcome, and this internal quality is all that matters when comparing outcoms, then IIA should hold.
don’t require or make intertheoretical comparisons.
rank all options (outcome distributions, including fixed outcomes) for any finite set of available options when each moral theory ranks them all, too. Maybe you can extend to arbitrary size sets by normalizing each theory relative to the infimum and supremum value across options according to that theory. So, I suspect they satisfy something that can reasonably be called universal domain.
I think formal bargain-theoretic approaches (here and here) satisfy 1 and 3, aren’t fanatical, but I’m not sure about 4 and violations of IIA.
On variance voting: yeah, I think 4 is the point here. I don’t think you can extend this approach to unbounded choice sets. I’m travelling atm so can’t be more formal, but hopefully tomorrow I can write up something a bit more detailed on this.
On bargaining-theory approaches, it actually isn’t clear that they avoid fanaticism: see pp.24-26 of the Greaves and Cotton-Barratt paper you link, especially their conclusion that ‘the Nash approach is not completely immune from fanaticism’. Again, I think constructive ambiguity in the way these theories get described often helps obscure their relationship to fanatical conclusions; but there’s an impossibility result here, so ultimately there’s no avoiding the choice.
The St. Petersburg paradox is not a paradox. The expected value of the wager varies as the square root of the sample size. Because we sum functions over infinity, we get infinite ev, but this is irrelevant to someone without infinite time. So the answer to the question, how much do we pay for the opportunity to wager, we say, something less than the square root of the number of times we get to run it. If you write a script to run this experiment, you can see how this is so.
Another approach to thinking about these difficulties could be to take counsel from the Maxwell Demon problem in thermodynamics. There, it looks like you can get a “repugnant conclusion” in which the second law is violated, if you don’t address the details of Maxwell’s demon directly and carefully.
I suspect there is a corresponding gap in analyses of situations at the edges of population ethics. Call it the “repugnant demon.” Meaning, in this hypothetical world full of trillions of people we’re being asked to create, what powers do we have to bestow on our demon which is responsible for enforcing barely livable conditions? These trillions of people want better lives, otherwise by definition they would not be suffering. So the demon must be given the power to prevent them from having those improved lives. How?
Pretty clearly, what we’re actually being asked is whether we want to create a totalitarian autocratic transgalactic prison state with total domination over its population. Is such a society one you wish to create or do you prefer to use the power of your demon which it would take to produce this result in a different way?
A much smaller scale check here is whether it is good to send altruistic donations to existing autocratic rulers or not. Their populations are not committing suicide, so the people must have positive life utility. The dictator can force the population to increase, so the implementation here would be finding dictators who will accept altruistic donations for setting up forced birth camps in their countries.
In other words, I suspect when you finish defining in detail what “repugnant demon” powers need to be created in world-building awful conditions for even very comparatively small populations, it becomes clear immediately where the “missing negative utility” is in these cases: it is that the utilitarian power in producing conditions of very low satisfaction are actually very large. Therefore using that power in the evil act of setting up a totalitarian prison camp instead of a different and morally preferable society is to be condemned.
You could create huge numbers of unlikely to be conscious beings or low moral weight beings who would not suffer at all and only experience pleasure, but each would only be barely above neutral in expected value. These beings may be more efficient to create and use to generate value, because their brains are simpler and more parallelizable.
The value of the far future seems vastly dominated by artificial sentience in expectation. The expected utility-maximizing artificial sentience could be huge numbers of beings with low average expected moral weight.
With current animals and ignoring the far future, it might be invertebrates, and the so-called Rebugnant Conclusion (https://jeffsebodotnet.files.wordpress.com/2021/08/the-rebugnant-conclusion-.pdf ). To be clear, though, I think many invertebrates are not very unlikely to be conscious, and it’s plausible they have similar or incomparable moral weight to humans, rather than much lower moral weight.
A bit repetitive to what I replied below but it isn’t clear to me that minimally -conscious beings can’t suffer (or be made to not be able to suffer).
On relatively more stable ground wrt power to choose between a world optimized for insects vs humans, I’m happy to report I’m a humanity partisan. :-)
In theory of mind terms , it sounds like we differ in estimating likelihood insects will be thought of as having conscious experience as we learn more. (Other invertebrates I think the analysis may be very different.) My sense given extraordinary capabilities of really-clearly-not-conscious ML systems is that pretty sophisticated behaviors are well within reach for unconscious organisms. Moreso than I might have thought a few years ago.
I think it’s very likely that we can stack the deck in favour of positive welfare, even if it’s still close to 0 due to low expected probability of consciousness or low moral weight. There are individual differences in average hedonic setpoints between humans that are influenced genetically and some extreme cases like Jo Cameron. The systems for pleasure and suffering don’t overlap fully, so we could cut parts selectively devoted to suffering out or reduce their sensitivity.
I agree with many sophisticated behaviors being well within range of reach for unconscious systems, but it’s not clear this counts that much against invertebrates. You can also come to it from the other side: it’s hard to pick out capacities that humans have that are with very high probability necessary for consciousness (e.g. theory of mind, self-awareness to the level of passing the mirror test and the capacity for verbal report don’t seem necessary), but aren’t present in (some) invertebrates. I’d recommend Rethink Priorities’ work on this topic (disclaimer: I work there, but didn’t work on this, and am not speaking for Rethink Priorities) and Luke Muehlhauser’s report for Open Phil.
Also, at what point would you start to worry about ML (or other AI) systems being conscious, especially ones that aren’t capable of verbal report?
Completely agree it is difficult to find “uniquely human” behaviors that seem indicative of consciousness as animals share so many of them.
Any animals which don’t rear young I am much more likely to believe have behaviors much more genetically determined and so therefore operating at time scales that don’t really satisfy what I think makes sense to call consciousness. I’m thinking of the famous Sphex wasp hacks for instance where complex behavior turns out to be pretty algorithmic and likely not indicative of anything approximating consciousness. Thanks for the pointer to the report!
WRT AI consciousness, I work on ML systems and have a lot of exposure to sophisticated models. My sense is that we are not close to that threshold, even with sophisticated systems that are obviously able to pass naive Turing tests (and have). My sense is we have a really powerful approach to world-model-building with unsupervised noise prediction now, and that current techniques (including RL) are just nowhere near enough to provide the kind of interiority that AI systems need to start me worrying there’s conscious elements there.
IOW, I’m not a “scale is all you need” person—I don’t think current ideas on memory/long-range augmentation or current planning type long-range state modeling is workable. I mean, maybe times 10^100 it is all you need? But that’s just sort of another way of saying it isn’t. :-) The sort of “self-talk” modularity that some LLMs are being experimented with strikes me as the most promising current direction for this (e.g. LAMDA paper) but currently the scale and ingredients are way too small for that to emerge IMO.
I do suspect that building conscious AI will teach us way more about non-verbal-report consciousness. We have some access to these mechanisms with neuroscience experiments but it is difficult going. My belief is we have enough of those to be quite certain many animals share something best called conscious experience.
Very interesting perspective I’ve never thought about.
Is your argument just that there do not exist choices with the logical structure of the repugnant conclusion, and that in any decision-situation that seems like the repugnant conclusion there is always a hidden source of negative utility to balance things out? Or is it more limited to the specific case that Parfit initially discussed?
If the latter, then I think your point is irrelevant to the philosophical questions about the repugnant conclusion, which are concerned only with its logical structure. (Consider Michael’s comment above for some other situations with similar structural features.) But if the former, then you’re going to need some general argument as to why no such structure ever exists, not just an argument as to why the specific case often proposed is a bit misleading. Just because Parfit’s original thought experiment might not pump your intuitions, so long as some set of choices has a structurally similar distribution of utility across time the question still arises. Do you have a more general argument why no choices exist with these distributions?
More like the former.
In other words, the moral weight of the choice we’re asked to make is about the use of power. An example that’s familiar and more successful because the power being exercised is much more clear is the drowning child example. The power here is going into the pond to rescue the child. Should one exercise that power, or are there reasons not to?
The powers bring appealed to in these population ethics scenarios are truly staggering. The question of how they should be used is (in my opinion) usually ignored in favor of treating them as preferences of states of affairs. I suspect this is the reason they end up being confusing—when you instead ask whether setting up forced reproduction camps is a morally acceptable use of the power to craft an arbitrary society, there’s just very little room for moral people to disagree any more.
Relative to creating large numbers of beings with the property of being unable to experience any negative utility but only small amounts of positive utility, it isn’t clear this power exists logically. (The same might be said about enforcing pan galactic totalitarianism but repugnant conclusion effects IMO start being noticeable on scales where we can be quite sure power does exist.)
If the power to create such beings exists, it implies a quite robust power to shape the minds and experiences of created beings. If it were used to prohibit the existence of beings with tremendous capacity for pleasure I think that would be an immoral application. Another scenario though might be the creation of large numbers of minimally-sentient beings who (in some sense) mildly “enjoy” being useful and supportive of such high-experience people. Do toasters and dishwashers and future helpful robots qualify here? It depends what kind of pan psychism ends up being like for hypothetical people with this kind of very advanced mind design power. I could see it being true that such a world is possible, but I think this framing in terms of power exercise removes the repugnance from the situation as well. Is a world of leisure supported by minimally-aware robots repugnant. Nah, not really. :-)
This confuses me. In the original context in which the Repugnant Conclusion was dreamed up (neo-Malthusian debates over population control), seeking a larger population was a kind of laissez-faire approach opposed to “population policy”, while advocates for smaller populations such as Garett Hardin explicitly embraced ‘coercion’. When Parfit originally formulated the Repugnant Conclusion, I don’t think he imagined ‘forced reproductive camps’!
So how about being more specific. Suppose you are living in 1968, and as a matter of fact you know that the claims made in Ehrlich and Ehrlich’s The Population Bomb are true. (This is a hypothetical, as those claims turned out to be false in the actual world.) And you have control over population policies—say, you are president of the United States. If you exercise coercive power over reproduction, you can ensure that the world will have a relatively small population of relatively happy people. If you don’t, and you simply let things be, then the world will have an enormous population of people whose lives are barely worth living.
This case has exactly the same structure as the Repugnant Conclusion, and not by accident: this is exactly the kind of question that Parfit and other population ethicists were thinking about in the 1960s and 1970s. But in this case, the larger population is not produced by the exercise of power; it is the smaller population that would be produced by coercion, and the larger population would be produced through laissez-faire. Thus, your argument about ‘the use of power’ does not support the claim that there do not exist choices with the logical structure of the Repugnant Conclusion.
In general, I think you have confused the Repugnant Conclusion itself with a weirdly specific variant of it, perhaps inspired by versions of the astronomical waste argument.
Thanks! This is a great set of context and a great way to ask for specifics. :-)
I think the situation is like this: I’m hypothetically in a position to exercise a lot of power over reproductive choices—perhaps by backing tax plans which either reward or punish having children. I think what you’re asking is “suppose you know that your plan to offer a child tax credit will result in a miserable population, should you stay with the plan because there’ll be so many miserable people that it’ll be better on utilitarian grounds”? The answer is no, I should not do that. I shouldn’t exercise power I have to make a world which I believe will contain a lot of miserable people.
I think a better power-inversion question is: “suppose you are given dictatorial control of one million miserable and hungry people. Should you slaughter 999,000 of them so the other 1000 can be well fed and happy.” My answer is, again, unsurprisingly, No. No, I shouldn’t use dictatorial power to genocide this unhappy group. Instead I should use it to implement policies I think will lead over time to a sustainable 1000-member happy population, perhaps by the same kind of anti-natalist policies that would in other happier circumstances be abhorrent.
My suspicion I think I share with you: that consequentialism’s advice is imperfect. My sense is it is imperfect mostly not because of unfamiliar galactic-scale reasons or other failures in reacting to odd situations involving unbelievably powerful political forces. If that’s where it broke down it’d be mostly immaterial to considering alternatives to consequentialism in everyday situations (IMO).
Forgive me for dropping a new and potentially shallow point in on this discussion. The intellectual stimulation from the different theoretical approaches, thought experiments and models is clear. It is great to stretch the mind and nurture new concepts – but otherwise I question their utility and priority, given our situation today.
We do not need to develop pure World A and World B style thought experiments on the application of EA concepts, for want of other opportunities to test the model. We (collectively, globally), have literally dozens of both broad and highly targeted issues where EA style thinking can be targeted and thereby may help unlock solutions. Climate change, ecosystem collapse, hybrid warfare, inequality, misinformation, internal division, fading democracy, self-regulated tech monopolies, biosecurity, pandemic responses etc… the individual candidate list is endless… We also need to consider the interconnected nature, and thereby the potential for interconnected solutions.
Surely, a sustained application of the minds demonstrated in this paper and comment train could both help on solutions, and as a bonus, provide some tough real life cases that can be mined for review and advancing the EA intellectual model.
I draw a parallel between the EA community and what I see in the foresight community (which is much older). The latter have developed an array of complex tools and ideas, (overly dependent on graphics rather than logic in some cases). They are intellectually satisfying to an elite, but some 30 years+ after the field evolved, foresight is simply not commonly used in practice. At present it remains a niche specialism – to date generally sold within a management consultancy package, or used in confidential defence wargaming.
With this type of benchmark in mind, I would argue that the value/utility of the EA models/theories/core arguments, (and perhaps even the larger forum) should be in part based on the practical or broader mindset changes that they have demonstrably driven or at least catalysed.
I have two gripes with this thought experiment. First, time is not modelled. Second, it’s left implicit why we should feel uneasy about the thought experiment. And that doesn’t work due to highly variable philosophical intuitions. I honestly don’t feel uneasy about the thought experiment at all (only slightly annoyed). But maybe I would have it been completely specified.
I can see two ways to add a time dimension to the problem. First, you could let all the presses be predetermined and in one go, where we get into Satan’s apple territory. Second, you could have 30 seconds pause between all presses. But in that case, we would accumulate massive amounts of utility in a very short time—just the seconds in-between presses would be invaluable! And who cares if the world ends in five minutes with probability 1−0.4910 when every second it survives is so sweet? :p
People should be allowed to destroy the button (aka “x-risk reduction”) ;-)
What is the “Satan’s apple” reference? I don’t get that, sorry.
I think I agree with you that the version of the example I offered did not fully specify the time dimension. But it’s crucial to the original St Petersburg game that your winnings at stage n are not collected until the game is finished, so that if you lose at stage n+1 you don’t get to enjoy your winnings. So presumably, in Hurka’s variant, the extra world(s) should not be created until after you walk away from the button, so the seconds in-between presses would not be valuable at all. But I take your point that this wasn’t specified well enough in OP.
Source: Satan, Saint Peter and Saint Petersburg
I got on a different train long ago. I am not a utilitarian; I am a contractualist who wants ‘maximize utility’ to be a larger part of the social contract (but certainly not the only one).
https://forum.effectivealtruism.org/posts/YRy48dpRbdNomnJb8/welfare-estimation-102-welfare-is-a-myth
I haven’t read enough on the topic yet, but my impression is that my train of belief would indeed be something somewhat like ‘a contractualist who wants to maximize utility’.
[I just quickly listened to the post and I’m not philosopher, nor I know deeply about Ergodicity Economics]
Maybe Ergodicity Economics (EE) [site, wikipedia] could be relevant here? It is relevant for the St. Petersburg experiment. It has to do with expected values of stochastic processes not being equal to their time averages (ensemble average ≠ time average).
I am sure there is much more to EE than this, but the one insight that I took from it when I got to know about EE is that when one of the outcomes of the game is to lose everything, expected value does not do a good job describing it. And, at least for x-risks this is exactly the case (and I consider catastrophic risks also in this domain).
It seems that EE is not very well known in the EA community, at least the term is almost not mentioned in the forum, so I thought I would mention it here in case anyone wants to dig in more. I’m for sure not the better trained to go deeper into it nor do I have the time to attempt to do it.
One post that seems to address the issue of EE within the EA is this one.
I hope this is a good lead!
Wow, this provided me with a lot of food for thought. For me personally, this was definitely one of the most simultaneously intriguing and relatable things I’ve ever read on the EA forum.
Reductionist utilitarian models are like play-dough. They’re fun and easy to work with, but useless for doing anything complicated and/or useful.
Perhaps in 100-200 years our understanding of neurobiology or psychometrics will be good enough for utilitarian modelling to become relevant to real life, but until then I don’t see any point getting on the train.
The fact that intelligent, well-meaning individuals are wasting their time thinking about the St Petersburg paradox is ironically un-utilitarian; that time could be used to accomplish tasks which actually generate wellbeing.
Re point 5: I would assume the subgroups would/could make moral trades to prevent this from happening.
I’m going to bite the bullet of absurdity, and say this already happened.
Imagine a noble/priest 500-1000 years ago trying to understand our western society, and they would likely find it absurd as well. Some norms have survived primarily due to the human baseline not changing through genetic engineering, but overall it would be weird and worrying.
For example, the idea that people are relatively equal would be absurd to a medieval noble, let alone our tolerance for outgroups/dissent.
The idea that religion isn’t an all-powerful panacea or even optional would be absurd to a priest.
The idea that there are positive sum trades that are the majority, rather than zero or negative sum trades, would again be absurd to the noble.
Science would be worrying to the noble.
And much more. In general I think people underestimate just how absurd things can get, so I’m not surprised.
The problem with this line of reasoning is that it applies to anything.
For example, suppose I have a metaethical view that says that the most important thing in life is doing a handstand. Morally, we ought to maximize the number of people who stand on their hands and we ought to maximize the length of time they do it, too. So universities are turned into handstand mills. Tax dollars are diverted from social spending and used to promote the public uptake of new handstand norms. And so on.
This view is, of course, counterintuitive in both its starting assumptions and its consequences. And that is a perfectly good reason for rejecting it. The fact that the medievals would have found modern physics and mathematics counterintuitive doesn’t change that.
So, in a similar way, it is conceivable that utilitarianism (or a related view) could have consequences so counterintuitive that they would give us grounds for rejecting utilitarianism.
I do not agree, in general. Now the exact example is incorrect more or less for humans because we don’t value it, though importantly this may not be universal in the future, but I do not generally accept this argument, especially about the long-term future.
As to why, I generally am not a fan of bullet-biting too much, but we should generally be suspicious of moralities that claim to do no wrong or have no weird or counterintuitive claims, because of the fact that we should generally be suspicious of straightforward, easy option A when that are just so clearly better than those uncomfortably costly tradeoff-y option B. This usually doesn’t exist in organizations with at least some efficiency. So that’s why bullet-biting in some cases is the best option.
(Also I have high likelihood that the long-term future in say 1,000 years will look absurd compared to us in technology and society.)
I’m not sure I understand your response to Adrian here? The claim is not that we should search for a view that has no weird or counterintuitive claims in it, only that some views might reasonably be rejected on the basis of their weird and counterintuitive claims. There might be no view that is completely un-weird, but handstandism is nonetheless obviously too weird!
Basically, my point is that coming back to my first comment is that the future can get really absurd, and arguably will.
We can pick which absurdities we choose and which to reject, but the option of a non-absurd future isn’t really on the table. It’s like claiming that gyroscopes won’t do weird things.
The arguments opening up this post, aren’t really biting or that relevant.
I made a “contest”, where I’ll write “answers” and challenge anyone to reply to me.
This seems related to this post.
I’m not sure, but my guess is that probably the post is framed/motivated in a way that makes its ideas seem much less tangential than it really is.
I think the Train to Crazytown is a result of mistaken utilitarian calculations, not an intrinsic flaw in utilitarianism. If we can’t help but make such mistakes, then perhaps utilitarianism would insist we take that risk into account when deciding whether or not to follow through on such calculations.
Take the St. Petersburg Paradox. A one-off button push has positive expected utility. But no rational gambler would take such a series of bets, even if they’re entirely motivated by money.
The Kelly criterion gives us the theoretically optimal bet size for an even money bet, which the St. Petersburg Paradox invokes (but for EV instead of money).
The Paradox proposes sizing the bet at 100% of bankroll. So to compute the proportion of the bet we’d have to win to make this an optimal bet, we plug it into the Kelly criteria and solve for B.
1 = .51 - .49/B
This gives B = −1, less than zero, so the Kelly criterion recommends not taking the bet.
The implicit utility function in Kelly (log of bankroll) amounts to rejecting additive aggregation/utilitarianism. That would be saying that doubling goodness from 100 to 200 would be of the same decision value as doubling from 100 billion to 200 billion, even though in the latter case the benefit conferred is a billion times greater.
It also absurdly says that loss goes to infinity as you go to zero. So it will reject any finite benefit of any kind to prevent even an infinitesimal chance of going to zero. If you say that the world ending has infinite disutility then of course you won’t press a button with any chance of the end of the world, but you’ll also sacrifice everything else to increment that probability downward, e.g. taking away almost everything good about the world for the last tiny slice of probability.
That’s a helpful correction/clarification, thank you!
I suppose this is why it’s important to be cautious about overapplying a particular utilitarian calculation—you (or in this case, I) might be wrong in how you’re going about it, even though the right ultimate conclusion is justified on the basis of a correct utilitarian calculus.
I don’t understand the relevance of the Kelly criterion. The wikipedia page for the Kelly criterion states that “[t]he Kelly bet size is found by maximizing the expected value of the logarithm of wealth,” but that’s not relevant here, is it?
Thanks a lot for this post. I have found it a superb piece and well worth meditating about, even if I have to say that I am probably biased because, a priori, I am not too inclined towards Utilitarianism in the first place. But I think the point you make is complex and not necessarily against consequentialism as such, and would probably go some way to accommodate the views of those who find a lot of it too alien and unpalatable.
This is way too many words trying to do morality without God. The effective altruist movement is best bracketed under the wider divine imperative of charity. Longtermism I think is best understood as a practical means for us to cooperate with the divine will that we should – and undying promise that we will – build a true Galactic Empire, as expressed in Genesis 15:4. Boom, done, ethical problems of EA philosophy solved.
FWIW, I think you might mean Genesis 15:5.
Surprisingly good take for the number of negative upvotes.