I feel like the issues with “How satisfied are you with your life, on a scale of 0 to 10?” run even a bit deeper than indicated. The series “The Americans” is about a married couple of Soviet spies living a painful and difficult life in America during the Cold War. The wife still believes in the mission, her husband not so much (but continues doing the job for the sake of his wife). Let’s say the wife rates her life satisfaction 9⁄10 because she convinced herself that she’s bravely doing highly important work. The husband rates his life satisfaction 1⁄10. They’d both score about the same in terms of more objective metrics like socioeconomic status, how much time they spend on hobbies or with their kids, how much stress they have, etc. But they interpret things differently because the wife ascribes meaning to her hardships and is proud of her accomplishments, while the husband feels trapped and like he wasted his life and endangered his children for no good reason.
Lukas_Gloor
“Between 1 and 10%” also feels surprisingly low to me for general AI-related catastrophes. I at least would have thought that experts are less optimistic than that.
But pending clarification, I wouldn’t put much weight on this estimate given that the interviews mentioned in the 80k problem area profile you link to seemed to be about informing the entire problem profile rather than this estimate specifically. So it’s not clear e.g. whether the interviews included a question about all-things-considered risk for AI-related catastrophe that was asked to Nick Bostrom, an anonymous leading professor of computer science, Jaan Tallinn, Jan Leike, Miles Brundage, Nate Soares, and Daniel Dewey.
Next to the counterpoints mentioned by Gregory Lewis, I think there is an additional reason why MCE seems less effective than more targeted interventions to improve the quality of the long-term future: Gains from trade between humans with different values become easier to implement as the reach of technology increases. As long as a non-trivial fraction of humans end up caring about animal wellbeing or digital minds, it seems likely it would be cheap for other coalitions to offer trades. So whether 10% of future people end up with an expanded moral circle or 100% may not make much of a difference to the outcome: It will be reasonably good either way if people reap the gains from trade.
One might object that it is unlikely that humans would be able to cooperate efficiently, given that we don’t see this type of cooperation happening today. However, I think it’s reasonable to assume that staying in control of technological progress beyond the AGI transition requires a degree of wisdom and foresight that is very far away from where most societal groups are at today. And if humans do stay in control, then finding a good solution for value disagreements may be the easier problem, or at worst similarly hard. So it feels to me that most likely, we either we get a future that goes badly for reasons related to lack of coordination and sophistication in the pre-AGI stage, or we get a future where humans set things up wisely enough to actually design an outcome that is nice (or at least not amongst the 10% of worst outcomes) by the lights of nearly everyone.
Brian Tomasik made the point that conditional on human values staying in control, we may be very unlikely to get something like broad moral reflection. Instead, values could be determined by a very small group of individuals who happened to be in power by the time AGI arrives (as opposed to individuals ending up there because they were unusually foresighted and also morally motivated). This feels possible too, but it seems to not be the likely default to me because I suspect that you’d need to necessarily increase your philosophical sophistication in order to stay in control of AGI, and that probably gives you more pleasant outcomes (correlational claim). Iterated amplification for instance, as an approach to AI alignment, has several uses for humans: Humans are not only where the resulting values come from, but they’re also in charge of keeping the bootstrapping process on track and corrigible. And as this post on factored cognition illustrates, this requires sophistication to set up. So if that’s the bar that AGI creators need to pass before they can determine how “human values” are to be extrapolated, maybe we shouldn’t be too pessimistic about the outcome. It seems kind of unlikely that someone would go through all of that only to be like “I’m going to implement my personal best guess about what matters to me, with little further reflection, and no other humans get a say here.” Similarly, it also feels unlikely that people would go through with all that and not find a way to make subparts of the population reasonably content about how sentient subroutines are going to be used.
Now, I feel a bit confused about the feasibility of AI alignment if you were to do it somewhat sloppily and with lower standards. I think that there’s a spectrum from “it just wouldn’t work at all and not be competitive” (and then people would have to try some other approach) to “it would produce a capable AGI but it would be vulnerable to failure modes like adversarial exploits or optimization daemons, and so it would end up with not human values”. These failure modes, to the very small degree I currently understand them, sound like they would not be sensitive to whether the human whose approval you tried to approximate had an expanded moral circle or not. I might be wrong about that. If people mostly want sophisticated alignment procedures because they care about preserving the option for philosophical reflection, rather than because they also think that you simply run into large failure modes otherwise, then it seems like (conditional on some kind of value alignment) whether we get an outcome with broad moral reflection is not so clear. If it’s technically easier to build value-aligned AI with very parochial values, then MCE could make a relevant difference to these non-reflection outcomes.
But all in all my argument is that it’s somewhat strange to assume that a group of people could succeed at building an AGI optimized for its creators’ values, without having to put in so much thinking about how to get this outcome right that they’d almost can’t help but become reasonably philosophically sophisticated in the process. And sure, philosophically sophisticated people can still have fairly strange values by your own lights, but it seems like there’s more convergence. Plus I’d at least be optimistic about their propensity to strive towards positive-sum outcomes, given how little scarcity you’d have if the transition does go well.
Of course, maybe value-alignment is going to work very differently from what people currently think. The main way I’d criticize my above points is that they’re based on heavy-handed inside-view thinking about how difficult I (and others I’m updating towards) expect the AGI transition to be. If AGI will be more like the Industrial Revolution rather than something that is even more difficult to stay remotely in control of, or if some other technology proves to be more consequential than AGI, then my argument has less force. I mainly see this as yet another reason to caveat that the ex ante plausible-seeming position that MCE can have a strong impact on AGI outcomes starts to feel more and more conjunctive the more you zoom in and try to identify concrete pathways.
Other reasons why someone competent at picking grants may not feel comfortable with the thought of having to write a report might be because writing specifically isn’t their strength or because exposing their thinking to public scrutiny might be anxiety-inducing.
There’s some related discussion here.
Lock in can also apply to “value-precursors” that determine how one goes about moral reflection, or which types of appeals one ends up finding convincing. I think these would get locked in to some degree (because something has to be fixed for it to be meaningful to talk about goalposts at all), and by affecting the precursors, moral or meta-philosophical reflection before aligned AGI can plausibly affect the outcomes post-AGI. It’s not very clear however whether that’s important, and from whose perspective it is important, because some of the things that mask as moral uncertainty might be humans having underdetermined values.
Yeah. I put it the following way in another post:
Especially when it comes to the prevention of s-risks affecting futures that otherwise contain a lot of happiness, it matters a great deal how the risk in question is being prevented. For instance, if we envision a future that is utopian in many respects except for a small portion of the population suffering because of problem x, it is in the interest of virtually all value systems to solve problem x in highly targeted ways that move probability mass towards even better futures. By contrast, only few value systems (ones that are strongly or exclusively about reducing suffering/bad things) would consider it overall good if problem x was “solved” in a way that not only prevented the suffering due to problem x, but also prevented all the happiness from the future scenario this suffering was embedded in.
So it’d be totally fine to address all sources of unnecessary suffering (and even “small” s-risks embedded in an otherwise positive future) if there are targeted ways to bring about uncontroversial improvements. :) In practice, it’s sometimes hard to find interventions that are targeted enough because affecting the future is very very difficult and we only have crude levers. Having said that, I think many things that we’re going to support with the fund are actually quite positive for positive-future-oriented value systems as well. So there certainly are some more targeted levers.
There are instances where it does feel justified to me to also move some probability mass away from s-risks towards extinction (or paperclip scenarios), but that should be reserved either for uncontroversially terrible futures, or for those futures where most of the disvalue for downside-focused value systems comes from. I doubt that this includes futures where 10^10x more people are happy than unhappy.
And of course positive-future-oriented EAs face analogous tradeoffs of cooperation with other value systems.
At least for me, this would be a pretty amazing outcome, and not something which should be prevented.
Yeah, we’re going to change the part that equates “worst case” with “s-risks”. Your view is common and reflects many ethical perspectives.
We were already thinking about changing the definition of “s-risk” based on similar feedback, to make it more intuitive and cooperative in the way you describe. It probably makes more sense to have it refer to only the few % of scenarios where most of the future’s expected suffering comes from (assuming s-risks are indeed heavy-tailed). These actual worst cases are what we want to focus on with the fund.
do I interpret it correctly that in the ethical system held by the fund human extinction is comparatively benign outcome in comparison with risks like creation of 10^25 unhappy minds even if they are offset by much larger [say 10^10x larger] number of happy minds?
No, that’s incorrect. Insofar as some fund managers hold this view personally (e.g., I do, while Jonas would agree with you that the latter outcome is vastly better), it won’t affect decisions because in any case, we want to avoid doing things that are weakly positive on some plausible moral views and very negative on others. But I can see why you were concerned, and thanks for raising this issue!
To turn the tables a bit here, I would say that to reject moral realism, on my account, one would need to say that there is no genuine normative force or property in, say, a state of extreme suffering (consider being fried in a brazen bull for concreteness).
Cool! I think the closest I’ll come to discussing this view is in footnote 18. I plan to have a post on moral realism via introspection about the intrinsic goodness (or badness) of certain conscious states.
I agree with reductionism about personal identity and I also find this to be one of the most persuasive arguments in favor of altruistic life goals. I would not call myself an open indvidualist though because I’m not sure what the position is exactly saying. For instance, I don’t understand how it differs from empty individualism. I’d understand if these are different framings or different metaphores, but if we assume that we’re talking about positions that can be true or false, I don’t understand what we’re arguing about when asking whether open individualism or true, or when discussing open vs. empty individualism.
Also, I think it’s perfectly coherent to have egoistic goals even under a reductionist view of personal identity. (It just turns out that egoism is not a well-defined concept either, and one has to make some judgment calls if one ever expects to encounter edge-cases for which our intuitions give no obvious answers about whether something is still “me.“)With respect to the One Compelling Axiology you mention, Lukas, I am not sure why you would set the bar so high in terms of specificity in order to accept a realist view. I mean, if “all philosophers or philosophically-inclined reasoners” found plausible a simple, yet inexhaustive principle like “reduce unnecessary suffering” why would that not be good enough to demonstrate its “realism” (on your account) when a more specific one would? It is unclear to me why greater specificity should be important, especially since even such an unspecific principle still would have plenty of practical relevance (many people can admit that they are not living in accordance with this principle, even as they do accept it).
Yeah, fair point. I mean, even Railton’s own view has plenty of practical relevance in the sense that it highlights that certain societal arrangements lead to more overall well-being or life satisfaction than others. (That’s also a point that Sam Harris makes.) But if that’s all we mean by “moral realism” then it would be rather trivial. Maybe my criteria are a bit too strict, and I would indeed already regard it as extremely surprising if you get something like One Compelling Axiology that agrees on population ethics while leaving a few other things underdetermined.
Inspired by another message of yours, there’s at least one important link here that I failed to mention: If moral discourse is about a, b, and c, and philosophers then say they want to make it about q and argue for realism about q, we can object that whatever they may have shown us regarding realism about q, it’s certainly not moral realism. And it looks like the Loeb paper also argues that if moral discourse is about mutually incompatible things, that looks quite bad for moral realism? Those are good points!
Do you think your argument also works against Railton’s moral naturalism, or does my One Compelling Axiology (OTA) proposal introduce something that breaks the idea? The way I meant it, OTA is just a more extreme version of Railton’s view.
I think I can see what you’re pointing to though. I wrote:
Note that this proposal makes no claims about the linguistic level: I’m not saying that ordinary moral discourse let’s us define morality as convergence in people’s moral views after philosophical reflection under ideal conditions. (This would be a circular definition.) Instead, I am focusing on the aspect that such convergence would be practically relevant: [...]
So yes, this would be a bad proposal for what moral discourse is about. But it’s meant like this: Railton claims that morality is about doing things that are “good for others from an impartial perspective.” I like this and wanted to work with that, so I adopt this assumption, and further add that I only want to call a view moral realism if “doing what is good for others from an impartial perspective” is well-specified. Then I give some account of what it would mean for it to be well-specified.
In my proposal, moral facts are not defined as that which people arrive at after reflection. Moral facts are still defined as the same thing Railton means. I’m just adding that maybe there are no moral facts in the way Railton means if we introduce the additional requirement that (strong) underdetermination is not allowed.
Probably intuitions about this issue depend on which type of moral or religious discourse one is used to. As someone who spent a year at a Christian private school in Texas where creationism was taught in Biology class and God and Jesus were a very tangible part of at least some people’s lives, I definitely got a strong sense that the metaphysical questions are extremely important.
By contrast, if the only type of religious claims I’d ever came into contact with had been moderate (picture the average level of religiosity of a person in, say, Zurich), then one may even consider it a bit of a strawman to assume that religious claims are to be taken literally.
I think this concern is somewhat relevant to the broader discussion, too, because you seem to imply that we can’t (or even shouldn’t?) make any advances on non-metaphysical claims before we haven’t figured out the metaphysical ones.
Just to be clear, all I’m saying is that I think it’s going to be less useful to discuss “what are moral claims usually about.” What we should instead do is instead what Chalmers describes (see the quote in footnote 4). Discussing what moral claims are usually about is not the same as making up one’s mind about normative ethics. I think it’s very useful to discuss normative ethics, and I’d even say that discussing whether anti-realism or realism is true might be slightly less important than making up one’s mind about normative ethics. Sure, it informs to some extent how to reason about morality, but as has been pointed out, you can make some progress about moral questions also from a lens of agnosticism about realism vs. anti-realism.
To go back to the religion analogy, what I’m recommending is to first figure out whether you believe in a God or an afterlife that would relevantly influence your priorities now, and not worry much about whether religious claims are “usually” or “best” to be taken literally or taken metaphorically(?).
Why should we care whether or not moral realism is true?
I plan to address this more in a future post, but the short answer is this that for some ways in which moral realism has been defined, it really doesn’t matter (much). But there are some versions of moral realism that would “change the game” for those people who currently reject them. And vice-versa, if one currently endorses a view that corresponds to the two versions of “strong moral realism” described in the last section of my post, one’s priorities could change noticeably if one changes one’s mind towards anti-realism.
What do you think are the implications of moral anti-realism for choosing altruistic activities?
It’s hard to summarize this succinctly because for most of the things that are straightforwardly important under moral realism (such as moral uncertainty or deferring judgment to future people who are more knowledgeable about morality), you can also make good arguments in favor of them going from anti-realist premises. Some quick thoughts:
– The main difference is that things become more “messy” with anti-realism.
– I think anti-realists should, all else equal, be more reluctant to engage in “bullet biting” where you abandon some of your moral intuitions in favor of making your moral view “simpler” or “more elegant.” The simplicity/elegance appeal is that if you have a view with many parameters that are fine-tuned for your personal intuitions, it seems extremely unlikely that other people would come up with the same parameters if they only thought about morality more. Moral realists may think that the correct answer to morality is one that everyone who is knowledgeable enough would endorse, whereas anti-realists may consider this a potentially impossible demand and therefore place more weight on finding something that feels very intuitively compelling on the individual level. Having said that, I think there are a number of arguments why even an anti-realist might want to adopt moral views that are “simple and elegant.” For instance, people may care about doing something meaningful that is “greater than their own petty little intuitions” – I think this is an intuition that we can try to cash out somehow even if moral realism turns out to be false (it’s just that it can be cashed out in different ways).
– “Moral uncertainty” works differently under anti-realism, because you have to say what you are uncertain about (it cannot be the one true morality because anti-realism says there is no such thing). One can be uncertain about what one would value after moral reflection under ideal conditions. This kind of “valuing moral reflection” seems like a very useful anti-realist alternative to moral uncertainty. The difference is that “valuing reflection” may be underdefined, so anti-realists have to think about how to distinguish having underdefined values from being uncertain about their values. This part can get tricky.
– There was recently a discussion about “goal drift” in the EA forum. I think it’s a bigger problem with anti-realism all else equal (unless one’s anti-realist moral view is egoism-related.) But again, there are considerations that go into both directions. :)
The idea is that morality is the set of rules that impartial, rational people would advocate as a public system.
Yes, this sounds like constructivism. I think this is definitely a useful framework for thinking about some moral/morality-related questions. I don’t think all of moral discourse is best construed as being about this type of hypothetical rule-making, but like I say in the post, I don’t think interpreting moral discourse should be the primary focus.
Rationality is understood, roughly speaking, as the set of things that virtually all rational agents would be averse to. This ends up being a list of basic harms—things like pain, death, disability, injury, loss of freedom, loss of pleasure.
Hm, this sounds like you’re talking about a substantive concept of rationality, as opposed to a merely “procedural” or “instrumental” concept of rationality (such as it’s common on Lesswrong and with anti-realist philosophers like Bernard Williams). Substantive concepts of rationally always go under moral non-naturalism, I think.
My post is a little confusing with respect to the distinction here, because you can be a constructivist in two different ways: Primarily as an intersubjectivist metaethical position, and “secondarily” as a form of non-naturalism. (See my comments on Thomas Sittler’s chart.)
People can be incorrect about whether a thing is harmful, just as they can be incorrect about whether a flower is red. But there’s nothing much more objective or “facty” about whether the plant is red than that ordinary human language users on earth are disposed to see and label it as red.
Yeah, it should be noted that “strong” versions of moral realism are not committed to silly views such as morality existing in some kind of supernatural realm. I often find it difficult to explain moral non-naturalism in a way that makes it sound as non-weird as when actual moral non-naturalists write about it, so I have to be careful to not strawman these positions. But what you describe may still qualify as “strong” because you’re talking about rationality as a substantive concept. (Classifying something as a “harm” is one thing if done in a descriptive sense, but probably you’re talking about classifying things as a harm in a sense that has moral connotations – and that gets into more controversial territory.)
The book title “normative bedrock” also sounds relevant because my next post will talk about “bedrock concepts” (Chalmers) at length, and specifically about “irreducible normativity” as a bedrock concept, which I think makes up the core of moral non-naturalism.
Cool! I think this is helpful both in itself as well as here as a complement to my post. I also thought about making a chart but was too lazy in the end. If I may, I’ll add some comments about how this chart relates to the distinctions I made/kept/found:
“Judgment-dependent cognitivism” corresponds to what I labelled subjectivism and intersubjectivism, and “judgment-_in_dependent cognitivism” corresponds to “objectivism.” (Terminology adopted from the Sayre-McCord essay; but see the Scanlon essay for the other terminology.)
I’m guessing “Kantian rationalism” refers to views such as Scanlon’s view. I’m didn’t go into detail in my post with explaining the difference between constructivism as an intersubjectivist position and constructivism as a version of non-naturalism. I tried to say something about that in footnote 7 but I fear it’ll only become more clear in my next post. Tl;dr is that non-naturalists think that we can have externalist reasons for doing something, reasons we cannot “shake off” by lacking internal buy-in or internal motivation. By contrast, someone who merely endorses constructivism as an intersubjectivist (or “judgment-dependent”) view, such as Korsgaard for instance, would reject these externalist reasons.
I agree with the way you draw the lines between the realist and the anti-realist camp. The only thing I don’t like about this (and this is a criticism not about your chart, but about the way philosophers have drawn these categories in the first place) is that it makes it seem as though we have to choose exactly one view. But by removing the entire discussion from the “linguistic level” (taking a stance on how we interpret moral discourse), we can acknowledge e.g. that subjectivism or intersubjectivism represent useful frameworks for thinking about morality-related questions, whether moral discourse is completely subjectivist or intersubjectivist in nature or not. And even if moral discourse was all subjectivist (which seems clearly wrong to me but let’s say it’s a hypothetical), for all we’d know that could still allow for the possibility that an objectivist moral reality exists in a meaningful and possibly action-relevant sense. I like Luke’s framing of “pluralistic moral reductionism” because it makes clear that there is more than one option.
1. What Is Moral Realism?
Talent gap—Middle (~50 people)
If the AI safety/alignment community is altogether around 50 people, that’s a large relative gap. Depending on how you count it might be bigger than 50 people, but the talent gap seems large in relative terms in either case. :)
It is much less clear whether this person would think they’ve made a mistake in allocating more of themselves away from EA, either at t2-now (they don’t regret they now have a family which takes their attention away from EA things), or at t1-past (if their previous EA-self could forecast them being in this situation, they would not be disappointed in themselves). If so, these would not be options that their t1-self should be trying to shut off, as (all things considered) the option might be on balance good. I am sure there are cases where ‘life gets in the way’ in a manner it is reasonable to regret. But I would be chary [...]
You discuss a case where there is regret from the perspective of both t1 and t2, and a case where there is regret from neither perspective. These are both plausible accounts. But there’s also a third option that I think happens a lot in practice: Regret at t1 about the projected future in question, and less/no regret at t2. So the t2-self may talk about “having become more wise” or “having learned something about myself,” while the t1-self would not be on board with this description and consider the future in question to be an unfortunate turn of events. (Or the t2-self could even acknowledge that some decisions in the sequence were not rational, but that from their current perspective, they really like the way things are.)
The distinction between moral insight and failure of goal preservation is fuzzy. Taking precautions against goal drift is a form of fanaticism and commonsense heuristics speak against that. OTOH, not taking precautions seems like not taking the things you currently care about seriously (at least insofar as there are things you care about that go beyond aspects related to your personal development).
Unfortunately I don’t think there is a safe default. Not taking precautions is tantamount to making the decision to be okay with potential value drift. And we cannot just say we are uncertain about our values, because that could result in mistaking uncertainty with underdetermination. There are meaningful ways of valuing further reflection about one’s own values, but those types of “indirect” values, where one values further reflection, they can also suffer from (more subtle) forms of goal drift.
There isn’t much more except that I got the impression that people in EA who have thought about this a lot think recovery is very likely, and I’m mostly deferring to them. The section about extinction risk is the part of my post where I feel the least knowledgeable. As for additional object-level arguments, I initially wasn’t aware of points such as crops and animals already being cultivated/domesticated, metals already mined, and there being alternatives to rapid growth induced by fossil fuels, one of which being slow but steady growth over longer time periods. The way cultural evolution works is that slight improvements from innovations (which are allowed to be disjunctive rather than having to rely on developing a very specific technology) spread everywhere, which makes me think that large populations + a lot of time should go far enough eventually. Note also that if all-out extinction is simply very unlikely to ever happen, then you have several attempts left to reach technological maturity again.
I think that there’s an inevitable tradeoff between wanting a reflection process to have certain properties and worries about this violating goal preservation for at least some people. This blogpost is not about MCE directly, but if you think of “BAAN thought experiment” as “we do moral reflection and the outcome is such a wide circle that most people think it is extremely counterintuitive” then the reasoning in large parts of the blogpost should apply perfectly to the discussion here.
That is not to say that trying to fine tune reflection processes is pointless: I think it’s very important to think about what our desiderata should be for a CEV-like reflection process. I’m just saying that there will be tradeoffs between certain commonly mentioned desiderata that people don’t realize are there because they think there is such a thing as “genuinely free and open-ended deliberation.”
The full analysis on AI impacts is also really clearly explained, or at least I got that impression when I was reading it (I normally can’t follow these types of protocols because I get bored or lose track of the relevance of different things). I really enjoyed reading it!