Position statement: I chose 36% disagreement. AMA!
My view is that Earth-originating civilisation, if we become spacefaring, will attain around 0.0001% of all value. This still makes extinction risk astronomically valuable (it’s equivalent to optimising a millionth of the whole cosmos!), but if we could increase the chance of optimising 1% of the universe by 1%, this would be 100x more valuable than avoiding extinction. (You’re not going to get an extremely well grounded explanation of these numbers from me, but I hope they make my position clearer.)
My view is that if Earth-originating civilisation becomes spacefaring, over the long term we will settle the entire universe and use nearly all of its energy. However, we could use this energy for many different things. I expect that by default we will mostly use this energy to create copies of ourselves (human brains) or digital slaves that allow us to learn and achieve new things, since humans love nothing more than humans and Earthly things. But these are inefficient media for realising moral value. We could instead use this energy to realise vastly more value, as I argue in Power Laws of Value. But our descendants basically won’t care at all about doing this. So I expect that we will miss out on nearly all value, despite using most of the universe’s energy.
I think that there will be significant path dependency in what Earth-originating civilisation chooses to make with our cosmic endowment. In particular, I expect that artificial intelligence will drive most growth and most choices after the 21st century. Several factors, such as the number of distinct agents, their values, and the processes by which their values evolve over time will therefore make a decisive difference to what our descendants choose to do with the cosmos.
So our choices about AI governance and architecture today are likely to make a significant and, at least on some paths we can choose, predictable difference to what our descendants do with all of the energy in the universe. If we do this well, it could make the difference between attaining almost no value and nearly all value.
Given that I expect us to miss out on almost all value by default, I view the value of avoiding extinction as smaller than those who think we will achieve almost all value by default.
My view is that Earth-originating civilisation, if we become spacefaring, will attain around 0.0001% of all value.
So you think: 1. People with your values control 1 in 1 millionth of future resources, or less? This seems pessimistic!
2. But maybe you think it’s just you who has your values and everyone else would converge on something subtly different—different enough to result in the loss of essentially all value. Then the 1-in-1-million would no longer seem so pessimistic.
But if so, then suppose I’m Galactic Emperor and about to turn everything into X, best by my lights… do you really take a 99.9% chance of extinction, and a 0.1% chance of stuff optimised by you, instead?
3. And if so, do you think that Tyler-now has different values than Tyler-2026? Or are you worried that he might have slightly different values, such that you should be trying to bind yourself to the mast in various ways?
4. Having such a low v(future) feels hard to maintain in light of model uncertainty and moral uncertainty.
E.g. what’s the probability you have that: i. People in general just converge on what’s right? ii. People don’t converge, but a significant enough fraction converge with you that you and others end up with more than 1milllionth of resources? iii. You are able to get most of what you want via trade with others?
Thank you, Will, excellent questions. And thanks for drawing out all of the implications here. Yeah I’m a super duper bullet biter. Age hasn’t dulled my moral senses like it has yours! xP
2. But maybe you think it’s just you who has your values and everyone else would converge on something subtly different—different enough to result in the loss of essentially all value. Then the 1-in-1-million would no longer seem so pessimistic.
Yes, I take (2) on the 1 vs 2 horn. I think I’m the only person who has my exact values. Maybe there’s someone else in the world, but not more than a handful at most. This is because I think our descendants will have to make razor-thin choices in computational space about what matters and how much, and these choices will amount to Power Laws of Value.
But if so, then suppose I’m Galactic Emperor and about to turn everything into X, best by my lights… do you really take a 99.9% chance of extinction, and a 0.1% chance of stuff optimised by you, instead?
I generally like your values quite a bit, but you’ve just admitted that you’re highly scope insensitive. So even if we valued the same matter equally as much, depending on the empirical facts it looks like I should value my own judgment potentially nonillions as much as yours, just on scope sensitivity grounds alone!
3. And if so, do you think that Tyler-now has different values than Tyler-2026? Or are you worried that he might have slightly different values, such that you should be trying to bind yourself to the mast in various ways?
Yup, I am worried about this and I am not doing much about it. I’m worried that the best thing that I could do would simply be to go into cryopreservation right now and hope that my brain is uploaded as a logically omniscient emulation with its values fully locked in and extrapolated. But I’m not super excited about making that sacrifice. Any tips on ways to tie myself to the mast?
what’s the probability you have that: i. People in general just converge on what’s right?
It would be something like: P(people converge on my exact tastes without me forcing them to) + [P(kind of moral or theistic realism I don’t understand)*P(the initial conditions are such that this convergence happens)*P(it happens quickly enough before other values are locked in)*P(people are very motivated by these values)]. To hazard an-off-the cuff guess, maybe 10^-8 + 10^-4*0.2*0.3*0.4, or about 2.4*10^-6.
ii. People don’t converge, but a significant enough fraction converge with you that you and others end up with more than 1milllionth of resources?
I should be more humble about this. Maybe it turns out there just aren’t that many free parameters on moral value once you’re a certain kind of hedonistic consequentialist who knows the empirical facts and those people kind of converge to the same things. Suppose that’s 1⁄30 odds vs my “it could be anything” modal view. Then suppose 1⁄20 elites become that kind of hedonistic consequentialist upon deliberation. Then it looks like we control 1/600th of the resources. I’m just making these numbers up, but hopefully they illustrate that this is a useful push that makes me a bit less pessimistic.
iii. You are able to get most of what you want via trade with others?
Maybe 1⁄20 that we do get to a suitably ideal kind of trade. I believe what I want is a pretty rivalrous good, i.e. stars, so at the advent of ideal trade I still won’t get very much of what I want. But it’s worth thinking about whether I could get most of what I want in other ways, such as by trading with digital slave-owners to make their slaves extremely happy, in a relatively non-rivalrous way.
I don’t have a clear view on this and think further reflection on this could change my views a lot.
Thanks! I appreciate you clarifying this, and for being clear about it. Views along these lines are what I always expect subjectivists to have, and they never do, and then I feel confused.
Thank you! IMO the best argument for subjectivists not having these views would be thinking that (1) humans generally value reasoning processes, (2) there are not that many different reasoning processes you could adopt or as a matter of biological or social fact we all value roughly the same reasoning processes, and (3) these processes have clear and determinate implications. Or, in short, Kant was right: if we reason from the standpoint of “reason”, which is some well-defined and unified thing that we all care about, we all end up in the same place. But I reject all of these premises.
The other argument is that our values are only determinate over Earthly things we are familiar with in our ancestral environment, and among Earthly things we empirically all kinda care about the same things. (I discuss this a bit here.)
How do these considerations affect what you are doing / spending resources on? Does it change the calculus if extinction is likely to happen sooner? (see also comment here).
Part of what it means that I try to support thinking on this issue, e.g. by seed-funding NYU MEP and doing this discussion, and doing my own thinking on it.
At this stage the thing I’m most excited about supporting is market-based mechanisms for democratic AI alignment like this. Also excited about trying to get more resources to work on AI welfare, utilitarianism, and to groups like Forethought: A new AI macrostrategy group.
In practice I spend more resources on extinction risk reduction. Part of this is just because I’d really prefer not to die in my 30s. When an EA cares for their family taking away time from extinction risk they’re valuing their family as much as 10^N people. I see myself as doing something similar here.
When an EA cares for their family taking away time from extinction risk they’re valuing their family as much as 10^N people.
No. I’ve said this before elsewhere, and it’s not directly relevant to most of this discussion, but I think it’s very worth reinforcing; EA is not utilitarianism, and the commitment to EA does not imply that you have any obligatory trade-off between yourself or your family’s welfare and your EA commitment. If, as is the generally accepted standard, a “normal” EA commitment is of 10% of your income and/or resources, it seems bad to suggest that such an EA should not ideally spend the other 90% of their time/effort on personal things like their family.
(Note that in addition to being a digression, this is a deontological rather than decision-theoretic point.)
When an EA cares for their family taking away time from extinction risk they’re valuing their family as much as 10^N people.
Not sure exactly what you mean here—do you mean attending to family matters (looking after family) taking away time from working on extinction risk reduction?
Position statement: I chose 36% disagreement. AMA!
My view is that Earth-originating civilisation, if we become spacefaring, will attain around 0.0001% of all value. This still makes extinction risk astronomically valuable (it’s equivalent to optimising a millionth of the whole cosmos!), but if we could increase the chance of optimising 1% of the universe by 1%, this would be 100x more valuable than avoiding extinction. (You’re not going to get an extremely well grounded explanation of these numbers from me, but I hope they make my position clearer.)
My view is that if Earth-originating civilisation becomes spacefaring, over the long term we will settle the entire universe and use nearly all of its energy. However, we could use this energy for many different things. I expect that by default we will mostly use this energy to create copies of ourselves (human brains) or digital slaves that allow us to learn and achieve new things, since humans love nothing more than humans and Earthly things. But these are inefficient media for realising moral value. We could instead use this energy to realise vastly more value, as I argue in Power Laws of Value. But our descendants basically won’t care at all about doing this. So I expect that we will miss out on nearly all value, despite using most of the universe’s energy.
I think that there will be significant path dependency in what Earth-originating civilisation chooses to make with our cosmic endowment. In particular, I expect that artificial intelligence will drive most growth and most choices after the 21st century. Several factors, such as the number of distinct agents, their values, and the processes by which their values evolve over time will therefore make a decisive difference to what our descendants choose to do with the cosmos.
So our choices about AI governance and architecture today are likely to make a significant and, at least on some paths we can choose, predictable difference to what our descendants do with all of the energy in the universe. If we do this well, it could make the difference between attaining almost no value and nearly all value.
Given that I expect us to miss out on almost all value by default, I view the value of avoiding extinction as smaller than those who think we will achieve almost all value by default.
So you think:
1. People with your values control 1 in 1 millionth of future resources, or less? This seems pessimistic!
2. But maybe you think it’s just you who has your values and everyone else would converge on something subtly different—different enough to result in the loss of essentially all value. Then the 1-in-1-million would no longer seem so pessimistic.
But if so, then suppose I’m Galactic Emperor and about to turn everything into X, best by my lights… do you really take a 99.9% chance of extinction, and a 0.1% chance of stuff optimised by you, instead?
3. And if so, do you think that Tyler-now has different values than Tyler-2026? Or are you worried that he might have slightly different values, such that you should be trying to bind yourself to the mast in various ways?
4. Having such a low v(future) feels hard to maintain in light of model uncertainty and moral uncertainty.
E.g. what’s the probability you have that:
i. People in general just converge on what’s right?
ii. People don’t converge, but a significant enough fraction converge with you that you and others end up with more than 1milllionth of resources?
iii. You are able to get most of what you want via trade with others?
Thank you, Will, excellent questions. And thanks for drawing out all of the implications here. Yeah I’m a super duper bullet biter. Age hasn’t dulled my moral senses like it has yours! xP
Yes, I take (2) on the 1 vs 2 horn. I think I’m the only person who has my exact values. Maybe there’s someone else in the world, but not more than a handful at most. This is because I think our descendants will have to make razor-thin choices in computational space about what matters and how much, and these choices will amount to Power Laws of Value.
I generally like your values quite a bit, but you’ve just admitted that you’re highly scope insensitive. So even if we valued the same matter equally as much, depending on the empirical facts it looks like I should value my own judgment potentially nonillions as much as yours, just on scope sensitivity grounds alone!
Yup, I am worried about this and I am not doing much about it. I’m worried that the best thing that I could do would simply be to go into cryopreservation right now and hope that my brain is uploaded as a logically omniscient emulation with its values fully locked in and extrapolated. But I’m not super excited about making that sacrifice. Any tips on ways to tie myself to the mast?
It would be something like: P(people converge on my exact tastes without me forcing them to) + [P(kind of moral or theistic realism I don’t understand)*P(the initial conditions are such that this convergence happens)*P(it happens quickly enough before other values are locked in)*P(people are very motivated by these values)]. To hazard an-off-the cuff guess, maybe 10^-8 + 10^-4*0.2*0.3*0.4, or about 2.4*10^-6.
I should be more humble about this. Maybe it turns out there just aren’t that many free parameters on moral value once you’re a certain kind of hedonistic consequentialist who knows the empirical facts and those people kind of converge to the same things. Suppose that’s 1⁄30 odds vs my “it could be anything” modal view. Then suppose 1⁄20 elites become that kind of hedonistic consequentialist upon deliberation. Then it looks like we control 1/600th of the resources. I’m just making these numbers up, but hopefully they illustrate that this is a useful push that makes me a bit less pessimistic.
Maybe 1⁄20 that we do get to a suitably ideal kind of trade. I believe what I want is a pretty rivalrous good, i.e. stars, so at the advent of ideal trade I still won’t get very much of what I want. But it’s worth thinking about whether I could get most of what I want in other ways, such as by trading with digital slave-owners to make their slaves extremely happy, in a relatively non-rivalrous way.
I don’t have a clear view on this and think further reflection on this could change my views a lot.
Thanks! I appreciate you clarifying this, and for being clear about it. Views along these lines are what I always expect subjectivists to have, and they never do, and then I feel confused.
Thank you! IMO the best argument for subjectivists not having these views would be thinking that (1) humans generally value reasoning processes, (2) there are not that many different reasoning processes you could adopt or as a matter of biological or social fact we all value roughly the same reasoning processes, and (3) these processes have clear and determinate implications. Or, in short, Kant was right: if we reason from the standpoint of “reason”, which is some well-defined and unified thing that we all care about, we all end up in the same place. But I reject all of these premises.
The other argument is that our values are only determinate over Earthly things we are familiar with in our ancestral environment, and among Earthly things we empirically all kinda care about the same things. (I discuss this a bit here.)
How do these considerations affect what you are doing / spending resources on? Does it change the calculus if extinction is likely to happen sooner? (see also comment here).
Part of what it means that I try to support thinking on this issue, e.g. by seed-funding NYU MEP and doing this discussion, and doing my own thinking on it.
At this stage the thing I’m most excited about supporting is market-based mechanisms for democratic AI alignment like this. Also excited about trying to get more resources to work on AI welfare, utilitarianism, and to groups like Forethought: A new AI macrostrategy group.
In practice I spend more resources on extinction risk reduction. Part of this is just because I’d really prefer not to die in my 30s. When an EA cares for their family taking away time from extinction risk they’re valuing their family as much as 10^N people. I see myself as doing something similar here.
Thanks for saying this. I feel likewise (but s/30s/40s :))
No. I’ve said this before elsewhere, and it’s not directly relevant to most of this discussion, but I think it’s very worth reinforcing; EA is not utilitarianism, and the commitment to EA does not imply that you have any obligatory trade-off between yourself or your family’s welfare and your EA commitment. If, as is the generally accepted standard, a “normal” EA commitment is of 10% of your income and/or resources, it seems bad to suggest that such an EA should not ideally spend the other 90% of their time/effort on personal things like their family.
(Note that in addition to being a digression, this is a deontological rather than decision-theoretic point.)
Not sure exactly what you mean here—do you mean attending to family matters (looking after family) taking away time from working on extinction risk reduction?
Yes. Which, at least on optimistic assumptions, means sacrificing lots of lives.
Fair point. But this applies to a lot of things in EA. We give what we can.