Nice! Consolidating some comments I had on a draft of this piece, many of them fairly pedantic:
Why would value be disributed over some suitable measure of world-states in a way that can be described as a power law specifically (vs some other functional form where the most valuable states are rare)? In particular, shouldn’t we think that there is a most valuable world state (or states)? So at least we need to say it’s a power law distribution with a max value.
“Then there is a powerful argument that the expected value of the future is very low.”
Very low as a fraction of the best futures, but not any lower relative to (e.g.) the world today, or the EV of the future. Indeed the future could be amazing by all existing measures. One framing on what you are saying is that it could be even better than we think, which is not a pessimistic result!
Another framing is more like “it’s easier than we thought to make amazing-seeming worlds far less valuable than they seem, by making mistakes like e.g. ignoring animal farming”. That is indeed bad news.
And of course the decision-relevance of MPL depends on the feasibility of the best futures, not just how they’re distributed.
One possibility that would support MPL is the possibility that value scales superlinearly with the amount of “optimised matter” — e.g. with brain size. The task of a risk-neutral classical utilitarian can then effectively be boiled down to “maximising the chance of getting ~the most optimized state possible”, as long as “~the most optimized state possible” is at all feasible.
“If you value pretty much anything (e.g. consciousness, desire satisfaction), there’s likely to be a sharp line in phase space where a tiny change to the property makes an all-or-nothing difference to value.” — this is true, but that the [arrangements of matter] → [value] function has discontinuities doesn’t imply that the very most valuable states are extremely rare and far more valuable than all the others. So I think it’s weak evidence for MPL.
Some distinctions which occur to me below. Assuming some measure over states:
EV of the world at a state, vs value of a state itself (where EV cares about future states)
Note that the [state]→[EV] function should be discontinuous, because the evolution of states over time is discontinuous, because that’s how the world works! E.g. in some cases you should evaluate a big difference in EV just by changing a vote counter after an election by one.
Fragility of value/EV, something like how much value tends to change with small changes in space of states
Rarity of value, something like what fraction of all states are >50% as valuable as the most valuable state(s)
Unity of value, something like whether all the most valuable states are clumped together in state space, or whether the ‘peaks’ of the landscape are far apart and separated by valleys of zero or negative value
I think it’s a true and important point that people currently converge on states they agree are high EV, because the option space is limited, and most good we value are still scarce instrumental goods — but when the option space grows, the latent disagreement becomes more important.
“I don’t know if my folk history is accurate, but my understanding is that early religions and cultures had a lot in common with each other”
I guess it depends on how you interpret ~indexical beliefs like “I want my group to win and the group over the hill to lose” — both sides can think that same thing, but might hate any compromise solution.
I think this is a reason for pessism about different values agreeing on the same states, and ∴ supportive of MPL.
Re brains, there are some (weak) reasons to expect finite optimal sizes, like speed of light. A ‘Jupiter brain’ is not very different from many smaller brains with high bandwidth (but laggy) communication.
I doubt how rare near-best futures are among desired futures is a strong guide to the expected value of the future. At least, you need to know more about e.g. the feasibility of near-best futures; whether deliberative processes and scientific progress converge on an understanding of which futures are near-best, etc.
There is an analogous argument which says: “most goals in the space of goals are bad and lead to AI scheming; AI will ~randomly initialise on a goal; so AI will probably scheme”. But obviously whenever we make things by design (like cars or whatever), we are creating things which are astronomically unlikely configurations in the “space of ways to organise matter”. And the likelihood that humans build cars just doesn’t have much to do with what fraction of matter state space they occupy. It’s just not an illuminating frame. The more interesting stuff is “will humans choose to make them”, and “how easy are they to make”. (I think Ben Garfinkel has made roughly this point, as has Joe Carlsmith more recently.)
These are excellent comments, and unfortunately they all have the virtue of being perspicuous and true so I don’t have that much to say about them.
I doubt how rare near-best futures are among desired futures is a strong guide to the expected value of the future. At least, you need to know more about e.g. the feasibility of near-best futures; whether deliberative processes and scientific progress converge on an understanding of which futures are near-best, etc.
Is the core idea here that human desires and the values people reach on deliberation come apart? That makes sense, though it also leaves open how much deliberation our descendants will actually do / how much their values will be based on a deliberating process. I guess I’ll just state my view without defending it that after a decade in philosophy I have become pretty pessimistic about convergence happening through deliberation rather than more divergence as more choice points are uncovered and reasoners either think they have a good loss function or just choose not to do backpropagation.
Nice! Consolidating some comments I had on a draft of this piece, many of them fairly pedantic:
Why would value be disributed over some suitable measure of world-states in a way that can be described as a power law specifically (vs some other functional form where the most valuable states are rare)? In particular, shouldn’t we think that there is a most valuable world state (or states)? So at least we need to say it’s a power law distribution with a max value.
“Then there is a powerful argument that the expected value of the future is very low.”
Very low as a fraction of the best futures, but not any lower relative to (e.g.) the world today, or the EV of the future. Indeed the future could be amazing by all existing measures. One framing on what you are saying is that it could be even better than we think, which is not a pessimistic result!
Another framing is more like “it’s easier than we thought to make amazing-seeming worlds far less valuable than they seem, by making mistakes like e.g. ignoring animal farming”. That is indeed bad news.
And of course the decision-relevance of MPL depends on the feasibility of the best futures, not just how they’re distributed.
One possibility that would support MPL is the possibility that value scales superlinearly with the amount of “optimised matter” — e.g. with brain size. The task of a risk-neutral classical utilitarian can then effectively be boiled down to “maximising the chance of getting ~the most optimized state possible”, as long as “~the most optimized state possible” is at all feasible.
“If you value pretty much anything (e.g. consciousness, desire satisfaction), there’s likely to be a sharp line in phase space where a tiny change to the property makes an all-or-nothing difference to value.” — this is true, but that the [arrangements of matter] → [value] function has discontinuities doesn’t imply that the very most valuable states are extremely rare and far more valuable than all the others. So I think it’s weak evidence for MPL.
Some distinctions which occur to me below. Assuming some measure over states:
EV of the world at a state, vs value of a state itself (where EV cares about future states)
Note that the [state]→[EV] function should be discontinuous, because the evolution of states over time is discontinuous, because that’s how the world works! E.g. in some cases you should evaluate a big difference in EV just by changing a vote counter after an election by one.
Fragility of value/EV, something like how much value tends to change with small changes in space of states
Rarity of value, something like what fraction of all states are >50% as valuable as the most valuable state(s)
Unity of value, something like whether all the most valuable states are clumped together in state space, or whether the ‘peaks’ of the landscape are far apart and separated by valleys of zero or negative value
I think it’s a true and important point that people currently converge on states they agree are high EV, because the option space is limited, and most good we value are still scarce instrumental goods — but when the option space grows, the latent disagreement becomes more important.
See: The Tails Coming Apart As Metaphor For Life
“I don’t know if my folk history is accurate, but my understanding is that early religions and cultures had a lot in common with each other”
I guess it depends on how you interpret ~indexical beliefs like “I want my group to win and the group over the hill to lose” — both sides can think that same thing, but might hate any compromise solution.
I think this is a reason for pessism about different values agreeing on the same states, and ∴ supportive of MPL.
Re brains, there are some (weak) reasons to expect finite optimal sizes, like speed of light. A ‘Jupiter brain’ is not very different from many smaller brains with high bandwidth (but laggy) communication.
I doubt how rare near-best futures are among desired futures is a strong guide to the expected value of the future. At least, you need to know more about e.g. the feasibility of near-best futures; whether deliberative processes and scientific progress converge on an understanding of which futures are near-best, etc.
There is an analogous argument which says: “most goals in the space of goals are bad and lead to AI scheming; AI will ~randomly initialise on a goal; so AI will probably scheme”. But obviously whenever we make things by design (like cars or whatever), we are creating things which are astronomically unlikely configurations in the “space of ways to organise matter”. And the likelihood that humans build cars just doesn’t have much to do with what fraction of matter state space they occupy. It’s just not an illuminating frame. The more interesting stuff is “will humans choose to make them”, and “how easy are they to make”. (I think Ben Garfinkel has made roughly this point, as has Joe Carlsmith more recently.)
These are excellent comments, and unfortunately they all have the virtue of being perspicuous and true so I don’t have that much to say about them.
Is the core idea here that human desires and the values people reach on deliberation come apart? That makes sense, though it also leaves open how much deliberation our descendants will actually do / how much their values will be based on a deliberating process. I guess I’ll just state my view without defending it that after a decade in philosophy I have become pretty pessimistic about convergence happening through deliberation rather than more divergence as more choice points are uncovered and reasoners either think they have a good loss function or just choose not to do backpropagation.