Ideally, I would include at this point some readings on how aggregation might work for building a utopia, since this seems like an obvious and important point. For instance, should the light cone be divided such that every person (or every moral patient more broadly, perhaps with the division taking moral weight into account) gets to live in a sliver of the light cone that’s optimized to fit their preferences? Should everybody’s preferences be aggregated somehow, so that everyone can live together happily in the overall light cone? Something else? However, I was unable to find any real discussion of this point. Let me know in the comments if there are writings I’m missing. For now, I’ll include the most relevant thing I could find as well as a more run-of-the-mill reading on preference aggregation theory.
It would probably be worth if for someone to write out the ethical
implications of K-complexity-weighted utilitarianism/UDASSA on how to think about far-future ethics.
A few things that come to mind about this question (these are all ~hunches and maybe only semi-related, sorry for the braindump):
The description length of earlier states of the universe is probably shorter, which means that the “claw” that locates minds earlier in a simple universe is also shorter. This implies that lives earlier in time in the universe would be more important, and that we don’t have to care about exact copies as much.
This is similar to the reasons why not to care too much about Boltzmann brains.
We might have to aggregate preferences of agents with different beliefs (possible) and different ontologies/metaphysical stances (not sure about this), probably across ontological crises.
I have some preliminary writings on this, but nothing publishable yet.
The outcomes of UDASSA is dependent on the choice of Turing machine. (People say it’s only up to a constant, but that constant can be pretty big).
So we either find a way of classifying Turing machines by simplicity without relying on a single Turing machine to give us that notion, or we start out with some probability distribution over Turing machines and do some “2-level-Solomonoff induction”, where we update both the probability of each Turing machine and the probabilities of each hypothesis for Turing machine.
This leads to selfishness for whoever is computing Solomonoff induction, because the Turing machine where the empty program just outputs their observations receives the highest posterior probability.
If we use UDASSA/K-ultilitarianism to weigh minds there’s a pressure/tradeoff to simplify one’s preferences to be simpler.
If we endorse some kind of total utilitarianism, and there are increasing marginal returns to energymatter or spacetime investment into minds with respect to degree of moral patienthood then we’d expect to end up with very few large minds, if there are decreasing marginal returns we end up with many small minds.
Theorems like Gibbard-Satterthwaite and Hylland imply that robust preference aggregation that resists manipulation is really hard. You can circumvent this by randomly selecting a dictator, but I think this would become unnecesary if we operate in an open-source game theory context, where algorithms can inspect each others’ reasons for a vote.
I’m surprised you didn’t mention reflective equilibrium! Formalising reflective equilibrium and value formation with meta-preferences would be major steps in a long reflection.
I have the intuition that Grand Futures talks about this problem somewhere[1], but I don’t remember/know where.
It would probably be worth if for someone to write out the ethical implications of K-complexity-weighted utilitarianism/UDASSA on how to think about far-future ethics.
A few things that come to mind about this question (these are all ~hunches and maybe only semi-related, sorry for the braindump):
The description length of earlier states of the universe is probably shorter, which means that the “claw” that locates minds earlier in a simple universe is also shorter. This implies that lives earlier in time in the universe would be more important, and that we don’t have to care about exact copies as much.
This is similar to the reasons why not to care too much about Boltzmann brains.
We might have to aggregate preferences of agents with different beliefs (possible) and different ontologies/metaphysical stances (not sure about this), probably across ontological crises.
I have some preliminary writings on this, but nothing publishable yet.
The outcomes of UDASSA is dependent on the choice of Turing machine. (People say it’s only up to a constant, but that constant can be pretty big).
So we either find a way of classifying Turing machines by simplicity without relying on a single Turing machine to give us that notion, or we start out with some probability distribution over Turing machines and do some “2-level-Solomonoff induction”, where we update both the probability of each Turing machine and the probabilities of each hypothesis for Turing machine.
This leads to selfishness for whoever is computing Solomonoff induction, because the Turing machine where the empty program just outputs their observations receives the highest posterior probability.
If we use UDASSA/K-ultilitarianism to weigh minds there’s a pressure/tradeoff to simplify one’s preferences to be simpler.
If we endorse some kind of total utilitarianism, and there are increasing marginal returns to energymatter or spacetime investment into minds with respect to degree of moral patienthood then we’d expect to end up with very few large minds, if there are decreasing marginal returns we end up with many small minds.
Theorems like Gibbard-Satterthwaite and Hylland imply that robust preference aggregation that resists manipulation is really hard. You can circumvent this by randomly selecting a dictator, but I think this would become unnecesary if we operate in an open-source game theory context, where algorithms can inspect each others’ reasons for a vote.
I’m surprised you didn’t mention reflective equilibrium! Formalising reflective equilibrium and value formation with meta-preferences would be major steps in a long reflection.
I have the intuition that Grand Futures talks about this problem somewhere[1], but I don’t remember/know where.
Which, given its length, isn’t that out there.