If we accept something like the total view of population ethics with linear aggregation, it follows that we should enrich the universe with as much goodness as possible. That means creating maximum pleasure, or eudaimonia, or whatever it is we consider valuable.
[...]
This vision does not depend on specific assumptions about what “flourishing” looks like. It could fit the hedonistic utilitarian idea of hedonium—matter that’s organized to efficiently produce simple but maximally happy beings, who have no functions other to experience happiness—but it could also look like something else.
The situation feels much bleaker to me than that, because for any very specific and plausible definition of “flourishing” X under a total view with linear aggregation, it seems to me that X would likely capture <<1% of the astronomical value under the vast majority of other plausible total/linear definitions of flourishing.
So it seems to me that if we have a fairly high bar of “existential win”, that naively doesn’t look too unreasonable (like “>1% of the value of the best possible utopia by Linch’s values under reflective equilibrium”), then there’s an intuitively compelling worldview that there’s a <1% probability of an existential win and thus x-risk is >99%, even if we get the other things right like AI alignment, large-scale coordination, long reflection, etc.
My impression is that this belief is pretty unusual, which leads me to think that I’m missing some important steps.
Although perhaps the most important consideration here is that all these questions can be deferred to until after the long reflection.
How many different plausible definitions of flourishing that differ significantly enough do you expect there to be?
One potential solution would be to divide the future spacetime (not necessarily into contiguous blocks) in proportion to our credences in them (or evenly), and optimize separately for the corresponding view in each. With equal weights, each of n views could get at least about 1/n of what it would if it had 100% weight (taking ratios of expected values), assuming there isn’t costly conflict between the views and no view (significantly) negatively values what another finds near optimal in practice. They could potentially do much better with some moral trades and/or if there’s enough overlap in what they value positively. One view going for a larger share would lead to zero sum work and deadweight loss as others respond to it.
I would indeed guess that a complex theory of flourishing (“complexity of value”, objective list theories, maybe), a preference/desire view and hedonism would assign <1% value to each other’s (practical) optima compared to their own. I think there could be substantial agreement between different complex theories of flourishing, though, since I expect them generally to overlap a lot in their requirements. I could also see hedonism and preference views overlapping considerably and having good moral trades, in case most of the resource usage is just to sustain consciousness (and not to instantiate preference satisfaction or pleasure in particular) and most of the resulting consciousness-sustaining structures/activity can shared without much loss on either view. However, this could just be false.
So I thought about this post a bit more, particularly the we never saturate the universe with maximally flourishing beings and impossibility of reflective equilibrium sections.
The situation feels much bleaker to me than that, because for any very specific and plausible definition of “flourishing” X under a total view with linear aggregation, it seems to me that X would likely capture <<1% of the astronomical value under the vast majority of other plausible total/linear definitions of flourishing.
So it seems to me that if we have a fairly high bar of “existential win”, that naively doesn’t look too unreasonable (like “>1% of the value of the best possible utopia by Linch’s values under reflective equilibrium”), then there’s an intuitively compelling worldview that there’s a <1% probability of an existential win and thus x-risk is >99%, even if we get the other things right like AI alignment, large-scale coordination, long reflection, etc.
My impression is that this belief is pretty unusual, which leads me to think that I’m missing some important steps.
Although perhaps the most important consideration here is that all these questions can be deferred to until after the long reflection.
How many different plausible definitions of flourishing that differ significantly enough do you expect there to be?
One potential solution would be to divide the future spacetime (not necessarily into contiguous blocks) in proportion to our credences in them (or evenly), and optimize separately for the corresponding view in each. With equal weights, each of n views could get at least about 1/n of what it would if it had 100% weight (taking ratios of expected values), assuming there isn’t costly conflict between the views and no view (significantly) negatively values what another finds near optimal in practice. They could potentially do much better with some moral trades and/or if there’s enough overlap in what they value positively. One view going for a larger share would lead to zero sum work and deadweight loss as others respond to it.
I would indeed guess that a complex theory of flourishing (“complexity of value”, objective list theories, maybe), a preference/desire view and hedonism would assign <1% value to each other’s (practical) optima compared to their own. I think there could be substantial agreement between different complex theories of flourishing, though, since I expect them generally to overlap a lot in their requirements. I could also see hedonism and preference views overlapping considerably and having good moral trades, in case most of the resource usage is just to sustain consciousness (and not to instantiate preference satisfaction or pleasure in particular) and most of the resulting consciousness-sustaining structures/activity can shared without much loss on either view. However, this could just be false.