Having settled most of the accessible universe we’ll have hundreds of billions or even trillions of years to try to keep improving how we’re using the matter and energy at our disposal.
Doesn’t it seems almost certain that over such a long time period our annual rate if improvement in the value generated by the best configuration would eventually asymptote towards zero? I think that’s all that’s necessary for safety to be substantially more attractive than speed-ups.
(BTW safety is never ‘infinitely’ preferred because even on a strict plateau view the accessible universe is still shrinking by about a billionth a year.)
Agreed. And even in the scenario where we could continue to find more valuable patterns of matter even billions of years in the future, I don’t think that efforts to accelerate things now would have any significant impact on the value we will create in the future, because it seems very likely that our future value creation will mostly depend on major events that won’t have much to do with the current state of things.
Let’s consider the launch of Von Neumann probes throughout the universe as such a possible major event: even if we could increase our current growth rate by 1% with a better allocation of resources, it doesn’t mean that the future launch of these probes will be 1% more efficient. Rather, the outcomes of this event seem largely uncorrelated with our growth rate prior to that moment. At best, accelerating our growth would hasten the launch by a tiny bit, but this is very different than saying “increasing our growth by 1% now will increase our whole future utility by 1%”.
Let me call X the statement: “our rate of improvement remains bounded away from zero far into the future”. If I understand correctly, you are saying that we have great difficulties imagining a scenario where X happens, therefore X is very unlikely.
Human imagination is very limited. For instance, most of human history shows very little change from one generation to the next; in other words, people were not able to imagine ways for future generations to do certain things in better ways than how they already knew. Here you ask our imagination to perform a spectacularly difficult task, namely to imagine what extremely advanced civilizations are likely to be doing in billions of years. I am not surprised if we do not manage to produce a credible scenario where X occurs. I do not take this as strong evidence against X.
Separately from this, I personally do not find it very likely that we will ultimately settle most of the accessible universe, as you suppose, because I would be surprised if human beings hold such a special position. (In my opinion, either advanced civilizations are not so interested in expanding in space; or else, we will at some point meet a much more advanced civilization, and our trajectory after this point will probably depend little on what we can do before it.)
Concerning the point you put in parentheses about safety being “infinitely” preferred, I meant to use phrases such as “virtually infinitely preferred” to convey that the preference is so strong that any actual empirical estimate is considered unnecessary. In footnote 5 above, I mentioned this 80k article intended to summarize the views of the EA community, where it is said that speedup interventions are “essentially morally neutral” (which, given the context, I take as being equivalent to saying that risk mitigation is essentially infinitely preferred).
As a first pass the rate of improvement should asymptote towards zero so long as there’s a theoretical optimum and declining returns to further research before the heat death of the universe, which seems like pretty mild assumptions.
As an analogy, there’s an impossibly wide range of configurations of matter you could in theory use to create a glass from which we can drink water. But we’ve already gotten most of the way towards the best glass for humans, I would contend. I don’t think we could keep improving glasses in any meaningful way using a galaxy’s resources for a trillion years.
Keep in mind eventually the light cone of each star shrinks so far it can’t benefit from research conducted elsewhere.
As a first pass the rate of improvement should asymptote towards zero so long as there’s a theoretical optimum and declining returns to further research before the heat death of the universe, which seems like pretty mild assumptions.
I think it would be really useful if this idea was explained in more details somewhere, preferably on the 80k website. Do you think there is a chance that this happens at some point? (hopefully not too far in the future ;-) )
Yes it needs to go in an explanation of how we score scale/importance in the problem framework! It’s on the list. :)
Alternatively I’ve been wondering if we need a standalone article explaining how we can influence the long term, and what are signs that something might be highly leveraged for doing that.
Having settled most of the accessible universe we’ll have hundreds of billions or even trillions of years to try to keep improving how we’re using the matter and energy at our disposal.
Doesn’t it seems almost certain that over such a long time period our annual rate if improvement in the value generated by the best configuration would eventually asymptote towards zero? I think that’s all that’s necessary for safety to be substantially more attractive than speed-ups.
(BTW safety is never ‘infinitely’ preferred because even on a strict plateau view the accessible universe is still shrinking by about a billionth a year.)
Agreed. And even in the scenario where we could continue to find more valuable patterns of matter even billions of years in the future, I don’t think that efforts to accelerate things now would have any significant impact on the value we will create in the future, because it seems very likely that our future value creation will mostly depend on major events that won’t have much to do with the current state of things.
Let’s consider the launch of Von Neumann probes throughout the universe as such a possible major event: even if we could increase our current growth rate by 1% with a better allocation of resources, it doesn’t mean that the future launch of these probes will be 1% more efficient. Rather, the outcomes of this event seem largely uncorrelated with our growth rate prior to that moment. At best, accelerating our growth would hasten the launch by a tiny bit, but this is very different than saying “increasing our growth by 1% now will increase our whole future utility by 1%”.
Let me call X the statement: “our rate of improvement remains bounded away from zero far into the future”. If I understand correctly, you are saying that we have great difficulties imagining a scenario where X happens, therefore X is very unlikely.
Human imagination is very limited. For instance, most of human history shows very little change from one generation to the next; in other words, people were not able to imagine ways for future generations to do certain things in better ways than how they already knew. Here you ask our imagination to perform a spectacularly difficult task, namely to imagine what extremely advanced civilizations are likely to be doing in billions of years. I am not surprised if we do not manage to produce a credible scenario where X occurs. I do not take this as strong evidence against X.
Separately from this, I personally do not find it very likely that we will ultimately settle most of the accessible universe, as you suppose, because I would be surprised if human beings hold such a special position. (In my opinion, either advanced civilizations are not so interested in expanding in space; or else, we will at some point meet a much more advanced civilization, and our trajectory after this point will probably depend little on what we can do before it.)
Concerning the point you put in parentheses about safety being “infinitely” preferred, I meant to use phrases such as “virtually infinitely preferred” to convey that the preference is so strong that any actual empirical estimate is considered unnecessary. In footnote 5 above, I mentioned this 80k article intended to summarize the views of the EA community, where it is said that speedup interventions are “essentially morally neutral” (which, given the context, I take as being equivalent to saying that risk mitigation is essentially infinitely preferred).
As a first pass the rate of improvement should asymptote towards zero so long as there’s a theoretical optimum and declining returns to further research before the heat death of the universe, which seems like pretty mild assumptions.
As an analogy, there’s an impossibly wide range of configurations of matter you could in theory use to create a glass from which we can drink water. But we’ve already gotten most of the way towards the best glass for humans, I would contend. I don’t think we could keep improving glasses in any meaningful way using a galaxy’s resources for a trillion years.
Keep in mind eventually the light cone of each star shrinks so far it can’t benefit from research conducted elsewhere.
I think it would be really useful if this idea was explained in more details somewhere, preferably on the 80k website. Do you think there is a chance that this happens at some point? (hopefully not too far in the future ;-) )
Yes it needs to go in an explanation of how we score scale/importance in the problem framework! It’s on the list. :)
Alternatively I’ve been wondering if we need a standalone article explaining how we can influence the long term, and what are signs that something might be highly leveraged for doing that.