Paul’s main argument in the second piece is that progress “doesn’t have much effect on very long-term outcomes” because he expects this quality-of-life curve to eventually reach a plateau. That is a somewhat different argument. As I show in Shaping Humanity’s Longterm Trajectory, the value of advancements can be extremely large as they scale with the instantaneous value of the future. e.g. it is plausible that if civilisation has spread to billions of billions of worlds that having an extra year of this is worth a lot (indeed this is the point of the first part of ‘Astronomical Waste’). But it doesn’t scale with the duration of our future, so can get beaten by things like reducing existential risk that scale with duration as well.
So the argument that we are likely to reach a plateau is really an argument that existential risk (or some other trajectory changes) will be even more important than progress, rather than that progress may be as likely to be negative value as positive value. For that you need to consider that the curve will end and that the timing of this could be endogenous.
(That said, I think arguments based on plateaus could be useful and powerful in this kind of longtermist reasoning, and encourage people to explore them. I don’t think it is obvious that we will reach a plateau (mainly as we might not make it that far), but they are a very plausible feature of the trajectory of humanity that may have important implications if it exists, and which would also simplify some of the analysis.)
Thanks Toby, this is a nice article, and I think more easily approachable than anything previous I knew on the topic.
For those interested in the topic, I wanted to add links to a couple of Paul Christiano’s classic posts:
How useful is “progress”?
(and maybe some other of the posts on that blog)
On Progress and Prosperity
(I think this is perhaps mostly relevant for the intellectual history.)
Interesting posts!
I don’t recall reading either of them before.
Paul’s main argument in the second piece is that progress “doesn’t have much effect on very long-term outcomes” because he expects this quality-of-life curve to eventually reach a plateau. That is a somewhat different argument. As I show in Shaping Humanity’s Longterm Trajectory, the value of advancements can be extremely large as they scale with the instantaneous value of the future. e.g. it is plausible that if civilisation has spread to billions of billions of worlds that having an extra year of this is worth a lot (indeed this is the point of the first part of ‘Astronomical Waste’). But it doesn’t scale with the duration of our future, so can get beaten by things like reducing existential risk that scale with duration as well.
So the argument that we are likely to reach a plateau is really an argument that existential risk (or some other trajectory changes) will be even more important than progress, rather than that progress may be as likely to be negative value as positive value. For that you need to consider that the curve will end and that the timing of this could be endogenous.
(That said, I think arguments based on plateaus could be useful and powerful in this kind of longtermist reasoning, and encourage people to explore them. I don’t think it is obvious that we will reach a plateau (mainly as we might not make it that far), but they are a very plausible feature of the trajectory of humanity that may have important implications if it exists, and which would also simplify some of the analysis.)