This is a really interesting post, thanks for writing it up.
I think I have two main models for thinking about these sorts of issues:
The accelerating view, where we have historically seen several big speed-ups in rate of change as a result of the introduction of more powerful methods of optimisation, and the introduction of human-level AGI is likely to be another. In this case the future is both potentially very valuable (because AGI will allow very rapid growth and world-optimisation) and endangered (because the default is that new optimisation forces do not respect the values or ‘values’ of previous modes.)
Physics/Chemistry/Plate Tectonics
Life/Evolution
Humanity/Intelligence/Culture/Agriculture
Enlightenment/Capitalism/Industrial Revolution
Recursively self-improving AGI?
The God of Straight Lines approach, where we’ll continue to see roughly 2% RGDP growth, because that is what always happens. AI will make us more productive, but not dramatically so, and at the same time previous sources of productivity growth will be exhausted, so overall trends will remain roughly intact. As such, the future is worth a lot less (perhaps we will colonise the stars, but only slowly, and growth rates won’t hit 50%/year) but also less endangered (because all progress will be incremental and slow, and humanity will remain in control). I think of this as being the epistemically modest approach.
As a result, my version of Clara thinks of AI Safety work as reducing risk in the worlds that happen to matter the most. It’s also possible that these are the worlds where we can have the most influence, if you thought that strong negative feedback mechanisms strongly limited action in the Straight Line world
Note that I was originally going to describe these as the inside and outside views, but I actually think that both have decent outside-view justifications.
Interesting view. It seems to me like it makes sense, but I also feel like it’d be valuable for it to be fleshed out and critiqued further to see how solid it is. (Perhaps this has already been done somewhere—I do feel like I’ve heard vaguely similar arguments here and there.)
Also, arriving at this thread 5 months late, I notice Toby Ord makes a similar argument in The Precipice. He writes about:
a subtle form of correlation—not between two risks, but between risks and the value of the future. There might be risks that are much more likely to occur in worlds with high potential. For example, if it is possible to create artificial intelligence that far surpasses humanity in every domain, this would increase the risk from misaligned AGI, but would also increase the value we could achieve using AGI that was aligned with human values. By ignoring this correlation, the total risk approach underweights the value of work on this risk.
This can be usefully understood in terms of there being a common cause for the risk and the benefit, producing the correlation. A high ceiling on technological capability might be another common cause between a variety of risks and extremely positive futures. I will set this possibility aside in the rest of the book, but it is an important issue for future work to explore.
Interesting view. It seems to me like it makes sense, but I also feel like it’d be valuable for it to be fleshed out and critiqued further to see how solid it is. (Perhaps this has already been done somewhere—I do feel like I’ve heard vaguely similar arguments here and there.)
Thanks! I’m not sure if I made it up or not. I will try to find some time to write more about it.
I think it’s worth noting that, for predictions concerning the next few decades, accelerating growth or “the god of straight lines” with 2% growth are not the only possibilities. There is for instance this piece by Tyler Cowen and Ben Southwood on the slowing down of scientific progress, which I find very good. Also, in Chapter 18 of Gordon’s book on “the rise and fall of American growth”, he predicts (under assumptions that I find reasonable) that the median disposable income per person in the US will grow by about 0.3% per year on average over the period 2015-2040. (This does not affect your argument though, as far as I understand it.)
That’s an interesting point of view. But notice that none of your two options is compatible with the “plateau” view that, as far as I understand, forms the basis of the recommendations on the 80k website. (See also Robert Wiblin’s comment below.)
In the ‘2% RGDP growth’ view, the plateau is already here, since exponential RGDP growth is probably subexponential utility growth. (I reckon this is a good example of confusion caused by using ‘plateau’ to mean ‘subexponential’ :) )
In the ‘accelerating view’, it seems that whether there is exponential utility growth in the long term comes down to the same intuitions about whether things keep accelerating forever that are discussed in other threads.
Ok, but note that this depends crucially on whether you decide that your utility looks more like log(GDP), or more like (GDP)^0.1, say. I don’t know how we can be confident that it is one and not the other.
This is a really interesting post, thanks for writing it up.
I think I have two main models for thinking about these sorts of issues:
The accelerating view, where we have historically seen several big speed-ups in rate of change as a result of the introduction of more powerful methods of optimisation, and the introduction of human-level AGI is likely to be another. In this case the future is both potentially very valuable (because AGI will allow very rapid growth and world-optimisation) and endangered (because the default is that new optimisation forces do not respect the values or ‘values’ of previous modes.)
Physics/Chemistry/Plate Tectonics
Life/Evolution
Humanity/Intelligence/Culture/Agriculture
Enlightenment/Capitalism/Industrial Revolution
Recursively self-improving AGI?
The God of Straight Lines approach, where we’ll continue to see roughly 2% RGDP growth, because that is what always happens. AI will make us more productive, but not dramatically so, and at the same time previous sources of productivity growth will be exhausted, so overall trends will remain roughly intact. As such, the future is worth a lot less (perhaps we will colonise the stars, but only slowly, and growth rates won’t hit 50%/year) but also less endangered (because all progress will be incremental and slow, and humanity will remain in control). I think of this as being the epistemically modest approach.
As a result, my version of Clara thinks of AI Safety work as reducing risk in the worlds that happen to matter the most. It’s also possible that these are the worlds where we can have the most influence, if you thought that strong negative feedback mechanisms strongly limited action in the Straight Line world
Note that I was originally going to describe these as the inside and outside views, but I actually think that both have decent outside-view justifications.
Interesting view. It seems to me like it makes sense, but I also feel like it’d be valuable for it to be fleshed out and critiqued further to see how solid it is. (Perhaps this has already been done somewhere—I do feel like I’ve heard vaguely similar arguments here and there.)
Also, arriving at this thread 5 months late, I notice Toby Ord makes a similar argument in The Precipice. He writes about:
Thanks! I’m not sure if I made it up or not. I will try to find some time to write more about it.
I think it’s worth noting that, for predictions concerning the next few decades, accelerating growth or “the god of straight lines” with 2% growth are not the only possibilities. There is for instance this piece by Tyler Cowen and Ben Southwood on the slowing down of scientific progress, which I find very good. Also, in Chapter 18 of Gordon’s book on “the rise and fall of American growth”, he predicts (under assumptions that I find reasonable) that the median disposable income per person in the US will grow by about 0.3% per year on average over the period 2015-2040. (This does not affect your argument though, as far as I understand it.)
That’s an interesting point of view. But notice that none of your two options is compatible with the “plateau” view that, as far as I understand, forms the basis of the recommendations on the 80k website. (See also Robert Wiblin’s comment below.)
In the ‘2% RGDP growth’ view, the plateau is already here, since exponential RGDP growth is probably subexponential utility growth. (I reckon this is a good example of confusion caused by using ‘plateau’ to mean ‘subexponential’ :) )
In the ‘accelerating view’, it seems that whether there is exponential utility growth in the long term comes down to the same intuitions about whether things keep accelerating forever that are discussed in other threads.
Ok, but note that this depends crucially on whether you decide that your utility looks more like log(GDP), or more like (GDP)^0.1, say. I don’t know how we can be confident that it is one and not the other.