Reaping the benefits of AGI later is pretty insignificant in my opinion. If we get aligned AGI utopia, we will have utopia for millions of years. Acceleration by a few years if negligible if it increase p(doom) by >1%.
This is not true depending on what you think AGI utopia will look like. There’s some math outlined in What We Owe the Future about this dilemma i.e. area under the curve of these hypothetical AGI utility functions.
∫∞02(x+1),dx∫∞02x,dx=2
Getting utopia 1 year faster creates a 2x better universe. (hypothetically)
I was reluctant to get into the weeds here but how can anything near this model be possible if 2^300 is around how many atoms there are in the universe and we already have conquered 2^150 of them. At some point, there will likely be no more growing and then there will be millions of stable utopia years.
This is not true depending on what you think AGI utopia will look like. There’s some math outlined in What We Owe the Future about this dilemma i.e. area under the curve of these hypothetical AGI utility functions.
∫∞02(x+1),dx∫∞02x,dx=2
Getting utopia 1 year faster creates a 2x better universe. (hypothetically)
I was reluctant to get into the weeds here but how can anything near this model be possible if 2^300 is around how many atoms there are in the universe and we already have conquered 2^150 of them. At some point, there will likely be no more growing and then there will be millions of stable utopia years.