Also, for me, they “stable galactic civilisation” doesn’t quite do the wildness there justice.
At that point, the consciousness might live lives that are as beyond our imagination as the the life of an archaeologist is to a cow.
To horrendously misquote CS Lewis: “ We are… like an ignorant child who wants to go on making mud pies in a slum because he cannot imagine what is meant by the offer of a holiday at the sea. We are far too easily pleased.”
I have just so little conception of what the life of the average stable galactic citizen might be like.
I think it’s wild if we’re living in the century (or even the 100,000 years) that will produce a misaligned AI whose values come to fill the galaxy for billions of years. That would just be quite a remarkable, high-leverage (due to the opportunity to avoid misalignment, or at least have some impact on what the values end up being) time period to be living in.
Also, for me, they “stable galactic civilisation” doesn’t quite do the wildness there justice.
At that point, the consciousness might live lives that are as beyond our imagination as the the life of an archaeologist is to a cow.
To horrendously misquote CS Lewis: “ We are… like an ignorant child who wants to go on making mud pies in a slum because he cannot imagine what is meant by the offer of a holiday at the sea. We are far too easily pleased.”
I have just so little conception of what the life of the average stable galactic citizen might be like.
I guess the point that gets me a bit is “is a stable galactic civilisation run by a single misaligned AI wild in the sense the author means?”
Pro wild:
building megastructures
galactic AI is pretty wild
Anti wild
Internal life of this AI might be trivial
Processes might be tremendously repetitive—if you have no aims, is turning all matter and energy into paperclips hard?
I guess for me the jury is still out, but that case does feel different to me, at least some times.
I think it’s wild if we’re living in the century (or even the 100,000 years) that will produce a misaligned AI whose values come to fill the galaxy for billions of years. That would just be quite a remarkable, high-leverage (due to the opportunity to avoid misalignment, or at least have some impact on what the values end up being) time period to be living in.