I think it’s wild if we’re living in the century (or even the 100,000 years) that will produce a misaligned AI whose values come to fill the galaxy for billions of years. That would just be quite a remarkable, high-leverage (due to the opportunity to avoid misalignment, or at least have some impact on what the values end up being) time period to be living in.
I guess the point that gets me a bit is “is a stable galactic civilisation run by a single misaligned AI wild in the sense the author means?”
Pro wild:
building megastructures
galactic AI is pretty wild
Anti wild
Internal life of this AI might be trivial
Processes might be tremendously repetitive—if you have no aims, is turning all matter and energy into paperclips hard?
I guess for me the jury is still out, but that case does feel different to me, at least some times.
I think it’s wild if we’re living in the century (or even the 100,000 years) that will produce a misaligned AI whose values come to fill the galaxy for billions of years. That would just be quite a remarkable, high-leverage (due to the opportunity to avoid misalignment, or at least have some impact on what the values end up being) time period to be living in.