Yes, I think we can go further and say that alignment of a superintelligent AGI even with a single individual human may well be impossible. Is such a thing mathematically verifiable as completely watertight, given the orthogonality thesis, basic AI drives and mesaoptimisation? And if it’s not watertight, then all the doom flows through the gaps of imperfect, thought to be “good enough”, alignment. We need a global moratorium on AGI development. This year.
Yes, I think we can go further and say that alignment of a superintelligent AGI even with a single individual human may well be impossible. Is such a thing mathematically verifiable as completely watertight, given the orthogonality thesis, basic AI drives and mesaoptimisation? And if it’s not watertight, then all the doom flows through the gaps of imperfect, thought to be “good enough”, alignment. We need a global moratorium on AGI development. This year.