I feel like a lot of what you’re describing is already encompassed by the concept of scalability, which would naturally include integration with existing social systems. However you are right in questioning whether this is a “relatively well-defined technical problem.”
An alternative taxonomy might be “technical” and “game-theoretic” alignment. The latter recognizes that competing visions for social organization exist and will not be solved within the scope of AI regulation. That in turn leads to more meta-theoretical discussions about how ambitious the AI safety agenda should be in order not to stifle market competition, which would be the ultimate insurance against extremist goalcraft.
Otherwise, engaging in these debates at the object level creates an open invitation for manipulation and bad faith.
would naturally include integration with existing social systems
I wouldn’t limit AI goalcraft to integration with existing social systems. It may be better to use the capabilites of AI to build fundamentally better preference aggregation engines. That’s the idea of CEV and its ilk.
I feel like a lot of what you’re describing is already encompassed by the concept of scalability, which would naturally include integration with existing social systems. However you are right in questioning whether this is a “relatively well-defined technical problem.”
An alternative taxonomy might be “technical” and “game-theoretic” alignment. The latter recognizes that competing visions for social organization exist and will not be solved within the scope of AI regulation. That in turn leads to more meta-theoretical discussions about how ambitious the AI safety agenda should be in order not to stifle market competition, which would be the ultimate insurance against extremist goalcraft.
Otherwise, engaging in these debates at the object level creates an open invitation for manipulation and bad faith.
I wouldn’t limit AI goalcraft to integration with existing social systems. It may be better to use the capabilites of AI to build fundamentally better preference aggregation engines. That’s the idea of CEV and its ilk.