I want to see a bargain solver for AI alignment to groups: a technical solution that would allow AI systems to solve the pie cutting problem for groups and get them the most of what they want, for AI alignment. The best solutions I’ve seen for maximizing long run value involve using a bargain solver to decide what ASI does, which preserves the richness and cardinality of people’s value functions and gives everyone as much of what they want as possible, weighted by importance. (See WWOTF Afterwards, the small literature on bargaining-theoretic approaches to moral uncertainty.) But existing democratic approaches to AI alignment seem to not be fully leveraging AI tools, and instead aligning AI systems to democratic processes that aren’t empowered with AI tools (e.g. CIPs and CAIS’S alignment to the written output of citizens’ assemblies.) Moreover, in my experience the best way to make something happen is just to build the solution. If you might be interested in building this tool and have the background, I would love to try to connect you to funding for it.
I don’t know! It’s possible that you can just solve a bargain and then align AI to that, like you can align AI to citizens assemblies. I want to be pitched.
I want to see a bargain solver for AI alignment to groups: a technical solution that would allow AI systems to solve the pie cutting problem for groups and get them the most of what they want, for AI alignment. The best solutions I’ve seen for maximizing long run value involve using a bargain solver to decide what ASI does, which preserves the richness and cardinality of people’s value functions and gives everyone as much of what they want as possible, weighted by importance. (See WWOTF Afterwards, the small literature on bargaining-theoretic approaches to moral uncertainty.) But existing democratic approaches to AI alignment seem to not be fully leveraging AI tools, and instead aligning AI systems to democratic processes that aren’t empowered with AI tools (e.g. CIPs and CAIS’S alignment to the written output of citizens’ assemblies.) Moreover, in my experience the best way to make something happen is just to build the solution. If you might be interested in building this tool and have the background, I would love to try to connect you to funding for it.
For deeper motivation see here.
Is the alignment motivation distinct from just using AI to solve general bargaining problems?
I don’t know! It’s possible that you can just solve a bargain and then align AI to that, like you can align AI to citizens assemblies. I want to be pitched.