It also seems a bit circular because if you want to build a Deep Democracy AGI, then that means you value Deep Democracy, so you’re still aligning AGI to your values
no, you’re aligning it to what everyone values.
J.C.Harsanyi, in a 2-page article involving no mathematics whatever [J.Political Economy 61,5 (1953) 434-435], came up with the following nice idea: “Optimizing social welfare” means “picking the state of the world all individuals would prefer if they were in a state of uncertainty about their identity.” I.e. if you are equally likely to be anybody, then your expected utility is the summed utility in the world divided by the number of people in it – i.e. average utility. Then by the linear-lottery property (Lin) of von Neumann utility, it follows that social utility is averaging.
no, you’re aligning it to what everyone values.
J.C.Harsanyi, in a 2-page article involving no mathematics whatever [J.Political Economy 61,5 (1953) 434-435], came up with the following nice idea: “Optimizing social welfare” means “picking the state of the world all individuals would prefer if they were in a state of uncertainty about their identity.” I.e. if you are equally likely to be anybody, then your expected utility is the summed utility in the world divided by the number of people in it – i.e. average utility. Then by the linear-lottery property (Lin) of von Neumann utility, it follows that social utility is averaging.
source