Agree with this post! It’s nice to see these concerns written down.
Deeply agree that “the thing we’re trying to maximize” is itself confusing/mysterious/partially unknown, and there is something slightly ridiculous and worrying about running around trying really hard to maximize it without knowing much about what it is. (Like, we hardly even have the beginnings of a science of conscious experience, yet conscious experience is the thing we’re trying to affect.).
And I don’t think this is just a vague philosophical concern — I really do think that we’re pretty terrible at understanding the experience of many different peoples across time/which combinations of experiences are good/bad and how to actually facilitate various experiences.
People seem to have really overconfident views about what counts as improvement in value, i.e., many people by default seem to think that GDP going up and Our World in Data numbers going up semi-automatically means that that things are largely improving. The real picture might be much more mixed — I think it’d be possible to have those numbers go way up while things simultaneously get worse for large tracts (or even a majority of) people. People are complicated, value is really complex and hard to understand, but often people act as if these things are mostly understood for intents and purposes. I think they’re mostly not understood.
More charitably, the situation could be described as, “we don’t know exactly what we’re trying to maximize, but it’s something in the vicinity of ‘over there’, and it seems like it would be bad if e.g. AI ran amok, since that would be quite likely to destroy whatever it is that actually is important to maximize”. I think this is a fine line of reasoning, but I think it’s really critical to be very consciously alert to the fact that we only have a vague idea of what we’re trying to maximize.
One potential approach could be to choose a maximand which is pluralistic. In other words:
Seek a large vector of seemingly-important things (e.g. include many many detailed aspects of human experience. Could even include things which you might not care about fundamentally but are important instrumental proxies for things that you do care about fundamentally, e.g. civic engagement, the strength of various prosocial norms, …)
Choose a value function over that vector which has a particular kind of shape: it goes way way down if even one or two elements of the vector end up close to zero. I.e., don’t treat things in the vector as substitutable with one another; having 1000x of item A isn’t necessarily enough to make up for having 0 of item B. To give a general idea: something like the product of the sqrts of the items in the vector.
Maintain a ton of uncertainty about what elements should be included in the vector, generally seek to be adding things, and try to run processes that pull deep information in from a TON of different people/perspectives about what should be in the vector. (Stuff like Polis can be kind of a way to do this; could likely be taken much further though. ML can probably help here. Consider applying techniques from next-gen governance — https://forum.effectivealtruism.org/posts/ue9qrxXPLfGxNssvX/cause-exploration-governance-design-and-formation)
Treat the question of “which items should be in the vector?” and “what value function should we run over the vector?” as open questions that need to be continually revisited. Answering that question is a whole-of-society project across all time!
First I’ve heard of utilitarian-leaning meta-pluralism! Sounds interesting — have any links?