I agree that people often have more intuitions about fragility in unipolar cases vs in multipolar cases. I’m not sure that I fully understand what’s going on here, but my current guesses are mostly in the vicinity of your “something good comes from people rubbing up against each other” point, and are something like:
To some extent, the intuitions about the multipolar case are providing evidence against value fragility, which people are failing to fully propagate through
People have the intuition about value fragility more when they think of specifying values as being about specifying precise goals (which is what they tend to picture when thinking of unipolar value specification), rather than about specifying the process that will lead to eventual outcomes in an emergent way (which is what they tend to picture when thinking of the multipolar case)
A thing that people are often worried about is “mind X has an idea of what values are correct and these get enacted” (for X being an AI or often a human), and they notice that this situation is fragile—if X’s values are a bit off, then there’s no external correcting force
A lot of the values people actually have are things that could loosely be called “prosocial”; in multipolar scenarios there are pressures to be prosocial (because these are reinforced by the game theory of everyone wanting other people to be prosocial and so rewarding behaviour which is and penalising behaviour which isn’t)
Multipolarity also permits for a diversity of viewpoints, which helps to avoid possible traps where a single mind fails to consider or care for something important
On this perspective, it’s not that value is extremely fragile, just that it’s kind of fragile—and so it helps a bunch to get more bites of the apple
I agree that people often have more intuitions about fragility in unipolar cases vs in multipolar cases. I’m not sure that I fully understand what’s going on here, but my current guesses are mostly in the vicinity of your “something good comes from people rubbing up against each other” point, and are something like:
To some extent, the intuitions about the multipolar case are providing evidence against value fragility, which people are failing to fully propagate through
People have the intuition about value fragility more when they think of specifying values as being about specifying precise goals (which is what they tend to picture when thinking of unipolar value specification), rather than about specifying the process that will lead to eventual outcomes in an emergent way (which is what they tend to picture when thinking of the multipolar case)
A thing that people are often worried about is “mind X has an idea of what values are correct and these get enacted” (for X being an AI or often a human), and they notice that this situation is fragile—if X’s values are a bit off, then there’s no external correcting force
A lot of the values people actually have are things that could loosely be called “prosocial”; in multipolar scenarios there are pressures to be prosocial (because these are reinforced by the game theory of everyone wanting other people to be prosocial and so rewarding behaviour which is and penalising behaviour which isn’t)
Multipolarity also permits for a diversity of viewpoints, which helps to avoid possible traps where a single mind fails to consider or care for something important
On this perspective, it’s not that value is extremely fragile, just that it’s kind of fragile—and so it helps a bunch to get more bites of the apple