I’ve had a chat with Justin Shovelain yesterday where we discussed a few more ways in which improved collaborative truth-seeking can backfire. These are, for me, rather early directions for further thought, so I’ll rather combine them in one answer for now.
They fall into two categories: Negative effects of better decision making and negative effects of collaborative methods.
Negative effects of better decision-making:
Valuable ambiguity. It might be that ambiguity plays an important role in social interactions. There is the stylized example where it is necessary to keep the number of rounds of an iterated game a secret or else that knowledge will distort the game. I’ve also read somewhere that there’s the theory that conflicts between two countries can be exacerbated if the countries have too low-quality intelligence about each other but also if they have too high-quality intelligence about each other. But I can’t find the source, so I’m likely to misremember something. Charity evaluators also benefit from ambiguity in that fewer charities would be willing to undergo their evaluation process if the only reason why a charity would either decline it or block the results from being published were reasons that reflect badly on the charity. But there are also good and neutral reasons, so charities will always have plausible deniability.
Negative effects of collaborative methods:
Centralized control. My earlier answer titled “Legibility” argued that collaborative methods will make it necessary to make considerations and values more legible than they are now so they can be communicated and quantified. This may also make them more transparent and thus susceptible to surveillance. That, in turn, may enable more powerful authoritarian governments, which may steer the world into a dystopian lock-in state.
Averaging effect. Maybe there are people who are particularly inclined toward outré opinions. These people will be either unusually right or unusually wrong for their time. Maybe there’s more benefit in being unusually right than there is harm in being unusually wrong (e.g., thanks to the law). And maybe innovation toward most of what we care about is carried by unusually right people. (I’m thinking of Newton here, whose bad ideas didn’t seem to have much of an effect compared to his good ideas.) Collaborative systems likely harness – explicitly or implicitly – some sort of wisdom of the crowds type of effect. But such an effect is likely to average away the unusually wrong and the unusually right opinions. So such systems might slow progress.
More power to the group. It might be that the behavior of groups (e.g., companies) is generally worse (e.g., more often antisocial) than that of individuals. Collaborative systems would shift more power from individuals to groups. So that may be undesirable.
Legible values. It may be very hard to model the full complexity of moral intuitions of people. In practice, people tend toward systems that greatly reduce the dimensionality of what people typically care about. The result are utils, DALYs, SWB, consumption, life years, probability of any existential catastrophe, etc. Collaborative systems would incentivize such low-dimensional measures of value, and through training, people may actually come to care about them more. It’s contentious and a bit circular to ask whether this is good or bad. But at least it’s not clearly neutral or good.
Assorted Risks
I’ve had a chat with Justin Shovelain yesterday where we discussed a few more ways in which improved collaborative truth-seeking can backfire. These are, for me, rather early directions for further thought, so I’ll rather combine them in one answer for now.
They fall into two categories: Negative effects of better decision making and negative effects of collaborative methods.
Negative effects of better decision-making:
Valuable ambiguity. It might be that ambiguity plays an important role in social interactions. There is the stylized example where it is necessary to keep the number of rounds of an iterated game a secret or else that knowledge will distort the game. I’ve also read somewhere that there’s the theory that conflicts between two countries can be exacerbated if the countries have too low-quality intelligence about each other but also if they have too high-quality intelligence about each other. But I can’t find the source, so I’m likely to misremember something. Charity evaluators also benefit from ambiguity in that fewer charities would be willing to undergo their evaluation process if the only reason why a charity would either decline it or block the results from being published were reasons that reflect badly on the charity. But there are also good and neutral reasons, so charities will always have plausible deniability.
Negative effects of collaborative methods:
Centralized control. My earlier answer titled “Legibility” argued that collaborative methods will make it necessary to make considerations and values more legible than they are now so they can be communicated and quantified. This may also make them more transparent and thus susceptible to surveillance. That, in turn, may enable more powerful authoritarian governments, which may steer the world into a dystopian lock-in state.
Averaging effect. Maybe there are people who are particularly inclined toward outré opinions. These people will be either unusually right or unusually wrong for their time. Maybe there’s more benefit in being unusually right than there is harm in being unusually wrong (e.g., thanks to the law). And maybe innovation toward most of what we care about is carried by unusually right people. (I’m thinking of Newton here, whose bad ideas didn’t seem to have much of an effect compared to his good ideas.) Collaborative systems likely harness – explicitly or implicitly – some sort of wisdom of the crowds type of effect. But such an effect is likely to average away the unusually wrong and the unusually right opinions. So such systems might slow progress.
More power to the group. It might be that the behavior of groups (e.g., companies) is generally worse (e.g., more often antisocial) than that of individuals. Collaborative systems would shift more power from individuals to groups. So that may be undesirable.
Legible values. It may be very hard to model the full complexity of moral intuitions of people. In practice, people tend toward systems that greatly reduce the dimensionality of what people typically care about. The result are utils, DALYs, SWB, consumption, life years, probability of any existential catastrophe, etc. Collaborative systems would incentivize such low-dimensional measures of value, and through training, people may actually come to care about them more. It’s contentious and a bit circular to ask whether this is good or bad. But at least it’s not clearly neutral or good.