Thank you! This seems importantly distinct from the “Benefiting Unscrupulous People” failure mode in that you put the emphasis not on intentional exploitation but on cooperation failures even among well-intentioned groups.
I’ll reuse this comment to bring up something related. The paper “Open Problems in Cooperative AI
” has a section The Potential Downsides of Cooperative AI.
The paper focuses on the cooperation aspect of the collaborative/cooperative truth-seeking, so the section on potential downsides focuses on downsides of cooperation and downsides of promoting cooperation rather than downsides of truth-seeking. That said, it might be the case that any tool that enables collaborative truth-seeking also promotes capacities that are needed for cooperation.
I currently think that promoting cooperation is one of the most robustly good things we can do, so challenges to that view are very interesting!
The paper has three:
Cooperation can enable collusion, cartels, and bribes. Various antisocial behaviors that wouldn’t be as easy if there were no way to trust the other party.
Promoting cooperation will go through promoting capacities that enable cooperation. Such capacities include understanding, recognizing honesty and deception, and commitment. Those same capacities can also be used to understand others’ vulnerabilities, deceive them, and commit to threats. Those, however, can also be prosocial again, e.g., in the case of laws that are enforced through a threat of a fine.
Finally, the authors note that competition can facilitate learning or training.
They also suggest mitigating factors when it comes to such cooperation failures:
Secondly, mutual gains in coercive capabilities tend to cancel each other out, like how mutual training in chess will tend to not induce a large shift in the balance of skill. To the extent, then, that research on Cooperative AI unavoidably also increases coercive skill, the hope is that those adverse impacts will largely cancel, whereas the increases in cooperative competence will be additive, if not positive complements. This argument is most true for coercive capabilities that are not destructive but merely lead to transfers of wealth between agents. Nevertheless, mutual increases in destructive coercive capabilities will also often cancel each other out through deterrence. The world has not experienced more destruction with the advent of nuclear weapons, because leaders possessing nuclear weapons have greatly moderated their aggression against each other. By contrast, cooperation and cooperative capabilities lead to positive feedback and are reinforcing; it is in one’s interests to help others learn to be better cooperators.
My thoughts, in the same order, on how these apply to collaborative truth-seeking:
This seems to depend on whether collaborative truth-seeking changes how easy it is to cooperate with others. While I’m very enthusiastic about promoting cooperation and so take this concern seriously, it also doesn’t seem to fit this post, or only very tenuously. It might be that tools that depend on cooperation enhance cooperation, but it might also not be the case. Or the effect might be very small.
A similar response applies here, but there is the wrinkle that enhanced truth-seeking may be used to improve what the author’s call “understanding.” I’m not very concerned about this, though, because your evolved intuitions for Theory of Mind are probably so good that a technical, collaborative tool can barely improve upon them. (Unlike AI, which the paper is about.) This applies less to groups of people than to individuals, so maybe this is again a danger once we’re dealing with companies or states.
This probably doesn’t apply. At least I can’t see a way in which collaborative tools would affect the level of competition. In fact it seems that it is hard to set the incentives of prediction markets such that people are not discouraged from information sharing (which is probably bad on balance).
Thank you! This seems importantly distinct from the “Benefiting Unscrupulous People” failure mode in that you put the emphasis not on intentional exploitation but on cooperation failures even among well-intentioned groups.
I’ll reuse this comment to bring up something related. The paper “Open Problems in Cooperative AI ” has a section The Potential Downsides of Cooperative AI.
The paper focuses on the cooperation aspect of the collaborative/cooperative truth-seeking, so the section on potential downsides focuses on downsides of cooperation and downsides of promoting cooperation rather than downsides of truth-seeking. That said, it might be the case that any tool that enables collaborative truth-seeking also promotes capacities that are needed for cooperation.
I currently think that promoting cooperation is one of the most robustly good things we can do, so challenges to that view are very interesting!
The paper has three:
Cooperation can enable collusion, cartels, and bribes. Various antisocial behaviors that wouldn’t be as easy if there were no way to trust the other party.
Promoting cooperation will go through promoting capacities that enable cooperation. Such capacities include understanding, recognizing honesty and deception, and commitment. Those same capacities can also be used to understand others’ vulnerabilities, deceive them, and commit to threats. Those, however, can also be prosocial again, e.g., in the case of laws that are enforced through a threat of a fine.
Finally, the authors note that competition can facilitate learning or training.
They also suggest mitigating factors when it comes to such cooperation failures:
My thoughts, in the same order, on how these apply to collaborative truth-seeking:
This seems to depend on whether collaborative truth-seeking changes how easy it is to cooperate with others. While I’m very enthusiastic about promoting cooperation and so take this concern seriously, it also doesn’t seem to fit this post, or only very tenuously. It might be that tools that depend on cooperation enhance cooperation, but it might also not be the case. Or the effect might be very small.
A similar response applies here, but there is the wrinkle that enhanced truth-seeking may be used to improve what the author’s call “understanding.” I’m not very concerned about this, though, because your evolved intuitions for Theory of Mind are probably so good that a technical, collaborative tool can barely improve upon them. (Unlike AI, which the paper is about.) This applies less to groups of people than to individuals, so maybe this is again a danger once we’re dealing with companies or states.
This probably doesn’t apply. At least I can’t see a way in which collaborative tools would affect the level of competition. In fact it seems that it is hard to set the incentives of prediction markets such that people are not discouraged from information sharing (which is probably bad on balance).