A system that improves collective decision making is likely value-neutral, so it can also be used by unscrupulous agents for their nefarious ends.
Moreover unscrupulous people may benefit from it more because they have fewer moral side-constraints. If set A is the set of all ethical, legal, cooperative methods of attaining a goal, and set B is the set of all methods of attaining the same goal, then A ⊆ B. So it should always be as easy or easier to attain a goal by any means necessary than only by ethical, legal, and cooperative means.
Three silver linings:
Unscrupulous people probably also have different goals from ours. Law enforcement will block them from attaining those goals, and better decision-making will hopefully not get them very far.
These systems are collaborative, so you can benefit from them more the more people collaborate on them (I’m not saying monotonically, just as a rough tendency). When you invite more people into some nefarious conspiracy, then the risk that one of them blows the whistle increases rapidly. (Though it may depend on the structure of the group. There are maybe some terrorist cells who don’t worry much about whistleblowing.)
If a group is headed by a narcissistic leader, the person may see a threat to their authority in a collaborative decision-making system, so that they won’t adopt it to begin with. (Though it might be that they like that collaborative systems can make it infeasible for individuals to use them to put their individual opinions to the test, so that they can silence individual dissenters. This will depend a lot on implementation details of the system.)
More speculatively, we can also promote and teach the system such that everyone who learns to use it also learns about multiverse-wide superrationality alias evidential cooperation in large worlds (ECL). Altruistic people with uncooperative agent-neutral goals will reason that they can now realize great gains from trade by being more cooperative or else lose out on them by continuing to defect.
We can alleviate the risk further by marketing the system mostly to people who run charities, social enterprises, prosocial research institutes, and democratic governments. Other people will still learn about the tools, and there are also a number of malevolent actors in those generally prosocial groups, but it may shift the power a bit toward more benevolent people. (The Benevolence, Intelligence, and Power framework may be helpful in this context.)
Finally, there is the option to make it hard to make models nonpublic. But that would have other downsides, and it’s also unlikely to be a stable equilibrium as others will just run a copy of the software on their private servers.
Benefitting Unscrupulous People
A system that improves collective decision making is likely value-neutral, so it can also be used by unscrupulous agents for their nefarious ends.
Moreover unscrupulous people may benefit from it more because they have fewer moral side-constraints. If set A is the set of all ethical, legal, cooperative methods of attaining a goal, and set B is the set of all methods of attaining the same goal, then A ⊆ B. So it should always be as easy or easier to attain a goal by any means necessary than only by ethical, legal, and cooperative means.
Three silver linings:
Unscrupulous people probably also have different goals from ours. Law enforcement will block them from attaining those goals, and better decision-making will hopefully not get them very far.
These systems are collaborative, so you can benefit from them more the more people collaborate on them (I’m not saying monotonically, just as a rough tendency). When you invite more people into some nefarious conspiracy, then the risk that one of them blows the whistle increases rapidly. (Though it may depend on the structure of the group. There are maybe some terrorist cells who don’t worry much about whistleblowing.)
If a group is headed by a narcissistic leader, the person may see a threat to their authority in a collaborative decision-making system, so that they won’t adopt it to begin with. (Though it might be that they like that collaborative systems can make it infeasible for individuals to use them to put their individual opinions to the test, so that they can silence individual dissenters. This will depend a lot on implementation details of the system.)
More speculatively, we can also promote and teach the system such that everyone who learns to use it also learns about multiverse-wide superrationality alias evidential cooperation in large worlds (ECL). Altruistic people with uncooperative agent-neutral goals will reason that they can now realize great gains from trade by being more cooperative or else lose out on them by continuing to defect.
We can alleviate the risk further by marketing the system mostly to people who run charities, social enterprises, prosocial research institutes, and democratic governments. Other people will still learn about the tools, and there are also a number of malevolent actors in those generally prosocial groups, but it may shift the power a bit toward more benevolent people. (The Benevolence, Intelligence, and Power framework may be helpful in this context.)
Finally, there is the option to make it hard to make models nonpublic. But that would have other downsides, and it’s also unlikely to be a stable equilibrium as others will just run a copy of the software on their private servers.