[Cause Exploration Prizes] Software Systems for Collective Intelligence
This anonymous essay was submitted to Open Philanthropy’s Cause Exploration Prizes contest and posted with the author’s permission.
If you’re seeing this in summer 2022, we’ll be posting many submissions in a short period. If you want to stop seeing them so often, apply a filter for the appropriate tag!
In this document I will attempt to describe a cause area that is relatively neglected, or rather, unexplored, given its potential importance. The issues in question, generally, are coordination and governance. I argue that while there are continued efforts to improve traditional institutional governance, too little attention is placed on inventive efforts toward methods of focusing collective intelligence in ways that can influence traditional systems, provide coordination across traditional systems, and have the long-term potential to replace parts of existing institutional infrastructure. Such changes may in fact be necessary in order to address a set of encroaching catastrophic risks.
The Problem
Traditional forms of governance have failure modes. Democracies can fail where expertise is required for public decisions. Authoritarian systems are overburdened at the top and subject to the whims of individuals who, at any given moment, may not serve as effective leaders. Consensus and collaboration efforts can be sluggish and fail at scale. Where institutions do at times succeed at enforcing and incentivizing systems of trust, those solutions will often be confined by physical or logical borders and are constrained from establishing trust and coordination across domains.
In some sense, the problem is simple: The world is facing a set of encroaching catastrophic and existential risks and people are bad at coordination. It could be argued that the greatest coordination failure at the moment is the failure to put serious effort into solving coordination. The question then is whether or not there are as-yet-unimagined solutions that might move us toward better and more effective collective intelligence systems focused on addressing those risks.
It seems inevitable that a technological solution to governance in general will become part of the landscape. After business, entertainment, and social, it is one of the last frontiers to remain largely offline. There may be an important opportunity at this stage to create a solution (constructed to define and accomplish EA goals) in the commons before corporate or state-sponsored solutions gain traction.
Existing Efforts
There are a number of positive moves in this direction. The vTaiwan project moves active democracy closer to the populace and, using tools like Pol.is and others, makes consensus building more fluid and transparent.
Early efforts toward decentralized governance in the form of DAOs continue to show promise. The Gitcoin DAO is an increasingly successful effort toward addressing a well-known coordination problem (underfunding of open source software projects).
There are a number of academic investigations into the necessity for improved collective intelligence as a means to address catastrophic risk: Yang and Sandberg, 2022 provides a current assessment of the state of the art. Promising investigations into adaptive and resilient methods of governance include: Choi et al., 2001; Walker et al., 2010; Haasnoot et al., 2013; Kwakkel et al., 2016; Folke et al., 2010; Farmer et al., 2010. These methods are evaluated and summarized here: Fisher, L. & Sandberg, A. 2022.
Investigation Area
Networked software solutions provide an opportunity to provide new organizational structures and behavioral enforcement systems based on our best ideas around optimizing collective intelligence toward risk mitigation and positive longterm outcomes. Further, progress has brought us to a place where AI assistance towards those ends could increasingly provide useful guidance and enforcement, perhaps preempting our attempts to foil cooperative collaboration for all the usual reasons.
While there are no direct paths to replace existing forms of government with borderless software systems, there are perhaps open paths to creating enough engagement to change the ways in which a given government legislates. Consider as a hypothetical example that Facebook or Twitter were a unified voting block and lobby, its users motivated by the need for alignment rather than combative engagement. Consider also that the optics of a non-governmental body effectively addressing problems, some of which are in areas poorly addressed by government, may provide an early enabling condition for changes in existing institutions.
What might such a system look like? While a social media site with the addition of issue voting is an interesting idea, it would not be sufficiently equipped to effectively coordinate the collective intelligence of its participants. A successful coordination system would seem to need, at least, the following qualities:
Users are impelled to address their reasoning and philosophical deficits in order to contribute and have tools available to address those deficits.
A system in place to give users feedback on their ability to contribute (forecasting, decision making, clarity of communication, etc.)
Users with better contribution abilities can be promoted and have increased weighting toward general decision making.
Users with proven subject matter expertise (and sufficient general contribution abilities) are given increased weighting toward subject matter decisions.
System structure is designed to limit the scope of relationships to a number that is conducive to collaboration (any given user’s graph node count should be under Dunbar’s number, for example).
System allows rapid creation of adaptive subnetworks to respond to issues at different levels of urgency and to identify important points of intervention.
System employs a cryptographically secure, flexible, and auditable voting system.
Everyone welcome at some level of participation.
The science on optimization of collaboration and collective intelligence is leveraged effectively.
Many small-scale iterations of invention and error correction may be required before a viable such system could be released in the wild. In my estimation, there are a number of variables that need to be thoughtfully considered and tested, including but not limited to:
Graph structure
Level of user anonymity
Voting methods
Evaluation and training methods
Incentives
As an example, suppose iteration one of such an effort had the following qualities:
A hierarchy of 5-member councils in which each user is both a member of one council and a chair of another. Top level council (level 0) is seeded with persons known to be capable thought leaders and collaborators, perhaps from EA or EA-adjacent organizations. The next level of councils (level 1) are divided into system-dictated areas of concern that match with the skills and interests of level 0 council members (acting as level 1 council chairs) and are populated with invitees from each chair. Invitations to level 1 lean toward participants with connection to existing academic/institutional organizations. Participation in deeper council levels (created as needed) will include avenues for membership from public participants of the system. Note that creation of new councils as issues move from coarse to granular could provide for a limited exponential growth. Approval processes would need to keep management of growth rate in mind.
Public is invited to comment/vote on issues transparently deliberated by councils a la Pol.is.
Public contribution is incorporated into voting council deliberations and decisions.
Financial incentives are provided to encourage participation in forecasting tournaments, problem-solving collaboration games, issue-specific essay contests, etc.
Training resources are provided to assist users in gaining greater skill and understanding of various forms of incentivized contests while integrating general instruction toward scientific thinking.
Various forms of evaluation (results of above contests, upvoting, narrow AI classification of contributions, etc) are used to make decisions about eligibility for voting councils and council promotion decisions. Such decisions will take into account the current science on the kind of diversity required to build effective collaboration. In essence, system participants shown to be the best forecasters, problem solvers, theorists, investigators, decision makers, etc. are placed within the system hierarchy in a manner that promotes more effective collective coordination.
Where long timelines are involved, high-level councils can create long deliberation/exploration/refinement cycles in assigned deep subnetworks (identified by areas of expertise, etc.) while more immediate concerns can be deliberated and decided quickly in pre-selected and adaptable subnetworks at or near the top of the network graph. -Note that, with time, top level councils become more populated with qualified public members that bubble up so the sense of meritocratic fairness should increase commensurately.
A charitable fund is associated with the system and voting council members will make decisions about which cause areas to fund and how to best fund them. Existing EA organizations are encouraged to contribute to the fund and participate in the system, perhaps incentived by the need for better coordination across organizational boundaries. As end user engagement increases and participants become more directly involved with problem areas and their details, public contributions to the fund may follow.
Further iterations of the system might include, for instance, AI-powered collaboration assistance that ensures deliberations do not fall into known pitfalls that impede successful cooperation.
The existence of a charitable fund requiring distribution and direction decisions provides users an incentive to deliberate and a meaningful reason to participate. A contest structure with financial and reputational stakes provides another incentive to participate and evolve within the institution.
The above example is provided not as a solution (there are probably a thousand better starting points) but to demonstrate that it may be possible to create a system that focuses collective intelligence while maintaining something akin to the appeal of current social media sites. Participants, beyond being provided new forms of community, will be provided engagement via the ability to contribute to problems they care about. Dunking, tribalism, and sideline criticism have the potential to be replaced by the necessity to collaborate and directly face well-explicated facets of issues at hand.
Possible Next Steps
A new nonprofit organization with a board composed of knowledgeable EA generalists and EA-supportive software founders could, for a relatively low investment, commission an investigation into the design of such a system. Funding decisions around the software development process could follow from the output of that effort.
Direct funding of such an effort by a philanthropist or a narrowly focused philanthropic organization has some potential to poison perceptions of the effort. Getting open-ended participation across organizations at the start would perhaps provide a better foundation for a successful project.
Uncertainties
The notion of building a software-based system of governance of this kind is one that is untried. There are myriad ways to approach it and likely infinite ways to fail. Good minds would need to be put to the task of design. Even so, opportunities exist to build such a system such that:
The wrong balance between direct participation and meritocracy creates a perception of inequality.
Complicated and/or slow processes around decision making, deliberation, promotion, etc. inhibit participation.
Failure to be adaptable to human issues (absences, flagging participation, unhelpful alliances, etc.) could inhibit or corrupt processes toward desired outcomes.
Resolution processes don’t adequately address conflicts and philosophical differences at crucial decision points.
Incentives toward participation aren’t carefully devised and pervert ultimate goals.
System processes aren’t carefully designed to thwart improperly motivated participation (party identification, greed, ego, etc.)
Perception issues keep the project from growing to a desired scale.
Designed processes make scaling too slow or too fast.
If, as I suspect, the best outcomes require initially seeding the network with a chosen set of people who have the ability to foster good collaboration and make reasoned choices, then there will exist the potential for the perception of elitism. As mentioned earlier, this perception could improve as public participants earn status. In the meantime, however, a very human seeding process will be required and without the kind of computer-assisted collaboration optimization that we would expect a smart, automated system to have over time.
Conclusion
A potential system of this kind at a large scale comes with many open questions and has many possible instantiations. Given the current set of technological tools, and given that the lack of solutions in this space could be filled by efforts with less equitable intentions, it seems the timing is good for embarking on such a project. Initial investment should be small and given the very difficult to see but somewhat possible future in which catastrophic/existential risk X might not otherwise be addressed, it makes sense to me to start the investigation.
Works Cited
Choi, T.Y., Dooley, K.J. & Rungtusanatham, M. (2001)
Supply networks and complex adaptive systems: control versus emergence.
Journal of Operations Management
Fisher, L. & Sandberg, A. (2022)
A Safe Governance Space for Humanity: Necessary Conditions for the Governance of Global Catastrophic Risks.
Global Policy
Folke, C., Carpenter, S.R., Walker, B., Scheffer, M., Chapin, T. & Rockström, J. (2010)
Resilience thinking: integrating resilience, adaptability and transformability.
Ecology and Society
Haasnoot, M., Kwakkel, J., Walker, W.E. & ter Maat, J. (2013)
Dynamic adaptive policy pathways: A method for crafting robust decisions for a deeply uncertain world.
Global Environmental Change
Kwakkel, J., Haasnoot, M. & Walker, W. (2016)
Coping with the wickedness of public policy problems: Approaches for decision-making under deep uncertainty.
Journal of Water Resources Planning and Management
Walker, W.E., Marchau, V.A.W.J. & Swanson, D. (2010)
Addressing deep uncertainty using adaptive policies.
Technological Forecasting & Social Change.
Chuqiao Yang, V. and Sandberg, A., (2022).
Collective Intelligence as Infrastructure for Reducing Broad Global Catastrophic Risks.
arXiv preprint arXiv:2205.03300
Attempted TL;DR [please comment if I missed something and I’ll fix it] :
Create software systems that will replace parts of government and perform better than them, such as interesting ways of voting which are both accessible and give citizens a different weighted voted based on what fits the situation.Edit: GG (who is the author, I think) says: It’s more about experimenting towards better solutions. Probably better to read that comment in full.
I’d say that the spirit of the post is less to suggest a solution than to point out the perhaps fruitful process of experimenting toward a solution. The idea of weighting a decision-making process toward more qualified decision makers (however that is determined) makes sense to me, as does experimentation with untried formal systems of managing collective intelligence and cooperative mitigation of catastrophic risk (some academic examples of which are cited in the post). Also, I’d say that the proposal is less about replacing aspects of governments than it is about providing a clear example of more effective governance, one that could perhaps influence existing means of governance in a number of ways.
My opinion:
“however that is determined” is where the magic happens, not a side point
As a naive example to explain what I’m talking about: if we give more weight to experts, there is the question of “how do we decide who are the experts”. Do we use expert experts? Who decides who those are?
The “Software systems” won’t be the challenging part of this project
Beyond, maybe, trying to give every person one vote (one “identity”) that can be used through the internet , which turns out to be really hard to do without huge problems, and the blockchain community is working really hard on already
Agree with these points. In the post, I give a toy example of a possible system in which various forms of contests can be used to assess a member’s ability to contribute. It seems as if identifying good generalists might be an easier task than identifying subject matter experts. I would imagine any process to identify expertise would include credentials and track records but I think it may be more important that a community of people that have already established trustworthiness are willing to take bets on a given individual. I think it’s very likely true that being a good generalist is a prerequisite to being an effective subject matter expert. But, yeah, lots of questions.
Thank you, updated.
Btw I personally really like how the Ethereum community experiments with things like this, seems super healthy to me
By voting, do you mean elections or referenda? If so, I don’t think that would be a good example, it really sells short the entire concept of software systems augmenting collective intelligence.
By referencing voting, I was referring to one possible input into a decision-making process. I wasn’t considering the notion of electing human representatives. Rather, the outputs of that collective decision-making process could be any number of things, and would be determined by the nature of the collective effort. In the post I suggest one possible such decision making body that controls the purse of a charitable organization. In such a case, the output of the process could include decisions around funding determinations, rule or policy decisions, the creation of a lobbying effort, etc.
It’s so cool seeing articles that align with what I’ve been wanting to see for years! Holden also recently wrote about governance but with a more external-to-EA lens (for example, to responsibly govern AI companies) rather than using better governance to solve issues in EA causes and within EA itself.
This work is very aligned with what Roote is working on (one of the organizations I help run) and our work on meta existential risk (although we don’t explicitly name collective intelligence as a potential part of the solution for this problem). We think that improving governance and collective intelligence is very important. We’re fans of the civic tech efforts in Taiwan and our building societal-level information and coordination software with our Civic Abundance project (we are looking for funding and trying to hire a project lead!). Among the things we’ve looked at is integrating with Manifold to experiment with futarchy (but in the context of an informational dashboard, not tied to the actual governance process).
We’re very aligned with Web3 for social good (like Greenpilled) and Gitcoin. I personally believe that EA itself needs a better mechanism to fund public goods within the EA community, given that EAIF seems to commonly ask public goods projects to become revenue positive which, well, is usually not possible for public goods to do without becoming private goods. Very unfortunate.
I was not aware that there was any research on the specific implications of collective intelligence on existential risk. I would have been excited to read a quick summary of the main points/findings or some hyperlinked articles.
Thanks for writing this!!
I am now wishing that this article had been written before I submitted my post. It points to new and existing efforts to put collective intelligence toward coordination and governance. It also makes the case for what I was trying to get across in a much more persuasive and rigorous way:
https://www.wired.com/story/collective-intelligence-democracy/
I suppose that if I could have seen this before submitting, I would have changed the spirit of my post to be something along the lines of “make use of collective intelligence systems to broadly manage EA giving, providing an example of efficient governing”, or something similar. I still think it makes sense to find a way to engage a large number of people committed to working together in new and experimental ways toward more coordinated and more useful outcomes.
I broadly agree that there’s ample opportunity for more digitally native public institutions and to improve government operations. Here is a post on some of the potential that I see there. Here is another looking at improving state capacity more broadly.
It seems like the Ethereum Foundations foray into a DAO would be worth exploring as a case study here? I like the idea of experimenting with novel approaches to governance with some subset of EA funding. I’m not sure though that this idea is fully fleshed out and might benefit from further fleshing out before convening a council to test the idea.
Replacing existing state institutions (or functional aspects of them) with digitally native systems will almost certainly have many benefits. That’s not really what I’m getting at here. I’m saying that it makes sense find better ways to govern and coordinate in general. The process of designing a software system with the general goal of focusing coordination could produce novel ways of doing that and/or it could leverage good ideas we already have around successful collaboration structures, focusing collective intelligence, etc. If there exists a subversive path in which we build such a system, for instance, to run a charitable organization and that process is successful enough to influence how other things are governed, then that is a win. My primary point is embarrassingly unfocused: getting better at coordination, given impending risks, is important and it might be good to start building experiments toward that goal.
Decentralized systems are attractive in that borders (of many varieties) inhibit coordination. I have some worry around the idea of using a crypto ecosystem as the basis for a something of this type. Whether the reasons are good or bad, the perception of cryptocurrency is divisive. I also worry that financial aspects around running a DAO could provide barriers to entry or could pervert incentives. I’m not really deep in that world so maybe my worries are overblown.
A lot of people have been hoping for software systems like these, for decades. I have no doubt that EA can oversee/fund extreme improvements on all sorts of existing systems and designs, and that reinventing the wheel might be necessary whenever existing systems are inaccessible and thus must be reinvented.
The post is very much about reinventing the wheel and that makes specifying a vision difficult because it is impossible to know what experimentation and invention will produce. -There are indeed a number of efforts to apply software to decision processes (or to improve voting). Some are good and have proven success. Most have not had great reach. The vision I intend to get across in the post is for a system that is accessible, provides the possibility of growth via feedback systems, and allows the best ideas to be promoted. I’ve seen academic work that investigates processes toward collective intelligence. The post attempts to make the point that it may be wise to move that to practice before state or corporate efforts contain the space and make something in the commons less possible.