Improving Institutional Decision-Making: a new working group

IanDavidMoss28 Dec 2020 5:47 UTC
157 points
24 comments10 min readEA link

By Ian David Moss, Vicky Clayton and Laura Green

Summary

  • This post describes recent and planned efforts to develop improving institutional decision-making (IIDM) as a cause area within and beyond the effective altruism movement.

  • Despite increasing interest in the topic over the past several years, IIDM remains underexplored compared to “classic” EA cause areas such as AI safety and animal welfare.

  • To help address some questions that have come up in our community-building work, we provide a working definition of IIDM, emphasizing its interdisciplinary nature and potential to bring together insights across professional, industry, and geographic boundaries.

  • We also describe a new meta initiative aiming to disentangle and make intellectual progress on IIDM over the next year. The initiative includes several research and community development projects intended to enable more confident funding recommendations and career guidance going forward.

  • You can get involved by volunteering to work on our projects, helping us secure funding, or giving us feedback on our plans.

Introduction

In 2017, 80,000 Hours published Jess Whittlestone’s problem profile on the topic of improving institutional decision-making (IIDM), which deemed the cause area “among the most pressing problems to work on” and suggested that “improving the quality of decision-making in important institutions could improve our ability to solve almost all other problems.” In the years since, we’ve seen signs of steadily increasing interest in IIDM within the EA community: IIDM-related talks, meetups and discussion channels have been included at most recent EA Global conferences, and a Facebook group founded to centralize discussion on the topic now has nearly 900 members. Today, 80,000 Hours continues to list IIDM among its priority problem areas and names “Building capacity to explore and solve problems,” a broad category that includes IIDM, as one of its top two overall priorities for career paths.

Still, IIDM remains underexplored compared to “classic” EA cause areas such as AI safety and animal welfare. Up until now, there has not been a formal, globally focused umbrella organization dedicated to IIDM within the effective altruism ecosystem, leaving a gap of coordination in the field. There are legitimate questions about the effectiveness and tractability of interventions in the space that need to be resolved in order to be able to direct donations or career tracks with confidence. And we know from conversations with others in the EA community that IIDM’s interdisciplinary nature can make the cause area feel fuzzy or overly broad to some.

For these reasons, the three of us are stepping up to act as a focal point for people interested in “disentangling” and making intellectual progress on IIDM in 2021. This work grows out of a year’s worth of informal exploration that has taken place since the first official IIDM meetup at EAG London 2019. In this article, we’ll share our working definition of IIDM and some key points from our recently developed operational plan.

What is improving institutional decision-making?

Decision-making at major institutions is shaped by a complex web of individual judgments, value systems, organizational structures and routines, leadership behaviors, incentives, social influences, and external conditions. As such, it’s worth taking a moment to define and explain what we mean by “improving institutional decision-making” a little more clearly.

Let’s focus first on the “decision-making” part. The classic decision problems taught in economics textbooks describe fairly straightforward analytical problems: given two or more mutually exclusive options, compare their costs and benefits and select the one that has the highest expected value. In practice, however, and especially in an institutional context, decision-making is rarely this simple. Theoretical work by decision professionals over the past half-century has yielded a useful framework for the concept of “decision quality” featuring the following six components:

  • Framing the decision: are we clear about the actual decision that needs to be made at this time?

  • Defining goals and values: what is this decision optimizing for, and how should we weigh competing goals against each other?

  • Generating options: what are the possible actions we could take, and what are the advantages and disadvantages of each of those options?

  • Gathering information: what evidence is relevant to this decision and has the potential to change our minds about the best way forward?

  • Sound reasoning: how should we integrate all of the various considerations together into a coherent, holistic judgment about what to do next?

  • Commitment to action: once the decision is made, can we expect that it will be fully implemented?

In practice, organizational decision-making is more complicated than this because many important decisions unfold implicitly within or across groups of stakeholders, often without awareness or acknowledgement that there is any decision being made.

Understanding the decision-making process itself more clearly helps us identify different levers for “improving” it. From an EA perspective, quality improvement can take place along several dimensions, such as:

  • Increasing clarity and shared awareness of the landscape of decisions to be made and who is responsible for making them

  • Making better choices about which decisions to prioritize (“deciding what to decide”)

  • Improving the accuracy of institutions’ judgments and predictions

  • Improving the range and promise of the options under consideration

  • Bringing institutions’ goals and values into greater alignment with effective altruist values

  • Ensuring that new institutions incorporate design principles that promote high-quality decisions

  • Shifting the environment in which an institution operates in ways that are likely to be beneficial to decision quality, e.g. by improving political incentives, regulatory expectations, or leadership selection processes

One of our priorities over the next year, as we’ll describe further below, will be to assess what existing research tells us about the general effectiveness of different decision-making interventions as well as what the greatest barriers to good decision-making seem to be in practice. Ultimately, we anticipate that efforts to improve decision-making at institutions will need to be tailored to the institution in question, using something like the COM-B model of behavioral change to develop a strategy that’s fit for purpose.

Finally, what do we mean by “institutions”? In her problem profile for 80,000 Hours, Whittlestone writes, “We think that improving the decision-making competence of key institutions may be particularly crucial, as the risks we face as a society are rapidly growing.” The article doesn’t indicate what is meant by “key institutions,” but we think that this is an important ambiguity to resolve. As a first step toward a working definition, we consider key institutions to be centrally managed bodies of one or more people in a direct position to allocate disproportionate funds and/​or set rules, incentives, and norms affecting the lives of many. Over the next year, we intend to quantify this definition further and launch an inquiry into what specific institutions in the world fall most clearly under this umbrella.

To sum up, then, “improving institutional decision-making” is about increasing both the technical quality and EA alignment of the most important decisions made by the world’s most important decision-making bodies. It is closely related to the project of effective altruism itself, but rather than focusing on making the world better through the promulgation of specific applied ideas (like ending factory farming), which may involve some IIDM as a means to an end, or recruiting specific people to the cause and advancing their careers, it emphasizes building sustained capacity to make high-quality and well-aligned decisions within powerful institutions themselves.

It’s important to emphasize that this definition is merely a starting point for our work. We received useful critical feedback on it prior to publication from several people working on IIDM-related initiatives in the EA community, some of which we’ve incorporated into the version you see here. Some important questions that remain open for us include how much to concern ourselves with ecosystems of interacting institutions in addition to dynamics within institutions; whether decentralized decision-making bodies such as legislatures and parliaments should be within our scope; and how far upstream of the decision-making process itself our scope extends. Our hope is that the disentanglement research yielded by the projects described below will help us come up with more precise and actionable ways of breaking down the challenges of IIDM.

IIDM within and outside of EA

While there is no organization at this time in the EA universe with the explicit mission of coordinating the IIDM community, there are several organizations and initiatives doing IIDM-related work that have strong EA roots. These include the All-Party Parliamentary Group for Future Generations, the Geneva Science-Policy Interface, the Quantified Uncertainty Research Institute, Czech Priorities, and IDinsight, among others. (If you run an IIDM-related initiative and are not already in contact with us, please get in touch!)

In the broader working world, “IIDM-like” initiatives and scholarship exist under many different names, including decision analysis, organizational behavior, operations research, management science, strategic learning, policy analysis, change management, knowledge management, and more. Each of these schools of thought and communities of practice have distinct intellectual histories which strongly shape their present-day spheres of influence. For example, “organizational behavior” are two words that rarely appear together outside of a university setting, while “strategic learning” is primarily a term of art among large staffed private foundations. Because of divergent socio-historical roots like these, key institutions in different fields and geographies frequently use different language and frameworks to describe similar phenomena, and miss out on theoretical and scientific advances achieved in parallel contexts.

The fact that “improving institutional decision-making” is not a widely used term outside of the EA community presents us with both a challenge and an opportunity. On the one hand, operating under a neologism might cause confusion in initial interactions with potential stakeholders due to its unfamiliarity. On the other hand, doing so frees both our growing community of interest and effective altruism in general to reach across disciplines, sectors, continents, and cultures in our efforts to improve the functioning of the world’s most important deliberative bodies. By treating IIDM as a meta-term that includes or relates to language that’s more familiar to our target audiences, our community can function as a kind of global connective tissue between IIDM-related initiatives and actors across boundaries that might not otherwise be bridged.

Key initiatives for 2021

Our operational plan for the coming year encompasses several projects that fit under two general categories: i) research and ii) development of the IIDM community. At present, we are an all-volunteer team, although we see acquiring funding as a tractable way to lift some constraints on our rate of progress.

The Most Important Institutions in the World

Our first research project will explore the optimal “target audiences” for IIDM-related efforts. While we come to this work with some initial intuitions about what kinds of institutions have outsized importance for world outcomes (large governments, foundations, multilateral agencies, etc.), we want to develop a more formal rubric for evaluating and ranking the potential for engaging productively with specific institutions. Open questions include: How do we become aware of candidate key institutions? How should we measure and compare the different types of influence (funds, norms etc) an institution can have? How do we weigh how much influence the institution has against how tractable it will be to influence its decision-making processes? How do we work with (or around) the constraints it might face? (To support our work on this project, we plan to assemble an advisory team of individuals with deep knowledge of or direct experience working in a wide range of institutions around the globe; if you feel you have such expertise and would like to be involved, please see the “How you can help” section below).

“What Works” in Institutional Decision-Making

Once we’ve identified and begun to prioritize key institutions, we will need to determine how we can engage with them (or help others engage with them) most productively. Accordingly, our other main research activity in 2021 will be to conduct a review of interventions to improve institutional decision-making. Drawing on multiple academic disciplines as well as practice knowledge from policymakers, think tanks, consultants, and other thought leaders, we will aim to identify interventions which offer strong evidence of improving decision-making outcomes in an institutional setting, promising approaches that may lack a robust evidence base but are grounded in compelling theory, and strategies to be avoided based on the evidence. The insights gained from this project could eventually feed into tailored strategies targeted at the key institutions identified from the first project. They will also begin to address one of the most important outstanding questions about IIDM as a cause area, which is whether there are tractable interventions that can reliably make a difference at scale. As with the “key institutions” analysis, we hope that this “what works in IIDM” project will be a living document and a core ongoing intellectual contribution of our community to the cause.

A Research Agenda for IIDM

We anticipate that our initial work to scope this cause area through the two projects above will yield a robust set of questions that demand further research. We’ll use these questions to form the basis of a research agenda that will guide our own future efforts and (we hope) serve as a source of inspiration for other researchers interested in these topics. Along the way, we’ll also initiate high-level discussions of how we think IIDM fits into theoretical questions in the EA space, such as how IIDM relates to ideas like near vs. long-termism and worldview diversification. We expect the agenda to be an evolving document and will seek input from people with a range of disciplinary and professional backgrounds.

IIDM Resource Directory

Concurrently with the above initiatives, we have already begun work on a directory of IIDM-related resources. It will include important publications, organizations working in the space, individual thought leaders, curricula and training programs, career guides, tools and techniques, and more. The resource directory will function not only as a reference guide for the broader community but also as a stakeholder map for our own efforts to build relevant alliances and advance the cause area both within and outside of the EA community.

Community Engagement and Development

On the community and organisational development side, our primary objective in the next year will be to establish a firmer infrastructure for our work and tighter coordination among the many actors operating in this space. We’ll seek to connect organizations and people pursuing highly aligned work but who may not be aware of each other’s efforts, accelerate the sharing of knowledge and promising practices across disciplines and geographic boundaries, and establish a circle of advisors to provide feedback on our projects. Our hypothesis is that this will result in more and better work on the above initiatives, improve collaboration on existing projects, highlight critical gaps in the field, and contribute to our own learning that will be taking place in parallel. A better understanding of who is currently working on related projects should also help with career mentoring—people interested in contributing to the field could be connected with people who have relevant experience.

Our chief goal in choosing these five initiatives is to put ourselves in a position by the end of 2021 to make initial recommendations about where and how IIDM-related efforts can accomplish the most good going forward. With that said, we expect our plans will evolve as we receive feedback on them and continue learning about the problem space, and will aim to post on the EA Forum or other venues at regular intervals to keep the community up to date about our progress.

How you can help

If you would like to get involved in any of the initiatives above, we would love to hear from you! Please fill in this form to express interest. We are particularly keen to hear from i) people with relevant research expertise to help develop the “What Works” for IIDM evidence synthesis and ii) people well placed to “own” responsibility for sections of the resource directory. We anticipate the volunteering to be remote, and to be scalable to the amount of time you’d like to dedicate. We would also welcome the possibility to talk to anyone about funding opportunities.

Beyond volunteering opportunities, we also value your feedback and ideas. Who should we talk to? What resources and initiatives should we be aware of, especially outside of the United States and Europe? What else should we be doing to develop IIDM as a cause area? We want to avoid reinventing the wheel to the extent possible and are keen to collaborate with and support well-aligned existing initiatives. We look forward to reading your comments on this post and subsequent conversations.

About the authors

Ian started the IIDM Facebook group in 2018. As an independent consultant, he works with foundations, government agencies, and large NGOs to help people around the world. His previous ventures include a think tank and a self-governed research community of practice. Ian’s articles about improving institutional decision-making have appeared in Stanford Social Innovation Review and the Chronicle of Philanthropy.

Vicky is a data science manager at the What Works Centre for Children’s Social Care, and brings expertise in impact evaluation, behavioural economics and decision science. Previously she co-founded a successful accelerator for social ventures.

Laura co-founded EA France and ran the organisation as executive director. She previously studied Political Science (undergrad) and Economics (Master’s) at Sciences Po, in France. She is currently involved in a variety of EA projects, mostly around community building, and on an EU policy path.

We are very grateful to those who provided comments on a draft of this post, including Nora Ammann, Tamara Borine, Ozzie Gooen, Sam Hilton, Habiba Islam, David Nash, Konrad Seifert, Maxime Stauffer, Stefan Torges, Ben West, and Jess Whittlestone.

IanDavidMoss28 Dec 2020 5:47 UTC
157 points
24 comments10 min readEA link