Survey on intermediate goals in AI governance

[MichaelA🔸]

It seems that a key bottleneck for the field of longtermism-aligned AI governance is limited strategic clarity (see Muehlhauser, 2020, 2021). As one effort to increase strategic clarity, in October-November 2022, we sent a survey to 229 people we had reason to believe are knowledgeable about longtermist AI governance, receiving 107 responses. We asked about:

• respondents’ “theory of victory” for AI risk (which we defined as the main, high-level “plan” they’d propose for how humanity could plausibly manage the development and deployment of transformative AI such that we get long-lasting good outcomes),

• how they’d feel about funding going to each of 53 potential “intermediate goals” for AI governance,^[1]

• what other intermediate goals they’d suggest,

• how high they believe the risk of existential catastrophe from AI is, and

• when they expect transformative AI (TAI) to be developed.

We hope the results will be useful to funders, policymakers, people at AI labs, researchers, field-builders, people orienting to longtermist AI governance, and perhaps other types of people. For example, the report could:

• Broaden the range of options people can easily consider

• Help people assess how much and in what way to focus on each potential “theory of victory”, “intermediate goal”, etc.

• Target and improve further efforts to assess how much and in what way to focus on each potential theory of victory, intermediate goal, etc.

You can see a summary of the survey results here. Note that we will expect readers to abide by the policy articulated in “About sharing information from this report” (for the reasons explained there).

Acknowledgments

This report is a project of Rethink Priorities–a think tank dedicated to informing decisions made by high-impact organizations and funders across various cause areas. The project was commissioned by Open Philanthropy. Full acknowledgements can be found in the linked “Introduction & summary” document.

If you are interested in RP’s work, please visit our research database and subscribe to our newsletter.

1. ^

   Here’s the definition of “intermediate goal” that we stated in the survey itself:

   By an intermediate goal, we mean any goal for reducing extreme AI risk that’s more specific and directly actionable than a high-level goal like ‘reduce existential AI accident risk’ but is less specific and directly actionable than a particular intervention. In another context (global health and development), examples of potential intermediate goals could include ‘develop better/​cheaper malaria vaccines’ and ‘improve literacy rates in Sub-Saharan Africa’.