Survey on intermediate goals in AI governance

It seems that a key bottleneck for the field of longtermism-aligned AI governance is limited strategic clarity (see Muehlhauser, 2020, 2021). As one effort to increase strategic clarity, in October-November 2022, we sent a survey to 229 people we had reason to believe are knowledgeable about longtermist AI governance, receiving 107 responses. We asked about:

  • respondents’ “theory of victory” for AI risk (which we defined as the main, high-level “plan” they’d propose for how humanity could plausibly manage the development and deployment of transformative AI such that we get long-lasting good outcomes),

  • how they’d feel about funding going to each of 53 potential “intermediate goals” for AI governance,[1]

  • what other intermediate goals they’d suggest,

  • how high they believe the risk of existential catastrophe from AI is, and

  • when they expect transformative AI (TAI) to be developed.

We hope the results will be useful to funders, policymakers, people at AI labs, researchers, field-builders, people orienting to longtermist AI governance, and perhaps other types of people. For example, the report could:

  • Broaden the range of options people can easily consider

  • Help people assess how much and in what way to focus on each potential “theory of victory”, “intermediate goal”, etc.

  • Target and improve further efforts to assess how much and in what way to focus on each potential theory of victory, intermediate goal, etc.

If you’d like to see a summary of the survey results, please request access to this folder. We expect to approve all access requests,[2] and will expect readers to abide by the policy articulated in “About sharing information from this report” (for the reasons explained there).


This report is a project of Rethink Priorities–a think tank dedicated to informing decisions made by high-impact organizations and funders across various cause areas. The project was commissioned by Open Philanthropy. Full acknowledgements can be found in the linked “Introduction & summary” document.

If you are interested in RP’s work, please visit our research database and subscribe to our newsletter.

  1. ^

    Here’s the definition of “intermediate goal” that we stated in the survey itself:

    By an intermediate goal, we mean any goal for reducing extreme AI risk that’s more specific and directly actionable than a high-level goal like ‘reduce existential AI accident risk’ but is less specific and directly actionable than a particular intervention. In another context (global health and development), examples of potential intermediate goals could include ‘develop better/​cheaper malaria vaccines’ and ‘improve literacy rates in Sub-Saharan Africa’.

  2. ^

    If two days after you request access you still haven’t received access, this is probably just due to a mistake or delay on our end, so please request access again.