This is a doc I made, and I suggest reading the doc rather than shortform version (assuming you want to read this at all). But here it is copied out anyway:
What is this doc, and why did I make it?
AI governance is a large, complex, important area that intersects with a vast array of other fields. Unfortunately, itās only fairly recently that this area started receiving substantial attention, especially from specialists with a focus on existential risks and/āor the long-term future. And as far as Iām aware there arenāt yet any canonical, high-quality textbooks or online courses on the topic.[1] It seems to me that this means this is an area where well-curated and well-structured reading lists, syllabi, or similar can be especially useful, helping to fill the role that textbooks otherwise could.[2]
Fortunately, when I started looking for relevant reading lists and syllabi, I was surprised by how many there were. So I decided to try to collect them all in one place. I also tried to put them in very roughly descending order of how useful Iād guess theyād be to a randomly chosen EA-aligned person interested in learning about AI governance.
I think this might help myself, my colleagues, and others who are trying to āget up to speedā, for the reasons given in the following footnote.[3]
I might later turn this doc into a proper post on the EA Forum.
This is more relevant to technical AI safety than to AI governance, but some categories are pretty relevant to AI governance, especially āAI strategy and policyā, āForecastingā, and āField buildingā
Tangential critique: I personally think that itās problematic and misleading that both The Precipice and this syllabus use the heading āunaligned artificial intelligenceā while seeming to imply that this covers all key aspects of AI risk, since I think this obscures some risk pathways.
My post Crucial questions for longtermists includes a structured list of questions related to the āValue of, and best approaches to, work related to AIā, and this associated doc contains readings related to each of those questions
I havenāt updated this much since 2020
Questions listed there include:
Is it possible to build an artificial general intelligence (AGI) and/āor transformative AI (TAI) system? Is humanity likely to do so?
What form(s) is TAI likely to take? What are the implications of that? (E.g., AGI agents vs comprehensive AI services)
What will the timeline of AI developments be?
How much should longtermistsā prioritise AI?
What forms might an AI catastrophe take? How likely is each?
What are the best approaches to reducing AI risk or increasing AI benefits?
I think many/āmost of these books were chosen for āseem[ing] likely to have passages relevant to the question of how well policy-makers will deal with AGIā
Many/āmost of these arenāt available as audiobooks; Luke turned them into audiobooks himself
You could also use research agendas related to AI governance as reading lists, by following the sources they cite on various topics. Relevant agendas include:
Longtermist AI policy projects for economistsāRisto Uuk (this doc was originally just made for Ristoās own use, so the ideas shouldnāt be taken as high-confidence recommendations to anyone else)
I think this is much more focused on technical AI safety than AI governance
Some Rethink Priorities staff may soon make a long, tiered reading list tailored to the AI governance project ideas we may work on. If it seems to me that this would be useful to other people, I might add a link to a version of it here.
I think there was also a short reading list associated with the EA In-Depth Fellowship
A related category to reading lists is newsletters that provide summaries and commentary of a bunch of research outputs. E.g.:
Rohinās
Jack Clarkeās
CSETās
ā¦
Mauricio Baker suggested that I or people reading this doc might also be interested in āsyllabi aimed at aspiring AI technical safety researchers, such as this one: Technical AI Safety Reading List. I have a vague sense that engaging with some of this content has been helpful for my having a better broad sense of whatās going on with AI safety, which seems helpful for governance.ā
Some parts of Krakovnaās AI safety resources and Mainiās AI Reading List may be quite useful for AI governance people, though I think theyāre more relevant for technical AI safety people
My thanks to everyone who made these lists, as well as to Mauricio Baker for pointing me to some of the lists.
Footnotes
[1] Though there are various presumably high-quality textbooks or courses with some relevance, some high-quality non-textbook books on the topic, some in-person courses that might be high-quality (I havenāt participated in them), and some things that fill somewhat similar roles (like EA seminar series, reading groups, or fellowships).
This collection should make it easier to find additional reading lists, syllabi, etc., and thus easier to find additional readings that have been evaluated as especially worth reading in general, especially worth reading on a given topic, and/āor especially good as introductory resources.
This collection should make it easier to find and focus on reading lists, syllabi, etc. that are better and/āor more relevant to oneās specific needs.
To help with this, please comment on this doc if you have opinions about anything listed.
Even before or without engaging with the actual items included in a given reading list, syllabus, or similar, engaging with the structure and commentary in that document itself could help one understand what the important components, divisions, concepts, etc. within AI governance are. And this collection should help people find more, better, and/āor more relevant such documents.
Collection of AI governance reading lists, syllabi, etc.
This is a doc I made, and I suggest reading the doc rather than shortform version (assuming you want to read this at all). But here it is copied out anyway:
What is this doc, and why did I make it?
AI governance is a large, complex, important area that intersects with a vast array of other fields. Unfortunately, itās only fairly recently that this area started receiving substantial attention, especially from specialists with a focus on existential risks and/āor the long-term future. And as far as Iām aware there arenāt yet any canonical, high-quality textbooks or online courses on the topic.[1] It seems to me that this means this is an area where well-curated and well-structured reading lists, syllabi, or similar can be especially useful, helping to fill the role that textbooks otherwise could.[2]
Fortunately, when I started looking for relevant reading lists and syllabi, I was surprised by how many there were. So I decided to try to collect them all in one place. I also tried to put them in very roughly descending order of how useful Iād guess theyād be to a randomly chosen EA-aligned person interested in learning about AI governance.
I think this might help myself, my colleagues, and others who are trying to āget up to speedā, for the reasons given in the following footnote.[3]
I might later turn this doc into a proper post on the EA Forum.
See also EA syllabi and teaching materials and Courses on longtermism.
How can you help
Please comment if you know of anything potentially relevant which I havenāt included!
Please comment if you have opinions on anything listed!
The actual collection
September AGI safety fundamentals curriculumāRichard Ngo
Alignment Newsletter DatabaseāRohin Shah
This is more relevant to technical AI safety than to AI governance, but some categories are pretty relevant to AI governance, especially āAI strategy and policyā, āForecastingā, and āField buildingā
AI Governance Reading ListāSERI 2021 SummerāMauricio Baker
Mauricio had also previously made a syllabus on the same topics: AI Governance Syllabus ā21.docx
Governance of AI Reading List ā Oxford Spring 2020 - Markus Anderljung
Reading Guide for the Global Politics of Artificial IntelligenceāAllan Dafoe
Iām guessing other lists made by people associated with GovAI already draw on and superseded this, but I donāt know
āResourcesā section from Guide to working in artificial intelligence policy and strategy ā 80,000 Hours
Note: I think the only book from there thatās available on Audible UK is The Second Machine Age.
But the description of the book sounds to me kind-of basic and not especially longtermism-relevant.
AI policy introductory reading listāNiel Bowerman (I think)
Governance of AIāSome suggested readings [v0.5, shared] - Ashwin Acharya
Drawn on for SERIās reading list
Artificial Intelligence and International Security Syllabus [public] - Remco Zwetsloot, 2018 (I think)
Books and lecture series relevant to AI governanceāme and commenters
Section on āUnaligned artificial intelligenceā from Syllabus ā The Precipice
Tangential critique: I personally think that itās problematic and misleading that both The Precipice and this syllabus use the heading āunaligned artificial intelligenceā while seeming to imply that this covers all key aspects of AI risk, since I think this obscures some risk pathways.
AI Policy Readings Draft.docxāEA Oxford
Drawn on for SERIās reading list
My post Crucial questions for longtermists includes a structured list of questions related to the āValue of, and best approaches to, work related to AIā, and this associated doc contains readings related to each of those questions
I havenāt updated this much since 2020
Questions listed there include:
Is it possible to build an artificial general intelligence (AGI) and/āor transformative AI (TAI) system? Is humanity likely to do so?
What form(s) is TAI likely to take? What are the implications of that? (E.g., AGI agents vs comprehensive AI services)
What will the timeline of AI developments be?
How much should longtermistsā prioritise AI?
What forms might an AI catastrophe take? How likely is each?
What are the best approaches to reducing AI risk or increasing AI benefits?
Good resources for getting a high-level understanding of AI riskāMichael Aird
AI governance intro readingsāFelipe Calero
Luke Muehlhauserās 2013 and 2014 lists of books heād listened to recently
I think many/āmost of these books were chosen for āseem[ing] likely to have passages relevant to the question of how well policy-makers will deal with AGIā
Many/āmost of these arenāt available as audiobooks; Luke turned them into audiobooks himself
A Contra AI FOOM Reading List ā Magnus Vinding
Described in SERIās reading list as a āList of arguments (of varied quality) against āfast takeoffāā
List of resources on AI and agencyāBen Pace
You could also use research agendas related to AI governance as reading lists, by following the sources they cite on various topics. Relevant agendas include:
(Note that I havenāt checked how well each of these agendas would work for this purpose. This list is taken from my central directory for open research questions.)
The Centre for the Governance of AIās research agenda ā 2018
Some AI Governance Research Ideasāthe Centre for the Governance of AI, 2021
Promising research projectsāAI Impacts, 2018
They also made a list in 2015; I havenāt checked how much they overlap
Cooperation, Conflict, and Transformative Artificial Intelligence (the Center on Long-Term Riskās research agenda) - Jesse Clifton, 2019
Open Problems in Cooperative AIāDafoe et al., 2020
Problems in AI Alignment that philosophers could potentially contribute toāWei Dai, 2019
Problems in AI risk that economists could potentially contribute toāMichael Aird, 2021
Technical AGI safety research outside AIāRichard Ngo, 2019
Artificial Intelligence and Global Security Initiative Research AgendaāCentre for a New American Security, no date
A survey of research questions for robust and beneficial AIāFuture of Life Institute, no date
āstudies which could illuminate our strategic situation with regard to superintelligenceāāLuke Muehlhauser, 2014 (he also made a list in 2012)
A shift in arguments for AI riskāTom Sittler, 2019
Longtermist AI policy projects for economistsāRisto Uuk (this doc was originally just made for Ristoās own use, so the ideas shouldnāt be taken as high-confidence recommendations to anyone else)
Annotated Bibliography of Recommended MaterialsāCHAI
I think this is much more focused on technical AI safety than AI governance
Some Rethink Priorities staff may soon make a long, tiered reading list tailored to the AI governance project ideas we may work on. If it seems to me that this would be useful to other people, I might add a link to a version of it here.
There may be additional relevant reading lists /ā syllabi /ā sections in the links given here: EA syllabi and teaching materialsāEA Forum
And here Courses on longtermism | Pabloās miscellany
I think there was also a short reading list associated with the EA In-Depth Fellowship
A related category to reading lists is newsletters that provide summaries and commentary of a bunch of research outputs. E.g.:
Rohinās
Jack Clarkeās
CSETās
ā¦
Mauricio Baker suggested that I or people reading this doc might also be interested in āsyllabi aimed at aspiring AI technical safety researchers, such as this one: Technical AI Safety Reading List. I have a vague sense that engaging with some of this content has been helpful for my having a better broad sense of whatās going on with AI safety, which seems helpful for governance.ā
Some parts of Krakovnaās AI safety resources and Mainiās AI Reading List may be quite useful for AI governance people, though I think theyāre more relevant for technical AI safety people
My thanks to everyone who made these lists, as well as to Mauricio Baker for pointing me to some of the lists.
Footnotes
[1] Though there are various presumably high-quality textbooks or courses with some relevance, some high-quality non-textbook books on the topic, some in-person courses that might be high-quality (I havenāt participated in them), and some things that fill somewhat similar roles (like EA seminar series, reading groups, or fellowships).
[2] See also Research Debt and Suggestion: EAs should post more summaries and collections.
[3]
This collection should make it easier to find additional reading lists, syllabi, etc., and thus easier to find additional readings that have been evaluated as especially worth reading in general, especially worth reading on a given topic, and/āor especially good as introductory resources.
This collection should make it easier to find and focus on reading lists, syllabi, etc. that are better and/āor more relevant to oneās specific needs.
To help with this, please comment on this doc if you have opinions about anything listed.
Even before or without engaging with the actual items included in a given reading list, syllabus, or similar, engaging with the structure and commentary in that document itself could help one understand what the important components, divisions, concepts, etc. within AI governance are. And this collection should help people find more, better, and/āor more relevant such documents.