What would you recommend as the best introduction to concerns (or lack thereof) about risks from AI?
If you have time and multiple recommendations, I would be interested in a taxonomy. (E.g. this is the best blog post for non-technical readers, this is the best book-length introduction for CS undergrads.)
I agree with Aidanās suggestion that Human Compatible is probably the best introduction to risks from AI (for both non-technical readers and readers with CS backgrounds). Itās generally accessible and engagingly written, itās up-to-date, and it covers a number of different risks. Relative to many other accounts, I think it also has the virtue of focusing less on any particular development scenario and expressing greater optimism about the feasibility of alignment. If someoneās too pressed for time to read Human Comptabile, the AI risk chapter in The Precipice would then be my next best bet. Another very readable option, mainly for non-CS people, would be the AI risk chapters in The AI Does Not Hate You: I think they may actually be the cleanest distillation of the āclassicā AI risk argument.
For people with CS backgrounds, hoping for a more technical understanding of the problems safety/āalignment researchers are trying to solve, I think that Concrete Problems in AI Safety, Scalable Agent Alignment Via Reward Modeling, and Rohin Shahās blog post sequence on āvalue learningā are especially good picks. Although none of these resources frames safety/āalignment research as something thatās intended to reduce existential risks.
I think that AI Governance: A Research Agenda would be the natural starting point for social scientists, especially if they have a substantial interest in risks beyond alignment.
Of course, for anyone interested in digging into arguments around AI risk, I think that Superintelligence is still a really important read. (Even beyond its central AI risk argument, it also has a ton of interesting ideas on the future of intelligent life, ethics, and the strategic landscape that other resources donāt.) But itās not where I think people should start.
FWIW, hereās an introduction to longtermism and AI risks I wrote for a friend. (My friend has some technical background, he had read Doing Good Better but not engaged further with EA, and I thought heād be a good fit for AI Policy research but not technical research.)
Longtermism: Future people matter, and there might be lots of them, so the moral value of our actions is significantly determined by their effects on the long-term future. We should prioritize reducing āexistential risksā like nuclear war, climate change, and pandemics that threaten to drive humanity to extinction, preventing the possibility of a long and beautiful future.
Academic paper arguing that future people matter morally, and we have tractable ways to help them, from the Doing Good Better philosopher
Best resource on this topic: The Precipice, a book explaining what risks could drive us to extinction and how we can combat them, released earlier this year by another Oxford philosophy professor
Artificial intelligence might transform human civilization within the next century, presenting incredible opportunities and serious potential problems
Elon Musk, Bill Gates, Stephen Hawking, and many leading AI researchers worry that extremely advanced AI poses an existential threat to humanity (Vox)
Best resource on this topic: Human Compatible, a book explaining the threats, existential and otherwise, posed by AI. Written by Stuart Russell, CS professor at UC Berkeley and author of the leading textbook on AI. Daniel Kahneman calls it āthe most important book I have read in quite some timeā. (Or this podcast with Russell)
CS paper giving the technical explanation of what could go wrong (from Google/āOpenAI/āBerkeley/āStanford)
(AI is less morally compelling if you donāt care about the long-term future. If you want to focus on the present, maybe focus on other causes: global poverty, animal welfare, grantmaking, or researching altruistic priorities.)
Generally, Iād like to hear more about how different people introduce the ideas of EA, longtermism, and specific cause areas. Thereās no clear cut canon, and effectively personalizing an intro can difficult, so Iād love to hear how others navigate it.
Generally, Iād like to hear more about how different people introduce the ideas of EA, longtermism, and specific cause areas. Thereās no clear cut canon, and effectively personalizing an intro can difficult, so Iād love to hear how others navigate it.
This seems like a promising topic for an EA Forum question. I would consider creating one and reposting your comment as an answer to it. A separate question is probably also a better place to collect answers than this thread, which is best reserved for questions addressed to Ben and for Benās answers to those questions.
More broadly, should AMA threads be reserved for direct questions to the respondent and the respondentās answers? Or should they encourage broader discussion of those questions and ideas by everyone?
Iād lean towards AMAs as a starting point for broader discussion, rather than direct Q&A. Good examples include the AMAs by Buck Shlegeris and Luke Muehlhauser. But it does seem that most AMAs are more narrow, focusing on direct question and answer.
[For example, this question isnāt really directed towards Ben, but Iām asking anyways because the context and motivations are clearer here than they would be elsewhere, making productive discussion more likely. But Iām happy to stop distracting if thereās consensus against this.]
I personally would lean towards the āmost AMAsā approach of having most dialogue be with the AMA-respondent. Itās not quite āquestions after a talkā, since question-askers have much more capacity to respond and have a conversation, but I feel like itās more in that direction than, say, a random EA social. Maybe something like the vibe of a post-talk mingling session?
I think this is probably more important early in a comment tree than later. Directly trying to answer someone elseās question seems odd/āout-of-place to me, whereas chiming in 4 levels down seems less so. I think this mirrors how the āpost-talk minglingā would work: if I was talking to a speaker at such an event, and I asked them a question, someone else answering before them would be odd/āannoying ā āsorry, I wasnāt talking to youā. Whereas someone else chiming in after a little back-and-forth would be much more natural.
Of course, you can have multiple parallel comment threads here, which alters things quite a bit. But thatās the kind of vibe that feels natural to me, and Pabloās comment above suggests Iām not alone in this.
What would you recommend as the best introduction to concerns (or lack thereof) about risks from AI?
If you have time and multiple recommendations, I would be interested in a taxonomy. (E.g. this is the best blog post for non-technical readers, this is the best book-length introduction for CS undergrads.)
I agree with Aidanās suggestion that Human Compatible is probably the best introduction to risks from AI (for both non-technical readers and readers with CS backgrounds). Itās generally accessible and engagingly written, itās up-to-date, and it covers a number of different risks. Relative to many other accounts, I think it also has the virtue of focusing less on any particular development scenario and expressing greater optimism about the feasibility of alignment. If someoneās too pressed for time to read Human Comptabile, the AI risk chapter in The Precipice would then be my next best bet. Another very readable option, mainly for non-CS people, would be the AI risk chapters in The AI Does Not Hate You: I think they may actually be the cleanest distillation of the āclassicā AI risk argument.
For people with CS backgrounds, hoping for a more technical understanding of the problems safety/āalignment researchers are trying to solve, I think that Concrete Problems in AI Safety, Scalable Agent Alignment Via Reward Modeling, and Rohin Shahās blog post sequence on āvalue learningā are especially good picks. Although none of these resources frames safety/āalignment research as something thatās intended to reduce existential risks.
I think that AI Governance: A Research Agenda would be the natural starting point for social scientists, especially if they have a substantial interest in risks beyond alignment.
Of course, for anyone interested in digging into arguments around AI risk, I think that Superintelligence is still a really important read. (Even beyond its central AI risk argument, it also has a ton of interesting ideas on the future of intelligent life, ethics, and the strategic landscape that other resources donāt.) But itās not where I think people should start.
FWIW, hereās an introduction to longtermism and AI risks I wrote for a friend. (My friend has some technical background, he had read Doing Good Better but not engaged further with EA, and I thought heād be a good fit for AI Policy research but not technical research.)
Longtermism: Future people matter, and there might be lots of them, so the moral value of our actions is significantly determined by their effects on the long-term future. We should prioritize reducing āexistential risksā like nuclear war, climate change, and pandemics that threaten to drive humanity to extinction, preventing the possibility of a long and beautiful future.
Quick intro to longtermism and existential risks from 80,000 Hours
Academic paper arguing that future people matter morally, and we have tractable ways to help them, from the Doing Good Better philosopher
Best resource on this topic: The Precipice, a book explaining what risks could drive us to extinction and how we can combat them, released earlier this year by another Oxford philosophy professor
Artificial intelligence might transform human civilization within the next century, presenting incredible opportunities and serious potential problems
Elon Musk, Bill Gates, Stephen Hawking, and many leading AI researchers worry that extremely advanced AI poses an existential threat to humanity (Vox)
Best resource on this topic: Human Compatible, a book explaining the threats, existential and otherwise, posed by AI. Written by Stuart Russell, CS professor at UC Berkeley and author of the leading textbook on AI. Daniel Kahneman calls it āthe most important book I have read in quite some timeā. (Or this podcast with Russell)
CS paper giving the technical explanation of what could go wrong (from Google/āOpenAI/āBerkeley/āStanford)
How you can help by working on US AI policy, explains 80,000 Hours
(AI is less morally compelling if you donāt care about the long-term future. If you want to focus on the present, maybe focus on other causes: global poverty, animal welfare, grantmaking, or researching altruistic priorities.)
Generally, Iād like to hear more about how different people introduce the ideas of EA, longtermism, and specific cause areas. Thereās no clear cut canon, and effectively personalizing an intro can difficult, so Iād love to hear how others navigate it.
This seems like a promising topic for an EA Forum question. I would consider creating one and reposting your comment as an answer to it. A separate question is probably also a better place to collect answers than this thread, which is best reserved for questions addressed to Ben and for Benās answers to those questions.
Good idea, thanks! Iāve posted a question here.
More broadly, should AMA threads be reserved for direct questions to the respondent and the respondentās answers? Or should they encourage broader discussion of those questions and ideas by everyone?
Iād lean towards AMAs as a starting point for broader discussion, rather than direct Q&A. Good examples include the AMAs by Buck Shlegeris and Luke Muehlhauser. But it does seem that most AMAs are more narrow, focusing on direct question and answer.
[For example, this question isnāt really directed towards Ben, but Iām asking anyways because the context and motivations are clearer here than they would be elsewhere, making productive discussion more likely. But Iām happy to stop distracting if thereās consensus against this.]
I personally would lean towards the āmost AMAsā approach of having most dialogue be with the AMA-respondent. Itās not quite āquestions after a talkā, since question-askers have much more capacity to respond and have a conversation, but I feel like itās more in that direction than, say, a random EA social. Maybe something like the vibe of a post-talk mingling session?
I think this is probably more important early in a comment tree than later. Directly trying to answer someone elseās question seems odd/āout-of-place to me, whereas chiming in 4 levels down seems less so. I think this mirrors how the āpost-talk minglingā would work: if I was talking to a speaker at such an event, and I asked them a question, someone else answering before them would be odd/āannoying ā āsorry, I wasnāt talking to youā. Whereas someone else chiming in after a little back-and-forth would be much more natural.
Of course, you can have multiple parallel comment threads here, which alters things quite a bit. But thatās the kind of vibe that feels natural to me, and Pabloās comment above suggests Iām not alone in this.