What would you recommend as the best introduction to concerns (or lack thereof) about risks from AI?
If you have time and multiple recommendations, I would be interested in a taxonomy. (E.g. this is the best blog post for non-technical readers, this is the best book-length introduction for CS undergrads.)
I agree with Aidan’s suggestion that Human Compatible is probably the best introduction to risks from AI (for both non-technical readers and readers with CS backgrounds). It’s generally accessible and engagingly written, it’s up-to-date, and it covers a number of different risks. Relative to many other accounts, I think it also has the virtue of focusing less on any particular development scenario and expressing greater optimism about the feasibility of alignment. If someone’s too pressed for time to read Human Comptabile, the AI risk chapter in The Precipice would then be my next best bet. Another very readable option, mainly for non-CS people, would be the AI risk chapters in The AI Does Not Hate You: I think they may actually be the cleanest distillation of the “classic” AI risk argument.
For people with CS backgrounds, hoping for a more technical understanding of the problems safety/alignment researchers are trying to solve, I think that Concrete Problems in AI Safety, Scalable Agent Alignment Via Reward Modeling, and Rohin Shah’s blog post sequence on “value learning” are especially good picks. Although none of these resources frames safety/alignment research as something that’s intended to reduce existential risks.
I think that AI Governance: A Research Agenda would be the natural starting point for social scientists, especially if they have a substantial interest in risks beyond alignment.
Of course, for anyone interested in digging into arguments around AI risk, I think that Superintelligence is still a really important read. (Even beyond its central AI risk argument, it also has a ton of interesting ideas on the future of intelligent life, ethics, and the strategic landscape that other resources don’t.) But it’s not where I think people should start.
FWIW, here’s an introduction to longtermism and AI risks I wrote for a friend. (My friend has some technical background, he had read Doing Good Better but not engaged further with EA, and I thought he’d be a good fit for AI Policy research but not technical research.)
Longtermism: Future people matter, and there might be lots of them, so the moral value of our actions is significantly determined by their effects on the long-term future. We should prioritize reducing “existential risks” like nuclear war, climate change, and pandemics that threaten to drive humanity to extinction, preventing the possibility of a long and beautiful future.
Academic paper arguing that future people matter morally, and we have tractable ways to help them, from the Doing Good Better philosopher
Best resource on this topic: The Precipice, a book explaining what risks could drive us to extinction and how we can combat them, released earlier this year by another Oxford philosophy professor
Artificial intelligence might transform human civilization within the next century, presenting incredible opportunities and serious potential problems
Elon Musk, Bill Gates, Stephen Hawking, and many leading AI researchers worry that extremely advanced AI poses an existential threat to humanity (Vox)
Best resource on this topic: Human Compatible, a book explaining the threats, existential and otherwise, posed by AI. Written by Stuart Russell, CS professor at UC Berkeley and author of the leading textbook on AI. Daniel Kahneman calls it “the most important book I have read in quite some time”. (Or this podcast with Russell)
CS paper giving the technical explanation of what could go wrong (from Google/OpenAI/Berkeley/Stanford)
(AI is less morally compelling if you don’t care about the long-term future. If you want to focus on the present, maybe focus on other causes: global poverty, animal welfare, grantmaking, or researching altruistic priorities.)
Generally, I’d like to hear more about how different people introduce the ideas of EA, longtermism, and specific cause areas. There’s no clear cut canon, and effectively personalizing an intro can difficult, so I’d love to hear how others navigate it.
Generally, I’d like to hear more about how different people introduce the ideas of EA, longtermism, and specific cause areas. There’s no clear cut canon, and effectively personalizing an intro can difficult, so I’d love to hear how others navigate it.
This seems like a promising topic for an EA Forum question. I would consider creating one and reposting your comment as an answer to it. A separate question is probably also a better place to collect answers than this thread, which is best reserved for questions addressed to Ben and for Ben’s answers to those questions.
More broadly, should AMA threads be reserved for direct questions to the respondent and the respondent’s answers? Or should they encourage broader discussion of those questions and ideas by everyone?
I’d lean towards AMAs as a starting point for broader discussion, rather than direct Q&A. Good examples include the AMAs by Buck Shlegeris and Luke Muehlhauser. But it does seem that most AMAs are more narrow, focusing on direct question and answer.
[For example, this question isn’t really directed towards Ben, but I’m asking anyways because the context and motivations are clearer here than they would be elsewhere, making productive discussion more likely. But I’m happy to stop distracting if there’s consensus against this.]
I personally would lean towards the “most AMAs” approach of having most dialogue be with the AMA-respondent. It’s not quite “questions after a talk”, since question-askers have much more capacity to respond and have a conversation, but I feel like it’s more in that direction than, say, a random EA social. Maybe something like the vibe of a post-talk mingling session?
I think this is probably more important early in a comment tree than later. Directly trying to answer someone else’s question seems odd/out-of-place to me, whereas chiming in 4 levels down seems less so. I think this mirrors how the “post-talk mingling” would work: if I was talking to a speaker at such an event, and I asked them a question, someone else answering before them would be odd/annoying – “sorry, I wasn’t talking to you”. Whereas someone else chiming in after a little back-and-forth would be much more natural.
Of course, you can have multiple parallel comment threads here, which alters things quite a bit. But that’s the kind of vibe that feels natural to me, and Pablo’s comment above suggests I’m not alone in this.
What would you recommend as the best introduction to concerns (or lack thereof) about risks from AI?
If you have time and multiple recommendations, I would be interested in a taxonomy. (E.g. this is the best blog post for non-technical readers, this is the best book-length introduction for CS undergrads.)
I agree with Aidan’s suggestion that Human Compatible is probably the best introduction to risks from AI (for both non-technical readers and readers with CS backgrounds). It’s generally accessible and engagingly written, it’s up-to-date, and it covers a number of different risks. Relative to many other accounts, I think it also has the virtue of focusing less on any particular development scenario and expressing greater optimism about the feasibility of alignment. If someone’s too pressed for time to read Human Comptabile, the AI risk chapter in The Precipice would then be my next best bet. Another very readable option, mainly for non-CS people, would be the AI risk chapters in The AI Does Not Hate You: I think they may actually be the cleanest distillation of the “classic” AI risk argument.
For people with CS backgrounds, hoping for a more technical understanding of the problems safety/alignment researchers are trying to solve, I think that Concrete Problems in AI Safety, Scalable Agent Alignment Via Reward Modeling, and Rohin Shah’s blog post sequence on “value learning” are especially good picks. Although none of these resources frames safety/alignment research as something that’s intended to reduce existential risks.
I think that AI Governance: A Research Agenda would be the natural starting point for social scientists, especially if they have a substantial interest in risks beyond alignment.
Of course, for anyone interested in digging into arguments around AI risk, I think that Superintelligence is still a really important read. (Even beyond its central AI risk argument, it also has a ton of interesting ideas on the future of intelligent life, ethics, and the strategic landscape that other resources don’t.) But it’s not where I think people should start.
FWIW, here’s an introduction to longtermism and AI risks I wrote for a friend. (My friend has some technical background, he had read Doing Good Better but not engaged further with EA, and I thought he’d be a good fit for AI Policy research but not technical research.)
Longtermism: Future people matter, and there might be lots of them, so the moral value of our actions is significantly determined by their effects on the long-term future. We should prioritize reducing “existential risks” like nuclear war, climate change, and pandemics that threaten to drive humanity to extinction, preventing the possibility of a long and beautiful future.
Quick intro to longtermism and existential risks from 80,000 Hours
Academic paper arguing that future people matter morally, and we have tractable ways to help them, from the Doing Good Better philosopher
Best resource on this topic: The Precipice, a book explaining what risks could drive us to extinction and how we can combat them, released earlier this year by another Oxford philosophy professor
Artificial intelligence might transform human civilization within the next century, presenting incredible opportunities and serious potential problems
Elon Musk, Bill Gates, Stephen Hawking, and many leading AI researchers worry that extremely advanced AI poses an existential threat to humanity (Vox)
Best resource on this topic: Human Compatible, a book explaining the threats, existential and otherwise, posed by AI. Written by Stuart Russell, CS professor at UC Berkeley and author of the leading textbook on AI. Daniel Kahneman calls it “the most important book I have read in quite some time”. (Or this podcast with Russell)
CS paper giving the technical explanation of what could go wrong (from Google/OpenAI/Berkeley/Stanford)
How you can help by working on US AI policy, explains 80,000 Hours
(AI is less morally compelling if you don’t care about the long-term future. If you want to focus on the present, maybe focus on other causes: global poverty, animal welfare, grantmaking, or researching altruistic priorities.)
Generally, I’d like to hear more about how different people introduce the ideas of EA, longtermism, and specific cause areas. There’s no clear cut canon, and effectively personalizing an intro can difficult, so I’d love to hear how others navigate it.
This seems like a promising topic for an EA Forum question. I would consider creating one and reposting your comment as an answer to it. A separate question is probably also a better place to collect answers than this thread, which is best reserved for questions addressed to Ben and for Ben’s answers to those questions.
Good idea, thanks! I’ve posted a question here.
More broadly, should AMA threads be reserved for direct questions to the respondent and the respondent’s answers? Or should they encourage broader discussion of those questions and ideas by everyone?
I’d lean towards AMAs as a starting point for broader discussion, rather than direct Q&A. Good examples include the AMAs by Buck Shlegeris and Luke Muehlhauser. But it does seem that most AMAs are more narrow, focusing on direct question and answer.
[For example, this question isn’t really directed towards Ben, but I’m asking anyways because the context and motivations are clearer here than they would be elsewhere, making productive discussion more likely. But I’m happy to stop distracting if there’s consensus against this.]
I personally would lean towards the “most AMAs” approach of having most dialogue be with the AMA-respondent. It’s not quite “questions after a talk”, since question-askers have much more capacity to respond and have a conversation, but I feel like it’s more in that direction than, say, a random EA social. Maybe something like the vibe of a post-talk mingling session?
I think this is probably more important early in a comment tree than later. Directly trying to answer someone else’s question seems odd/out-of-place to me, whereas chiming in 4 levels down seems less so. I think this mirrors how the “post-talk mingling” would work: if I was talking to a speaker at such an event, and I asked them a question, someone else answering before them would be odd/annoying – “sorry, I wasn’t talking to you”. Whereas someone else chiming in after a little back-and-forth would be much more natural.
Of course, you can have multiple parallel comment threads here, which alters things quite a bit. But that’s the kind of vibe that feels natural to me, and Pablo’s comment above suggests I’m not alone in this.