Introducing the Existential Risks Introductory Course (ERIC)
Crossposted to LessWrong.
Introduction
We (the Cambridge Existential Risks Initiative) ran an Existential Risks Introductory Course (ERIC) in the first quarter of 2022, aiming to introduce the field of existential risks, without being explicitly associated with any particular philosophy. We expect the programme to be most useful to people who are new to this field, and we hypothesised that we may be able to reach a different target audience by not explicitly branding it as EA.
The full curriculum we used for the programme, along with exercises and organisation spotlights, can be found here. This was primarily designed by Callum McDougall, with some inputs from the rest of the CERI team.
If you are interested in joining the next iteration of the course in Winter 2022 (either as a participant or as a facilitator), please fill out this interest form.
This post contains an overview of the course, which is followed by an abbreviated version of the syllabus for the ease of gathering feedback. The weekly summaries may also be helpful for community builders looking for summaries of any of the core readings from our syllabus.
We welcome any feedback on the content, exercises or anything else pertaining to the course, either here publicly on the Forum, or you can also reach out to us privately if you prefer that.
Course overview
The course consists of 8 weeks of reading (split into core and applied). Some weeks also include exercises, which participants are encouraged to complete and discuss in the session. Each week, participants will meet for 1.5 hour sessions where they will discuss the material and exercises with a facilitator.
The topics for each week are as follows:
Week 1: Introduction to Existential Risks
Provides an introduction to x-risks, why they might be both highly important and neglected, and introduces some important terminology.
Week 2: Natural & Anthropogenic Risks
Discusses natural risks, and risks from nuclear war and climate change.
Week 3: Biosecurity, And How To Think About Future Risks
Discusses risks from engineered pandemics, as well as a broader look at future risks in general and how we can reason about them and prepare for them.
Week 4: Unaligned Artificial Intelligence
Discusses risks from unaligned AI, and provides a brief overview of the different approaches that are being taken to try and solve the problem.
Week 5: Dystopias, Lock-in & Unknown Unknowns
Concludes the discussion of specific risks by discussing some more neglected risks. Also includes a discussion of the “unknown unknowns” problem, and how we can categorise and assess probabilities of risks.
Week 6: Forecasting & Decision-making
Moves away from specific risks, and discusses broad strategies that can help mitigate a variety of risks, with a focus on improving forecasting and decision-making (both at the institutional and individual level).
Week 7: Different Frameworks for Existential Risk
Further explores some alternative frameworks for x-risks than those found in The Precipice, e.g. FHI’s origin/scaling/endgame model, and the “Democratising Risk” paper.
Week 8: Next Steps
Concludes the fellowship with a lookback on the key themes in the material, and a discussion of how the fellows plan to put what they’ve learned into action (e.g. in their future careers).
Abbreviated curriculum (Core readings)
Week 1: Introduction to Existential Risks
The first group of core materials here outlines the key ideas of Toby Ord’s book The Precipice, that we may be living in a uniquely important and dangerous time thanks to the threat of existential risks.
What are the most important moral problems of our time? (10 mins.)
The Precipice: Introduction, Chapters 1 & 2 (pages 3 − 64) (100 mins.)
All Possible Views About Humanity’s Future Are Wild (15 mins.)
However, it is important to know that not everyone in the existential risks field shares these views, and there have been alternative framings proposed. The paper below discusses the drawbacks with the “Techno-Utopian Approach” to x-risks, as exemplified by books like The Precipice. We will read more of it in later sessions, but for now it is important to be aware that there are other ways of thinking about these issues.
Democratising Risk: In Search of a Methodology to Study Existential Risk(Introduction) (5 mins.)
Finally, please check out these, which relate to how we’d like you to approach our discussion sessions:
Why “scout mindset” is crucial to good judgement | Julia Galef | TEDxPSU (video − 12 mins.)
Week 2: Natural & Anthropogenic Risks
In Week 2 we start to investigate specific existential risks.
We will focus on anthropogenic risks, which arise from unique features of human society or current technology. Unlike natural risks, we can’t point to the historical record as evidence for their probability being small, meaning we could plausibly be facing more risk from them than we have at any previous time in history.
The Precipice: Chapters 3 & 4 summary notes (30 mins.)
80,000 Hours problem profiles: Climate Change (40 mins.)
80,000 Hours problem profiles: Nuclear Security (15 mins.)
Week 3: Biosecurity, And How To Think About Future Risks
During Week 3, we will focus on the first of two particularly significant risks from future technology: engineered pathogens. We discuss some past examples of bioweapon misuse or bioresearch accidents, as well as the kind of organisations and protocols that exist to make these events less likely.
The Precipice: Chapter 5, Future Risks (just pages 121 − 137) (30 mins.)
80,000 Hours problem profiles: Global catastrophic biological risks
We will also discuss the different ways we can think about risks from future technology, for instance through the lens of the unilateralist’s curse (the idea that in any group, decisions made by individuals that have a significant impact on the rest of the group will be systematically made more than they should, and this problem grows as the group grows).
The Vulnerable World Hypothesis, pages 1-8 | Nick Bostrom (35 mins.)
Information Hazards in Biotechnology | Greg Lewis (25 mins.)
Week 4: Unaligned Artificial Intelligence
In Week 4, we turn our attention to the second major risk from future technology: unaligned artificial intelligence.
AI safety is a large field (it has 2 separate 8-week fellowships at Cambridge alone!). In this session, we hope to give you an overview of the key ideas: why AI could be transformative to society, why it poses risks which humanity may be currently unsuited to handle, and some of the different approaches that are being taken to try and make it go well.
Superintelligence, Chapter 7: The Superintelligent Will | Nick Bostrom (25 mins.)
What Failure Looks Like | Paul Christiano (15 mins.)
Specification Gaming: the flipside of AI ingenuity | DeepMind Safety Team (15 mins.)
Week 5: Dystopias, Lock-in & Unknown Unknowns
This week, we’ll conclude our discussion of specific risks. We will cover dystopias (defined as a world with a functioning civilisation, but is locked into a terrible form), and extreme suffering risks (or s-risks).
The Precipice: Chapter 5, Future Risks (just pages 153-162) (20 mins.)
The Precipice: Chapter 6, The Risk Landscape (pages 165 − 186) (35 mins.)
The Vulnerable World Hypothesis, pages 8-16 | Nick Bostrom (40 mins.)
S-risks: Why they are the worst existential risks, and how to prevent them | EAG Boston 2017 (20 mins.)
Additionally, many of the risks we have discussed so far wouldn’t have occurred to researchers 50 years ago, which could make us suspect the existence of other currently unknown risks, which make themselves known as technology advances. We will discuss the possibility of such “unknown unknowns”, and what they imply for work on existential risks.
The Importance of Unknown Existential Risks (15 mins.)
Finally, we will also provide an overview of the risk landscape, and discuss the probabilities that Ord puts on each of the risks he discusses.
Week 6: Forecasting & Decision-making
Week 6 marks a movement away from discussing specific risks, and towards the kinds of broad strategies that can help mitigate a variety of risks. We will focus on how to improve forecasting & decision-making — both within organisations and at the individual level.
The Precipice: Chapter 7, Safeguarding Humanity (pages 187 − 216) (50 mins.)
Global priorities research (20 mins.)
A Better Crystal Ball | Scoblic & Tetlock (20 mins.)
Evidence, cluelessness, and the long term | Hilary Greaves (30 mins.) (or this YouTube talk)
Week 7: Different Frameworks for Existential Risk
This week, we will discuss some different responses to the problem of existential risks. One influential school of thought is longtermism—the idea that the very long-run future impact of our actions should represent our dominant moral consideration. However, not all frameworks for thinking about x-risks need to go hand-in-hand with longtermism; in fact an over-focus on longtermism may pose some serious problems. The recently-published Democratising Risk paper offers an alternative path for the field.
Other Frameworks
Longtermism
Our Potential
Letter from Utopia | Nick Bostrom (15 mins.)
Existential Risk and Existential Hope: Definitions | Owen Cotton-Barratt, Toby Ord(10 mins.)
Week 8: Next Steps
One of the main ways in which we can affect the world for the better is through our careers. For this final week we hope to help you think about potential next steps for applying the ideas of existential risk reduction or longtermism to your own life and career.
Click here for the full version of the curriculum, which contains additional readings, exercises, and organisation spotlights. If you are planning to run a similar programme at your local group, please do reach out to us, as we may also be able to share our facilitators’ guide and other resources.
Ways in which you can help
Call for facilitators
If you would be interested in facilitating this course when it is run again in Winter 2022, we would love to hear from you.
The course will be virtual as default. Familiarity with the core concepts of x-risks is desirable, but you don’t have to be an expert in order to facilitate. This is designed as an introductory fellowship, so far more important are good communication skills, and an ability to stimulate productive and interesting conversations among the fellows. Overall, being a facilitator is a great way to help with outreach and field-building, as well as improve your communication skills. I can say that I’ve personally thoroughly enjoyed the experience!
If this sounds like it might be a good fit for you or someone you know, the expression of interest form can be found here.
Call for fellows
If you or anyone you know might be interested in participating in this course, please fill out the expression of interest form here. We welcome people from all backgrounds and career stages!
Call for feedback
We would be particularly interested in hearing if you have suggestions for:
How to improve the balance of the curriculum by reducing the focus on Toby Ord and Nick Bostrom’s writings, and introducing some different perspectives,
Material discussing climate change as an existential risk,
Material focusing on solutions as well as problems (i.e. which research directions in x-risk reduction seem promising, and what work in the field tends to look like)
Suggestions for how we could improve the marketing of this course (i.e. places it could be advertised)
- CERI Research Symposium Presentations (incl. Youtube links) by (24 Sep 2022 3:45 UTC; 55 points)
- Announcing ERA: a spin-off from CERI by (13 Dec 2022 20:58 UTC; 55 points)
- Future Matters #5: supervolcanoes, AI takeover, and What We Owe the Future by (14 Sep 2022 13:02 UTC; 31 points)
- Monthly Overload of EA—September 2022 by (1 Sep 2022 13:43 UTC; 15 points)
- Introducing the Existential Risks Introductory Course (ERIC) by (LessWrong; 19 Aug 2022 15:54 UTC; 9 points)
Added to the list of courses here.
Thank you for making this! This looks great. I’ve added it to the list of AI safety courses.
It’s not on just technical AI safety but I feel like it’s related enough that anybody looking at the list will also be interested in this resource.
Suggestion: If the goal is to attract non-EAs, I might change the title to be more legible to people who don’t know what an existential risk is.
That’s something we’ve definitely considered, but the idea is for this course to be marketed mainly via CERI, and since they already have existential risks in their name plus define it in a lot of their promo material, we felt like it would probably be more appropriate to stick with that terminology.
Thank you for taking the time to make this publicly available.
Greetings
This course seems interesting.
Thank you for making this.
And lastly I filled the interest form but still didn’t receive any developments about the course or my acceptance?
Regards
Hi, thanks for sharing this!
One clarification: given that the course is almost 100% EA/longtermist in content and structure (with the exception of just under half of week 7), does the mention of introducing existential risk without being explicitly associated with any particular philosophy refer to 1) intending to provide an even-handed introduction to the field, or 2) using the concept of existential risk as an EA/longtermism recruitment approach?
I see trade-offs with using either approach. 2) may lead to further impact down the line through career-alignment, but will necessarily reduce the quality of the course by narrowing the range of acceptable topics, readings, and approaches.
In practice the “create something which is ideologically independent from EA” wasn’t really what we went for, it’s more like “really hone in on this one area that lots of EAs care about”. We could have phrased it better in the post.
Yeah +1 to Nandini’s point, I think we should have been made this clearer in the post. I think people have a lot of misconceptions about EA (e.g. lots of people just think EA is about effective charitable giving), and we wanted to emphasise this particular part rather than trying to construct the whole tower of assumptions.
That being said, I do think that the abundance of writing from Ord/Bostrom is something that we could have done a better job of toning down, and different perspectives could have been included. If you have any specific recommendations for reading material you think would positively contribute in any week (or reading material already in the course that you think could be removed), we’d be really grateful!
Why? Doesn’t a focus on particular risks assume that we will be able to mitigate not only particular risks, but all such existential risks? What difference does it make if we solve one problem if we are then destroyed by another?
Is it credible for us to propose that we can succeed in mitigating each and every existential threat which will emerge from an ever accelerating knowledge explosion which is generating ever more, ever larger powers, at an ever accelerating rate?
Wouldn’t such an assumption be a failure of holistic thinking which fails to take in to account that the maturity of the human beings who are to manage such an assembly line of risks will not advance at the same pace as the challenge being presented?
Thanks for the comment! I first want to register strong agreement with many of your points, e.g. the root of the problem isn’t necessarily technology inherently, but rather our inability to do things like coordinate well and think in a long-term way. I also think that focusing too much on individual risks while avoiding the larger picture is a failure mode that some in the community fall into, and Ord’s book might have done well to spend some time taking this perspective (he does talk about risk factors which is part of the way to a more systemic perspective, but he doesn’t really address the fundamental drivers of many of these risks, which I agree seems like a missed opportunity).
That being said, I think I have a few main disagreements here:
Lack of good opportunities for more general longtermist interventions. I think if there were really promising avenues for advancing along the frontiers you suggest (e.g. trying to encourage cultural philosophical perspective shifts, if I’m understanding your point here correctly) then I’d probably change my mind here. But it still seems imo like these kinds of interventions aren’t as promising as direct work on individual risks, which is still super neglected in cases like bio/AI.
Work on individual risks does (at least partially) generalise. For instance, in the case of work on specific future risks e.g. bio and AI, it doesn’t seem like we can draw useful lessons about what kinds of strategies work (e.g. regulation/slowing research, better public materials and education about the risks, integrating more with the academic community) unless we actually try out these strategies.
Addressing some risks might directly reduce others. For instance, getting AI alignment right would probably be a massive boon for our ability to handle other natural risks. This is pretty speculative though, because we don’t really know what a future where we get AI right looks like.
Hi Callum, thanks for the response, much appreciated.
I would describe it as an unwillingness or inability to think holistically, to consider human limitations as one of the factors that must be taken in to account when designing our technological future. The “more is better” relationship with knowledge which science culture is built upon seems to largely ignore this. “More is better” without limit is not going to work when one of the components of the “machine” is limited.
Yes, that’s a good summary. I’m not against working on particular risks, I just see that as a failed strategy if we don’t also bring focus to the knowledge explosion which is generating all the risks. More here:
https://forum.effectivealtruism.org/posts/F76dnd5xvQHdqBPd8/what-is-the-most-effective-way-to-look-at-existential-risk
Fair point. No complaints. I don’t object to such work. I’m objecting to what I perceive to be an almost exclusive focus on particular risks, at the cost of largely ignoring the source of the risks. Example: After seventy years we still have basically no idea how to escape the nuclear weapons threat, and yet we keep piling on more and more risks, faster and faster.
So long as the risk pipeline is generating new risks faster than our ability to respond, working on particular risks will ultimately fail. You know, when we’re talking about existential risks, we have to win every time, and can’t afford to lose once. It won’t matter if we solve AI, if some other existential risk destroys the system anyway.
Meaning no disrespect, I see this notion that intellectual elites in prestigious universities can solve the AI problem as a self serving mythology. How do such well meaning elites intend to manage the Russians, Chinese, North Koreans, drug gangs, terror groups, hacker boys on Reddit etc?
On the genetic engineering front, leaders like Jennifer Doudna respond to concerns by discussing governing mechanisms. Industry leaders would like us to believe that they will be able to control the spread and development of genetic engineering, but that’s just silly. It’s almost insulting that they keep expecting us to believe that.
To me, the central question is, how do we take control of the knowledge explosion so that it is proceeding at a rate which human beings can successfully manage?
The challenge here is that the “more is better” relationship with knowledge has been with us since the beginning, and has delivered many miracles, so people have a very hard wrapping their minds around the fact that the success of the knowledge explosion has created a very different new environment which we are required to adapt to, like it or not.
This problem is amplified by the fact that, generally speaking, the science community seems not to grasp this, and they are the ones with most of the cultural authority on such issues. To a significant degree we are being led by people who are living in the past.
Ok, that’s enough words for now, too many really. Thanks again for engaging, and for the work you are doing. I look forward to more exchanges as your time and interest permit.
My suggestion, as always, would be to shift some focus from particular technological threats such as AI and genetic engineering, to the ever accelerating knowledge explosion which is the source of these threats.
Imagine if you will that a perfect solution is found to the threats presented by AI and genetic engineering. That sounds great at first, but really, so what?
The ever accelerating knowledge explosion will continue to generate ever more powers, of ever greater scale, at an ever faster pace. So long as the focus is on managing particular threats, and not the mechanism which is generating all the threats, then we are engaged in a game of wack-a-mole which we will sooner or later lose.
Imho, the underlying problem uniting all these threats is not technical, but philosophical. As a culture we’re clinging to a “more is better” relationship with knowledge which was entirely rational in the long era of knowledge scarcity, and blindly assuming that this “more is better” paradigm is still rational in an entirely different new era characterized by knowledge exploding in every direction at an ever accelerating rate.
We’re failing to adapt to the revolutionary new environment created by the success of the knowledge explosion. Continuing to push the knowledge explosion forward faster and faster without limit is not adapting to the future, it’s clinging to the past.
I’ll take a lot more interest in EA if I’m able to find anyone discussing the threats we face from this perspective.
Well done for the hard work in putting together the course, and I am sure many people will benefit from it! Nonetheless, I do have a number of critiques (sorry!) as many of the choices seem somewhat odd. Firstly, relying on the Precipice so much, particularly for cause area work, seems a little odd. For example, on climate change, the precipice really is nowhere near the best work done on it, and for example Beard et al 2020 or Kemp et al 2022 may be considerably better. I certainly think a lot of the climate stuff you have could be improved. I don’t know how genralisable such a suggestion is to other cause areas.
Secondly in the forecasting and decision making, you have nothing on foresight or horizon scanning, important techniques used in decision making with the future, despite what you have on forecasting. This decision seems odd given the former is often seen as more appropriate in situations of deep uncertainty. I think it’s a major mistake not to include foresight work in here, as in oractice this is vital to xrisk work.
Thirdly, your week on alternative paradigms really hardly explores the systems/complexity approach to xrisk (although defense in depth and democratising risk is a start!). I would recommend some of what’s written in Beard & Torres 2020 for an introduction to this “3rd paradigm” of X-Risk studies, and maybe to look from there. A lot of work coming out of CSER maybe useful on this, as might some work Seth Baum and other have done. Other useful papers may be “7 questions for ERS Scholars” and some of the talks at CCCR 2022.
Your choices to lump climate, nuclear and anthropogenic and natural risks into one week seems a little strange, and I really don’t think you do any of these justice by such a small time spent on them.
The choice to have the Vulnerable World Hypothesis without critique seems potentially dangerous, despite its utility (it sort of Advocates for a global surveillance state!) I also think some critical views on AI risk (eg Ben Garfinkel) maybe useful in that week.
More generally, whilst it is only an intro course, I really do worry that some of the decisions made look in some common assumptions of how xrisk will manifest and be that seems quite common in the ea community buildingspaces that I don’t see as very common in the more research orientated xrisk spaces. I don’t think all these criticisms bite. Some of it is legitimately the compromises you have to make with an intro course. And maybe you have different intentions to me and you want to convince the participants of a sort of 2nd wave xrisk approach (one that is common in ea, pretty mechanistic and modernist, close to what kemp and cramer describe as the TUA), but if you don’t and rather want to give people an intro into xrisk that introduces them to the state of the academic debate more generally, I think a number of the changes I suggested should be incorporated.