New Princeton course on longtermism

This semester (Fall 2023), Prof Adam Elga and I will be co-instructing Longtermism, Existential Risk, and the Future of Humanity, an upper div undergraduate philosophy seminar at Princeton. (Yes, I did shamelessly steal half of our title from The Precipice.) We are grateful for support from an Open Phil course development grant and share the reading list here for all who may be interested.

[Edit as of 19 Sept 2023: link to full syllabus—which is a bit different than the reading list below: available here]

Part 1: Setting the stage

Week 1: Introduction to longtermism and existential risk

Core
- Ord, Toby. 2020. The Precipice: Existential Risk and the Future of Humanity. London: Bloomsbury. Read introduction, chapter 1, and chapter 2 (pp. 49-56 optional); chapters 4-5 optional but highly recommended.
Optional
- Roser (2022) “The Future is Vast: Longtermism’s perspective on humanity’s past, present, and future” Our World in Data
- Karnofsky (2021) ‘This can’t go on’ Cold Takes (blog)
- Kurzgesagt (2022) “The Last Human—A Glimpse into the Far Future”

Week 2: Introduction to decision theory

Core
- Weisberg, J. (2021). Odds & Ends.https://jonathanweisberg.org/vip/_main.pdf Read chapters 8, 11, and 14.
- Ord, T., Hillerbrand, R., & Sandberg, A. (2010). “Probing the improbable: Methodological challenges for risks with low probabilities and high stakes.” Journal of Risk Research, 13(2), 191–205. https://doi.org/10.1080/13669870903126267 Read sections 1-2.
Optional
- Weisberg, J. (2021). Odds & Ends chapters 5-7 (these may be helpful background for understanding chapter 8, if you don’t have much background in probability).
- Titelbaum, M. G. (2020) Fundamentals of Bayesian Epistemology chapters 3-4

Week 3: Introduction to population ethics

Core
- Parfit, Derek. 1984. Reasons and Persons. Oxford: Oxford University Press. Read sections 4.16.120-23, 125, and 127 (pp. 355-64; 366-71, and 377-79).
- Parfit, Derek. 1986. “Overpopulation and the Quality of Life.” In Applied Ethics, ed. P. Singer, 145–164. Oxford: Oxford University Press. Read sections 1-3.
Optional
- Remainders of Part IV of Reasons and Persons and “Overpopulation and the Quality of Life”
- Greaves (2017) “Population Axiology” Philosophy Compass
- McMahan (2022) “Creating People and Saving People” section 1, first page of section 4, and section 8
- Temkin (2012) Rethinking the Good 12.2 pp. 416-17 and section 12.3 (esp. pp. 422-27)
- Harman (2004) “Can We Harm and Benefit in Creating?”
- Roberts (2019) “The Nonidentity Problem” SEP
- Frick (2022) “Context-Dependent Betterness and the Mere Addition Paradox”
- Mogensen (2019) “Staking our future: deontic long-termism and the non-identity problem” sections 4-5

Week 4: Longtermism: for and against

Core
- Greaves, Hilary and William MacAskill. 2021. “The Case for Strong Longtermism.” Global Priorities Institute Working Paper No.5-2021. Read sections 1-6 and 9.
- Curran, Emma J. 2023. “Longtermism and the Complaints of Future People”. Forthcoming in Essays on Longtermism, ed. H. Greaves, J. Barrett, and D. Thorstad. Oxford: OUP. Read section 1.
Optional
- Thorstad (2023) “High risk, low reward: A challenge to the astronomical value of existential risk mitigation.”Focus on sections 1-3.
- Curran, E. J. (2022). “Longtermism, Aggregation, and Catastrophic Risk” (GPI Working Paper 18–2022). Global Priorities Institute.
- Beckstead (2013) “On the Overwhelming Importance of Shaping the Far Future” Chapter 3
- “Toby Ord on why the long-term future of humanity matters more than anything else, and what we should do about it” 80,000 Hours podcast
- Frick (2015) “Contractualism and Social Risk” sections 7-8

Part 2: Philosophical problems

Week 5: Fanaticism

Core
- Bostrom, N. (2009). “Pascal’s mugging.” Analysis, 69 (3): 443–445.
- Russell, J. S. “On two arguments for fanaticism.” Noûs, forthcoming. Read sections 1, 2.1, and 2.2.
- Temkin, L. S. (2022). “How Expected Utility Theory Can Drive Us Off the Rails.” In L. S. Temkin (Ed.), Being Good in a World of Need. Oxford University Press.
Optional
- Wilkinson, H. (2022). In Defense of Fanaticism. Ethics, 132(2), 445–477. https://doi.org/10.1086/716869 Prioritize sections I, III, and VI.A.
- Beckstead and Thomas (2023) “A Paradox for Tiny Probabilities and Enormous Values” Prioritize from the beginning through and including section 2.
- Tarsney (2023) “The epistemic challenge to longtermism”
- Tarsney (2020) “Exceeding Expectations: Stochastic Dominance as a General Decision Theory”
- Balfour (2021) “Pascal’s Mugger Strikes Again”
- Alexander (2022) “‘Longtermism’ vs. ‘Existential Risk’” blog post on the Effective Altruism Forum

Week 6: Cluelessness

Core
- Lenman, J. (2000). “Consequentialism and Cluelessness.” Philosophy and Public Affairs, 29(4), 342–370.Read sections I, II, and VI.
- Greaves, H. (2016). “Cluelessness.” Proceedings of the Aristotelian Society, 116 (3): 311–339.Read sections V and VI (and also section III if you would like more background about the principle of indifference in order to understand section V).
- Mogensen, A., & MacAskill, W. (2021). “The Paralysis Argument.” Philosophers’ Imprint, 21(15).Read sections 1-2.
Optional
- Mogensen (2021) “Maximal Cluelessness”
- Unruh (2023) “The Constraint Against Doing Harm and Long-term Consequences”
- Re-read section on cluelessness in Greaves and MacAskill “The Case for Strong Longtermism”

Week 7: The Asymmetry

Core
- Frick, Johann. 2020. “Conditional Reasons and the Procreation Asymmetry.” Philosophical Perspectives 34: 53-87. Read sections 1-6 and 9.
- Chappell, Richard Yetter. 2017. “Rethinking the Asymmetry.” Canadian Journal of Philosophy 47 (2-3): 167-77.

Part 3: Our place in history and what we ought to do

Week 8: Moral uncertainty

Core
- MacAskill, W., Bykvist, K., & Ord, T. (2020). Moral Uncertainty. Oxford: Oxford University Press. Read Introduction pp. 1-2, Chapter 1 sections I-II, and Conclusion pp. 213-14.
- Harman, Elizabeth. (2015). “The Irrelevance of Moral Uncertainty.” In R. Shafer-Landau (Ed.), Oxford Studies in Metaethics, Volume 10. Oxford University Press.Read sections 3.1 − 3.3.
- Weatherson, Brian. (2014). “Running risks morally.” Philosophical Studies, 167(1), 141–163.Read sections 1, 3, and 4.
Optional
- MacAskill, Bykvist, and Ord (2020) Moral Uncertainty chapter 2 pp. 33-35
- Barnett, Z. (2021). “Rational Moral Ignorance.” Philosophy and Phenomenological Research, 102(3), 645–664. https://doi.org/10.1111/phpr.12684
- Weatherson, B. (2019). Normative Externalism. Oxford University Press. https://doi.org/10.1093/oso/9780199696536.001.0001 Chapter 3 (“Against asymmetry”)
- Weatherson (2019) Normative Externalism chapter 7 (“Level-crossing principles”) and chapter 8 (“Higher-order evidence”)
- Tarsney (2021) review of Weatherson Normative Externalism in Mind

Week 9: Moral uncertainty and stakes-sensitivity

Core
- Williams, Evan G. 2015. “The Possibility of an Ongoing Moral Catastrophe.” Ethical Theory and Moral Practice 18: 971-82. Read sections 1-2.
- MacAskill, William and Toby Ord. 2020. “Why Maximize Expected Choiceworthiness?” Noûs 54 (2): 327-353. Read sections 1, 5, 6, and 7.iv-v.
- Greaves, Hilary and Toby Ord. 2017. “Moral Uncertainty about Population Axiology.” Journal of Ethics and Social Philosophy 12 (2): 135-67. Read sections 3, 4.1-4.5 (4.3 optional), and 8.
- Beckstead, Nick and Teruji Thomas. 2023. “A Paradox for Tiny Probabilities and Enormous Values.” Noûs. DOI: 10.1111/nous.12462. Read section 6.
Optional
- MacAskill (2019) “Practical Ethics Given Moral Uncertainty”

Week 10: Transformative Artificial Intelligence

Core
- Carlsmith, J. (2022). “Is Power-Seeking AI an Existential Risk?” (arXiv:2206.13353). arXiv. Read Introduction (page 1) and section 8.
- Carlsmith, J. (forthcoming). “Existential risk from power-seeking AI.” In J. Barrett, H. Greaves, & D. Thorstad (Eds.), Essays on Longtermism. Oxford University Press.Read sections 1 − 5.3.3.
- Lazar, S., Howard, J., & Narayanan, A. (2023, May 30). “Is Avoiding Extinction from AI Really an Urgent Priority?” Fast.ai
Optional
- Responses to Carlsmith
- Thorstad (2023) ‘Exaggerating the risks’ parts 6-8 (see also Carlsmith’s response in the comments on part 8)
- Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety (arXiv:1606.06565). arXiv. http://arxiv.org/abs/1606.06565
- Critch, A., & Russell, S. (2023). TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI (arXiv:2306.06924). arXiv. https://doi.org/10.48550/arXiv.2306.06924
- Cotra (2021) “Why AI alignment could be hard with modern deep learning”
- Ord (2020) The Precipice chapter 5 section on AI
- Piper (2020) “The case for taking AI seriously as a threat to humanity” Vox
- Russell (2019) Human Compatible chapters 6-10
- Christian (2020) The Alignment Problem chapters 7-9
- John et al. (2023) “Dead rats, dopamine, performance metrics, and peacock tails: proxy failure is an inherent risk in goal-oriented systems”
- Hadfield-Menell, Dragan, Abbeel, & Russell (2017) “The Off-Switch Game”
- Chiang (2002) “Understand” in Stories of Your Life and Others
- Branwen, G. (2022). “It Looks Like You’re Trying To Take Over The World.”

Week 11: Possibilities for our future and what we can do

Core
- Ord, Toby. 2020. The Precipice: Existential Risk and the Future of Humanity. London: Bloomsbury. Read chapter 8.
- MacAskill, William. 2022. What We Owe the Future. New York: Basic Books. Read chapter 10.
Optional
- 80,000 Hours career guide sections 1,4, and 5
- Bostrom, “Humanity’s biggest problems aren’t what you think they are” Ted
- Bostrom (2020) “Letter from Utopia”

Further readings (all optional)

Discounting

Cowen and Parfit (1992) “Against the Social Discount Rate” (in Peter Laslett & James S. Fishkin (eds.) Justice Between Age Groups and Generations pp. 144–161)
Mogensen (2022) “The only ethical argument for positive δ? Partiality and pure time preference”
Russell (2022) “Problems for Intergenerational Equity” Parfit Memorial Lecture
Ord (2020) The Precipice Appendix A
Greaves (2017) “Discounting for Public Policy” section 7 (pp. 404-09)
Kelleher (2017) “Pure time preference in intertemporal welfare economics”

The hinge of history hypothesis

MacAskill (2022) “Are We Living at the Hinge of History?”
Mogensen (fc.) “The hinge of history hypothesis: reply to MacAskill”
Fisher (2020) “Are We Living at the ‘Hinge of History’?” BBC
Karnofsky (2021) “the most important century” blog post series
Morris (2010) Why the West Rules—For Now concluding chapter on the 21st century

Global catastrophic biological risks

Ord (2020) The Precipice chapter 5 section on pandemics
Piper (2022) “Why experts are terrified of a human-made pandemic — and what we can do to stop it” Vox
Lewis (2020) “Reducing global catastrophic biological risks” 80,000 Hours

Coordination problems and great power conflict

Nordstrom (2020) Inquiry-Based Introduction to Game Theory chapters 2 and 4
Alexander (2014) “Meditations on Moloch”
Allison (2017) Destined for War: Can America and China Escape Thucydides’s Trap?
Clare (2021) “Great Power Conflict” pp. 1-6, 59-97
Blattman (2022) Why We Fight

Human enhancement

Buchanan (2011) Beyond Humanity?: The Ethics of Biomedical Enhancement (especially chapter 2)
Bostrom (2013) “Why I want to be a posthuman when I grow up”
Bostrom and Ord (2006) “The Reversal Test: Eliminating Status Quo Bias in Applied Ethics”
Chiang (2019) “It’s 2059, and the Rich Kids Are Still Winning” NYT

Non-consequentialist concern for the future

Scheffler (2013) Death and the Afterlife
Scheffler (2018) Why Worry about Future Generations?
Caney (2018) “Justice and Future Generations”
Meyer (2021) “Intergenerational Justice” SEP
Finneron-Burns (2016) “Contractualism and the Non-Identity Problem”
Kumar (2018) “Risking Future Generations”

Impossibility results in population axiology

Arrhenius (2011) “The Impossibility of a Satisfactory Population Ethics”
Thomas (2018) “Some Possibilities in Population Axiology”
Thornley (2021) “The impossibility of a satisfactory population prospect axiology (independently of Finite Fine-Grainedness)”
Zuber et al. (2021) “What Should We Agree on about the Repugnant Conclusion?”
Huemer (2008) “In Defence of Repugnance”