A few years ago, I made a outline of Evan G. Williams’ excellent philosophy paper, for a local discussion group. It slowly got circulated on the EA internet. Somebody recently recommended that I make the summary more widely known, so here it is.
The paper is readable and not behind a paywall, so I’d highly recommend reading the original paper if you have the time.
Summary
I. Core claim
Assuming moral objectivism (or a close approximation), we are probably unknowingly guilty of serious, large-scale wrong-doing (“ongoing moral catastrophe”).
II. Definition: What is a moral catastrophe? Three criteria:
Must be a serious wrong-doing (closer to wrongful death or slavery than mild insults or inconveniences).
Must be large-scale (instead of a single wrongful execution, or a single man tortured)
Broad swathes of society are responsible through action or inaction (can’t be unilateral unavoidable actions by a single dictator).
III. Why we probably have unknown moral catastrophes. Two core arguments:
The Inductive Argument
Assumption: It’s possible to engage in great moral wrongdoings even while acting in accordance to your own morals, and that of your society.
Basic motivation: an honest, sincere Nazi still seems to be acting wrongly in important ways.
It’s not relevant whether this wrongdoing is due to mistaken empirical beliefs (All Jews are part of a major worldwide conspiracy) or wrong values (Jews are subhuman and have no moral value).
Given that assumption in mind, pretty much every major society in history has acted catastrophically wrongly.
Consider conquistadores, crusaders, caliphates, Aztecs etc. who conquered in the name of God(s), who they called good and just.
It’s unlikely that all of these people in history only professed such a belief, and that all of them were liars instead of true believers.
Existence proof: People can (and in fact do) do great evil without being aware of this.
Us having ongoing moral catastrophes isn’t just a possibility, but probable.
We are not that different from past generations: Literally hundreds of generations have thought that they actually were right and had figured out the One True Morality
As recent as our parents’ generation, it was a common belief that some people have more rights than others because of race, sexuality etc.
We live in a time of moral upheaval, where our morality is very different from our grandparents’.
Even if some generation would eventually figure out All of Morality, the generation that gets everything right is probably a generation whose parents gets almost everything right.
The Disjunctive Argument
Activists are not exempt. Even if all your pet causes come to fruition, this doesn’t mean our society is good, because there are still unknown moral catastrophes.
There are so many different ways that a society could get things very wrong, that it’s almost impossible to get literally everything right.
This isn’t just a minor concern, we could be wrong in ways that are a sizable proportion of how bad the Holocaust is.
There are many different kinds of ways that society could be wrong.
We could be wrong about who has moral standing.(eg. fetuses, animals)
We could be empirically wrong about what harms or hurts people who morally matter (eg. religious indoctrination of children)
We could be right about some obligations but not others.
We can act immorally in paying too much attention and using resources on false moral obligations (a la crusaders)
We could be right about what’s wrong and should be fixed, but wrong at how to prioritize different fixes.
We could be right about what’s wrong, but wrong about what is and is not our responsibility to fix. (eg. poverty, borders)
We could be wrong about the far future (natalism, existential risk)
Within each category, there are multiple ways to go wrong.
Further, some are mutually exclusive. Eg. Pro-lifers could be right and abortion is a great sin, or fetuses don’t matter and it’s greatly immoral to deprive women of their freedom in eg. third trimester abortions.
Unlikely that we’re currently at the golden mean for all of these trade-offs.
Disjunction comes into play.
Even if you believe that we’re 95% right at each major issue, and there are maybe 15 of them, the total probability that we are right is maybe ~.95^15~=46% (LZ: Assumes independence)
In practice, 95% sure we’re right at each major issue seems way too confident, and 15 items too low.
IV. What should we do about it?
Discarded possibility: hedging. If you’re not sure, play it “safe”, morally speaking.
Eg. even if you think farmed animals probably aren’t sentient, or sentience doesn’t morally matter, you can go vegetarian “just in case”
This does NOT generally work well enough because it’s not robust: as noted, too many things can go wrong, some in contradictory directions.
Recognition of Wrongdoing
Actively try to figure out which catastrophic wrongs we’re committing
Research more into practical fields (eg. animal consciousness) where we can be critically wrong
Research more into moral philosophy
Critical: bad to have increased technological knowledge w/o increased moral wisdom
imagine Genghis Khan w/nuclear weapons
These fields must interact
Not enough for philosophers to say that animals are important if they are conscious and for scientists to say that dolphins are conscious but don’t know if this is important...our society must be able to integrate this.
Need marketplace of ideas where true ideas win out
Rapid intellectual progress is critical.
If it’s worth fighting literal wars to defeat the Nazis or end slavery, it’s worth substantial material investment and societal loss to figure out what we’re currently doing wrong.
Implementation of improved values
Once we figure out what great moral wrongs we’ve committed, we want to be able to make moral reparations for past harms, or at least stop doing future harms in that direction as quickly as possible.
To do this, we want to maximize flexibility in material conditions
Extremely poor/war-torn societies would be unable to make rapid moral changes as needed
LZ example: Complex systems built along specific designs are less resilient to shocks, and also harder to change, cf. Antifragile.
In the same way we stock up resources for war preparation, we might want to save up resources for future moral emergencies, so we can eg. pay reparations, or at least quickly make the relevant changes.
LZ: Unsure how this is actually possible in practice. Eg, individuals usually save by investing, and governments save by buying other government’s debt or by investing in the private sector, but it’s unclear how the world “saves” as a whole.
We want to maximize flexibility in social conditions
Even if it’s materially possible to make large changes, society might make such changes very difficult, because inertia and conservatism bias.
Constitutional amendments, for example, are suspect.
V. Conclusion/Other remarks
Counterconsideration One: Building a society that can correct moral catastrophes isn’t the same as actually correcting moral catastrophes.
Counterconsideration Two : Many of the measures suggested above to prepare for correcting moral catastrophes may themselves be evil
e.g. money spent on moral research could have instead been spent on global poverty, building a maximally flexible society might involve draconian restrictions on current people’s rights
However, this is still worth doing in the short term.
This work is licensed under a Creative Commons Attribution 4.0 International License.
I considered Evan Williams’ paper one of the most important papers in cause prioritization at the time, and I think I still broadly buy this. As I mention in this answer, there are at least 4 points his paper brought up that are nontrivial, interesting, and hard to refute.
If I were to write this summary again, I think I’d be noticeably more opinionated. In particular, a key disagreement I have with him (which I remember having at the time I was making the summary, but this never making it into my notes) is on the importance of the speed of moral progress vs the sustainability of continued moral progress. In “implementation of improved values”, the paper focuses a lot on the flexibility of setting up society to be able to make moral progress quickly, but naively I feel about as worried or more worried that society can make anti-progress and do horrifyingly dumb and new things in the name of good. So I’d be really worried about trajectory changes for the worse, especially longer-lasting ones (“lock-in” is a phrase that’s in vogue these days).
I’ve also updated significantly on both the moral cost and the emprical probability of near-term extinction risks, and of course extinction is the archetypal existential risk that will dramatically curtail the value of the far future.
It feels weird getting my outline into the EA decade review, instead of the original paper, though I wouldn’t be very surprised if at this point more EAs have read my outline than the paper itself.
I vaguely feel like Williams should get a lot more credit than he has received for this paper. Like EA should give him a prize or something, maybe help him identify more impactful research areas, etc.
Strong upvoting because I want to incentivize people to write and share more summaries.
Summaries are awesome and allow me to understand the high level of papers that I would not have read otherwise. This summary in particular is well written and well-formatted.
Thanks for writing it and sharing it!
Agreed! 5 years on and now we have LLM summarizers. Perhaps with LLM summarizer + human proof-reader there could be something interesting here.
We absolutely welcome summaries! People getting more ways to access good material is one of the reasons the Forum exists.
That said, did you consider copying the summary into a Forum post, rather than linking it? That’s definitely more work, but my impression is that it usually leads to more discussion when people don’t have to click away into another page. I don’t have strong evidence to back that up, though.
Also: because the title is long and long titles are cut short in some views of the Forum, I’d recommend that summaries of pieces be something like “The Possibility of an Ongoing Moral Catastrophe (Summary)”.
Yay!
Yes. I did a lot of non-standard formatting tricks in Google Docs when I first wrote it (because I wasn’t expecting to ever need to port it over to a different format). So when I first tried to copy it over, the whole thing looked disastrously unreadable.
Changed the title. :)
I’m entering philosophy grad school now, but in a few years I’m going to have to start thinking about designing courses, and I’m thinking of designing an intro course around this paper. Would it be alright if I used your summary as course material?
You may also like our discussion sheets for this topic:
https://drive.google.com/drive/u/1/folders/0B3C8dkkHYGfqeXh1SE5kLVJPdGM
Thanks!
Sure! In general you can assume that anything I write publicly is freely available for academic purposes. I’d also be interested in seeing the syllabus if/when you end up designing it.
Definitely, I’ll send it along when I design it. Since intro ethics at my institution is usually taught as applied ethics, the basic concept would be to start by introducing the students to the moral catastrophes paper/concept, then go through at least some of the moral issues Williams brings up in the disjunctive portion of the argument to examine how likely they are to be moral catastrophes. I haven’t picked particular readings yet though as I don’t know the literatures yet. Other possible topics: a unit on historical moral catastrophes (e.g. slavery in the South, the Holocaust); a unit on biases related to moral catastrophes; a unit on the psychology of evil (e.g. Baumeister’s work on the subject, which I haven’t read yet); a unit on moral uncertainty; a unit on whether antirealism can escape or accommodate the possibility of moral catastrophes.
Assignment ideas:
pick one of the potential moral catastophes Williams mentions, which you think is least likely to actually be a moral catastrophe. Now, imagine that you are yourself five years from now and you’ve been completely convinced that it is in fact a moral catastrophe. What convinced you? Write a paper trying to convince your current self that it is a moral catastrophe after all.
Come up with a potential moral catastrophe that Williams didn’t mention, and write a brief (maybe 1-2 pages?) argument for why it is or isn’t one (whatever you actually believe). Further possibility: Once these are collected, I observe how many people argued that the one they picked was not a moral catastrophe, and if it’s far over 50%, discuss with the class where that bias might come from (e.g. status quo bias, etc.).
This is all still in the brainstorming stage at the moment, but feel free to use any of this if you’re ever designing a course/discussion group for this paper.
For #2, Ideological Turing Tests could be cool too.
Thanks for the summary. I have two takeaways:
1. EA is (in part) claiming that there are several ongoing moral catastrophes caused by inaction against global poverty, animal suffering, x-risk,… (some of them are definitely caused by action, but that does not matter as much on consequentialist grounds). Unknown ongoing moral catastrophes are cause-X.
2. The possibility of working to increase our capability to handle undiscovered ongoing moral catastrophe in the future as a major goal. The idea I saw here was to reserve resources, which is a very interesting argument to invest in economic growth.
The statement, “Need marketplace of ideas where true ideas win out”, indicates the ineviatable failure to this approach. There will never be agreement about the identity, or even the existence of, “true ideas”.
I don’t think this is particularly true. Government debt is not solely owned by other governments—otherwise it would be strange that all governments (as far as I know) have positive debt. Generally, I believe government debt is owned by the private sector (individual people, businesses, etc.). If we’re talking about all governments having money saved up (because we would trust a government to pay large amounts of money to avert a crisis, whereas we would not normally trust the public to do so voluntarily), then that is possible. A good way of achieving that is having governments ensure their debt doesn’t get too high (though austerity measures can have negative consequences too) - in fact this is advice that some give to governments: “don’t have too much debt—otherwise if there’s a crisis, you won’t be able to afford to spend as much extra money”. If you believe that it is morally crucial that governments are able to spend large amounts of money to avert a newfound moral crisis, then you might propose that they keep debt low, so they could spend extra large amounts if necessary. If you believe this to a very strong extent (which I do not), then you might even advocate that they have “negative” debt (which could be investing in the private sector).
Even though I hadn’t considered this point before, and even though I consider it valid, I don’t intuitively think it would have a large effect on the optimal level for debt—my instinctive guess is that this consideration might be worth ~5pp less of debt. I think it is sufficient for governments, when an extremely urgent cause is discovered, to increase taxes, decrease domestic spending, and send this extra money towards this new cause. I believe a more pressing imperative is that (rich) governments significantly increase their international aid payments now. The most effective charities are pretty good, and there’s not too much risk of accidentally doing harm (I often argue that political causes are risky, though, because people are notoriously overconfident about their political views). I think it’s unjustifiable that e.g. US healthcare expenditure could save many many more lives if partially diverted toward a poorer country.
About the main topic of the article, I do mostly agree, and especially so in cases involving non-human sentient beings. Given how much more time is devoted to human well-being, I think it is more difficult (but still, of course, very possible) for a particular human-centered catastrophe to slip under the radar. I nonetheless think advocating for an increase in “crisis awareness research” is very sensible in either case.
The idea of reparations doesn’t have instrinsic value in my utilitarian morality, though I suppose it can often be incidentally relevant (in the same vein as money being more effective if spend on people who are more in need). The main priorities, in my opinion, should be reducing the chance of an ongoing catastrophe, and quickly (but also cautiously) stopping one when it is discovered.
Side Note: this is a linkpost, so maybe I shouldn’t have commented here? This is my first non-trivial comment, and I got a bit carried away—apologies.
I can highly recommend this mockumentary about how future people might judge our treatment of non-human animals: Carnage—Swallowing the Past
https://vimeo.com/798189243
You should sit in with some friends of Bill W. Tell them you’re visitor, then just sit there and listen. They’ll teach you about a pure altruism. It’s pretty damn awesome.
We could build up large stockpiles of resources, or capacity for making resources in the future, that could be flexibility used for a wide variety of purposes depending on where moral philosophy lead us. Essentially we are trying to predict what future resources moral philosophy might demand and get the ready. For example, we might spend less on consumption now and invest more effort in getting prepared to produce huge quantities of semiconductors and spaceships.
In short: the world saves by consuming less and investing more.