I thought The Precipice was a fantastic book; I’d highly recommend it. And I agree with a lot about Chivers’ review of it for The Spectator. I think Chivers captures a lot of the important points and nuances of the book, often with impressive brevity and accessibility for a general audience. (I’ve also heard good things about Chivers’ own book.)
But there are three parts of Chivers’ review that seem to me to like they’re somewhat un-nuanced, or overstate/oversimplify the case for certain things, or could come across as overly alarmist.
I think Ord is very careful to avoid such pitfalls in The Precipice, and I’d guess that falling into such pitfalls is an easy and common way for existential risk related outreach efforts to have less positive impacts than they otherwise could, or perhaps even backfire. I understand that a review gives on far less space to work with than a book, so I don’t expect anywhere near the level of nuance and detail. But I think that overconfident or overdramatic statements of uncertain matters (for example) can still be avoided.
I’ll now quote and comment on the specific parts of Chivers’ review that led to that view of mine.
Firstly, in my view, there are three flaws with the opening passage of the review:
Humanity has come startlingly close to destroying itself in the 75 or so years in which it has had the technological power to do so. Some of the stories are less well known than others. One, buried in Appendix D of Toby Ord’s splendid The Precipice, I had not heard, despite having written a book on a similar topic myself. During the Cuban Missile Crisis, a USAF captain in Okinawa received orders to launch nuclear missiles; he refused to do so, reasoning that the move to DEFCON 1, a war state, would have arrived first.
Not only that: he sent two men down the corridor to the next launch control centre with orders to shoot the lieutenant in charge there if he moved to launch without confirmation. If he had not, I probably would not be writing this — unless with a charred stick on a rock.
First issue: Toby Ord makes it clear that “the incident I shall describe has been disputed, so we cannot yet be sure whether it occurred.” Ord notes that “others who claimed to have been present in the Okinawa missile bases at the time” have since challenged this account, although there is also “some circumstantial evidence” supporting the account. Ultimately, Ord concludes “In my view this alleged incident should be taken seriously, but until there is further confirmation, no one should rely on it in their thinking about close calls.” I therefore think Chivers should’ve made it clear that this is a disputed story.
Second issue: My impression from the book is that, even in the account of the person claiming this story is true, the two men sent down the corridor did not turn out to be necessary to avert the launch. (That said, the book isn’t explicit on the point, so I’m unsure.) Ord writes that Bassett “telephoned the Missile Operations Centre, asking the person who radioed the order to either give the DEFCON 1 order or issue a stand-down order. A stand-down order was quickly given and the danger was over.” That is the end of Ord’s retelling of the account itself (rather than discussion of the evidence for or against it).
Third issue: I think it’s true that, if a nuclear launch had occurred in that scenario, a large-scale nuclear war probably would’ve occurred (though it’s not guaranteed, and it’s hard to say). And if that happened, it seems technically true that Chivers probably would’ve have written this review. But I think that’s primarily because history would’ve just unfolded very, very difficulty. Chivers seems to imply this is because civilization probably would’ve collapsed, and done so so severely than even technologies such as pencils would be lost and that they’d still be lost all these decades on (such that, if he was writing this review, he’d do so with “a charred stick on a rock”).
This may seem like me taking a bit of throwaway rhetoric or hyperbole too seriously, and that may be so. But I think among the key takeaways of the book were vast uncertainties around whether certain events would actually lead to major catastrophes (e.g., would a launch lead to a full-scale nuclear war?), whether catastrophes would lead to civilizational collapse (e.g., how severe and long-lasting would the nuclear winter be, and how well would we adapt?), how severe collapses would be (e.g., to pre-industrial or pre-agricultural levels?), and how long-lasting collapses would be (from memory, Ord seems to think recovery is in fact fairly likely).
So I worry that a sentence like that one makes the book sound somewhat alarmist, doomsaying, and naive/simplistic, whereas in reality it seems to me quite nuanced and open about the arguments for why existential risk from certain sources may be “quite low”—and yet still extremely worth attending to, given the stakes.
To be fair, or to make things slightly stranger, Chivers does later say:
Perhaps surprisingly, [Ord] doesn’t think that nuclear war would have been an existential catastrophe. It might have been — a nuclear winter could have led to sufficiently dreadful collapse in agriculture to kill everyone — but it seems unlikely, given our understanding of physics and biology.
(Also, as an incredibly minor point, I think the relevant appendix was Appendix C rather than D. But maybe that was different in different editions or in an early version Chivers saw.)
Secondly, Chivers writes:
[Ord] points out that although the difference between a disaster that kills 99 per cent of us and one that kills 100 per cent would be numerically small, the outcome of the latter scenario would be vastly worse, because it shuts down humanity’s future.
I don’t recall Ord ever saying something like that the death of 1 percent of the population would be “numerically small”. Ord very repeatedly emphasises and reminds the reader that something really can count as deeply or even unprecedently awful, and well worth expending resources to avoid, even if it’s not an existential catastrophe. This seems to me a valuable thing to do, otherwise the x-risk community could easily be seen as coldly dismissive of any sub-existential catastrophes. (Plus, such catastrophes really are very bad and well worth expending resources to avoid—this is something I would’ve said anyway, but seems especially pertinent in the current pandemic.)
I think saying “the difference between a disaster that kills 99 per cent of us and one that kills 100 per cent would be numerically small” cuts against that goal, and again could paint Ord as more simplistic or extremist than he really is.
Finally (for the purpose of my critiques), Chivers writes:
We could live for a billion years on this planet, or billions more on millions of other planets, if we manage to avoid blowing ourselves up in the next century or so.
To me, “avoid blowing ourselves up” again sounds quite informal or naive or something like that. It doesn’t leave me with the impression that the book will be a rigorous and nuanced treatment of the topic. Plus, Ord isn’t primarily concerned with us “blowing ourselves up”—the specific risks he sees as the largest are unaligned AI, engineered pandemics, and “unforeseen anthropogenic risk”.
And even in the case of nuclear war, Ord is quite clear that it’s the nuclear winter that’s the largest source of existential risk, rather than the explosions themselves (though of course the explosions are necessary for causing such a winter). In fact, Ord writes “While one often hears the claim that we have enough nuclear weapons to destroy the world may times over, this is loose talk.” (And he explains why this is loose talk.)
So again, this seems like a case where Ord actively separates his clear-headed analysis of the risks from various naive, simplistic, alarmist ideas that are somewhat common among some segments of the public, but where Chivers’ review makes it sound (at least to me) like the book will match those sorts of ideas.
All that said, I should again note that I thought the review did a lot right. In fact, I have no quibbles at all with anything from that last quote onwards.
This was an excellent meta-review! Thanks for sharing it.
I agree that these little slips of language are important; they can easily compound into very stubborn memes. (I don’t know whether the first person to propose a paperclip AI regrets it, but picking a different example seems like it could have had a meaningful impact on the field’s progress.)
These seem to often be examples of hedge drift, and their potential consequences seem like examples of memetic downside risks.
Civilization Re-Emerging After a Catastrophe—Karim Jebari [EAGx Nordics]
Civilizational Collapse: Scenarios, Prevention, Responses—Denkenberger, Jeffrey Ladish [talks + Q&A]
Update on civilizational collapse research—Ladish [EA Forum] (I found his talk more useful, personally)
The long-term significance of reducing global catastrophic risks—Nick Beckstead [GiveWell/OPP] (Beckstead never actually writes “collapse”, but has very relevant discussion of probability of “recovery” and trajectory changes following non-extinction catastrophes)
Toby Ord on the precipice and humanity’s potential futures [80k podcast] (the first directly relevant part is in the section on nuclear war)
Various EA Forum posts by Dave Denkenberger (see also ALLFED’s site)
Long-Term Trajectories of Human Civilization—Baum et al. [open access paper] (the authors never actually write “collapse”, but their section 4 is very relevant to the topic, and the paper is great in general)
Defence in Depth Against Human Extinction: Prevention, Response, Resilience, and Why They All Matter—Cotton-Barratt, Daniel, Sandberg [open access paper] (collapse is only explicitly addressed briefly, but the paper as a whole still seems quite relevant and useful)
Civilization: Institutions, Knowledge and the Future—Samo Burja [Foresight talk]
Causal diagrams of the paths to existential catastrophe—me [EA Forum]
Things I haven’t properly read/watched/listened to yet but which might be relevant
Feeding Everyone No Matter What—Denkenberger [book]
Why and how civilisations collapse—Kemp [CSER]
https://en.wikipedia.org/wiki/The_Knowledge:_How_to_Rebuild_Our_World_from_Scratch—Dartnell [book] (there’s also this TEDx Talk by the author, but I didn’t find that very useful from a civilizational collapse perspective)
Secret of Our Success—Henrich [book] (I don’t think this book is intended to be about civilizational collapse or recovery, but after reading this review I feel like the book might have some applicable insights)
I intend to add to this list over time. If you know of other relevant work, please mention it in a comment.
Bostrom & Ćirković (pages 1 and 2):
The term ‘global catastrophic risk’ lacks a sharp definition. We use it to refer, loosely, to a risk that might have the potential to inflict serious damage to human well-being on a global scale.
[...] a catastrophe that caused 10,000 fatalities or 10 billion dollars worth of economic damage (e.g., a major earthquake) would not qualify as a global catastrophe. A catastrophe that caused 10 million fatalities or 10 trillion dollars worth of economic loss (e.g., an influenza pandemic) would count as a global catastrophe, even if some region of the world escaped unscathed. As for disasters falling between these points, the definition is vague. The stipulation of a precise cut-off does not appear needful at this stage. [emphasis added]
Open Philanthropy Project/GiveWell:
risks that could be bad enough to change the very long-term trajectory of humanity in a less favorable direction (e.g. ranging from a dramatic slowdown in the improvement of global standards of living to the end of industrial civilization or human extinction).
Global Challenges Foundation:
threats that can eliminate at least 10% of the global population.
Wikipedia (drawing on Bostrom’s works):
a hypothetical future event which could damage human well-being on a global scale, even endangering or destroying modern civilization. [...]
any risk that is at least “global” in scope, and is not subjectively “imperceptible” in intensity.
Beckstead (writing for Open Philanthropy Project/GiveWell):
the Open Philanthropy Project’s work on global catastrophic risks focuses on both potential outright extinction events and global catastrophes that, while not threatening direct extinction, could have deaths amounting to a significant fraction of the world’s population or cause global disruptions far outside the range of historical experience.
[Note that Beckstead might not be saying that global catastrophes are defined as those that “could have deaths amounting to a significant fraction of the world’s population or cause global disruptions far outside the range of historical experience”. He might instead mean that OPP is focused on the relatively extreme subset of global catastrophes which fit that description. It may be worth noting that he later quotes OPP’s other, earlier definition of GCRs, which I listed above.]
My impression is that, at least in EA-type circles, the term “global catastrophic risk” is typically used for events substantially larger than things which cause “10 million fatalities or 10 trillion dollars worth of economic loss (e.g., an influenza pandemic)”.
E.g., the Global Challenges Foundation’s definition implies that the catastrophe would have to be able to eliminate at least ~750 million people, which is 75 times higher than the number Bostrom & Ćirković give. And I’m aware of at least some x-risk focused EAs whose impression is that the rough cutoff would be 100 million fatalities.
With that in mind, I also find it interesting to note that Bostrom & Ćirković gave the “10 million fatalities” figure as indicating something clearly is a GCR, rather than as the lower threshold that a risk must clear in order to be a GCR. From their loose definition, it seems entirely plausible that, for example, a risk with 1 million fatalities might be a GCR.
That said, I do agree that “The stipulation of a precise cut-off does not appear needful at this stage.” Personally, I plan to continue to use the term in a quite loose way, but probably primarily for risks that could cause much more than 10 million fatalities.
Some more definitions, from or quoted in 80k’s profile on reducing global catastrophic biological risks
Gregory Lewis, in that profile itself:
Global catastrophic risks (GCRs) are roughly defined as risks that threaten great worldwide damage to human welfare, and place the long-term trajectory of humankind in jeopardy. Existential risks are the most extreme members of this class.
Open Philanthropy Project:
[W]e use the term “global catastrophic risks” to refer to risks that could be globally destabilising enough to permanently worsen humanity’s future or lead to human extinction.
Schoch-Spana et al. (2017), on GCBRs, rather than GCRs as a whole:
The Johns Hopkins Center for Health Security’s working definition of global catastrophic biological risks (GCBRs): those events in which biological agents—whether naturally emerging or reemerging, deliberately created and released, or laboratory engineered and escaped—could lead to sudden, extraordinary, widespread disaster beyond the collective capability of national and international governments and the private sector to control. If unchecked, GCBRs would lead to great suffering, loss of life, and sustained damage to national governments, international relationships, economies, societal stability, or global security.
Questions: Is a change in the offence-defence balance part of why interstate (and intrastate?) conflict appears to have become less common? Does this have implications for the likelihood and trajectories of conflict in future (and perhaps by extension x-risks)?
Epistemic status: This post is unpolished, un-researched, and quickly written. I haven’t looked into whether existing work has already explored questions like these; if you know of any such work, please comment to point me to it.
Background/elaboration: Pinker argues in The Better Angels of Our Nature that many types of violence have declined considerably over history. I’m pretty sure he notes that these trends are neither obviously ephemeral nor inevitable. But the book, and other research pointing in similar directions, seems to me (and I believe others?) to at least weakly support the ideas that:
if we avoid an existential catastrophe, things will generally continue to get better
apart from the potential destabilising effects of technology, conflict seems to be trending downwards, somewhat reducing the risks of e.g. great power war, and by extension e.g. malicious use of AI (though of course a partial reduction in risks wouldn’t necessarily mean we should ignore the risks)
But How Does the Offense-Defense Balance Scale? (by Garfinkel and Dafoe, of the Center for the Governance of AI; summary here) says:
It is well-understood that technological progress can impact offense-defense balances. In fact, perhaps the primary motivation for developing the concept has been to understand the distinctions between different eras of military technology.
For instance, European powers’ failure to predict the grueling attrition warfare that would characterize much of the First World War is often attributed to their failure to recognize that new technologies, such as machine guns and barbed wire, had shifted the European offense-defense balance for conquest significantly toward defense.
holding force sizes fixed, the conventional wisdom holds that a conflict with mid-nineteenth century technology could be expected to produce a better outcome for the attacker than a conflict with early twentieth century technology. See, for instance, Van Evera, ‘Offense, Defense, and the Causes of War’.
The paper tries to use these sorts of ideas to explore how emerging technologies will affect trajectories, likelihood, etc. of conflict. E.g., the very first sentence is: “The offense-defense balance is a central concept for understanding the international security implications of new technologies.”
But it occurs to me that one could also do historical analysis of just how much these effects have played a role in the sort of trends Pinker notes. From memory, I don’t think Pinker discusses this possible factor in those trends. If this factor played a major role, then perhaps those trends are substantially dependent on something “we” haven’t been thinking about as much—perhaps we’ve wondered about whether the factors Pinker discusses will continue, whereas they’re less necessary and less sufficient than we thought for the overall trend (decline in violence/interstate conflict) that we really care about.
And at a guess, that might mean that that trend is more fragile or “conditional” than we might’ve thought. It might mean that we really really can’t rely on that “background trend” continuing, or at least somewhat offsetting the potentially destabilising effects of new tech—perhaps a lot of the trend, or the last century or two of it, was largely about how tech changed things, so if the way tech changes things changes, the trend could very easily reverse entirely.
I’m not at all sure about any of that, but it seems it would be important and interesting to explore. Hopefully someone already has, in which case I’d appreciate someone pointing me to that exploration.
(Also note that what the implications of a given offence-defence balance even are is apparently somewhat complicated/debatable matter. Eg., Garfinkel and Dafoe write: “While some hold that shifts toward offense-dominance obviously favor conflict and arms racing, this position has been challenged on a number of grounds. It has even been suggested that shifts toward offense-dominance can increase stability in a number of cases.”)
This is adapted from this comment, and I may develop it into a proper post later. I welcome feedback on whether it’d be worth doing so, as well as feedback more generally.
Epistemic status: During my psychology undergrad, I did a decent amount of reading on topics related to the “continued influence effect” (CIE) of misinformation. My Honours thesis (adapted into this paper) also partially related to these topics. But I’m a bit rusty (my Honours was in 2017, and I haven’t reviewed the literature since then).
This is a quick attempt to summarise some insights from psychological findings on the continued influence effect of misinformation (and related areas) that (speculatively) might suggest downsides to some of EA’s epistemic norms (e.g., just honestly contributing your views/data points to the general pool and trusting people will update on them only to the appropriate degree, or clearly acknowledging counterarguments even when you believe your position is strong).
From memory, this paper reviews research on CIE, and I perceived it to be high-quality and a good intro to the topic.
From this paper’s abstract:
Information that initially is presumed to be correct, but that is later retracted or corrected, often continues to influence memory and reasoning. This occurs even if the retraction itself is well remembered. The present study investigated whether the continued influence of misinformation can be reduced by explicitly warning people at the outset that they may be misled. A specific warning—giving detailed information about the continued influence effect (CIE)--succeeded in reducing the continued reliance on outdated information but did not eliminate it. A more general warning—reminding people that facts are not always properly checked before information is disseminated—was even less effective. In an additional experiment, a specific warning was combined with the provision of a plausible alternative explanation for the retracted information. This combined manipulation further reduced the CIE but still failed to eliminate it altogether. (emphasis added)
This seems to me to suggest some value in including “epistemic status” messages up front, but that this don’t make it totally “safe” to make posts before having familiarised oneself with the literature and checked one’s claims.
Here’s a couple other seemingly relevant quotes from papers I read back then:
“retractions [of misinformation] are less effective if the misinformation is congruent with a person’s relevant attitudes, in which case the retractions can even backfire [i.e., increase belief in the misinformation].” (source) (see also this source)
“we randomly assigned 320 undergraduate participants to read a news article presenting either claims both for/against an autism-vaccine link [a “false balance”], link claims only, no-link claims only or non-health-related information. Participants who read the balanced article were less certain that vaccines are safe, more likely to believe experts were less certain that vaccines are safe and less likely to have their future children vaccinated. Results suggest that balancing conflicting views of the autism-vaccine controversy may lead readers to erroneously infer the state of expert knowledge regarding vaccine safety and negatively impact vaccine intentions.” (emphasis added) (source)
This seems relevant to norms around “steelmanning” and explaining reasons why one’s own view may be inaccurate. Those overall seem like very good norms to me, especially given EAs typically write about issues where there truly is far less consensus than there is around things like the autism-vaccine “controversy” or climate change. But it does seem those norms could perhaps lead to overweighting of the counterarguments when they’re actually very weak, perhaps especially when communicating to wider publics who might read and consider posts less carefully than self-identifying EAs/rationalists would. But that’s all my own speculative generalisations of the findings on “falsely balanced” coverage.
Movement collapse scenarios—Rebecca Baron
Why do social movements fail: Two concrete examples. - NunoSempere
What the EA community can learn from the rise of the neoliberals—Kerry Vaughan
Some of the Sentience Institute’s research, such as its “social movement case studies” and the post How tractable is changing the course of history?
These aren’t quite “EA analyses”, but Slate Star Codex has several relevant book reviews and other posts, such as https://slatestarcodex.com/2019/03/18/book-review-inventing-the-future/
It appears Animal Charity Evaluators did relevant research, but I haven’t read it, they described it as having been “of variable quality”, and they’ve discontinued it.
Also, I’m aware that there are a lot of non-EA analyses of these topics. The reasons I’m collecting only EA analyses here are that:
their precise focuses or methodologies may be more relevant to other EAs than would be the case with non-EA analyses
links to non-EA work can be found in most of the things I list here
I’d guess that many collections of non-EA analyses of these topics already exist (e.g., in reference lists)
(See also my/Convergence’s posts on the topic.)
Information hazards [EA concepts]
Information Hazards in Biotechnology—Lewis et al. − 2019 - Risk Analysis [open access paper]
Bioinfohazards [EA Forum]
Information Hazards [Bostrom’s original paper; open access]
Terrorism, Tylenol, and dangerous information [LessWrong]
Lessons from the Cold War on Information Hazards: Why Internal Communication is Critical [LessWrong]
Horsepox synthesis: A case of the unilateralist’s curse? [Lewis]
The Precipice (particularly pages 135-137)- Toby Ord, 2020
Information hazard [LW Wiki]
Informational hazards and the cost-effectiveness of open discussion of catastrophic risks [EA Forum]
A point of clarification on infohazard terminology [LessWrong]
Somewhat less directly relevant
The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse? [open access paper] (commentary here)
The Vulnerable World Hypothesis [open access paper] (footnotes 39 and 41 in particular)
Managing risk in the EA policy space [EA Forum] (touches briefly on information hazards)
Strategic Implications of Openness in AI Development [open access paper] (sort-of relevant, though not explicitly about information hazards)
[Review] On the Chatham House Rule (Ben Pace, Dec 2019) [LessWrong]
Interesting example: Leo Szilard and cobalt bombs
In The Precipice, Toby Ord mentions the possibility of “a deliberate attempt to destroy humanity by maximising fallout (the hypothetical cobalt bomb)” (though he notes such a bomb may be beyond our current abilities). In a footnote, he writes that “Such a ‘doomsday device’ was first suggested by Leo Szilard in 1950″. Wikipedia similarly says:
The concept of a cobalt bomb was originally described in a radio program by physicist Leó Szilárd on February 26, 1950. His intent was not to propose that such a weapon be built, but to show that nuclear weapon technology would soon reach the point where it could end human life on Earth, a doomsday device. Such “salted” weapons were requested by the U.S. Air Force and seriously investigated, but not deployed. [...]
The Russian Federation has allegedly developed cobalt warheads for use with their Status-6 Oceanic Multipurpose System nuclear torpedoes. However many commentators doubt that this is a real project, and see it as more likely to be a staged leak to intimidate the United States.
That’s the extent of my knowledge of cobalt bombs, so I’m poorly placed to evaluate that action by Szilard. But this at least looks like it could be an unusually clear-cut case of one of Bostrom’s subtypes of information hazards:
Attention hazard: The mere drawing of attention to some particularly potent or relevant ideas or data increases risk, even when these ideas or data are already “known”.
Because there are countless avenues for doing harm, an adversary faces a vast search task in finding out which avenue is most likely to achieve his goals. Drawing the adversary’s attention to a subset of especially potent avenues can greatly facilitate the search. For example, if we focus our concern and our discourse on the challenge of defending against viral attacks, this may signal to an adversary that viral weapons—as distinct from, say, conventional explosives or chemical weapons—constitute an especially promising domain in which to search for destructive applications. The better we manage to focus our defensive deliberations on our greatest vulnerabilities, the more useful our conclusions may be to a potential adversary.
It seems that Szilard wanted to highlight how bad cobalt bombs would be, that no one had recognised—or at least not acted on—the possibility of such bombs until he tried to raise awareness of them, and that since he did so there may have been multiple government attempts to develop such bombs.
I was a little surprised that Ord didn’t discuss the potential information hazards angle of this example, especially as he discusses a similar example with regards to Japanese bioweapons in WWII elsewhere in the book.
I was also surprised by the fact that it was Szilard who took this action. This is because one of the main things I know Szilard for is being arguably one of the earliest (the earliest?) examples of a scientist bucking standard openness norms due to, basically, concerns of information hazards potentially severe enough to pose global catastrophic risks. E.g., a report by MIRI/Katja Grace states:
Leó Szilárd patented the nuclear chain reaction in 1934. He then asked the British War Office to hold the patent in secret, to prevent the Germans from creating nuclear weapons (Section 2.1). After the discovery of fission in 1938, Szilárd tried to convince other physicists to keep their discoveries secret, with limited success.
Ways people trying to do good accidentally make things worse, and how to avoid them—Rob Wiblin and Howie Lempel [80,000 Hours]
How to Avoid Accidentally Having a Negative Impact with your Project—Max Dalton and Jonas Vollmer [EAG]
Sources that seem somewhat relevant
https://en.wikipedia.org/wiki/Unintended_consequences (in particular, “Unexpected drawbacks” and “Perverse results”, not “Unintended benefits”)
(See also my “shortform comment” lists of sources related to information hazards, differential progress, and the unilateralist’s curse.)
Unilateralist’s curse [EA Concepts]
Horsepox synthesis: A case of the unilateralist’s curse? [Lewis] (usefully connects the curse to other factors)
The Unilateralist’s Curse and the Case for a Principle of Conformity [Bostrom et al.’s original paper]
Hard-to-reverse decisions destroy option value [CEA]
Framing issues with the unilateralist’s curse—Linch, 2020
Managing risk in the EA policy space [EA Forum] (touches briefly on the curse)
Ways people trying to do good accidentally make things worse, and how to avoid them [80k] (only one section on the curse)
Some concepts/posts/papers I find myself often wanting to direct people to
(Will likely be expanded as I find and remember more)
Differential Intellectual Progress as a Positive-Sum Project [FRI]
Differential technological development: Some early thinking [GiveWell]
Differential progress [EA Concepts]
Differential technological development [Wikipedia]
On Progress and Prosperity [EA Forum]
Differential intellectual progress [LW Wiki]
Existential Risks: Analyzing Human Extinction Scenarios [open access paper] (section 9.4) (introduced the term differential technological development, I think)
Intelligence Explosion: Evidence and Import [MIRI] (section 4.2) (introduced the term differential intellectual development, I think)
Some sources that are quite relevant but that don’t explicitly use those terms
Strategic Implications of Openness in AI Development [open access paper]
The growth of our “power” (or “science and technology”) vs our “wisdom” (see, e.g., page 34 of The Precipice)
The “pacing problem” (see, e.g., footnote 57 in Chapter 1 of The Precipice)
Each of the following works show or can be read as showing a different model/classification scheme/taxonomy:
Defence in Depth Against Human Extinction:Prevention, Response, Resilience, and Why They All Matter—Cotton-Barratt, Daniel, and Sandberg, 2020
The same model is also discussed in Toby Ord’s The Precipice.
Classifying global catastrophic risks—Avin et al., 2018
Causal diagrams of the paths to existential catastrophe—Michael Aird, 2020
Conflict of interest statement: I am the aforementioned human.
This might not quite “belong” in this list. But one could classify risks by which of the different “paths” they might follow (e.g., those that would vs wouldn’t “pass through” a distinct collapse stage).
Typology of human extinction risks—Alexey Turchin, ~2015
Related LessWrong post
Personally, I think the model/classification scheme in Defence in Depth is probably the most useful. But I think at least a quick skim of the above sources is useful; I think they each provide an additional useful angle or tool for thought.
Wait, exactly what are you actually collecting here?
The scope of this collection is probably best revealed by checking out the above sources.
But to further clarify, here are two things I don’t mean, which aren’t included in the scope:
Classifications into things like “AI risk vs biorisk”, or “natural vs anthropogenic”
Such categorisation schemes are clearly very important, but they’re also well-established and you probably don’t need a list of sources that show them.
Classifications into different “types of catastrophe”, such as Ord’s distinction between extinction, unrecoverable collapse, and unrecoverable dystopia
This is also very important, and maybe I should make such a collection at some point, but it’s a separate matter to this.
Things I’ve written
My thoughts on Toby Ord’s existential risk estimates
My Google Play review
My review of Tom Chivers’ review of Toby Ord’s The Precipice
If a typical mammalian species survives for ~1 million years, should a 200,000 year old species expect another 800,000 years, or another million years?
Working titles of things I plan/vaguely hope to write
Note: If you might be interested in writing about similar ideas, feel very free to reach out to me. It’s very unlikely I’ll be able to write all of these posts by myself, so potentially we could collaborate, or I could just share my thoughts and notes with you and let you take it from there.
Defining existential risks and existential catastrophes
My thoughts on Toby Ord’s policy & research recommendations
Civilizational collapse and recovery: Toby Ord’s views and my doubts
The Terrible Funnel: Estimating odds of each step on the x-risk causal path (working title)
The idea here would be to adapt something like the “Great Filter” or “Drake Equation” reasoning to estimating the probability of existential catastrophe, using how humanity has fared in prior events that passed or could’ve passed certain “steps” on certain causal chains to catastrophe.
E.g., even though we’ve never faced a pandemic involving a bioengineered pathogen, perhaps our experience with how many natural pathogens have moved from each “step” to the next one can inform what would likely happen if we did face a bioengineered pathogen, or if it did get to a pandemic level.
This idea seems sort of implicit in the Precipice, but isn’t really spelled out there. Also, as is probably obvious, I need to do more to organise my thoughts on it myself.
This may include discussion of how Ord distinguishes natural and anthropogenic risks, and why the standard arguments for an upper bound for natural extinction risks don’t apply to natural pandemics. Or that might be a separate post.
Developing—but not deploying—drastic backup plans
“Macrostrategy”: Attempted definitions and related concepts
This would relate in part to Ord’s concept of “grand strategy for humanity”
Collection of notes
A post summarising the ideas of existential risk factors and existential security factors?
I suspect I won’t end up writing this, but I think someone should. For one thing, it’d be good to have something people can reference/link to that explains that idea (sort of like the role EA Concepts serves).
Some selected Precipice-related works by others
80,000 Hours’ interview with Toby Ord
Slate Star Codex’s review of the book
FLI Podcast interview with Toby Ord
tl;dr I think it’s “another million years”, or slightly longer, but I’m not sure.
In The Precipice, Toby Ord writes:
How much of this future might we live to see? The fossil record provides some useful guidance. Mammalian species typically survive for around one million years before they go extinct; our close relative, Homo erectus, survived for almost two million. If we think of one million years in terms of a single, eighty-year life, then today humanity would be in its adolescence—sixteen years old, just coming into our power; just old enough to get ourselves into serious trouble.
(There are various extra details and caveats about these estimates in the footnotes.)
Ord also makes similar statements on the FLI Podcast, including the following:
If you think about the expected lifespan of humanity, a typical species lives for about a million years [I think Ord meant “mammalian species”]. Humanity is about 200,000 years old. We have something like 800,000 or a million or more years ahead of us if we play our cards right and we don’t lead to our own destruction. The analogy would be 20% of the way through our life[...]
I think this is a strong analogy from a poetic perspective. And I think that highlighting the typical species’ lifespan is a good starting point for thinking about how long we might have left. (Although of course we could also draw on many other facts for that analysis, as Ord discusses in the book.)
But I also think that there’s a way in which the lifespan analogy might be a bit misleading. If a human is 70, we expect they have less time less to live than if a human is 20. But I’m not sure whether, if a species if 700,000 years old, we should expect that species to go extinct sooner than a species that is 200,000 years old will.
My guess would be that a ~1 million year lifespan for a typical mammalian species would translate into a roughly 1 in a million chance of extinction each year, which doesn’t rise or fall very much in a predictable way over most of the species’ lifespan. Specific events, like changes in a climate or another species arriving/evolving, could easily change the annual extinction rate. But I’m not aware of an analogy here to how ageing increases the annual risk of humans dying from various causes.
I would imagine that, even if a species has been around for almost or more than a million years, we should still perhaps expect a roughly 1 in a million chance of extinction each year. Or perhaps we should even expect them to have a somewhat lower annual chance of extinction, and thus a higher expected lifespan going forwards, based on how long they’ve survived so far?
(But I’m also not an expert on the relevant fields—not even certain what they would be—and I didn’t do extra research to inform this shortform comment.)
I don’t think that Ord actually intends to imply that species’ “lifespans” work like humans’ lifespans do. But the analogy does seem to imply it. And in the FLI interview, he does seem to briefly imply that, though of course there he was speaking off the cuff.
I’m also not sure how important this point is, given that humans are very atypical anyway. But I thought it was worth noting in a shortform comment, especially as I expect that, in the wake of The Precipice being great, statements along these lines may be quoted regularly over the coming months.
The Precipice—Toby Ord (Chapter 5 has a full section on Dystopian Scenarios)
The Totalitarian Threat—Bryan Caplan (a link to a Word doc version can be found on this page) (some related discussion on the 80k podcast here; use the “find” function)
The Centre for the Governance of AI’s research agenda—Allan Dafoe (this contains discussion of “robust totalitarianism”, and related matters)
A shift in arguments for AI risk—Tom Sittler (this has a brief but valuable section on robust totalitarianism) (discussion of the overall piece here)
Existential Risk Prevention as Global Priority—Nick Bostrom (this discusses the concepts of “permanent stagnation” and “flawed realisation”, and very briefly touches on their relevance to e.g. lasting totalitarianism)