I make a newsletter, called the Anti-Apocalyptus newsletter, that every week links to five articles about topics related to EA, among other things X-risk, disease, great power wars and emerging technologies.
In it I also occasionally interview people who work in these fields, one of which is the below talk I had with Thomas Moynihan from the Future of Humanity Institute and author of X-Risk: How Humanity Discovered Its Own Extinction. Since my previous interview with Tom Chivers generated some positive interest here, I thought this one might be interesting for the forum as well.
In the interview we mainly discuss how the Enlightenment discovered existential risk, where it was previously absent from our thinking. And why this realisation is a cornerstone of modernity.
---
Could you introduce your current work and position, and how you ended up there?
“I have just finished a book on the history of humans thinking about human extinction, entitled X-Risk: How Humanity Discovered its Extinction. The book argues that this is a distinctly modern idea. In other words, prior to modernity, people could not even think about it. The book also argues that the discovery of this idea can in many ways be seen as one of the philosophical centrepieces, or crowning achievements, of what we call the modern epoch. In other words, it is one of the most important conceptual discoveries our species has ever made.
The book stages the drama of its discovery across the past few centuries, and, through this, aims to unveil today’s research into X-risk as part and parcel of a much wider undertaking of human learning. It is aimed at a wider audience: the goal being to introduce a larger demographic the central motivating ideas behind longtermism and existential risk.
At the moment, I am working with Oxford’s Future of Humanity Institute with a grant from Berkeley’s Existential Risk Initiative. Before that, I was doing a PhD at Oriel College, Oxford. My PhD thesis was also on the topic of the history of ideas about human extinction: a topic of research I first started following almost eight years ago, back in 2012.
Though what I do is the ‘history of ideas’, a mixture of history of science, cultural history, and history of philosophy, my own conviction is that historical investigation should have transformative effects on the present: potentially realigning our current priorities and attitudes. I don’t think history is just the collecting and constellating of an inert and value-neutral set of facts or sources, it is the forging of an active argument for actions in the present and thus contains value-laden contestations and claims.”
Why is it important for humanity to look at and combat existential risk? Particularly compared to other cause areas?
“Following on from the above, we humans are historical beings. What I mean by this is that we correct ourselves and ameliorate errors in our beliefs over time. Uniquely, we acknowledge that we can be wrong. Through this we improve our ideas about the world. Moreover, we pass these improvements onto the next generations. It is because of this ‘ratcheting effect’ that we accumulate increasing practical and predictive knowledge over the generations and, thus, come to increasingly transform our material conditions at the same time as incrementally alleviating ignorances and injustices. No other animals have a history in this cumulative sense.
“there was once a time when people couldn’t think about human extinction nor the extinction of human values within the wider universe.”
But because we acknowledge we can be wrong, and thus have the potential to be better, we also acknowledge that, as historical beings, we may not know exactly yet what our true potential is, or, our true potential to do good things or create value in the world. This means that there is vast value in ‘keeping history going’: so that we can further learn what is potentially at stake in our existing or not existing. Anything else seems brash, because we don’t know all of the facts of the matter yet, so we can’t really make an informed decision.
To take a case in point, there was once a time when people couldn’t think about human extinction nor the extinction of human values within the wider universe. The prerequisite of all ethical action—which is the existence of ethical agents—was taken as an unquestionable given: i.e. not subject to hazard or risk. Then, after centuries of painstaking inquiry concerning not only the objective world but also our epistemic biases, this prerequisite for all practical action gradually came under scrutiny due to the dawning discovery that ethical agents could, in fact, go extinct—irreversibly so. The existence of ethical action itself came under threat. Ever since, various voices have slowly come to realise that our primary ethical obligation must therefore be to protect the very foothold, within the physical universe, of all otherethical obligations, no matter what they may be: that is, to safeguard the existence of beings like ourselves—ethically recognisant agents, or, agents that can follow ethical argumentation. I like to call this the imperative behind all imperatives. We have progressively realised that there is a reason to our existence because we have progressively become sensitive to the fact that the very existence of reason itself might depend upon it, at least, in our corner of the cosmos.
After people discovered this, all prior ethical outlooks—though often still compelling and compulsory in their own domains of application—were revealed to be somewhat limited or parochial in scope. As with classical mechanics in physics, the elder ethical outlooks, all of which took the existence of ethics itself for granted, were revealed to be regional: often still correct in their own way, but nested within this wider imperative behind all imperatives.
In a certain sense I think we were duty-bound to discover this idea, inasmuch as it a discovery that any agent that submits itself to acting ethically or altruistically must eventually come to acknowledge. The pathways may be different, but, at the very least, there is clear survival value in knowing about the value of survival. But the point here is that there will still be other discoveries and shocks on the road ahead for us, equally seismic and paradigm-shifting in scope: we do not know everything yet…
So, yes, I think ‘keeping history going’, i.e. minimizing and mitigating history-ending disasters, should be our priority not only because of all the things that we do know about this imperative but also because of all of the things which we as yet do not. Nick Bostrom refers to this as the ‘option value’ of civilization, and I think it is a deeply important point.
All other ethical causes focus on the things we know and already acknowledge as requiring alleviation, which is utterly legitimate, but ‘keeping history going’ allows us to discover the causes that we can’t even yet imagine. This gives it a unique position, as an ethical priority, in my estimation.”
Why are you personally interested in something as ‘dark’ as existential risk?
“Because I feel that it is, almost by definition, the most important topic of the modern era. If modernity is the loss of external moral assurances and hierarchies, and is thus also humanity’s assumption of increasing self-responsibility for its own institutions and opinions, then the demand to think about existential risk is an unavoidable threshold in this still-unfolding process. Because, in the discovery of X-risk, is the understanding that our existence matters because independent nature will not necessarily uphold our own values or morals should we cease to exist. It is the ultimate expression of the loss of external assurances, and thus is one of the culminating features of modern history.
Moreover, as the twentieth-century futurologist and palaeontologist Pierre Teilhard de Chardin once sagely said: ‘There are no summits without abysses’. That is to say, we need to focus on the perils to put the promises into relief and vice versa. This is the central lesson of ‘coming of age’, both for each individual but also for civilization itself: one does not assume responsibility without coming to carefully acknowledge the risks one faces.”
You trace the intellectual history of existential risk back to the Enlightenment. How did people think about the end of humanity before that, and what changed then?
“As mentioned, modernity is the loss of external assurances, and the concomitant assumption of self-responsibility for the human race. That is to say, elder worldviews tended to believe that, independently of our actions and beliefs, value was secure and safe in the external universe. Both mythological and theistic worldviews tended to assume that moral law is in some sense woven into the very fabric of the cosmos at large. Indeed, medieval cosmology makes barely any distinction between ethics and physics: moral hierarchy just is the celestial architecture. There was thus nowhere in the cosmos that was not peopled with moral forces or value structures. Even the centre of the earth was thought to be populated, but with devilish disvalue rather than angelic jubilation. So, looking at the world like this, value is secure regardless of what happens to humans or what humans do.
In line with this, Aristotle once said that everything useful, valuable, and knowable has been discovered infinitely many times by the ‘wisdom of the ages’. That means that not only are there are no risks or challenges beyond what has been previously recorded, but also that maximizing value is just returning to some previous maximum and, moreover, that epistemic inquiry is essentially just remembering or rediscovering old stuff in an edifying way. This background assumption, which influenced many other subsequent thinkers, meant that the material conditions of human existence could not be improved upon and neither could they fail.
“After centuries of presuming that value was indestructible, scientists and philosophers slowly woke up to the fact that everything that we find valuable could plausibly be destroyed.”
Moreover, this went hand-in-hand with another belief—commonly called the Principle of Plenitude—which was pretty much omniprevalent up until the Enlightenment. This held that nothing can truly be lost from nature. From Plato to the Stoics, people thought that whatever was destroyed was eventually, inevitably returned in the fullness of time. This applied not only to species but to entire worlds and also to values. Anything valuable that is decimated will one day replenish or return. So, value was indestructible. It was utterly conserved throughout its myriad destructions—so, again, we could do nothing to maximize it in the world nor could we be culpable for not being vigilant by allowing it to be diminished or damaged. So why bother?
When the scientific revolution rolled around, ethics and physics became separated and it was revealed we lived on just one planet in the midst of a vast interstellar void. But people simply continued to assume that value was indestructible. Around Galileo’s time, people were already acknowledging suns and entire planetary systems could be destroyed, but they were totally convinced that this would not lead to any ‘loss’ or ‘damage’ in nature, because everything destroyed would return on another world. In the cosmic infinity, nothing can truly go extinct. People explicitly said that this was why the destruction of the Earth would not matter: because humans and their values will either continue to exist elsewhere or will re-evolve on another planet.
This perpetuated a sense that value was indestructible and that we could do nothing to change the amount of it in the universe. Believing that value is interminably conserved, one could accommodate any catastrophe or disaster, no matter how titanic and calamitous, whilst still stripping it of any genuine moral consequences. It led to a sense of cosmic nonchalance. Philosophers explicitly said that, because the universe is maximally and recurrently populated with humanoid beings, we ought not care about the threats facing Earth.
Around the enlightenment, however, intellectuals started realising that we ought to be careful not to contaminate our objective theories of the world with our moral prejudices and biases. The presumption that independent nature aligns with our ideas of what is dignified or valuable or purposive was shown to be naïve and unfounded. It might seem like a massive waste of space for these other worlds to not be populated with interminable humanoids, but ‘waste’ is a value judgement and we should not let our value judgements distort our objective theories. So, from cosmic nonchalance, people started waking up to a sense of cosmic fragility. Indeed, it was only around the turn of the 1800s that people first started entertaining the idea that most of the cosmos is sterile, inhospitable, and inactive, and not rife with overabundant life and civilized complexity. This reached a head, in the previous century, with Enrico Fermi’s famous paradox regarding the conspicuous absence of extraterrestrial activities.
Note that this has nothing to do with the fact of how populous the galaxy actually turns out to be nor how many other ‘humanoids’ might actually be out there. It has nothing to do with truth statements. It is, instead, entirely to do with the dismantling of the presumption ahead of evidence that the universe is fully populated. Though it may seem strange to us now, this assumption was basically the default until the late Enlightenment. After critiquing the idea, we now realise that, even if we have high credence in other populated planets, this is a matter of probability and not yet one of certainty. So, using it as an excuse to act nonchalantly with our own civilization is revealed as extremely rash. But, as mentioned, this nonchalance was pretty much the only game in town up until the Enlightenment.
In tandem with this, across the seventeenth and eighteenth centuries, a number of new sciences emerged that further highlighted our precariousness. These were fields like probabilism, demography, and geology. For instance, geology revealed our planet was not designed with maximal and perpetual habitability in mind, as had previously been assumed by theists and deists, and that it had, in fact, been the scene of multitudes of prehistoric extinctions and paleontological holocausts. Geology also revealed that complex life was not a permanent feature on our own planet, so why should it be elsewhere in the cosmos? Scientists started tentatively thinking about questions of habitability and inhabitability.
Complex things like life, mind, and civilization started to look more and more contingent and precarious. In lockstep with this, people started thinking of these things as more precious. After centuries of presuming that value was indestructible, scientists and philosophers slowly woke up to the fact that everything that we find valuable could plausibly be destroyed.Finally, people started to think of our extinction, clearly and vocally, as a moral tragedy: tragic because it would represent the permanent and irreversible termination of humanity and its values. By thus coming to acknowledge the moral stakes involved in our extinction, people at last started to genuinely think about it as an objective event that can be predicted and even mitigated. It was learning the moral stakes that motivated people to start assessing it objectively. Indeed, by the 1770s people were already using probability theory to calculate the odds of cometary impact and some were even suggesting mitigation strategies in the form of ballistics and deflection technologies.
But this all remained the parlour speculation of natural philosophers and scientists. It did not seem a pressing issue, it seemed a remote plausibility. Moreover, with the ascendance of evolutionary worldviews across the 1800s, there was a mood of general confidence. Even Darwin himself fell foul to some progressivist presumptions: combined with his ideas of gradualist and continuous change, early evolutionary theory implied that a species as adaptive and successful as ours would not die off without a lineage, but would merely be replaced by its more adaptive and thus more ‘progressed’ successors. Darwin himself was confident that this postpartum ‘progress’ would apply to moral, as well as physical, affairs So, the myth of inexorable progress in the natural world—inevitably taking place independently of our actions and decisions—again created a sense of existential nonchalance. This was prevalent during the Victorian era. People ultimately became more concerned with our species degenerating into something less worthy than it dying out entirely and without recompense or lineage.
Around mid-century, thermodynamics was formulated and heat death theorised, but this again remained far-off and distant: not a pressing issue. At the turn of the 1900s, scientists began acknowledging cosmic disasters and global risks, but they continued to think that they were so improbable as to not be an object of massive concern. They pointed to how long life had persisted on Earth without such a biosphere-ending disaster, awareness of observer selection effects was not yet on the menu. After World War 1, the idea of inexorable progress took a battering and many intellectuals and politicians became concerned that our species may one day wipe out itself, whether with engineered pandemic or mechanized warfare. But, again, this remained remote: there was, as yet, no tangible technology that would actuate this fate. Then, in World War 2, the H-bomb was developed. There were fears that it would ignite the atmosphere and, later, there was talk of developing a ‘cobalt bomb’ that could fatally irradiate the biosphere. For the first time, extinction became a policy issue. Scientists turned their attention to it as a serious topic. The myth of inevitable progress, and the existential nonchalance it perpetuated, took a further battering. Moreover, the nuke changed how we relate to technology in a cosmic setting: it seemed less an inevitability because of its preeminent adaptiveness, but began to look more like a potential fluke that is prone to destroy itself should it not be vigilant. Moreover, the inflammation of Fermi’s Paradox, ever since the 1950s, and the continuing lack of evidence for other advanced civilizations in SETI searches has only exasperated such issues. Ever since the Cold War, there has been a building scientific and policy concern in such issues, building into the millennium with the articulation of a rigorous analytic framework to think about ‘existential risk’ as promoted by Nick Bostrom and others.
So, I like to think about this history as having four phases. First is the phase of ‘indestructible value’, from prehistory to the 1600s. This was when people couldn’t even think of a possible world without human minds. Then there is a phase of ‘cosmic nonchalance’ from the 1600s to the 1800s. This was when people thought about planet-destroying catastrophes, but believed that humans would reappear elsewhere or elsewhen in the cosmos. Then, there is a phase of growing ‘cosmic loneliness’ from the 1800s to 1950. This saw growing awareness that the universe is not, by default, maximally habitable or fully inhabited. Then, after 1950 to the present day, is a phase of emerging awareness of ‘astronomical value’: that is, an building sense that, finding ourselves within a corner of the cosmos that appears sterile, unpopulated, and undeveloped, there is therefore a potential for us to make an genuinely significant difference on the amount of value in the universe by surviving X-risks and expanding outwards. These are the landmark phases in thinking about the existential fate of humanity across the ages.”
What’s the relation between religious notions of apocalypse, and modern-day ideas about existential risk?
“To be blunt, none. A lot of people believe that there is nothing ever truly new discovered and that all knowledge is just a rediscovery of the wisdom of the ancients. They would be in the illustrious company of Aristotle, but this opinion is totally wrong. You can find ideas about the end of the world across all times and places, but these are all nested within cycles of return and reconstitution or within a framework of divine justice and final judgement. Therefore, they have nothing to do with extinction as the permanent and irreversible loss of our species and its activities.
I like to talk about these things as ‘false friends’ in the history of ideas. I borrow the term from linguistics, where it refers to ideas that misleadingly sound similar but mean entirely different things in different languages. For instance, ‘das Gift’ in German does not mean a nice present, it means poison.
A similar thing applies to ancient notions of apocalypse and modern ideas of human extinction. They are false friends. They may look similar with their melodrama and pyrotechny, but these are just aesthetic accoutrements, and on a deeper level the notions are actually totally conceptually incompatible.
Old notions of apocalypse conceptualize the end of the world as the revelation of the ultimate meaningfulness of time. Even if it is divine and inscrutable, the moral lesson of cosmic chronology is revealed. Judgement day means that we can rest assured knowing that moral law will get the last word in the universe. Extinction, contrarily, anticipates the end of meaningfulness within time. It acknowledges that moral action can be terminated and the physical world continues, for aimless aeons, without it.
I like to sum it up like this: where apocalypse secures a sense of an ending, extinction anticipates the ending of sense.
This goes back to what I was talking about above. Apocalypse and judgement day allows a certain sense of nonchalance, because regardless of what we do, moral law is the structuring principle of the cosmos. It’s the same for the ancient cyclical cosmologies too: there may be times when there are no humans, but humans will always return no matter what, so why worry? Why fret? Why bother correcting your forecasts with more accurate, predictive ones?
“Apocalypse coddles us with certainty, but extinction calls us to action”
At the entrance of the modern world lay the discovery of the stakes in our ignorance. It was learning that our ignorance may have genuinely disastrous consequences that first motivated early modern societies to correct their opinions about the world, rather than relying on the wisdom of tradition, in order to reach more predictive knowledge. The problem with apocalyptic prophecies and religious revelation is that they do not acknowledge that they can be wrong. Again, this is why there is a vast difference between ‘forecast’ and ‘prophecy’ as there is between ‘extinction’ and ‘apocalypse’.
Again, this is why the Enlightenment was important. The great Enlightenment philosophers liked to define enlightening as humanity’s emergence from immaturity. They thought of it as our ‘coming of age’ as a species. Of course, there were problems with this: the universalism espoused by these people was often only extended to certain demographics and this led to tragic and criminal consequences and complicities. However, the basic ideal remains sound: ‘assuming maturity’ as a civilization and a species just means asserting more control over our means of existence, thus eliminating need, privation, and suffering, and undertaking more accountability over our institutions and opinions, thus eradicating injustice and ignorance. Just because prior espousers of this programme have previously failed to live up to their own ideals is no indictment against the ideals they (fallibly) pursued. It is the same with our own fallibility. To think otherwise would be to reduce all the promise of humankind to the scale of its former errancies. In my opinion, this is not only smallminded but small of heart.
Moreover, the ideal of ‘coming of age’ is not dead. It has just moved elsewhere. Carl Sagan and others used to talk about ours as an age of ‘technological adolescence’, where we have the power to kill ourselves but not the foresight or resilience to prevent this outcome. Nick Bostrom continues to talk about ‘technological maturity’ as a state of maximal insulation against risk coupled with the achievement of the greatest energetic capture and economic productivity that is physically feasible.
Talking in lofty terms about diffuse cultural attitudes that span centuries, we can think of existential nonchalance as a sign of minority, or immaturity, and the shift toward growing concern for existential risk as a move toward majority, or maturity. Apocalypse is a sign of minority, and coming to understand extinction—and its unique moral stakes—is a requirement of ‘coming of age’.
Apocalypse coddles us with certainty, but extinction calls us to action, but it is precisely because of this that it is the path to civilizational maturity: because what could be more ethically responsible than coming to care for the very existence of ethics and responsibility itself? What could be more altruistic than coming to safeguard the very anchorage of altruistic action in an otherwise indifferent universe? As historical beings, we no doubt have a lot left to learn, a lot of maturing left to do, but at least we now acknowledge this.
This, again, is why the discovery of extinction is, in many ways, underappreciated as a philosophical centrepiece of the modern age. It is a total break with prior traditions of apocalypse, revelation, judgement day, and end times.”
How can a history of the idea help us think about existential risk today?
“It can provide us with a novel argument as to why this topic is so essential and compelling. Because it demonstrates that the topic emerges from the wider sweep of human history, that it is not just some contingent ‘trend’ or ‘fad’ of the present moment, but part of a building and historically consistent and continuous process: it is part of that process of ‘coming of age’ as a global community, whatever that may mean, and regardless of how fallible we have been in the past concerning this ideal, and regardless of how fallible and wayward we will continue to be in relation to such an goal.
Assuming responsibility for our species very obviously remains an unfinished process, and there is much left to learn, but the history undeniably shows that thinking about X-risk emerges from this undertaking as it spreads across history, in all its waywardness and across all its setbacks and detours. Looking at the history, moreover, shows how uniquely important it is as a culmination of centuries of inquiry, spanning multiple continents and cultures.
Today many people interested in X-risk rightly care about it because of the weight of the future consequences of such a disaster, and this is utterly convincing and compelling, but we ought not take for granted all the hard work that got us here in the first place. Recounting the history shows how central these ideas are to the founding ideas of modernity itself. It is something that people care about now because of the immense consequences, but it was also something that people were historically bound to discover ever since humans first defined themselves as creatures that attempt to assume ever greater accountability for the consequence of their actions.
More tangibly, I believe that this history dispels some of the suspicions that this is just the newest face of perennial doom-mongering. Only by retracing the history, of the emergence of extinction as separate to religious notions, do we truly disambiguate apocalypse and X-risk. Elsewhere, it is a nice object lesson in how biases have afflicted inquiry into existential matters. In geology and cosmology, reigning faith in uniformitarianism and gradualism long obstructed acknowledgement of the existence of abrupt and sudden catastrophes. The belief was that the cosmos was a steady and balanced system. This was revealed to be less about actual evidence, and more about an, almost aesthetic, predilection for parsimony and equilibrium. In other quarters, such as biology, the belief in progressivism long led people to believe that a species as adaptive as humanity will only be replaced by its better descendants. This assumption, whose prevalence had more to do with moral bias than objective evidence, occluded the true severity of the plausible extinction of our species. As already mentioned, for many Victorian era biologists, it seemed that our extinction might even be the price of progress. The same applies to the widely-held conviction that something like Homo sapiens is the inevitable outcome of ‘replaying the tape’ on evolution, on other planets or after some global catastrophe here on Earth. This conviction only really began to come under scrutiny in the mid-1900s. Its prior prevalence was, again, to do with our tendency to prefer flattering theories, where human-like intelligence is inevitable. The point is this: the direction of travel has been that, as we progressively overcome our pre-theoretical biases, we have come to realise, more and more, how precious and precarious we may potentially be and, thus, how cautious it is sensible for us to be. I believe that this ‘direction of travel’ is not going to slow, or let up, any time soon: thus, understanding its historical precedent is a useful lesson for the present. We ought to continue to be wary about the ways that our inherited biases distort our objective theories about the risks and obstruct our appreciation of the true moral stakes of our actions and decisions.
Another insight from history is that of the progress we have made, but how contingent and fragile it remains. It is easy to forget how far we have come in our knowledge of the world, but it is also easy to become complacent that continued progress is inevitable. There are naïve optimists and naïve pessimists, but it is increasingly undeniable that neither good nor bad outcomes are independently guaranteed: it is up to us to produce the fate that we want.”
What do you think are the most urgent existential risks humanity should focus on?
“Urgency is ambiguous: does it mean most probable or most assuredly fatal? Ever since the opening of the 1900s people have rightly realised that anthropogenic risks are the true ones to fear in terms of omni-fatality. After World War 1’s mechanized conflagration, many intellectuals and politicians made noises about universal species suicide, but this still seemed far-off and remote. Then the H-bomb was produced, and suddenly it became a policy issue. Afterward, yet more technological risks have emerged: nanotech, synthetic biology or AI. This recent paper assesses the background risks from nature to Homo sapiens, taking into account selection effects, and backs up this view: the most urgent existential risks are anthropogenic and technological ones.
Additionally, I think an area that possibly requires more thought is systemic risks. There are plenty of diffuse issues, exploits, and flaws, in our institutions, our technologies, and our behaviours, that are not X-risks in their own right but that could nonetheless massively compound the likelihood of an existential catastrophe: think not only of how our actions increase the likelihood of zoonosis and world-perturbing pandemics, but also of things like spread of misinformation and political instability. I know some very interesting work is being done on the latter.
It is not an existential risk, but I think more research should be done on the topic of how likely recovery is after a civilizational collapse. How much infrastructure and knowledge needs to be destroyed for this recovery to become improbable? This would be useful to gauge. More dramatically, what is the likelihood of some other species ‘picking up where we left off’ should we go extinct. Historically speaking, this has been the go-to method of deescalating the severity of our extinction. It has proven a very attractive way of placating ourselves in the face of such a tragedy. Throughout the ages, people would often say ‘ah, yes, but some other species that is left behind will evolve a big brain and take our place’. Monkeys have often been elected for the task, think of the Planet of the Apes, but some people have even imagined weasels or dogs or lizards. Statements of inevitably re-evolving intelligence extend from Denis Diderot to Edwin Hubble. Statements in the opposite direction begin around 1800, but only really start to take hold with George Gaylord Simpson’s 1964 paper ‘The Nonprevalence of Humanoids’, and then reach a head with Stephen Jay Gould’s 1989 declaration that ‘Homo sapiens is an entity, not a tendency’. Again, historically speaking, the direction of travel has been a migration from cosmic omniprevalence to heightened contingence for human-like intelligence. Nonetheless, given that people have often assumed the recurrence of humanoids without much genuine evidence, it would be interesting to assess the facts of the matter further, given growing insights from fields like astrobiology, and how they interact with how severe we ought to consider an X-risk and, thus, how much more we should be working to prevent such an event.”
Interview Thomas Moynihan: “The discovery of extinction is a philosophical centrepiece of the modern age”
Hi everyone,
I make a newsletter, called the Anti-Apocalyptus newsletter, that every week links to five articles about topics related to EA, among other things X-risk, disease, great power wars and emerging technologies.
In it I also occasionally interview people who work in these fields, one of which is the below talk I had with Thomas Moynihan from the Future of Humanity Institute and author of X-Risk: How Humanity Discovered Its Own Extinction. Since my previous interview with Tom Chivers generated some positive interest here, I thought this one might be interesting for the forum as well.
In the interview we mainly discuss how the Enlightenment discovered existential risk, where it was previously absent from our thinking. And why this realisation is a cornerstone of modernity.
---
Could you introduce your current work and position, and how you ended up there?
“I have just finished a book on the history of humans thinking about human extinction, entitled X-Risk: How Humanity Discovered its Extinction. The book argues that this is a distinctly modern idea. In other words, prior to modernity, people could not even think about it. The book also argues that the discovery of this idea can in many ways be seen as one of the philosophical centrepieces, or crowning achievements, of what we call the modern epoch. In other words, it is one of the most important conceptual discoveries our species has ever made.
The book stages the drama of its discovery across the past few centuries, and, through this, aims to unveil today’s research into X-risk as part and parcel of a much wider undertaking of human learning. It is aimed at a wider audience: the goal being to introduce a larger demographic the central motivating ideas behind longtermism and existential risk.
At the moment, I am working with Oxford’s Future of Humanity Institute with a grant from Berkeley’s Existential Risk Initiative. Before that, I was doing a PhD at Oriel College, Oxford. My PhD thesis was also on the topic of the history of ideas about human extinction: a topic of research I first started following almost eight years ago, back in 2012.
Though what I do is the ‘history of ideas’, a mixture of history of science, cultural history, and history of philosophy, my own conviction is that historical investigation should have transformative effects on the present: potentially realigning our current priorities and attitudes. I don’t think history is just the collecting and constellating of an inert and value-neutral set of facts or sources, it is the forging of an active argument for actions in the present and thus contains value-laden contestations and claims.”
Why is it important for humanity to look at and combat existential risk? Particularly compared to other cause areas?
“Following on from the above, we humans are historical beings. What I mean by this is that we correct ourselves and ameliorate errors in our beliefs over time. Uniquely, we acknowledge that we can be wrong. Through this we improve our ideas about the world. Moreover, we pass these improvements onto the next generations. It is because of this ‘ratcheting effect’ that we accumulate increasing practical and predictive knowledge over the generations and, thus, come to increasingly transform our material conditions at the same time as incrementally alleviating ignorances and injustices. No other animals have a history in this cumulative sense.
But because we acknowledge we can be wrong, and thus have the potential to be better, we also acknowledge that, as historical beings, we may not know exactly yet what our true potential is, or, our true potential to do good things or create value in the world. This means that there is vast value in ‘keeping history going’: so that we can further learn what is potentially at stake in our existing or not existing. Anything else seems brash, because we don’t know all of the facts of the matter yet, so we can’t really make an informed decision.
To take a case in point, there was once a time when people couldn’t think about human extinction nor the extinction of human values within the wider universe. The prerequisite of all ethical action—which is the existence of ethical agents—was taken as an unquestionable given: i.e. not subject to hazard or risk. Then, after centuries of painstaking inquiry concerning not only the objective world but also our epistemic biases, this prerequisite for all practical action gradually came under scrutiny due to the dawning discovery that ethical agents could, in fact, go extinct—irreversibly so. The existence of ethical action itself came under threat. Ever since, various voices have slowly come to realise that our primary ethical obligation must therefore be to protect the very foothold, within the physical universe, of all otherethical obligations, no matter what they may be: that is, to safeguard the existence of beings like ourselves—ethically recognisant agents, or, agents that can follow ethical argumentation. I like to call this the imperative behind all imperatives. We have progressively realised that there is a reason to our existence because we have progressively become sensitive to the fact that the very existence of reason itself might depend upon it, at least, in our corner of the cosmos.
After people discovered this, all prior ethical outlooks—though often still compelling and compulsory in their own domains of application—were revealed to be somewhat limited or parochial in scope. As with classical mechanics in physics, the elder ethical outlooks, all of which took the existence of ethics itself for granted, were revealed to be regional: often still correct in their own way, but nested within this wider imperative behind all imperatives.
In a certain sense I think we were duty-bound to discover this idea, inasmuch as it a discovery that any agent that submits itself to acting ethically or altruistically must eventually come to acknowledge. The pathways may be different, but, at the very least, there is clear survival value in knowing about the value of survival. But the point here is that there will still be other discoveries and shocks on the road ahead for us, equally seismic and paradigm-shifting in scope: we do not know everything yet…
So, yes, I think ‘keeping history going’, i.e. minimizing and mitigating history-ending disasters, should be our priority not only because of all the things that we do know about this imperative but also because of all of the things which we as yet do not. Nick Bostrom refers to this as the ‘option value’ of civilization, and I think it is a deeply important point.
All other ethical causes focus on the things we know and already acknowledge as requiring alleviation, which is utterly legitimate, but ‘keeping history going’ allows us to discover the causes that we can’t even yet imagine. This gives it a unique position, as an ethical priority, in my estimation.”
Why are you personally interested in something as ‘dark’ as existential risk?
“Because I feel that it is, almost by definition, the most important topic of the modern era. If modernity is the loss of external moral assurances and hierarchies, and is thus also humanity’s assumption of increasing self-responsibility for its own institutions and opinions, then the demand to think about existential risk is an unavoidable threshold in this still-unfolding process. Because, in the discovery of X-risk, is the understanding that our existence matters because independent nature will not necessarily uphold our own values or morals should we cease to exist. It is the ultimate expression of the loss of external assurances, and thus is one of the culminating features of modern history.
Moreover, as the twentieth-century futurologist and palaeontologist Pierre Teilhard de Chardin once sagely said: ‘There are no summits without abysses’. That is to say, we need to focus on the perils to put the promises into relief and vice versa. This is the central lesson of ‘coming of age’, both for each individual but also for civilization itself: one does not assume responsibility without coming to carefully acknowledge the risks one faces.”
You trace the intellectual history of existential risk back to the Enlightenment. How did people think about the end of humanity before that, and what changed then?
“As mentioned, modernity is the loss of external assurances, and the concomitant assumption of self-responsibility for the human race. That is to say, elder worldviews tended to believe that, independently of our actions and beliefs, value was secure and safe in the external universe. Both mythological and theistic worldviews tended to assume that moral law is in some sense woven into the very fabric of the cosmos at large. Indeed, medieval cosmology makes barely any distinction between ethics and physics: moral hierarchy just is the celestial architecture. There was thus nowhere in the cosmos that was not peopled with moral forces or value structures. Even the centre of the earth was thought to be populated, but with devilish disvalue rather than angelic jubilation. So, looking at the world like this, value is secure regardless of what happens to humans or what humans do.
In line with this, Aristotle once said that everything useful, valuable, and knowable has been discovered infinitely many times by the ‘wisdom of the ages’. That means that not only are there are no risks or challenges beyond what has been previously recorded, but also that maximizing value is just returning to some previous maximum and, moreover, that epistemic inquiry is essentially just remembering or rediscovering old stuff in an edifying way. This background assumption, which influenced many other subsequent thinkers, meant that the material conditions of human existence could not be improved upon and neither could they fail.
Moreover, this went hand-in-hand with another belief—commonly called the Principle of Plenitude—which was pretty much omniprevalent up until the Enlightenment. This held that nothing can truly be lost from nature. From Plato to the Stoics, people thought that whatever was destroyed was eventually, inevitably returned in the fullness of time. This applied not only to species but to entire worlds and also to values. Anything valuable that is decimated will one day replenish or return. So, value was indestructible. It was utterly conserved throughout its myriad destructions—so, again, we could do nothing to maximize it in the world nor could we be culpable for not being vigilant by allowing it to be diminished or damaged. So why bother?
When the scientific revolution rolled around, ethics and physics became separated and it was revealed we lived on just one planet in the midst of a vast interstellar void. But people simply continued to assume that value was indestructible. Around Galileo’s time, people were already acknowledging suns and entire planetary systems could be destroyed, but they were totally convinced that this would not lead to any ‘loss’ or ‘damage’ in nature, because everything destroyed would return on another world. In the cosmic infinity, nothing can truly go extinct. People explicitly said that this was why the destruction of the Earth would not matter: because humans and their values will either continue to exist elsewhere or will re-evolve on another planet.
This perpetuated a sense that value was indestructible and that we could do nothing to change the amount of it in the universe. Believing that value is interminably conserved, one could accommodate any catastrophe or disaster, no matter how titanic and calamitous, whilst still stripping it of any genuine moral consequences. It led to a sense of cosmic nonchalance. Philosophers explicitly said that, because the universe is maximally and recurrently populated with humanoid beings, we ought not care about the threats facing Earth.
Around the enlightenment, however, intellectuals started realising that we ought to be careful not to contaminate our objective theories of the world with our moral prejudices and biases. The presumption that independent nature aligns with our ideas of what is dignified or valuable or purposive was shown to be naïve and unfounded. It might seem like a massive waste of space for these other worlds to not be populated with interminable humanoids, but ‘waste’ is a value judgement and we should not let our value judgements distort our objective theories. So, from cosmic nonchalance, people started waking up to a sense of cosmic fragility. Indeed, it was only around the turn of the 1800s that people first started entertaining the idea that most of the cosmos is sterile, inhospitable, and inactive, and not rife with overabundant life and civilized complexity. This reached a head, in the previous century, with Enrico Fermi’s famous paradox regarding the conspicuous absence of extraterrestrial activities.
Note that this has nothing to do with the fact of how populous the galaxy actually turns out to be nor how many other ‘humanoids’ might actually be out there. It has nothing to do with truth statements. It is, instead, entirely to do with the dismantling of the presumption ahead of evidence that the universe is fully populated. Though it may seem strange to us now, this assumption was basically the default until the late Enlightenment. After critiquing the idea, we now realise that, even if we have high credence in other populated planets, this is a matter of probability and not yet one of certainty. So, using it as an excuse to act nonchalantly with our own civilization is revealed as extremely rash. But, as mentioned, this nonchalance was pretty much the only game in town up until the Enlightenment.
In tandem with this, across the seventeenth and eighteenth centuries, a number of new sciences emerged that further highlighted our precariousness. These were fields like probabilism, demography, and geology. For instance, geology revealed our planet was not designed with maximal and perpetual habitability in mind, as had previously been assumed by theists and deists, and that it had, in fact, been the scene of multitudes of prehistoric extinctions and paleontological holocausts. Geology also revealed that complex life was not a permanent feature on our own planet, so why should it be elsewhere in the cosmos? Scientists started tentatively thinking about questions of habitability and inhabitability.
Complex things like life, mind, and civilization started to look more and more contingent and precarious. In lockstep with this, people started thinking of these things as more precious. After centuries of presuming that value was indestructible, scientists and philosophers slowly woke up to the fact that everything that we find valuable could plausibly be destroyed. Finally, people started to think of our extinction, clearly and vocally, as a moral tragedy: tragic because it would represent the permanent and irreversible termination of humanity and its values. By thus coming to acknowledge the moral stakes involved in our extinction, people at last started to genuinely think about it as an objective event that can be predicted and even mitigated. It was learning the moral stakes that motivated people to start assessing it objectively. Indeed, by the 1770s people were already using probability theory to calculate the odds of cometary impact and some were even suggesting mitigation strategies in the form of ballistics and deflection technologies.
But this all remained the parlour speculation of natural philosophers and scientists. It did not seem a pressing issue, it seemed a remote plausibility. Moreover, with the ascendance of evolutionary worldviews across the 1800s, there was a mood of general confidence. Even Darwin himself fell foul to some progressivist presumptions: combined with his ideas of gradualist and continuous change, early evolutionary theory implied that a species as adaptive and successful as ours would not die off without a lineage, but would merely be replaced by its more adaptive and thus more ‘progressed’ successors. Darwin himself was confident that this postpartum ‘progress’ would apply to moral, as well as physical, affairs So, the myth of inexorable progress in the natural world—inevitably taking place independently of our actions and decisions—again created a sense of existential nonchalance. This was prevalent during the Victorian era. People ultimately became more concerned with our species degenerating into something less worthy than it dying out entirely and without recompense or lineage.
Around mid-century, thermodynamics was formulated and heat death theorised, but this again remained far-off and distant: not a pressing issue. At the turn of the 1900s, scientists began acknowledging cosmic disasters and global risks, but they continued to think that they were so improbable as to not be an object of massive concern. They pointed to how long life had persisted on Earth without such a biosphere-ending disaster, awareness of observer selection effects was not yet on the menu. After World War 1, the idea of inexorable progress took a battering and many intellectuals and politicians became concerned that our species may one day wipe out itself, whether with engineered pandemic or mechanized warfare. But, again, this remained remote: there was, as yet, no tangible technology that would actuate this fate. Then, in World War 2, the H-bomb was developed. There were fears that it would ignite the atmosphere and, later, there was talk of developing a ‘cobalt bomb’ that could fatally irradiate the biosphere. For the first time, extinction became a policy issue. Scientists turned their attention to it as a serious topic. The myth of inevitable progress, and the existential nonchalance it perpetuated, took a further battering. Moreover, the nuke changed how we relate to technology in a cosmic setting: it seemed less an inevitability because of its preeminent adaptiveness, but began to look more like a potential fluke that is prone to destroy itself should it not be vigilant. Moreover, the inflammation of Fermi’s Paradox, ever since the 1950s, and the continuing lack of evidence for other advanced civilizations in SETI searches has only exasperated such issues. Ever since the Cold War, there has been a building scientific and policy concern in such issues, building into the millennium with the articulation of a rigorous analytic framework to think about ‘existential risk’ as promoted by Nick Bostrom and others.
So, I like to think about this history as having four phases. First is the phase of ‘indestructible value’, from prehistory to the 1600s. This was when people couldn’t even think of a possible world without human minds. Then there is a phase of ‘cosmic nonchalance’ from the 1600s to the 1800s. This was when people thought about planet-destroying catastrophes, but believed that humans would reappear elsewhere or elsewhen in the cosmos. Then, there is a phase of growing ‘cosmic loneliness’ from the 1800s to 1950. This saw growing awareness that the universe is not, by default, maximally habitable or fully inhabited. Then, after 1950 to the present day, is a phase of emerging awareness of ‘astronomical value’: that is, an building sense that, finding ourselves within a corner of the cosmos that appears sterile, unpopulated, and undeveloped, there is therefore a potential for us to make an genuinely significant difference on the amount of value in the universe by surviving X-risks and expanding outwards. These are the landmark phases in thinking about the existential fate of humanity across the ages.”
What’s the relation between religious notions of apocalypse, and modern-day ideas about existential risk?
“To be blunt, none. A lot of people believe that there is nothing ever truly new discovered and that all knowledge is just a rediscovery of the wisdom of the ancients. They would be in the illustrious company of Aristotle, but this opinion is totally wrong. You can find ideas about the end of the world across all times and places, but these are all nested within cycles of return and reconstitution or within a framework of divine justice and final judgement. Therefore, they have nothing to do with extinction as the permanent and irreversible loss of our species and its activities.
I like to talk about these things as ‘false friends’ in the history of ideas. I borrow the term from linguistics, where it refers to ideas that misleadingly sound similar but mean entirely different things in different languages. For instance, ‘das Gift’ in German does not mean a nice present, it means poison.
A similar thing applies to ancient notions of apocalypse and modern ideas of human extinction. They are false friends. They may look similar with their melodrama and pyrotechny, but these are just aesthetic accoutrements, and on a deeper level the notions are actually totally conceptually incompatible.
Old notions of apocalypse conceptualize the end of the world as the revelation of the ultimate meaningfulness of time. Even if it is divine and inscrutable, the moral lesson of cosmic chronology is revealed. Judgement day means that we can rest assured knowing that moral law will get the last word in the universe. Extinction, contrarily, anticipates the end of meaningfulness within time. It acknowledges that moral action can be terminated and the physical world continues, for aimless aeons, without it.
I like to sum it up like this: where apocalypse secures a sense of an ending, extinction anticipates the ending of sense.
This goes back to what I was talking about above. Apocalypse and judgement day allows a certain sense of nonchalance, because regardless of what we do, moral law is the structuring principle of the cosmos. It’s the same for the ancient cyclical cosmologies too: there may be times when there are no humans, but humans will always return no matter what, so why worry? Why fret? Why bother correcting your forecasts with more accurate, predictive ones?
At the entrance of the modern world lay the discovery of the stakes in our ignorance. It was learning that our ignorance may have genuinely disastrous consequences that first motivated early modern societies to correct their opinions about the world, rather than relying on the wisdom of tradition, in order to reach more predictive knowledge. The problem with apocalyptic prophecies and religious revelation is that they do not acknowledge that they can be wrong. Again, this is why there is a vast difference between ‘forecast’ and ‘prophecy’ as there is between ‘extinction’ and ‘apocalypse’.
Again, this is why the Enlightenment was important. The great Enlightenment philosophers liked to define enlightening as humanity’s emergence from immaturity. They thought of it as our ‘coming of age’ as a species. Of course, there were problems with this: the universalism espoused by these people was often only extended to certain demographics and this led to tragic and criminal consequences and complicities. However, the basic ideal remains sound: ‘assuming maturity’ as a civilization and a species just means asserting more control over our means of existence, thus eliminating need, privation, and suffering, and undertaking more accountability over our institutions and opinions, thus eradicating injustice and ignorance. Just because prior espousers of this programme have previously failed to live up to their own ideals is no indictment against the ideals they (fallibly) pursued. It is the same with our own fallibility. To think otherwise would be to reduce all the promise of humankind to the scale of its former errancies. In my opinion, this is not only smallminded but small of heart.
Moreover, the ideal of ‘coming of age’ is not dead. It has just moved elsewhere. Carl Sagan and others used to talk about ours as an age of ‘technological adolescence’, where we have the power to kill ourselves but not the foresight or resilience to prevent this outcome. Nick Bostrom continues to talk about ‘technological maturity’ as a state of maximal insulation against risk coupled with the achievement of the greatest energetic capture and economic productivity that is physically feasible.
Talking in lofty terms about diffuse cultural attitudes that span centuries, we can think of existential nonchalance as a sign of minority, or immaturity, and the shift toward growing concern for existential risk as a move toward majority, or maturity. Apocalypse is a sign of minority, and coming to understand extinction—and its unique moral stakes—is a requirement of ‘coming of age’.
Apocalypse coddles us with certainty, but extinction calls us to action, but it is precisely because of this that it is the path to civilizational maturity: because what could be more ethically responsible than coming to care for the very existence of ethics and responsibility itself? What could be more altruistic than coming to safeguard the very anchorage of altruistic action in an otherwise indifferent universe? As historical beings, we no doubt have a lot left to learn, a lot of maturing left to do, but at least we now acknowledge this.
This, again, is why the discovery of extinction is, in many ways, underappreciated as a philosophical centrepiece of the modern age. It is a total break with prior traditions of apocalypse, revelation, judgement day, and end times.”
How can a history of the idea help us think about existential risk today?
“It can provide us with a novel argument as to why this topic is so essential and compelling. Because it demonstrates that the topic emerges from the wider sweep of human history, that it is not just some contingent ‘trend’ or ‘fad’ of the present moment, but part of a building and historically consistent and continuous process: it is part of that process of ‘coming of age’ as a global community, whatever that may mean, and regardless of how fallible we have been in the past concerning this ideal, and regardless of how fallible and wayward we will continue to be in relation to such an goal.
Assuming responsibility for our species very obviously remains an unfinished process, and there is much left to learn, but the history undeniably shows that thinking about X-risk emerges from this undertaking as it spreads across history, in all its waywardness and across all its setbacks and detours. Looking at the history, moreover, shows how uniquely important it is as a culmination of centuries of inquiry, spanning multiple continents and cultures.
Today many people interested in X-risk rightly care about it because of the weight of the future consequences of such a disaster, and this is utterly convincing and compelling, but we ought not take for granted all the hard work that got us here in the first place. Recounting the history shows how central these ideas are to the founding ideas of modernity itself. It is something that people care about now because of the immense consequences, but it was also something that people were historically bound to discover ever since humans first defined themselves as creatures that attempt to assume ever greater accountability for the consequence of their actions.
More tangibly, I believe that this history dispels some of the suspicions that this is just the newest face of perennial doom-mongering. Only by retracing the history, of the emergence of extinction as separate to religious notions, do we truly disambiguate apocalypse and X-risk. Elsewhere, it is a nice object lesson in how biases have afflicted inquiry into existential matters. In geology and cosmology, reigning faith in uniformitarianism and gradualism long obstructed acknowledgement of the existence of abrupt and sudden catastrophes. The belief was that the cosmos was a steady and balanced system. This was revealed to be less about actual evidence, and more about an, almost aesthetic, predilection for parsimony and equilibrium. In other quarters, such as biology, the belief in progressivism long led people to believe that a species as adaptive as humanity will only be replaced by its better descendants. This assumption, whose prevalence had more to do with moral bias than objective evidence, occluded the true severity of the plausible extinction of our species. As already mentioned, for many Victorian era biologists, it seemed that our extinction might even be the price of progress. The same applies to the widely-held conviction that something like Homo sapiens is the inevitable outcome of ‘replaying the tape’ on evolution, on other planets or after some global catastrophe here on Earth. This conviction only really began to come under scrutiny in the mid-1900s. Its prior prevalence was, again, to do with our tendency to prefer flattering theories, where human-like intelligence is inevitable. The point is this: the direction of travel has been that, as we progressively overcome our pre-theoretical biases, we have come to realise, more and more, how precious and precarious we may potentially be and, thus, how cautious it is sensible for us to be. I believe that this ‘direction of travel’ is not going to slow, or let up, any time soon: thus, understanding its historical precedent is a useful lesson for the present. We ought to continue to be wary about the ways that our inherited biases distort our objective theories about the risks and obstruct our appreciation of the true moral stakes of our actions and decisions.
Another insight from history is that of the progress we have made, but how contingent and fragile it remains. It is easy to forget how far we have come in our knowledge of the world, but it is also easy to become complacent that continued progress is inevitable. There are naïve optimists and naïve pessimists, but it is increasingly undeniable that neither good nor bad outcomes are independently guaranteed: it is up to us to produce the fate that we want.”
What do you think are the most urgent existential risks humanity should focus on?
“Urgency is ambiguous: does it mean most probable or most assuredly fatal? Ever since the opening of the 1900s people have rightly realised that anthropogenic risks are the true ones to fear in terms of omni-fatality. After World War 1’s mechanized conflagration, many intellectuals and politicians made noises about universal species suicide, but this still seemed far-off and remote. Then the H-bomb was produced, and suddenly it became a policy issue. Afterward, yet more technological risks have emerged: nanotech, synthetic biology or AI. This recent paper assesses the background risks from nature to Homo sapiens, taking into account selection effects, and backs up this view: the most urgent existential risks are anthropogenic and technological ones.
Additionally, I think an area that possibly requires more thought is systemic risks. There are plenty of diffuse issues, exploits, and flaws, in our institutions, our technologies, and our behaviours, that are not X-risks in their own right but that could nonetheless massively compound the likelihood of an existential catastrophe: think not only of how our actions increase the likelihood of zoonosis and world-perturbing pandemics, but also of things like spread of misinformation and political instability. I know some very interesting work is being done on the latter.
It is not an existential risk, but I think more research should be done on the topic of how likely recovery is after a civilizational collapse. How much infrastructure and knowledge needs to be destroyed for this recovery to become improbable? This would be useful to gauge. More dramatically, what is the likelihood of some other species ‘picking up where we left off’ should we go extinct. Historically speaking, this has been the go-to method of deescalating the severity of our extinction. It has proven a very attractive way of placating ourselves in the face of such a tragedy. Throughout the ages, people would often say ‘ah, yes, but some other species that is left behind will evolve a big brain and take our place’. Monkeys have often been elected for the task, think of the Planet of the Apes, but some people have even imagined weasels or dogs or lizards. Statements of inevitably re-evolving intelligence extend from Denis Diderot to Edwin Hubble. Statements in the opposite direction begin around 1800, but only really start to take hold with George Gaylord Simpson’s 1964 paper ‘The Nonprevalence of Humanoids’, and then reach a head with Stephen Jay Gould’s 1989 declaration that ‘Homo sapiens is an entity, not a tendency’. Again, historically speaking, the direction of travel has been a migration from cosmic omniprevalence to heightened contingence for human-like intelligence. Nonetheless, given that people have often assumed the recurrence of humanoids without much genuine evidence, it would be interesting to assess the facts of the matter further, given growing insights from fields like astrobiology, and how they interact with how severe we ought to consider an X-risk and, thus, how much more we should be working to prevent such an event.”