I don’t think longtermism necessarily needs new priorities to be valuable if it offers a better perspective on existing ones (although I don’t think it does this well either).
Understanding what the far future might need is very difficult. If you’d asked someone 1000 years what they should focus on to benefit us, you’d get answers largely irrelevant to our needs today.[1] If you asked someone a little over 100 years ago their ideas might seem more intelligible and one guy was even perceptive enough to imagine nuclear weapons, although his optimism about what became known as mutually assured destruction setting the world free looks very wrong now, and people 100 years ago that did boring things focused on the current world did more for us than people dreaming of post-work utopias.
To that extent, the focus on x-risk seems quite reasonable: still existing is something we actually can reasonably believe will be valued by humans in a million years time[2] Of course, there are also over 8 billion reasons to try to avoid human extinction alive today (and most non-longtermists consider at least as far as their children) but longtermism makes arguments for it being more important than we think. This logically leads to willingness to allocate more money to x-risk causes, and consider more unconventional and highly unlikely approaches x-risk. This is a consideration, but in practice I’m not sure that it leads to better outcomes: some of the approaches to x-risk seeking funding make directionally different assumptions about whether more or less AGI is crucial to survival: they can’t both be right and the ‘very long shot’ proposals that only start to make sense if we introduce fantastically large numbers of humans to the benefit side of the equation look suspiciously like Pascal’s muggings.[3]
Plus people making longtermist arguments typically seem to attach fairly high probabilities to stuff like AGI that they’re working on in their own estimations, which if true would make their work entirely justifiable even focusing only on humans living today.
(A moot point but I’d have also thought that although the word ‘longtermist’ wasn’t coined until much later, Bostrom and to a lesser extent Parfit fit in with the description of longtermist philosophy. Of course they also weren’t the first people to write about x-risk)
Notwithstanding fringe possibilities like the possibility humans in a million years might be better off not existing, or for impartial total utilitarians humanity might be displacing something capable of experiencing much higher aggregate welfare.
Not just superficially in that someone is asking to suspend scepticism by invoking huge reward, but also that the huge rewards themselves make sense only if you believe in very specific claims about x-risk over the long term future being highly concentrated in the present (very large numbers of future humans in expectation or x-risk being nontrivial for any extended period of time might seem superficially uncontroversial possibilities but they’re actually strongly in conflict with each other).
One of the more excellent comments I’ve ever read on the EA Forum. Perceptive and nimbly expressed. Thank you.
people 100 years ago that did boring things focused on the current world did more for us than people dreaming of post-work utopias.
Very well said!
To that extent, the focus on x-risk seems quite reasonable: still existing is something we actually can reasonably believe will be valued by humans in a million years time
I totally agree. To be clear, I support mitigation of existential risks, global catastrophic risks, and all sorts of low-probability, high-impact risks, including those on the scale of another pandemic like covid-19 or large volcanic eruptions. I love NASA’s NEO Surveyor.
I think, though, we may need to draw a line between acts of nature — asteroids, volcanoes, and natural pandemics — and acts of humankind — nuclear war, bioterror, etc.
The difference between nature and humankind is that nature does not respond to what we do. Asteroids don’t try to foul our defenses. In a sense, viruses “try” to beat our vaccines and so on, but that’s already baked-in to our idea what viruses have been for a long time, and it isn’t the same thing as what humans do when we’re in an adversarial relationship with them.
I certainly think we should still try our absolute best to protect humanity against acts of humankind like nuclear war and bioterror. But it’s much harder, if not outright impossible, to get good statistical evidence for the probability of events that depend on what humans decide to do, using all their intelligence and creativity, as opposed to a natural phenomenon like an asteroid or a virus (or a volcano). We might need to draw a line between nature and humankind and say that rigorous, cost-effectiveness estimates on the other side of the line may not be possible, and at the very least are much more speculative and uncertain.
I don’t think that’s an argument against doing a lot about them, but it’s an important point nonetheless.
With AI, the uncertainty that exists with nuclear war and bioterror is cranked up to 11. We’re talking about fundamentally new technologies based on, most likely, new science yet to be discovered, and even new theoretical concepts in the science yet to be developed, if not an outright new theoretical paradigm. This is quite different from bombs that already exist and have been exploded before. With bioterror, we already know natural viruses are quite dangerous (e.g. just releasing natural smallpox could be bad), and I believe there have been proofs-of-concept of techniques bioterrorists could use. So, it’s more speculative, but not all that speculative.
Imagine this idea: someday in the hazy future, we invent the first AGI. Little do we know, this AGI is perfectly “friendly”, aligned, safe, benevolent, wise, nonviolent, and so on. It will be a wonderful ally to humanity, like Data from Star Trek or Samantha from Her. Until… we decide to apply the alignment techniques we’ve been developing to it. Oh no, what a mistake! Our alignment techniques actually do the opposite of what we wanted, and turn a friendly, aligned, safe, benevolent AI into an unfriendly, misaligned, unsafe, rogue, dangerous AI. We caused the very disaster we were trying to prevent!
How likely is such a scenario? There’s no way to know. We simply have no idea, and we have no way of finding out.
This helps illustrate, I hope, one of the problems (out of multiple distinct problems) with precautionary arguments about AGI, particularly if back-of-the-envelope cost-effectiveness calculations are used to justify spending on precautionary research. There is no completely agnostic way to reduce risk. You have to make certain technical and scientific assumptions to justify funding AI alignment research. And how well-thought-out, or well-studied, or well-scrutinized are those assumptions?
I don’t think longtermism necessarily needs new priorities to be valuable if it offers a better perspective on existing ones (although I don’t think it does this well either).
Understanding what the far future might need is very difficult. If you’d asked someone 1000 years what they should focus on to benefit us, you’d get answers largely irrelevant to our needs today.[1] If you asked someone a little over 100 years ago their ideas might seem more intelligible and one guy was even perceptive enough to imagine nuclear weapons, although his optimism about what became known as mutually assured destruction setting the world free looks very wrong now, and people 100 years ago that did boring things focused on the current world did more for us than people dreaming of post-work utopias.
To that extent, the focus on x-risk seems quite reasonable: still existing is something we actually can reasonably believe will be valued by humans in a million years time[2] Of course, there are also over 8 billion reasons to try to avoid human extinction alive today (and most non-longtermists consider at least as far as their children) but longtermism makes arguments for it being more important than we think. This logically leads to willingness to allocate more money to x-risk causes, and consider more unconventional and highly unlikely approaches x-risk. This is a consideration, but in practice I’m not sure that it leads to better outcomes: some of the approaches to x-risk seeking funding make directionally different assumptions about whether more or less AGI is crucial to survival: they can’t both be right and the ‘very long shot’ proposals that only start to make sense if we introduce fantastically large numbers of humans to the benefit side of the equation look suspiciously like Pascal’s muggings.[3]
Plus people making longtermist arguments typically seem to attach fairly high probabilities to stuff like AGI that they’re working on in their own estimations, which if true would make their work entirely justifiable even focusing only on humans living today.
(A moot point but I’d have also thought that although the word ‘longtermist’ wasn’t coined until much later, Bostrom and to a lesser extent Parfit fit in with the description of longtermist philosophy. Of course they also weren’t the first people to write about x-risk)
I suspect the main answers would be to do with religious prophecies or strengthening their no-longer-extant empire/state
Notwithstanding fringe possibilities like the possibility humans in a million years might be better off not existing, or for impartial total utilitarians humanity might be displacing something capable of experiencing much higher aggregate welfare.
Not just superficially in that someone is asking to suspend scepticism by invoking huge reward, but also that the huge rewards themselves make sense only if you believe in very specific claims about x-risk over the long term future being highly concentrated in the present (very large numbers of future humans in expectation or x-risk being nontrivial for any extended period of time might seem superficially uncontroversial possibilities but they’re actually strongly in conflict with each other).
One of the more excellent comments I’ve ever read on the EA Forum. Perceptive and nimbly expressed. Thank you.
Very well said!
I totally agree. To be clear, I support mitigation of existential risks, global catastrophic risks, and all sorts of low-probability, high-impact risks, including those on the scale of another pandemic like covid-19 or large volcanic eruptions. I love NASA’s NEO Surveyor.
I think, though, we may need to draw a line between acts of nature — asteroids, volcanoes, and natural pandemics — and acts of humankind — nuclear war, bioterror, etc.
The difference between nature and humankind is that nature does not respond to what we do. Asteroids don’t try to foul our defenses. In a sense, viruses “try” to beat our vaccines and so on, but that’s already baked-in to our idea what viruses have been for a long time, and it isn’t the same thing as what humans do when we’re in an adversarial relationship with them.
I certainly think we should still try our absolute best to protect humanity against acts of humankind like nuclear war and bioterror. But it’s much harder, if not outright impossible, to get good statistical evidence for the probability of events that depend on what humans decide to do, using all their intelligence and creativity, as opposed to a natural phenomenon like an asteroid or a virus (or a volcano). We might need to draw a line between nature and humankind and say that rigorous, cost-effectiveness estimates on the other side of the line may not be possible, and at the very least are much more speculative and uncertain.
I don’t think that’s an argument against doing a lot about them, but it’s an important point nonetheless.
With AI, the uncertainty that exists with nuclear war and bioterror is cranked up to 11. We’re talking about fundamentally new technologies based on, most likely, new science yet to be discovered, and even new theoretical concepts in the science yet to be developed, if not an outright new theoretical paradigm. This is quite different from bombs that already exist and have been exploded before. With bioterror, we already know natural viruses are quite dangerous (e.g. just releasing natural smallpox could be bad), and I believe there have been proofs-of-concept of techniques bioterrorists could use. So, it’s more speculative, but not all that speculative.
Imagine this idea: someday in the hazy future, we invent the first AGI. Little do we know, this AGI is perfectly “friendly”, aligned, safe, benevolent, wise, nonviolent, and so on. It will be a wonderful ally to humanity, like Data from Star Trek or Samantha from Her. Until… we decide to apply the alignment techniques we’ve been developing to it. Oh no, what a mistake! Our alignment techniques actually do the opposite of what we wanted, and turn a friendly, aligned, safe, benevolent AI into an unfriendly, misaligned, unsafe, rogue, dangerous AI. We caused the very disaster we were trying to prevent!
How likely is such a scenario? There’s no way to know. We simply have no idea, and we have no way of finding out.
This helps illustrate, I hope, one of the problems (out of multiple distinct problems) with precautionary arguments about AGI, particularly if back-of-the-envelope cost-effectiveness calculations are used to justify spending on precautionary research. There is no completely agnostic way to reduce risk. You have to make certain technical and scientific assumptions to justify funding AI alignment research. And how well-thought-out, or well-studied, or well-scrutinized are those assumptions?