I think Transformative AI is unusually powerful and dangerous relative to other things that can plausibly kill us or otherwise drastically affect human trajectories, and many of us believe AI doom is not inevitable.
I think it’s probably correct for EAs to focus on AI more than other things.
Other plausible contenders (some of which I’ve worked on) include global priorities research, biorisk mitigation, and moral circle expansion. But broadly a) I think they’re less important or tractable than AI, b) many of them are entangled with AI (e.g. global priorities research that ignores AI is completely missing the most important thing).
I largely agree with Linch’s answer (primarily: that AI is really likely very dangerous), and want to point out a couple of relevant resources in case a reader is less familiar with some foundations for these claims:
[I’ll be assuming a consequentialist moral framework in this response, since most EAs are in fact consequentialists. I’m sure other moral systems have their own arguments for (de)prioritizing AI.]
Almost all the disputes on prioritizing AI safety are really epistemological, rather than ethical; the two big exceptions being a disagreement about how to value future persons, and one on ethics with very high numbers of people (Pascal’s Mugging-adjacent situations).
I’ll use the importance-tractability-neglectedness (ITN) framework to explain what I mean. The ITN framework is meant to figure out whether an extra marginal dollar to cause 1 will have more positive impact than a dollar to cause 2; in any consequentialist ethical system, that’s enough reason to prefer cause 1. Importance is the scale of the problem, the (negative) expected value in the counterfactual world where nothing is done about it—I’ll note this as CEV, for counterfactual expected value. Tractability is the share of the problem which can be solved with a given amount of resources; percent-solved-per-dollar, which I’ll note as %/$. Neglectedness is comparing cause 1 to other causes with similar importance-times-tractability, and seeing which cause currently has more funding. In an equation:
CEV1∗(%/$)1>CEV2∗(%/$)2?
Now let’s do ITN for AI risk specifically:
Tractability—This is entirely an epistemological issue, and one which changes the result of any calculations done by a lot. If AI safety is 15% more likely to be solved with a billion more dollars to hire more grad students (or maybe Terence Tao), few people who are really worried about AI risk would object to throwing that amount of money at it. But there are other models under which throwing an extra billion dollars at the problem would barely increase AI safety progress at all, and many are skeptical of using vast amounts of money which could otherwise help alleviate global poverty on an issue with so much uncertainty.
Neglectedness—Just about everyone agrees that if AI safety is indeed as important and tractable as safety advocates say, it currently gets less resources than other issues on the same or smaller scales, like climate change and nuclear war prevention.
Importance—Essentially, importance is probability-of-doom [p(doom)] multiplied by how-bad-doom-actually-is [v(doom)], which gives us expected-(negative)-value [CEV] in the counterfactual universe where we don’t do anything about AI risk.
p(doom)∗v(doom)=CEV
The obvious first issue here is an epistemological one; what is p(doom)? 10% chance of everyone dying is a lot different from 1%, which is in turn very different from 0.1%. And some people think p(doom) is over 50%, or almost nonexistent! All of these numbers have very different implications regarding how much money we should put into AI safety.
Alright, let’s briefly take a step back before looking at how to calculate v(doom). Our equation now looks like this:
p(doom)∗v(doom)∗(%/$)1>CEV2∗(%/$)2?
Assuming that the right side of the equation is constant, we now have 3 variables that can move around: p(doom),(%/$)1, and v(doom). I’ve shown that the first two have a lot of variability, which can lead to multiple orders of magnitude difference in the results.
The ‘longtermist’ argument for AI risk is, plainly, that v(doom) is so unbelievably large that the variations in p(doom) and (%/$)1 are too small to matter. This is based on an epistemological claim and two ethical claims.
Epistemological claim: the expected amount of people (/sentient beings) in the future is huge. An OWID article estimates it at between 800 trillion and 625 quadrillion given a stable population of 11 billion on Earth, while some longtermists, assuming things like space colonization and uploaded minds, go up to 10^53 or something like that. This is the Astronomical Value Thesis.
This claim, at its core, is based on an expectation that existential risk will effectively cease to exist soon (or at least drop to very very low levels), because of something like singularity-level technology. If x-risk stays at something like 1% per century, after all, it’s very unlikely that we ever reach anything like 800 trillion people, let alone some of the higher numbers. This EA Forum post does a great job of explaining the math behind it.
Moral claim 1: We should care about potential future sentient beings; 800 trillion humans existing is 100,000 times better than 8 billion, and the loss of 800 trillion future potential lives should be counted as 100,000 times as bad as the loss of today’s 8 billion lives. This is a very non-intuitive moral claim, but many total utilitarians will agree with it.
If we combine the Astronomical Value Thesis with moral claim 1, we get to the conclusion that v(doom) is so massive that it overwhelms nearly everything else in the equation. To illustrate, I’ll use the lowball estimate of 800 trillion lives:
You don’t need advanced math to know that the side with that many zeroes is probably larger. But valid math is not always valid philosophy, and it has often been said that ethics gets weird around very large numbers. Some people say that this is in fact invalid reasoning, and that it resembles the case of Pascal’s mugging, which infamously ‘proves’ things like that you should exchange 10$ for a one-in-a-quadrillion chance of getting 50 quadrillion dollars (after all, the expected value is $50).
So, to finish, moral claim 2: at least in this case, reasoning like this with very large numbers is ethically valid.
And there you have it! If you accept the Astronomical Value Thesis and both moral claims, just about any spending which decreases x-risk at all will be worth prioritizing. If you reject any of those three claims, it can still be entirely reasonable to prioritize AI risk, if your p(doom) and tractability estimates are high enough. Plugging in the current 8 billion people on the planet:
Thank you for asking this! Some fascinating replies!
A related question:
Considering other existential risks like engineered pandemics, etc., is there an ethical case for continuing to escalate the advancement of AI development despite the possibly-pressing risk of unaligned AGI for addressing/mitigating other risks, such as developing better vaccines, increasing the rate of progress in climate technology research, etc.?
Reasonable people think it has the most chance of killing all of us and ending future conscious life. Compared to other risks it is bigger, compared to other cause areas it will extinguish more lives.
“Reasonable people think”—this sounds like a very weak way to start an argument. Who are those people—would be the next question. So let’s skip the differing to authority argument. Then we have “the most chance”—what are the probabilities and how soon in the future? Cause when we talk about deprioritizing other cause areas for the next X years, we need to have pretty good probabilities and timelines, right? So yeah, I would not consider differing to authorities a strong argument. But thanks for taking the time to reply.
A survey of ML researchers (not necessarily AI Safety, or EA) gave the following
That seems much higher than the corresponding sample in any other field I can think of.
I think that an “extremely bad outcome” is probably equivalent to 1Bn or more people dying.
Do a near majority of those who work in green technology (what feels like the right comparison class) feel that climate change has a 10% chance of 1 Bn deaths?
Personally, I think there is like a 7% chance of extinction before 2050, which is waaay higher than anything else.
FYI—subsamples of that survey were asked about this in other ways, which gave some evidence that “extremely bad outcome” was ~equivalent to extinction.
Explicit P(doom)=5-10% The levels of badness involved in that last question seemed ambiguous in retrospect, so I added two new questions about human extinction explicitly. The median respondent’s probability of x-risk from humans failing to control AI [1]was 10%, weirdly more than median chance of human extinction from AI in general,[2] at 5%. This might just be because different people got these questions and the median is quite near the divide between 5% and 10%. The most interesting thing here is probably that these are both very high—it seems the ‘extremely bad outcome’ numbers in the old question were not just catastrophizing merely disastrous AI outcomes.
Or, ‘human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species’
What is the strongest ethical argument you know for prioritizing AI over other cause areas?
I’d also be very interested in the reverse of this. Is there anyone who has thought very hard about AI risk and decided to de-prioritise it?
I think Transformative AI is unusually powerful and dangerous relative to other things that can plausibly kill us or otherwise drastically affect human trajectories, and many of us believe AI doom is not inevitable.
I think it’s probably correct for EAs to focus on AI more than other things.
Other plausible contenders (some of which I’ve worked on) include global priorities research, biorisk mitigation, and moral circle expansion. But broadly a) I think they’re less important or tractable than AI, b) many of them are entangled with AI (e.g. global priorities research that ignores AI is completely missing the most important thing).
I largely agree with Linch’s answer (primarily: that AI is really likely very dangerous), and want to point out a couple of relevant resources in case a reader is less familiar with some foundations for these claims:
The 80,000 Hours problem profile for AI is pretty good, and has lots of other useful links
This post is also really helpful, I think: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
More broadly, you can explore a lot of discussion on the AI risk topic page in the EA Forum Wiki
[I’ll be assuming a consequentialist moral framework in this response, since most EAs are in fact consequentialists. I’m sure other moral systems have their own arguments for (de)prioritizing AI.]
Almost all the disputes on prioritizing AI safety are really epistemological, rather than ethical; the two big exceptions being a disagreement about how to value future persons, and one on ethics with very high numbers of people (Pascal’s Mugging-adjacent situations).
I’ll use the importance-tractability-neglectedness (ITN) framework to explain what I mean. The ITN framework is meant to figure out whether an extra marginal dollar to cause 1 will have more positive impact than a dollar to cause 2; in any consequentialist ethical system, that’s enough reason to prefer cause 1. Importance is the scale of the problem, the (negative) expected value in the counterfactual world where nothing is done about it—I’ll note this as CEV, for counterfactual expected value. Tractability is the share of the problem which can be solved with a given amount of resources; percent-solved-per-dollar, which I’ll note as %/$. Neglectedness is comparing cause 1 to other causes with similar importance-times-tractability, and seeing which cause currently has more funding. In an equation:
CEV1∗(%/$)1>CEV2∗(%/$)2?
Now let’s do ITN for AI risk specifically:
Tractability—This is entirely an epistemological issue, and one which changes the result of any calculations done by a lot. If AI safety is 15% more likely to be solved with a billion more dollars to hire more grad students (or maybe Terence Tao), few people who are really worried about AI risk would object to throwing that amount of money at it. But there are other models under which throwing an extra billion dollars at the problem would barely increase AI safety progress at all, and many are skeptical of using vast amounts of money which could otherwise help alleviate global poverty on an issue with so much uncertainty.
Neglectedness—Just about everyone agrees that if AI safety is indeed as important and tractable as safety advocates say, it currently gets less resources than other issues on the same or smaller scales, like climate change and nuclear war prevention.
Importance—Essentially, importance is probability-of-doom [p(doom)] multiplied by how-bad-doom-actually-is [v(doom)], which gives us expected-(negative)-value [CEV] in the counterfactual universe where we don’t do anything about AI risk.
p(doom)∗v(doom)=CEV
The obvious first issue here is an epistemological one; what is p(doom)? 10% chance of everyone dying is a lot different from 1%, which is in turn very different from 0.1%. And some people think p(doom) is over 50%, or almost nonexistent! All of these numbers have very different implications regarding how much money we should put into AI safety.
Alright, let’s briefly take a step back before looking at how to calculate v(doom). Our equation now looks like this:
p(doom)∗v(doom)∗(%/$)1>CEV2∗(%/$)2?
Assuming that the right side of the equation is constant, we now have 3 variables that can move around: p(doom),(%/$)1, and v(doom). I’ve shown that the first two have a lot of variability, which can lead to multiple orders of magnitude difference in the results.
The ‘longtermist’ argument for AI risk is, plainly, that v(doom) is so unbelievably large that the variations in p(doom) and (%/$)1 are too small to matter. This is based on an epistemological claim and two ethical claims.
Epistemological claim: the expected amount of people (/sentient beings) in the future is huge. An OWID article estimates it at between 800 trillion and 625 quadrillion given a stable population of 11 billion on Earth, while some longtermists, assuming things like space colonization and uploaded minds, go up to 10^53 or something like that. This is the Astronomical Value Thesis.
This claim, at its core, is based on an expectation that existential risk will effectively cease to exist soon (or at least drop to very very low levels), because of something like singularity-level technology. If x-risk stays at something like 1% per century, after all, it’s very unlikely that we ever reach anything like 800 trillion people, let alone some of the higher numbers. This EA Forum post does a great job of explaining the math behind it.
Moral claim 1: We should care about potential future sentient beings; 800 trillion humans existing is 100,000 times better than 8 billion, and the loss of 800 trillion future potential lives should be counted as 100,000 times as bad as the loss of today’s 8 billion lives. This is a very non-intuitive moral claim, but many total utilitarians will agree with it.
If we combine the Astronomical Value Thesis with moral claim 1, we get to the conclusion that v(doom) is so massive that it overwhelms nearly everything else in the equation. To illustrate, I’ll use the lowball estimate of 800 trillion lives:
p(doom)∗800,000,000,000,000 lives∗(%/$)1>CEV2∗(%/$)2?
You don’t need advanced math to know that the side with that many zeroes is probably larger. But valid math is not always valid philosophy, and it has often been said that ethics gets weird around very large numbers. Some people say that this is in fact invalid reasoning, and that it resembles the case of Pascal’s mugging, which infamously ‘proves’ things like that you should exchange 10$ for a one-in-a-quadrillion chance of getting 50 quadrillion dollars (after all, the expected value is $50).
So, to finish, moral claim 2: at least in this case, reasoning like this with very large numbers is ethically valid.
And there you have it! If you accept the Astronomical Value Thesis and both moral claims, just about any spending which decreases x-risk at all will be worth prioritizing. If you reject any of those three claims, it can still be entirely reasonable to prioritize AI risk, if your p(doom) and tractability estimates are high enough. Plugging in the current 8 billion people on the planet:
p(doom)∗8,000,000,000 lives∗(%/$)1>CEV2∗(%/$)2?
That’s still a lot of zeroes!
Thank you for asking this! Some fascinating replies!
A related question:
Considering other existential risks like engineered pandemics, etc., is there an ethical case for continuing to escalate the advancement of AI development despite the possibly-pressing risk of unaligned AGI for addressing/mitigating other risks, such as developing better vaccines, increasing the rate of progress in climate technology research, etc.?
Reasonable people think it has the most chance of killing all of us and ending future conscious life. Compared to other risks it is bigger, compared to other cause areas it will extinguish more lives.
“Reasonable people think”—this sounds like a very weak way to start an argument. Who are those people—would be the next question. So let’s skip the differing to authority argument. Then we have “the most chance”—what are the probabilities and how soon in the future? Cause when we talk about deprioritizing other cause areas for the next X years, we need to have pretty good probabilities and timelines, right? So yeah, I would not consider differing to authorities a strong argument. But thanks for taking the time to reply.
A survey of ML researchers (not necessarily AI Safety, or EA) gave the following
That seems much higher than the corresponding sample in any other field I can think of.
I think that an “extremely bad outcome” is probably equivalent to 1Bn or more people dying.
Do a near majority of those who work in green technology (what feels like the right comparison class) feel that climate change has a 10% chance of 1 Bn deaths?
Personally, I think there is like a 7% chance of extinction before 2050, which is waaay higher than anything else.
FYI—subsamples of that survey were asked about this in other ways, which gave some evidence that “extremely bad outcome” was ~equivalent to extinction.
Or, ‘human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species’
That is, ‘future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species’
There is a big gap between killing all of us and ending future conscious life (on earth, in our galaxy, entire universe/multiverse?)
Yes, but it’s a much smaller gap than any other cause doing this.
You’re right, conscious life will probably be fine. But it might not be.