[I’ll be assuming a consequentialist moral framework in this response, since most EAs are in fact consequentialists. I’m sure other moral systems have their own arguments for (de)prioritizing AI.]
Almost all the disputes on prioritizing AI safety are really epistemological, rather than ethical; the two big exceptions being a disagreement about how to value future persons, and one on ethics with very high numbers of people (Pascal’s Mugging-adjacent situations).
I’ll use the importance-tractability-neglectedness (ITN) framework to explain what I mean. The ITN framework is meant to figure out whether an extra marginal dollar to cause 1 will have more positive impact than a dollar to cause 2; in any consequentialist ethical system, that’s enough reason to prefer cause 1. Importance is the scale of the problem, the (negative) expected value in the counterfactual world where nothing is done about it—I’ll note this as CEV, for counterfactual expected value. Tractability is the share of the problem which can be solved with a given amount of resources; percent-solved-per-dollar, which I’ll note as %/$. Neglectedness is comparing cause 1 to other causes with similar importance-times-tractability, and seeing which cause currently has more funding. In an equation:
CEV1∗(%/$)1>CEV2∗(%/$)2?
Now let’s do ITN for AI risk specifically:
Tractability—This is entirely an epistemological issue, and one which changes the result of any calculations done by a lot. If AI safety is 15% more likely to be solved with a billion more dollars to hire more grad students (or maybe Terence Tao), few people who are really worried about AI risk would object to throwing that amount of money at it. But there are other models under which throwing an extra billion dollars at the problem would barely increase AI safety progress at all, and many are skeptical of using vast amounts of money which could otherwise help alleviate global poverty on an issue with so much uncertainty.
Neglectedness—Just about everyone agrees that if AI safety is indeed as important and tractable as safety advocates say, it currently gets less resources than other issues on the same or smaller scales, like climate change and nuclear war prevention.
Importance—Essentially, importance is probability-of-doom [p(doom)] multiplied by how-bad-doom-actually-is [v(doom)], which gives us expected-(negative)-value [CEV] in the counterfactual universe where we don’t do anything about AI risk.
p(doom)∗v(doom)=CEV
The obvious first issue here is an epistemological one; what is p(doom)? 10% chance of everyone dying is a lot different from 1%, which is in turn very different from 0.1%. And some people think p(doom) is over 50%, or almost nonexistent! All of these numbers have very different implications regarding how much money we should put into AI safety.
Alright, let’s briefly take a step back before looking at how to calculate v(doom). Our equation now looks like this:
p(doom)∗v(doom)∗(%/$)1>CEV2∗(%/$)2?
Assuming that the right side of the equation is constant, we now have 3 variables that can move around: p(doom),(%/$)1, and v(doom). I’ve shown that the first two have a lot of variability, which can lead to multiple orders of magnitude difference in the results.
The ‘longtermist’ argument for AI risk is, plainly, that v(doom) is so unbelievably large that the variations in p(doom) and (%/$)1 are too small to matter. This is based on an epistemological claim and two ethical claims.
Epistemological claim: the expected amount of people (/sentient beings) in the future is huge. An OWID article estimates it at between 800 trillion and 625 quadrillion given a stable population of 11 billion on Earth, while some longtermists, assuming things like space colonization and uploaded minds, go up to 10^53 or something like that. This is the Astronomical Value Thesis.
This claim, at its core, is based on an expectation that existential risk will effectively cease to exist soon (or at least drop to very very low levels), because of something like singularity-level technology. If x-risk stays at something like 1% per century, after all, it’s very unlikely that we ever reach anything like 800 trillion people, let alone some of the higher numbers. This EA Forum post does a great job of explaining the math behind it.
Moral claim 1: We should care about potential future sentient beings; 800 trillion humans existing is 100,000 times better than 8 billion, and the loss of 800 trillion future potential lives should be counted as 100,000 times as bad as the loss of today’s 8 billion lives. This is a very non-intuitive moral claim, but many total utilitarians will agree with it.
If we combine the Astronomical Value Thesis with moral claim 1, we get to the conclusion that v(doom) is so massive that it overwhelms nearly everything else in the equation. To illustrate, I’ll use the lowball estimate of 800 trillion lives:
You don’t need advanced math to know that the side with that many zeroes is probably larger. But valid math is not always valid philosophy, and it has often been said that ethics gets weird around very large numbers. Some people say that this is in fact invalid reasoning, and that it resembles the case of Pascal’s mugging, which infamously ‘proves’ things like that you should exchange 10$ for a one-in-a-quadrillion chance of getting 50 quadrillion dollars (after all, the expected value is $50).
So, to finish, moral claim 2: at least in this case, reasoning like this with very large numbers is ethically valid.
And there you have it! If you accept the Astronomical Value Thesis and both moral claims, just about any spending which decreases x-risk at all will be worth prioritizing. If you reject any of those three claims, it can still be entirely reasonable to prioritize AI risk, if your p(doom) and tractability estimates are high enough. Plugging in the current 8 billion people on the planet:
[I’ll be assuming a consequentialist moral framework in this response, since most EAs are in fact consequentialists. I’m sure other moral systems have their own arguments for (de)prioritizing AI.]
Almost all the disputes on prioritizing AI safety are really epistemological, rather than ethical; the two big exceptions being a disagreement about how to value future persons, and one on ethics with very high numbers of people (Pascal’s Mugging-adjacent situations).
I’ll use the importance-tractability-neglectedness (ITN) framework to explain what I mean. The ITN framework is meant to figure out whether an extra marginal dollar to cause 1 will have more positive impact than a dollar to cause 2; in any consequentialist ethical system, that’s enough reason to prefer cause 1. Importance is the scale of the problem, the (negative) expected value in the counterfactual world where nothing is done about it—I’ll note this as CEV, for counterfactual expected value. Tractability is the share of the problem which can be solved with a given amount of resources; percent-solved-per-dollar, which I’ll note as %/$. Neglectedness is comparing cause 1 to other causes with similar importance-times-tractability, and seeing which cause currently has more funding. In an equation:
CEV1∗(%/$)1>CEV2∗(%/$)2?
Now let’s do ITN for AI risk specifically:
Tractability—This is entirely an epistemological issue, and one which changes the result of any calculations done by a lot. If AI safety is 15% more likely to be solved with a billion more dollars to hire more grad students (or maybe Terence Tao), few people who are really worried about AI risk would object to throwing that amount of money at it. But there are other models under which throwing an extra billion dollars at the problem would barely increase AI safety progress at all, and many are skeptical of using vast amounts of money which could otherwise help alleviate global poverty on an issue with so much uncertainty.
Neglectedness—Just about everyone agrees that if AI safety is indeed as important and tractable as safety advocates say, it currently gets less resources than other issues on the same or smaller scales, like climate change and nuclear war prevention.
Importance—Essentially, importance is probability-of-doom [p(doom)] multiplied by how-bad-doom-actually-is [v(doom)], which gives us expected-(negative)-value [CEV] in the counterfactual universe where we don’t do anything about AI risk.
p(doom)∗v(doom)=CEV
The obvious first issue here is an epistemological one; what is p(doom)? 10% chance of everyone dying is a lot different from 1%, which is in turn very different from 0.1%. And some people think p(doom) is over 50%, or almost nonexistent! All of these numbers have very different implications regarding how much money we should put into AI safety.
Alright, let’s briefly take a step back before looking at how to calculate v(doom). Our equation now looks like this:
p(doom)∗v(doom)∗(%/$)1>CEV2∗(%/$)2?
Assuming that the right side of the equation is constant, we now have 3 variables that can move around: p(doom),(%/$)1, and v(doom). I’ve shown that the first two have a lot of variability, which can lead to multiple orders of magnitude difference in the results.
The ‘longtermist’ argument for AI risk is, plainly, that v(doom) is so unbelievably large that the variations in p(doom) and (%/$)1 are too small to matter. This is based on an epistemological claim and two ethical claims.
Epistemological claim: the expected amount of people (/sentient beings) in the future is huge. An OWID article estimates it at between 800 trillion and 625 quadrillion given a stable population of 11 billion on Earth, while some longtermists, assuming things like space colonization and uploaded minds, go up to 10^53 or something like that. This is the Astronomical Value Thesis.
This claim, at its core, is based on an expectation that existential risk will effectively cease to exist soon (or at least drop to very very low levels), because of something like singularity-level technology. If x-risk stays at something like 1% per century, after all, it’s very unlikely that we ever reach anything like 800 trillion people, let alone some of the higher numbers. This EA Forum post does a great job of explaining the math behind it.
Moral claim 1: We should care about potential future sentient beings; 800 trillion humans existing is 100,000 times better than 8 billion, and the loss of 800 trillion future potential lives should be counted as 100,000 times as bad as the loss of today’s 8 billion lives. This is a very non-intuitive moral claim, but many total utilitarians will agree with it.
If we combine the Astronomical Value Thesis with moral claim 1, we get to the conclusion that v(doom) is so massive that it overwhelms nearly everything else in the equation. To illustrate, I’ll use the lowball estimate of 800 trillion lives:
p(doom)∗800,000,000,000,000 lives∗(%/$)1>CEV2∗(%/$)2?
You don’t need advanced math to know that the side with that many zeroes is probably larger. But valid math is not always valid philosophy, and it has often been said that ethics gets weird around very large numbers. Some people say that this is in fact invalid reasoning, and that it resembles the case of Pascal’s mugging, which infamously ‘proves’ things like that you should exchange 10$ for a one-in-a-quadrillion chance of getting 50 quadrillion dollars (after all, the expected value is $50).
So, to finish, moral claim 2: at least in this case, reasoning like this with very large numbers is ethically valid.
And there you have it! If you accept the Astronomical Value Thesis and both moral claims, just about any spending which decreases x-risk at all will be worth prioritizing. If you reject any of those three claims, it can still be entirely reasonable to prioritize AI risk, if your p(doom) and tractability estimates are high enough. Plugging in the current 8 billion people on the planet:
p(doom)∗8,000,000,000 lives∗(%/$)1>CEV2∗(%/$)2?
That’s still a lot of zeroes!