Related to/expandng upon @huw ’s objection, you not only need to show that longtermist organizations make a difference in the probability of extinction, but that it lasts for a long time. Even if we assume that your $10 Billion would reduce the probability of extinction in 2026, we could counterfactually face extinction in 2027, or 2028, or… etc.
In effect, your calculation assumes that your reduction in existential risk lasts forever. This is a classic example of the “1% fallacy,” where the tiny probability is really hard to achieve.
I would say any intervention that moves us from the path of extinction to the path of never going extinct ever for the next 10^(big number) years is exceptionally valuable. But that’s a different category of interventions than simply “reducing the probability of extinction.”
Even if you believe that the value of utility in the future is equal to the value of utility in the present, you should always discount by a minimum of your probability of extinction per unit time.
This has some really counterintuitive implications.
Let’s consider the value of preventing extinction entirely for a year, which would be an absurdly large claim for any organization to make by orders of magnitude.
Maybe the probability of extinction per year is really high. That means that reducing it to zero is a massive change in our odds of survival (yay!). It also means that our probability that everyone will die in the next few years anyway is also large (darn!).
Maybe the probability of extinction per year is really low. That means reducing it to zero is unlikely to actually do anything (darn!). But if it does, we’ll probably stick around for a long time (yay!).
It turns out these effects exactly cancel out mathematically. Using an annual probability of extinction x, the value of preventing extinction is:
That means that preventing extinction for 1 year is equal to 1 year of utility. Granted, 1 year of utility is a lot, but it is very different from 10^40 or 10^58 lives.
Using the numbers you used in your post,[1] each dollar reduces the probability of extinction by one in a hundred trillion. We should multiply that by the value of one year of utility. I’ll use 10 Billion DALYs (representing 1 year of life for ~10B people) for simplicity, which results in a cost-effectiveness of $10,000 per DALY. That’s orders of magnitude worse than Global Health, your “worst” EA Cause Area.
Caveats/objections:
Accounting for population growth/improving livelihoods, the value of each year does increase over time, but unless you assume unconstrained exponential growth and a very small probability of extinction every year, that effect is relatively mild compared to the discounting and that won’t shift the endline result much.
You could claim that the extinction probabilities are heavily correlated so by preventing extinction in one year you’re also preventing it every year afterwards. I think there’s a reasonable argument to be made that probabilities are correlated across short time spans, but I find it much less plausible across long time spans. As a result, this could shift the value up, but I don’t think it resolves the underlying issue.
You could claim that the probability of extinction is particularly high this year but will drop off later. However, that requires a very specific view of probabilities that is not argued for here in your post. Even if you believe we are at a “hinge of history,” you would need to make the claim that the duration of the hinge is very short and the probability of extinction after the hinge is tiny.
I acknowledge that the numbers in your post are intended to be very conservative. I also have my own reasons to object to your choice of conceptual approach to considering what cause is the best cause area. However, I am considering your numbers as listed for the sake of argument.
Even if you think conditional on preventing a catastrophe there’s a 99.9% chance we’d still go extinct later, before achieving some vast intergalactic future, that’s still only 3 OOMs.
I DO believe that preventing extinction is valuable and there are some projects promising enough to prioritize over work in other cause areas.
I DON’T believe that longtermism dominates all other cause areas in expectation, or that the argument you have made in favor of longtermism is convincing.
This seems to me to be another instance of the 1% fallacy (or the 0.1% fallacy, or the 10^-18 fallacy).
In your post, you talk about being skeptical of arguments where infinities cancel. I would argue that uncanceled infinities are generally a sign of a model being applied beyond its range of efficacy.
If you start off with a rough model and extrapolate it out to get a big enough number, all you have to do is come up with a set of conditions under which that number could plausibly be achieved, no matter how improbable. Then, say “Zero isn’t a probability, and you don’t have enough evidence to show that the probability cancels my big number.” from the start and suddenly you have a large expected value.
However, when you refine the model (considering higher order effects, more thoroughly treating counterfactuals, turning exponential curves into more accurate s-curves), the big number drops out.
The base rate for the infinite is 0. As a result, I am much more skeptical of models where infinities don’t cancel.
Could you please provide any concrete grounding for the probability of counterfactually shifting from extinction to a vast future (not delaying extinction temporarily) that is not based on a very small subjectively “conservative” probability?
Related to/expandng upon @huw ’s objection, you not only need to show that longtermist organizations make a difference in the probability of extinction, but that it lasts for a long time. Even if we assume that your $10 Billion would reduce the probability of extinction in 2026, we could counterfactually face extinction in 2027, or 2028, or… etc.
In effect, your calculation assumes that your reduction in existential risk lasts forever. This is a classic example of the “1% fallacy,” where the tiny probability is really hard to achieve.
I would say any intervention that moves us from the path of extinction to the path of never going extinct ever for the next 10^(big number) years is exceptionally valuable. But that’s a different category of interventions than simply “reducing the probability of extinction.”
Even if you believe that the value of utility in the future is equal to the value of utility in the present, you should always discount by a minimum of your probability of extinction per unit time.
This has some really counterintuitive implications.
Let’s consider the value of preventing extinction entirely for a year, which would be an absurdly large claim for any organization to make by orders of magnitude.
Maybe the probability of extinction per year is really high. That means that reducing it to zero is a massive change in our odds of survival (yay!). It also means that our probability that everyone will die in the next few years anyway is also large (darn!).
Maybe the probability of extinction per year is really low. That means reducing it to zero is unlikely to actually do anything (darn!). But if it does, we’ll probably stick around for a long time (yay!).
It turns out these effects exactly cancel out mathematically. Using an annual probability of extinction x, the value of preventing extinction is:
That means that preventing extinction for 1 year is equal to 1 year of utility. Granted, 1 year of utility is a lot, but it is very different from 10^40 or 10^58 lives.
Using the numbers you used in your post,[1] each dollar reduces the probability of extinction by one in a hundred trillion. We should multiply that by the value of one year of utility. I’ll use 10 Billion DALYs (representing 1 year of life for ~10B people) for simplicity, which results in a cost-effectiveness of $10,000 per DALY. That’s orders of magnitude worse than Global Health, your “worst” EA Cause Area.
Caveats/objections:
Accounting for population growth/improving livelihoods, the value of each year does increase over time, but unless you assume unconstrained exponential growth and a very small probability of extinction every year, that effect is relatively mild compared to the discounting and that won’t shift the endline result much.
You could claim that the extinction probabilities are heavily correlated so by preventing extinction in one year you’re also preventing it every year afterwards. I think there’s a reasonable argument to be made that probabilities are correlated across short time spans, but I find it much less plausible across long time spans. As a result, this could shift the value up, but I don’t think it resolves the underlying issue.
You could claim that the probability of extinction is particularly high this year but will drop off later. However, that requires a very specific view of probabilities that is not argued for here in your post. Even if you believe we are at a “hinge of history,” you would need to make the claim that the duration of the hinge is very short and the probability of extinction after the hinge is tiny.
I acknowledge that the numbers in your post are intended to be very conservative. I also have my own reasons to object to your choice of conceptual approach to considering what cause is the best cause area. However, I am considering your numbers as listed for the sake of argument.
Agree with the first point, was going to go into that in more detail, but thought it would just make things more confusing to a general audience. So I just threw that under the bucket of “things covered by the arbitrarily shaving off 18 OOMs.” I talk more about this here https://benthams.substack.com/p/three-mistakes-in-three-mistakes?utm_source=publication-search
Even if you think conditional on preventing a catastrophe there’s a 99.9% chance we’d still go extinct later, before achieving some vast intergalactic future, that’s still only 3 OOMs.
I figured this was important to keep separate:
I DO believe that preventing extinction is valuable and there are some projects promising enough to prioritize over work in other cause areas.
I DON’T believe that longtermism dominates all other cause areas in expectation, or that the argument you have made in favor of longtermism is convincing.
This seems to me to be another instance of the 1% fallacy (or the 0.1% fallacy, or the 10^-18 fallacy).
In your post, you talk about being skeptical of arguments where infinities cancel. I would argue that uncanceled infinities are generally a sign of a model being applied beyond its range of efficacy.
If you start off with a rough model and extrapolate it out to get a big enough number, all you have to do is come up with a set of conditions under which that number could plausibly be achieved, no matter how improbable. Then, say “Zero isn’t a probability, and you don’t have enough evidence to show that the probability cancels my big number.” from the start and suddenly you have a large expected value.
However, when you refine the model (considering higher order effects, more thoroughly treating counterfactuals, turning exponential curves into more accurate s-curves), the big number drops out.
The base rate for the infinite is 0. As a result, I am much more skeptical of models where infinities don’t cancel.
Could you please provide any concrete grounding for the probability of counterfactually shifting from extinction to a vast future (not delaying extinction temporarily) that is not based on a very small subjectively “conservative” probability?