This post seeks to estimate how much we should expect a highly cost-effective charity to spend on reducing existential risk by a certain amount. By setting a threshold for cost-effectiveness, we can be selective about which longtermist charities to recommend to donors.
We appreciate feedback. We would like for this post to be the first in a sequence about cost-effectiveness thresholds for giving, and your feedback will help us write better posts.
How many beings does extinction destroy?
This chart gives six estimates for the size of the moral universe that would be lost in an extinction event on Earth this century. There is a truly incredible range in the possible size of the moral universe, and the value you see in the future depends on the moral weights you put on different types of moral patients.
(Click this link to explore the value of the future. You can change the moral weights on humans, vertebrates, invertebrates, and beings of the future; the number of years you expect humans and animals to live without inevitable extinction; and the number of animals and humans you expect to exist at one time, from conservative to liberal estimates. The value at the bottom, T, shows you the total value you put on the future. You can then see how changing your moral weights radically influences the value you put on the future – and you can recalculate the cost-effectiveness thresholds below based on that.)
If the universe could have between 8 x 10^9 and 5 x 10^55 morally valuable beings, then a 0.01% absolute reduction in cumulative x-risk is “equivalent” (given several assumptions) to saving 8 x 10^5 to 5 x 10^51 lives. This is a reduction in risk of one basis point, or bp. Note that this is a much greater accomplishment than reducing per-century x-risk by one basis point, which we will discuss later.
How much should we pay to prevent this destruction?
Using near-termist thresholds as a starting point
Using the SoGive Gold Standard for cost per life saved in a near-termist framework (£5,000), this would mean a “good cost” for a 0.01% absolute reduction in the cumulative risk of extinction would be between £4 billion and £2.5 * 10^55. For context, all the money in the world is probably between £10^13 and £10^14.
Are either of these the thresholds we should be using? The cost-effectiveness of longtermist interventions is different from near-termist interventions:
The interventions we are analyzing are of a different character.
The evidence of their cost-effectiveness is more speculative.
It might be much cheaper, or much more expensive, to prevent the loss of future life than to save a person now. It would not be useful to set a threshold that ~all highly effective longtermist charities would be above or below, when the purpose of a threshold is to help us select the top tier of existing cost-effective interventions within a cause area.
Furthermore, we might never have enough evidence to say whether an intervention has reduced cumulative x-risk by a certain amount. It might be more manageable to set a threshold based on reduction in per-century x-risk.
Per-century risk
Epistemic status: Uncertain. Please leave feedback and help us improve the math here.
Briefly looking at this, let’s assume that humans will last some time into the future, and that those humans are our moral patients. (If we didn’t put value on future humans, then cumulative risk would be equal to the risk of extinction for present beings. This would give us a lower bound on a good cost for a 0.01% absolute reduction in per-century risk of £4 billion.)
Let’s also assume for now that we are not in a time of perils, so that we have a simpler relationship between cumulative risk and per-century risk. This will give us a lower bound for what we should spend on a reduction in per-century risk under our assumption above. (Given the chance we are in a time of perils, a “good cost” for a reduction in per-century risk this century would only be higher. If we are in an extreme time of perils where a reduction in per-century risk is about equivalent to a reduction in cumulative risk, then we already have our upper bound on a good cost for a 0.01% reduction in per-century risk – £2.5 * 10^55.)
According to David Thorstad, assuming a future of 1 billion years, an absolute reduction in cumulative x-risk of only 1⁄100,000,000requires that we reduce per-century x-risk down from its current figure (estimated by Toby Ord at 1⁄6) at least down to below 16⁄10,000,000.[1]
Let’s try to use the same math. On the low end of the size of the future, suppose humans could exist for a maximum of 800,000 more years before inevitable extinction. Then, an absolute reduction in cumulative risk of 1⁄10,000 (1 bp) would require us to drive per-century risk at least down to below 11⁄10,000.
Bringing per-century risk from ⅙ to 11⁄10,000 is an absolute reduction of 1656 basis points. We might say that if the future is 800,000 years and there are 10^14 moral patients (low end) that can exist in that time, then we should pay 10^14 * 0.01% * £5000 / 1656 = £30.19 billion[2] to reduce per-century x-risk by 1 bp.
On the high end of the size of the future, suppose humans could exist for a maximum of 100 billion years before inevitable extinction. Then, an absolute reduction in cumulative risk of 1⁄10,000 (1 bp) would require us to drive per-century risk at least down to below 92⁄10,000,000,000.
Bringing per-century risk from ⅙ to 92⁄10,000,000,000 is an absolute reduction of 1667 basis points. We might say that if the future is 100 billion years and there are 5 x 10^55 moral patients (high end) that can exist in that time, then we should pay 5 * 10^55 * 0.01% * £5000 / 1667 = £1.50 × 10^52 to reduce per-century x-risk by 1 bp.
Clearly, this is a very difficult task in either situation. What does this mean for our threshold? Do we think that reducing cumulative x-risk by one basis point is worth a different amount, when we know that it means reducing per-century x-risk so extremely?
Should we tolerate a higher cost threshold for a more difficult task?
Should we reduce our tolerance for the cost, since the task may be less tractable?
Overall, our range for what we should spend on a reduction in a 0.01% absolute reduction in x-risk for this century is £4 billion, if you only put moral value on present humans; £30.19 billion, if you believe the future is small, the class of moral patients is small, and we are not in a time of perils; £1.50 × 10^52, if you predict a very large future and a large class of moral patients, and believe we are not in a time of perils; and £2.5 * 10^55, if you predict a very large future and a large class of moral patients, and believe we are in an extreme time of perils.
Using benchmarks for cost-effectiveness from current longtermist charities
We are currently seeking estimates of how much longtermist charities are reducing x-risk. These estimates can give us a benchmark for cost-effectiveness. Benchmarking is another method we can use to set a threshold. However, estimates we get from charities for their own cost-effectiveness are likely to be biased.
Using the estimates of others
Now let’s look at cost-effectiveness thresholds from others, taken from this post by Linch and this post by Vasco and their comments sections, which use USD. The table below is adapted and expanded from Vasco’s post.
Most commenters seem to have been thinking of an absolute reduction of one basis point this century (it’s unclear) – but one commenter, NunoSempere, gave a range for an absolute reduction by one basis point in the yearly x-risk for one century, and another range for an absolute reduction by one basis point in the x-risk this century. The commenters used a range of methods to answer the question, but their answers clustered around $1 billion.
Please note Linch’s disclaimer, which likely also applies to the estimates of others:
“EDIT 2022/09/21: The 100M-1B estimates are relatively off-the-cuff and very not robust, I think there are good arguments to go higher or lower. I think the numbers aren’t crazy, partially because others independently come to similar numbers (but some people I respect have different numbers). I don’t think it’s crazy to make decisions/defer roughly based on these numbers given limited time and attention. However, I’m worried about having too much secondary literature/large decisions based on my numbers, since it will likely result ininformation cascades. My current tentative guess as of 2022/09/21 is that there are more reasons to go higher (think averting x-risk is more expensive) than lower. However, overspending on marginal interventions is more -EV than underspending, which pushes us to bias towards conservatism.”
“I think we are currently funding projects that are definitely more cost-effective than that”
“My very rough gut estimate says something like” $1 billion per bp
“Probably more on the margin”
NunoSempere
$99 billion per bp, lower bound assuming a “one-off 0.01% existential risk reduction over a century”
$16 trillion per bp, lower bound per yearly reduction of 1bp … to $330T per bp, upper bound assuming a “one-off 0.01% existential risk reduction over a century”
$3.8 x 10^15 per bp, upper bound per yearly reduction of 1bp
Anonymous person from Vasco’s post
$1 trillion per existential catastrophe averted, or $100 million per bp
$100 trillion per existential catastrophe averted, or $10 billion per bp
Simon Skade
“Note that I do think that there are even more effective ways to reduce x-risk, and in fact I suspect most things longtermist EA is currently funding have a higher expected x-risk reduction than 0.01% per 154M$.”
$154 million per bp
“I just don’t think that it is likely that the 50 billionth 2021 dollar EA spends has a much higher effectiveness than 0.01% per 154M$, so I think we should grant everything that has a higher expected effectiveness.”
William Kiely
$34 million per bp
“I thought it would be interesting to answer this using a wrong method (writing the bottom line first).”
~ $100 million per bp
$340 million per bp
Zach Stein-Perlman
$25 million per bp
$50 million per bp
“Unlike Linch, I would be quite sad about trading $100M for a single measly basis point — $100M (granted reasonably well) would make a bigger difference, I think.” $100 million per bp
Median
$100 million per bp
$1 billion per bp
$5.17 billion per bp
This suggests that a reasonable range for the cost of reducing absolute x-risk by 0.01% this century could be between USD $100 million and $5.17 billion, or £75 million and £3.88 billion.[3]
Earlier, we put a “good cost” of a 1 bp absolute reduction in per-century x-risk, based on SoGive’s near-termist Gold Standard, between £4 billion and £2.5 * 10^55, depending on how much you value the future what parts of the future you value, and your forecast for the future. Note that the estimates given in this table are all below that range.
Why is that?
Were these commenters expecting it to be much cheaper to save a life by preventing the loss of potential in an extinction, than to save a life using near-termist interventions?
Were they envisioning a relatively small future?
Were they discounting the future?
Probably, instead of these possibilities, the commenters were giving an implicit haircut to their estimates based on uncertainty.
What does this say for our threshold?
Conclusion
For near-termist interventions, when SoGive has high confidence that the intervention is effective, we use a Gold Standard. A tentative threshold for a “SoGive Longtermist Gold Standard” could be £750 million / bp. This is the median estimate given by the commenters above.
Having a threshold for cost-effectiveness can help us decide whether to give now or give later. If all the available funding opportunities cost more than our threshold, then we can hold onto our donations until a more cost-effective opportunity comes along. We may come back to the nuanced relationship between thresholds and give now/give later in a future post.
There remains a practical problem with using a cost-effectiveness threshold for deciding where to give within longtermism. Most longtermist projects do not publish cost-effectiveness estimates – and if they did, they could be as flawed as misleading cost-effectiveness estimates from near-termist charities. For third-party evaluators, there is no clearly reliable method for estimating how much x-risk has been reduced by one project.
Thus, a threshold for longtermist cost-effectiveness could be better applied to the average cost-effectiveness of categories, e.g., “AI risk-reducing projects in Europe,” rather than to individual projects. (It is not clear that cost-effectiveness for projects varies less within geographic regions than across regions – this is just an example of a category of funding opportunities.) Analyzing cost-effectiveness over a larger category of funding opportunities, instead of per-intervention or per-project, could help us avoid false precision.
These estimates are not robust enough to make the most important decisions we face. We recommend conducting a survey of funders, charities, and experts to get a stronger picture of what the standard should be and the cost-effectiveness of different types of work.
This represents a reduction from a cumulative risk of 1 to a cumulative risk of 0.9999. If we are starting with a cumulative risk below 1, then we need to reduce per-century risk somewhat more, but likely not orders of magnitude more.
(number of moral patients) * (% saved in expectation given 0.01% reduction in cumulative risk) * (“good cost” to save a life) / (conversion from 0.01% reduction in cumulative risk to 0.01% reduction in per-century risk)
Thresholds #1: What does good look like for longtermism?
Introduction
This post seeks to estimate how much we should expect a highly cost-effective charity to spend on reducing existential risk by a certain amount. By setting a threshold for cost-effectiveness, we can be selective about which longtermist charities to recommend to donors.
We appreciate feedback. We would like for this post to be the first in a sequence about cost-effectiveness thresholds for giving, and your feedback will help us write better posts.
How many beings does extinction destroy?
This chart gives six estimates for the size of the moral universe that would be lost in an extinction event on Earth this century. There is a truly incredible range in the possible size of the moral universe, and the value you see in the future depends on the moral weights you put on different types of moral patients.
(Click this link to explore the value of the future. You can change the moral weights on humans, vertebrates, invertebrates, and beings of the future; the number of years you expect humans and animals to live without inevitable extinction; and the number of animals and humans you expect to exist at one time, from conservative to liberal estimates. The value at the bottom, T, shows you the total value you put on the future. You can then see how changing your moral weights radically influences the value you put on the future – and you can recalculate the cost-effectiveness thresholds below based on that.)
If the universe could have between 8 x 10^9 and 5 x 10^55 morally valuable beings, then a 0.01% absolute reduction in cumulative x-risk is “equivalent” (given several assumptions) to saving 8 x 10^5 to 5 x 10^51 lives. This is a reduction in risk of one basis point, or bp. Note that this is a much greater accomplishment than reducing per-century x-risk by one basis point, which we will discuss later.
How much should we pay to prevent this destruction?
Using near-termist thresholds as a starting point
Using the SoGive Gold Standard for cost per life saved in a near-termist framework (£5,000), this would mean a “good cost” for a 0.01% absolute reduction in the cumulative risk of extinction would be between £4 billion and £2.5 * 10^55. For context, all the money in the world is probably between £10^13 and £10^14.
Are either of these the thresholds we should be using? The cost-effectiveness of longtermist interventions is different from near-termist interventions:
The interventions we are analyzing are of a different character.
The evidence of their cost-effectiveness is more speculative.
It might be much cheaper, or much more expensive, to prevent the loss of future life than to save a person now. It would not be useful to set a threshold that ~all highly effective longtermist charities would be above or below, when the purpose of a threshold is to help us select the top tier of existing cost-effective interventions within a cause area.
Furthermore, we might never have enough evidence to say whether an intervention has reduced cumulative x-risk by a certain amount. It might be more manageable to set a threshold based on reduction in per-century x-risk.
Per-century risk
Epistemic status: Uncertain. Please leave feedback and help us improve the math here.
Briefly looking at this, let’s assume that humans will last some time into the future, and that those humans are our moral patients. (If we didn’t put value on future humans, then cumulative risk would be equal to the risk of extinction for present beings. This would give us a lower bound on a good cost for a 0.01% absolute reduction in per-century risk of £4 billion.)
Let’s also assume for now that we are not in a time of perils, so that we have a simpler relationship between cumulative risk and per-century risk. This will give us a lower bound for what we should spend on a reduction in per-century risk under our assumption above. (Given the chance we are in a time of perils, a “good cost” for a reduction in per-century risk this century would only be higher. If we are in an extreme time of perils where a reduction in per-century risk is about equivalent to a reduction in cumulative risk, then we already have our upper bound on a good cost for a 0.01% reduction in per-century risk – £2.5 * 10^55.)
According to David Thorstad, assuming a future of 1 billion years, an absolute reduction in cumulative x-risk of only 1⁄100,000,000 requires that we reduce per-century x-risk down from its current figure (estimated by Toby Ord at 1⁄6) at least down to below 16⁄10,000,000.[1]
Let’s try to use the same math. On the low end of the size of the future, suppose humans could exist for a maximum of 800,000 more years before inevitable extinction. Then, an absolute reduction in cumulative risk of 1⁄10,000 (1 bp) would require us to drive per-century risk at least down to below 11⁄10,000.
Bringing per-century risk from ⅙ to 11⁄10,000 is an absolute reduction of 1656 basis points. We might say that if the future is 800,000 years and there are 10^14 moral patients (low end) that can exist in that time, then we should pay 10^14 * 0.01% * £5000 / 1656 = £30.19 billion[2] to reduce per-century x-risk by 1 bp.
On the high end of the size of the future, suppose humans could exist for a maximum of 100 billion years before inevitable extinction. Then, an absolute reduction in cumulative risk of 1⁄10,000 (1 bp) would require us to drive per-century risk at least down to below 92⁄10,000,000,000.
Bringing per-century risk from ⅙ to 92⁄10,000,000,000 is an absolute reduction of 1667 basis points. We might say that if the future is 100 billion years and there are 5 x 10^55 moral patients (high end) that can exist in that time, then we should pay 5 * 10^55 * 0.01% * £5000 / 1667 = £1.50 × 10^52 to reduce per-century x-risk by 1 bp.
Clearly, this is a very difficult task in either situation. What does this mean for our threshold? Do we think that reducing cumulative x-risk by one basis point is worth a different amount, when we know that it means reducing per-century x-risk so extremely?
Should we tolerate a higher cost threshold for a more difficult task?
Should we reduce our tolerance for the cost, since the task may be less tractable?
Overall, our range for what we should spend on a reduction in a 0.01% absolute reduction in x-risk for this century is £4 billion, if you only put moral value on present humans; £30.19 billion, if you believe the future is small, the class of moral patients is small, and we are not in a time of perils; £1.50 × 10^52, if you predict a very large future and a large class of moral patients, and believe we are not in a time of perils; and £2.5 * 10^55, if you predict a very large future and a large class of moral patients, and believe we are in an extreme time of perils.
Using benchmarks for cost-effectiveness from current longtermist charities
We are currently seeking estimates of how much longtermist charities are reducing x-risk. These estimates can give us a benchmark for cost-effectiveness. Benchmarking is another method we can use to set a threshold. However, estimates we get from charities for their own cost-effectiveness are likely to be biased.
Using the estimates of others
Now let’s look at cost-effectiveness thresholds from others, taken from this post by Linch and this post by Vasco and their comments sections, which use USD. The table below is adapted and expanded from Vasco’s post.
Most commenters seem to have been thinking of an absolute reduction of one basis point this century (it’s unclear) – but one commenter, NunoSempere, gave a range for an absolute reduction by one basis point in the yearly x-risk for one century, and another range for an absolute reduction by one basis point in the x-risk this century. The commenters used a range of methods to answer the question, but their answers clustered around $1 billion.
Please note Linch’s disclaimer, which likely also applies to the estimates of others:
“EDIT 2022/09/21: The 100M-1B estimates are relatively off-the-cuff and very not robust, I think there are good arguments to go higher or lower. I think the numbers aren’t crazy, partially because others independently come to similar numbers (but some people I respect have different numbers). I don’t think it’s crazy to make decisions/defer roughly based on these numbers given limited time and attention. However, I’m worried about having too much secondary literature/large decisions based on my numbers, since it will likely result in information cascades. My current tentative guess as of 2022/09/21 is that there are more reasons to go higher (think averting x-risk is more expensive) than lower. However, overspending on marginal interventions is more -EV than underspending, which pushes us to bias towards conservatism.”
“$200 trillion per world saved”
Or $20 billion per bp
“I thought it would be interesting to answer this using a wrong method (writing the bottom line first).”
~ $100 million per bp
This suggests that a reasonable range for the cost of reducing absolute x-risk by 0.01% this century could be between USD $100 million and $5.17 billion, or £75 million and £3.88 billion.[3]
Earlier, we put a “good cost” of a 1 bp absolute reduction in per-century x-risk, based on SoGive’s near-termist Gold Standard, between £4 billion and £2.5 * 10^55, depending on how much you value the future what parts of the future you value, and your forecast for the future. Note that the estimates given in this table are all below that range.
Why is that?
Were these commenters expecting it to be much cheaper to save a life by preventing the loss of potential in an extinction, than to save a life using near-termist interventions?
Were they envisioning a relatively small future?
Were they discounting the future?
Probably, instead of these possibilities, the commenters were giving an implicit haircut to their estimates based on uncertainty.
What does this say for our threshold?
Conclusion
For near-termist interventions, when SoGive has high confidence that the intervention is effective, we use a Gold Standard. A tentative threshold for a “SoGive Longtermist Gold Standard” could be £750 million / bp. This is the median estimate given by the commenters above.
Having a threshold for cost-effectiveness can help us decide whether to give now or give later. If all the available funding opportunities cost more than our threshold, then we can hold onto our donations until a more cost-effective opportunity comes along. We may come back to the nuanced relationship between thresholds and give now/give later in a future post.
There remains a practical problem with using a cost-effectiveness threshold for deciding where to give within longtermism. Most longtermist projects do not publish cost-effectiveness estimates – and if they did, they could be as flawed as misleading cost-effectiveness estimates from near-termist charities. For third-party evaluators, there is no clearly reliable method for estimating how much x-risk has been reduced by one project.
Thus, a threshold for longtermist cost-effectiveness could be better applied to the average cost-effectiveness of categories, e.g., “AI risk-reducing projects in Europe,” rather than to individual projects. (It is not clear that cost-effectiveness for projects varies less within geographic regions than across regions – this is just an example of a category of funding opportunities.) Analyzing cost-effectiveness over a larger category of funding opportunities, instead of per-intervention or per-project, could help us avoid false precision.
These estimates are not robust enough to make the most important decisions we face. We recommend conducting a survey of funders, charities, and experts to get a stronger picture of what the standard should be and the cost-effectiveness of different types of work.
This represents a reduction from a cumulative risk of 1 to a cumulative risk of 0.9999. If we are starting with a cumulative risk below 1, then we need to reduce per-century risk somewhat more, but likely not orders of magnitude more.
(number of moral patients) * (% saved in expectation given 0.01% reduction in cumulative risk) * (“good cost” to save a life) / (conversion from 0.01% reduction in cumulative risk to 0.01% reduction in per-century risk)
Converting at an average exchange rate of 0.75.