I cannot help notice the cost-effectiveness of the project on preventing a natural disaster is of the same order of magnitude as that of the 2 projects on AI alignement. For readers’ context, AI safety technical research is 80,000 Hours’ top career path, whereas one could argue extinction risk from natural disasters is astronomically low. I understand the results of the CCM are tentative, but this result makes me think I should put very little trust on the default parameters of the projects aiming to reduce existential risk.
I think you should put very little trust in the default parameters of the projects. It was our initial intention to create defaults that reflected the best evidence and expert opinion, but we had difficulty getting consensus on what these values should be and decided instead to explicitly stand back from the defaults. The parameter settings are adjustable to suit your views, and we encourage people to think about what those parameter settings should be and not take the defaults too seriously.
For readers’ context, AI safety technical research is 80,000 Hours’ top career path, whereas one could argue extinction risk from natural disasters is astronomically low.
The parameters allow you to control how far into the future you look and the outcomes include not just effects on the long-term future from the extinction / preservation of the species but also on the probabilities of near-term catastrophes that cause large numbers of death but don’t cause extinction. Depending on your settings, near-term catastrophes can dominate the expected value. For the default settings for natural disasters and bio-risk, much of the value of mitigation work (at least over the next 1000 years) comes from prevention of relatively small-scale disasters. I don’t see anything obviously wrong about this result and I expect that 80K’s outlook is based on a willingness to consider effects more than 1000 years in the future.
I happen to disagree with these numbers because I think that numbers for effectiveness of x-risk projects are too low. E.g., for the “Small-scale AI Misalignment Project”: “we expect that it reduces absolute existential risk by a factor between 0.000001 and 0.000055″, these seem like many zeroes to me.
Ditto for the “AI Misalignment Megaproject”: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
Ditto for the “AI Misalignment Megaproject”: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
I think we’re somewhat bearish on the ability of money by itself to solve problems. The technical issues around alignment appear quite challenging, especially given the pace of development, so it isn’t clear that any amount of money will be able to solve them. If the issues are too easy on the other hand, then your investment of money is unlikely to be needed and so your expenditure isn’t going to reduce extinction risk.
Even if the technical issues are in the goldilocks spot of being solvable but not trivially so, the political challenges around getting those solutions adopted seem extremely daunting. There is a lot we don’t explicitly specify in these parameter settings: if the money is coming from a random billionaire unaffiliated with AI scene then it might be harder to get expertise and buy in then if it is coming from insiders or the federal government.
All that said, it is plausible to me that we should have a somewhat higher chance of having an impact coupled with a lower chance of a positive outcome. A few billion dollars is likely to shake things up even if the outcome isn’t what we hoped for.
Is that 3% an absolute percentage point reduction in risk? If so, that doesn’t seem very low if your baseline risk estimate is low, like 5-20%, or you’re as pessimistic about aligning AI as MIRI is.
No, 3% is “chance of success”. After adding a bunch of multipliers, it comes to about 0.6% reduction in existential risk over the next century, for $8B to $20B.
Hello,
Rethink Priorities has noted CCM’s estimates are not resilient. However, just for reference, here they are in descending order in DALY/k$[1]:
Global health and development:
Good GHD Intervention: 39.
GiveWell Bar: 21.
Open Philanthropy Bar: 21.
Best HIV Intervention: 5.
Direct Cash Transfers: 2.
Standard HIV Intervention: 1.
US Gov GHD Intervention: 1.
Weak GHD Intervention: 1.
Ineffective GHD Intervention: 0.
Very Ineffective GHD Intervention: 0.
Animal Welfare:
Cage-free Chicken Campaign: 714.
Generic Chicken Campaign: 714.
Shrimp Ammonia Intervention: 397.
Generic Black Soldier Fly Intervention: 46.
Generic Carp Intervention: 37.
Shrimp Slaughter Intervention: 10.
Generic Shrimp Intervention: 9.
Existential risk (the results change from run to run, but I think the values below represent the right order of magnitude):
Small-scale AI Misuse Project: 269 k.
Small-scale Nuclear Safety Project: 48 k.
Small-scale Nanotech Safety Project: 16 k.
Small-scale Biorisk Project: 8,915.
Portfolio of Biorisk Projects: 5,919.
Nanotech Safety Megaproject: 3,199.
Small-scale AI Misalignment Project: 1,718.
AI Misalignment Megaproject: 1,611.
Small-scale Natural Disaster Prevention Project: 1,558.
Exploratory Research into Unknown Risks: 30.
I cannot help notice the cost-effectiveness of the project on preventing a natural disaster is of the same order of magnitude as that of the 2 projects on AI alignement. For readers’ context, AI safety technical research is 80,000 Hours’ top career path, whereas one could argue extinction risk from natural disasters is astronomically low. I understand the results of the CCM are tentative, but this result makes me think I should put very little trust on the default parameters of the projects aiming to reduce existential risk.
For draws, I present the estimates by alphabetical order.
I think you should put very little trust in the default parameters of the projects. It was our initial intention to create defaults that reflected the best evidence and expert opinion, but we had difficulty getting consensus on what these values should be and decided instead to explicitly stand back from the defaults. The parameter settings are adjustable to suit your views, and we encourage people to think about what those parameter settings should be and not take the defaults too seriously.
The parameters allow you to control how far into the future you look and the outcomes include not just effects on the long-term future from the extinction / preservation of the species but also on the probabilities of near-term catastrophes that cause large numbers of death but don’t cause extinction. Depending on your settings, near-term catastrophes can dominate the expected value. For the default settings for natural disasters and bio-risk, much of the value of mitigation work (at least over the next 1000 years) comes from prevention of relatively small-scale disasters. I don’t see anything obviously wrong about this result and I expect that 80K’s outlook is based on a willingness to consider effects more than 1000 years in the future.
I happen to disagree with these numbers because I think that numbers for effectiveness of x-risk projects are too low. E.g., for the “Small-scale AI Misalignment Project”: “we expect that it reduces absolute existential risk by a factor between 0.000001 and 0.000055″, these seem like many zeroes to me.
Ditto for the “AI Misalignment Megaproject”: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
I think we’re somewhat bearish on the ability of money by itself to solve problems. The technical issues around alignment appear quite challenging, especially given the pace of development, so it isn’t clear that any amount of money will be able to solve them. If the issues are too easy on the other hand, then your investment of money is unlikely to be needed and so your expenditure isn’t going to reduce extinction risk.
Even if the technical issues are in the goldilocks spot of being solvable but not trivially so, the political challenges around getting those solutions adopted seem extremely daunting. There is a lot we don’t explicitly specify in these parameter settings: if the money is coming from a random billionaire unaffiliated with AI scene then it might be harder to get expertise and buy in then if it is coming from insiders or the federal government.
All that said, it is plausible to me that we should have a somewhat higher chance of having an impact coupled with a lower chance of a positive outcome. A few billion dollars is likely to shake things up even if the outcome isn’t what we hoped for.
Is that 3% an absolute percentage point reduction in risk? If so, that doesn’t seem very low if your baseline risk estimate is low, like 5-20%, or you’re as pessimistic about aligning AI as MIRI is.
No, 3% is “chance of success”. After adding a bunch of multipliers, it comes to about 0.6% reduction in existential risk over the next century, for $8B to $20B.
2 nitpicks that end up arguing in favor of your high-level point
2.7% (which you’re rounding up to 3%) is chance of having an effect, and 70% x 2.7% = 1.9% is chance of positive effect (‘success’ by your wording)
your Squiggle calc doesn’t include the CCM’s ‘intervention backfiring’ part of the calc