I cannot help notice the cost-effectiveness of the project on preventing a natural disaster is of the same order of magnitude as that of the 2 projects on AI alignement. For readersâ context, AI safety technical research is 80,000 Hoursâ top career path, whereas one could argue extinction risk from natural disasters is astronomically low. I understand the results of the CCM are tentative, but this result makes me think I should put very little trust on the default parameters of the projects aiming to reduce existential risk.
I think you should put very little trust in the default parameters of the projects. It was our initial intention to create defaults that reflected the best evidence and expert opinion, but we had difficulty getting consensus on what these values should be and decided instead to explicitly stand back from the defaults. The parameter settings are adjustable to suit your views, and we encourage people to think about what those parameter settings should be and not take the defaults too seriously.
For readersâ context, AI safety technical research is 80,000 Hoursâ top career path, whereas one could argue extinction risk from natural disasters is astronomically low.
The parameters allow you to control how far into the future you look and the outcomes include not just effects on the long-term future from the extinction /â preservation of the species but also on the probabilities of near-term catastrophes that cause large numbers of death but donât cause extinction. Depending on your settings, near-term catastrophes can dominate the expected value. For the default settings for natural disasters and bio-risk, much of the value of mitigation work (at least over the next 1000 years) comes from prevention of relatively small-scale disasters. I donât see anything obviously wrong about this result and I expect that 80Kâs outlook is based on a willingness to consider effects more than 1000 years in the future.
I happen to disagree with these numbers because I think that numbers for effectiveness of x-risk projects are too low. E.g., for the âSmall-scale AI Misalignment Projectâ: âwe expect that it reduces absolute existential risk by a factor between 0.000001 and 0.000055âł, these seem like many zeroes to me.
Ditto for the âAI Misalignment Megaprojectâ: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
Ditto for the âAI Misalignment Megaprojectâ: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
I think weâre somewhat bearish on the ability of money by itself to solve problems. The technical issues around alignment appear quite challenging, especially given the pace of development, so it isnât clear that any amount of money will be able to solve them. If the issues are too easy on the other hand, then your investment of money is unlikely to be needed and so your expenditure isnât going to reduce extinction risk.
Even if the technical issues are in the goldilocks spot of being solvable but not trivially so, the political challenges around getting those solutions adopted seem extremely daunting. There is a lot we donât explicitly specify in these parameter settings: if the money is coming from a random billionaire unaffiliated with AI scene then it might be harder to get expertise and buy in then if it is coming from insiders or the federal government.
All that said, it is plausible to me that we should have a somewhat higher chance of having an impact coupled with a lower chance of a positive outcome. A few billion dollars is likely to shake things up even if the outcome isnât what we hoped for.
Is that 3% an absolute percentage point reduction in risk? If so, that doesnât seem very low if your baseline risk estimate is low, like 5-20%, or youâre as pessimistic about aligning AI as MIRI is.
No, 3% is âchance of successâ. After adding a bunch of multipliers, it comes to about 0.6% reduction in existential risk over the next century, for $8B to $20B.
2 nitpicks that end up arguing in favor of your high-level point
2.7% (which youâre rounding up to 3%) is chance of having an effect, and 70% x 2.7% = 1.9% is chance of positive effect (âsuccessâ by your wording)
your Squiggle calc doesnât include the CCMâs âintervention backfiringâ part of the calc
Hello,
Rethink Priorities has noted CCMâs estimates are not resilient. However, just for reference, here they are in descending order in DALY/âk$[1]:
Global health and development:
Good GHD Intervention: 39.
GiveWell Bar: 21.
Open Philanthropy Bar: 21.
Best HIV Intervention: 5.
Direct Cash Transfers: 2.
Standard HIV Intervention: 1.
US Gov GHD Intervention: 1.
Weak GHD Intervention: 1.
Ineffective GHD Intervention: 0.
Very Ineffective GHD Intervention: 0.
Animal Welfare:
Cage-free Chicken Campaign: 714.
Generic Chicken Campaign: 714.
Shrimp Ammonia Intervention: 397.
Generic Black Soldier Fly Intervention: 46.
Generic Carp Intervention: 37.
Shrimp Slaughter Intervention: 10.
Generic Shrimp Intervention: 9.
Existential risk (the results change from run to run, but I think the values below represent the right order of magnitude):
Small-scale AI Misuse Project: 269 k.
Small-scale Nuclear Safety Project: 48 k.
Small-scale Nanotech Safety Project: 16 k.
Small-scale Biorisk Project: 8,915.
Portfolio of Biorisk Projects: 5,919.
Nanotech Safety Megaproject: 3,199.
Small-scale AI Misalignment Project: 1,718.
AI Misalignment Megaproject: 1,611.
Small-scale Natural Disaster Prevention Project: 1,558.
Exploratory Research into Unknown Risks: 30.
I cannot help notice the cost-effectiveness of the project on preventing a natural disaster is of the same order of magnitude as that of the 2 projects on AI alignement. For readersâ context, AI safety technical research is 80,000 Hoursâ top career path, whereas one could argue extinction risk from natural disasters is astronomically low. I understand the results of the CCM are tentative, but this result makes me think I should put very little trust on the default parameters of the projects aiming to reduce existential risk.
For draws, I present the estimates by alphabetical order.
I think you should put very little trust in the default parameters of the projects. It was our initial intention to create defaults that reflected the best evidence and expert opinion, but we had difficulty getting consensus on what these values should be and decided instead to explicitly stand back from the defaults. The parameter settings are adjustable to suit your views, and we encourage people to think about what those parameter settings should be and not take the defaults too seriously.
The parameters allow you to control how far into the future you look and the outcomes include not just effects on the long-term future from the extinction /â preservation of the species but also on the probabilities of near-term catastrophes that cause large numbers of death but donât cause extinction. Depending on your settings, near-term catastrophes can dominate the expected value. For the default settings for natural disasters and bio-risk, much of the value of mitigation work (at least over the next 1000 years) comes from prevention of relatively small-scale disasters. I donât see anything obviously wrong about this result and I expect that 80Kâs outlook is based on a willingness to consider effects more than 1000 years in the future.
I happen to disagree with these numbers because I think that numbers for effectiveness of x-risk projects are too low. E.g., for the âSmall-scale AI Misalignment Projectâ: âwe expect that it reduces absolute existential risk by a factor between 0.000001 and 0.000055âł, these seem like many zeroes to me.
Ditto for the âAI Misalignment Megaprojectâ: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
I think weâre somewhat bearish on the ability of money by itself to solve problems. The technical issues around alignment appear quite challenging, especially given the pace of development, so it isnât clear that any amount of money will be able to solve them. If the issues are too easy on the other hand, then your investment of money is unlikely to be needed and so your expenditure isnât going to reduce extinction risk.
Even if the technical issues are in the goldilocks spot of being solvable but not trivially so, the political challenges around getting those solutions adopted seem extremely daunting. There is a lot we donât explicitly specify in these parameter settings: if the money is coming from a random billionaire unaffiliated with AI scene then it might be harder to get expertise and buy in then if it is coming from insiders or the federal government.
All that said, it is plausible to me that we should have a somewhat higher chance of having an impact coupled with a lower chance of a positive outcome. A few billion dollars is likely to shake things up even if the outcome isnât what we hoped for.
Is that 3% an absolute percentage point reduction in risk? If so, that doesnât seem very low if your baseline risk estimate is low, like 5-20%, or youâre as pessimistic about aligning AI as MIRI is.
No, 3% is âchance of successâ. After adding a bunch of multipliers, it comes to about 0.6% reduction in existential risk over the next century, for $8B to $20B.
2 nitpicks that end up arguing in favor of your high-level point
2.7% (which youâre rounding up to 3%) is chance of having an effect, and 70% x 2.7% = 1.9% is chance of positive effect (âsuccessâ by your wording)
your Squiggle calc doesnât include the CCMâs âintervention backfiringâ part of the calc