I happen to disagree with these numbers because I think that numbers for effectiveness of x-risk projects are too low. E.g., for the “Small-scale AI Misalignment Project”: “we expect that it reduces absolute existential risk by a factor between 0.000001 and 0.000055″, these seem like many zeroes to me.
Ditto for the “AI Misalignment Megaproject”: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
Ditto for the “AI Misalignment Megaproject”: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
I think we’re somewhat bearish on the ability of money by itself to solve problems. The technical issues around alignment appear quite challenging, especially given the pace of development, so it isn’t clear that any amount of money will be able to solve them. If the issues are too easy on the other hand, then your investment of money is unlikely to be needed and so your expenditure isn’t going to reduce extinction risk.
Even if the technical issues are in the goldilocks spot of being solvable but not trivially so, the political challenges around getting those solutions adopted seem extremely daunting. There is a lot we don’t explicitly specify in these parameter settings: if the money is coming from a random billionaire unaffiliated with AI scene then it might be harder to get expertise and buy in then if it is coming from insiders or the federal government.
All that said, it is plausible to me that we should have a somewhat higher chance of having an impact coupled with a lower chance of a positive outcome. A few billion dollars is likely to shake things up even if the outcome isn’t what we hoped for.
Is that 3% an absolute percentage point reduction in risk? If so, that doesn’t seem very low if your baseline risk estimate is low, like 5-20%, or you’re as pessimistic about aligning AI as MIRI is.
No, 3% is “chance of success”. After adding a bunch of multipliers, it comes to about 0.6% reduction in existential risk over the next century, for $8B to $20B.
I happen to disagree with these numbers because I think that numbers for effectiveness of x-risk projects are too low. E.g., for the “Small-scale AI Misalignment Project”: “we expect that it reduces absolute existential risk by a factor between 0.000001 and 0.000055″, these seem like many zeroes to me.
Ditto for the “AI Misalignment Megaproject”: $8B+ expenditure to only have a 3% chance of success (?!), plus some other misc discounting factors. Seems like you could do better with $8B.
I think we’re somewhat bearish on the ability of money by itself to solve problems. The technical issues around alignment appear quite challenging, especially given the pace of development, so it isn’t clear that any amount of money will be able to solve them. If the issues are too easy on the other hand, then your investment of money is unlikely to be needed and so your expenditure isn’t going to reduce extinction risk.
Even if the technical issues are in the goldilocks spot of being solvable but not trivially so, the political challenges around getting those solutions adopted seem extremely daunting. There is a lot we don’t explicitly specify in these parameter settings: if the money is coming from a random billionaire unaffiliated with AI scene then it might be harder to get expertise and buy in then if it is coming from insiders or the federal government.
All that said, it is plausible to me that we should have a somewhat higher chance of having an impact coupled with a lower chance of a positive outcome. A few billion dollars is likely to shake things up even if the outcome isn’t what we hoped for.
Is that 3% an absolute percentage point reduction in risk? If so, that doesn’t seem very low if your baseline risk estimate is low, like 5-20%, or you’re as pessimistic about aligning AI as MIRI is.
No, 3% is “chance of success”. After adding a bunch of multipliers, it comes to about 0.6% reduction in existential risk over the next century, for $8B to $20B.
2 nitpicks that end up arguing in favor of your high-level point
2.7% (which you’re rounding up to 3%) is chance of having an effect, and 70% x 2.7% = 1.9% is chance of positive effect (‘success’ by your wording)
your Squiggle calc doesn’t include the CCM’s ‘intervention backfiring’ part of the calc