Ah yeah, you’re right—I think basically I put in the percent rather than the probability. So it would indeed be very expensive to be competitive with AMF. Though so is everything else, so that’s not hugely surprising.
As for the numbers, yeah, it does just strike me as really, really unlikely that we can solve AI x-risk right now. 1⁄10,000 does feel about right to me. I certainly wouldn’t expect everyone else to agree though! I think some people would put the odds much higher, and others (like Tyler Cowen maybe?) would put them a bit lower. Probably the 1% step is the step I’m least confident in—wouldn’t surprise me if the (hard to find, hard to execute) solutions that are findable would reduce risk significantly more.
EDIT: tried to fix the math and switched the “relative risk reduction term” to 10%. I feel like among findable, executable interventions there’s probably a lot of variance, and it’s plausible some of the best ones do reduce risk by 10% or so. And 1/1000 feels about as plausible as 1/10000 to me. So, somewhere in there.
it does just strike me as really, really unlikely that we can solve AI x-risk right now
I think Erik wasn’t commenting so much on this number, but rather its combination with the assumption that there is a 94% chance things are fine by default.
I.e. you are assuming that there is a 94% chance it’s trivially easy, and 6% chance it’s insanely hard.
Very few problems have such a bimodal nature, and I also would be interested to understand what’s generating it for you.
Ah yeah, you’re right—I think basically I put in the percent rather than the probability. So it would indeed be very expensive to be competitive with AMF. Though so is everything else, so that’s not hugely surprising.
As for the numbers, yeah, it does just strike me as really, really unlikely that we can solve AI x-risk right now. 1⁄10,000 does feel about right to me. I certainly wouldn’t expect everyone else to agree though! I think some people would put the odds much higher, and others (like Tyler Cowen maybe?) would put them a bit lower. Probably the 1% step is the step I’m least confident in—wouldn’t surprise me if the (hard to find, hard to execute) solutions that are findable would reduce risk significantly more.
EDIT: tried to fix the math and switched the “relative risk reduction term” to 10%. I feel like among findable, executable interventions there’s probably a lot of variance, and it’s plausible some of the best ones do reduce risk by 10% or so. And 1/1000 feels about as plausible as 1/10000 to me. So, somewhere in there.
I think Erik wasn’t commenting so much on this number, but rather its combination with the assumption that there is a 94% chance things are fine by default.
I.e. you are assuming that there is a 94% chance it’s trivially easy, and 6% chance it’s insanely hard.
Very few problems have such a bimodal nature, and I also would be interested to understand what’s generating it for you.