In general I quite like this post, I think it elucidates some disagreements quite well. However, as an out and proud default-success guy, I’m not sure it represents the default-success argument on uncertainty well. To see why, let’s translate default-failure and default-success into their full beliefs:
Default-failure is the stance that, by default, highly intelligent AI systems will be developed, at least one of which rebels and successfully defeats and then murders/enslaves all of humanity.
Default-success is referring to a stance that, by default, that chain of events above won’t happen.
I think the argument of the default-success people is that uncertainty about the future means that you shouldn’t be default-failure. We’re saying that the uncertainty about the future should translate into uncertainty about:
1.whether AI systems are developed,
2. whether they rebel
3.whether they beat us
4.whether, having defeated us, they decide to kill/enslave us all.
In order to get to a 90% chance of doom, you need to estimate a 97% certainty of every single step in that process. Of course, it’s completely fine to have 97% confidence in something if you have a lot of evidence for it. I do not feel that doomers have anywhere close to this standard of evidence. I do agree that discussion is better pointed to discussing this evidence than gesturing to uncertainty, but I don’t think it’s a useless point to make.
In general I quite like this post, I think it elucidates some disagreements quite well.
Thanks!
I’m not sure it represents the default-success argument on uncertainty well.
I haven’t tried to make an object-level argument for either “AI risk is default-failure” or “AI risk is default-success” (sorry if that was unclear). See Nate’s post for the former.
Re your argument for default-success, you only need to have 97% certainty for 1-4 if every step was independent, which they aren’t.
I do agree that discussion is better pointed to discussing this evidence than gesturing to uncertainty
Re your argument for default-success, you only need to have 97% certainty for 1-4 if every step was independent, which they aren’t.
I’m pretty sure this isn’t true. To be clear, I was talking about conditional probabilities, the probability of each occurring, given that the previous steps had already occurred.
Consider me making an estimate like “theres a 90% chance you complete this triathlon (without dropping out)”. In order to complete the triathlon as a whole, I need to complete the swimming, cycling and running in turn.
To get to 90% probability overall, I might estimate that you have a 95% chance of completing the swimming portion, a 96% chance of completing the cycling portion given that you finished the swimming portion, and a 99% chance of you completing the running portion, given that you finished the swimming and running portion. Total probability is 0.95*0.96*0.99=0.90.
The different events are correlated (a fit person will find all three easier than an unfit person), but that’s taken care of in the conditional nature of the calculation. It’s also possible that uncertainty is correlated (If I find out you have a broken leg, all three of my estimates will probably go down, even though they are conditional).
With regards to the doomsday scenario, the point is that there are several possible exit ramps (the AI doesn’t get built, it isn’t malevolent, it can’t kill us all). If you want to be fairly certain that no exit ramps are taken, you have to be very certain that each individual exit ramp won’t get taken.
That’s fair; upon re-reading your comment it’s actually pretty obvious you meant the conditional probability, in which case I agree multiplying is fine.
I think the conditional statements are actually straightforward—e.g. once we’ve built something far more capable than humanity, and that system “rebels” against us, it’s pretty certain that we lose, and point (2) is the classic question of how hard alignment is. Your point (1) about whether we build far-superhuman AGI in the next 30 years or so seems like the most uncertain one here.
Yeah, no worries, I was afraid I’d messed up the math for a second there!
It’s funny, I think my estimates are the opposite of yours, I think 1 is probably the most likely, whereas I view 3 as vastly unlikely. None of the proposed takeover scenarios seem within the realm of plausibility, at least in the near future. But I’ve already stated my case elsewhere.
In general I quite like this post, I think it elucidates some disagreements quite well. However, as an out and proud default-success guy, I’m not sure it represents the default-success argument on uncertainty well. To see why, let’s translate default-failure and default-success into their full beliefs:
Default-failure is the stance that, by default, highly intelligent AI systems will be developed, at least one of which rebels and successfully defeats and then murders/enslaves all of humanity.
Default-success is referring to a stance that, by default, that chain of events above won’t happen.
I think the argument of the default-success people is that uncertainty about the future means that you shouldn’t be default-failure. We’re saying that the uncertainty about the future should translate into uncertainty about:
1.whether AI systems are developed,
2. whether they rebel
3.whether they beat us
4.whether, having defeated us, they decide to kill/enslave us all.
In order to get to a 90% chance of doom, you need to estimate a 97% certainty of every single step in that process. Of course, it’s completely fine to have 97% confidence in something if you have a lot of evidence for it. I do not feel that doomers have anywhere close to this standard of evidence. I do agree that discussion is better pointed to discussing this evidence than gesturing to uncertainty, but I don’t think it’s a useless point to make.
Thanks!
I haven’t tried to make an object-level argument for either “AI risk is default-failure” or “AI risk is default-success” (sorry if that was unclear). See Nate’s post for the former.
Re your argument for default-success, you only need to have 97% certainty for 1-4 if every step was independent, which they aren’t.
Agreed.
I’m pretty sure this isn’t true. To be clear, I was talking about conditional probabilities, the probability of each occurring, given that the previous steps had already occurred.
Consider me making an estimate like “theres a 90% chance you complete this triathlon (without dropping out)”. In order to complete the triathlon as a whole, I need to complete the swimming, cycling and running in turn.
To get to 90% probability overall, I might estimate that you have a 95% chance of completing the swimming portion, a 96% chance of completing the cycling portion given that you finished the swimming portion, and a 99% chance of you completing the running portion, given that you finished the swimming and running portion. Total probability is 0.95*0.96*0.99=0.90.
The different events are correlated (a fit person will find all three easier than an unfit person), but that’s taken care of in the conditional nature of the calculation. It’s also possible that uncertainty is correlated (If I find out you have a broken leg, all three of my estimates will probably go down, even though they are conditional).
With regards to the doomsday scenario, the point is that there are several possible exit ramps (the AI doesn’t get built, it isn’t malevolent, it can’t kill us all). If you want to be fairly certain that no exit ramps are taken, you have to be very certain that each individual exit ramp won’t get taken.
That’s fair; upon re-reading your comment it’s actually pretty obvious you meant the conditional probability, in which case I agree multiplying is fine.
I think the conditional statements are actually straightforward—e.g. once we’ve built something far more capable than humanity, and that system “rebels” against us, it’s pretty certain that we lose, and point (2) is the classic question of how hard alignment is. Your point (1) about whether we build far-superhuman AGI in the next 30 years or so seems like the most uncertain one here.
Yeah, no worries, I was afraid I’d messed up the math for a second there!
It’s funny, I think my estimates are the opposite of yours, I think 1 is probably the most likely, whereas I view 3 as vastly unlikely. None of the proposed takeover scenarios seem within the realm of plausibility, at least in the near future. But I’ve already stated my case elsewhere.
I’m curious what 3 (defeat) might look like without 4 happening?