Re your argument for default-success, you only need to have 97% certainty for 1-4 if every step was independent, which they aren’t.
I’m pretty sure this isn’t true. To be clear, I was talking about conditional probabilities, the probability of each occurring, given that the previous steps had already occurred.
Consider me making an estimate like “theres a 90% chance you complete this triathlon (without dropping out)”. In order to complete the triathlon as a whole, I need to complete the swimming, cycling and running in turn.
To get to 90% probability overall, I might estimate that you have a 95% chance of completing the swimming portion, a 96% chance of completing the cycling portion given that you finished the swimming portion, and a 99% chance of you completing the running portion, given that you finished the swimming and running portion. Total probability is 0.95*0.96*0.99=0.90.
The different events are correlated (a fit person will find all three easier than an unfit person), but that’s taken care of in the conditional nature of the calculation. It’s also possible that uncertainty is correlated (If I find out you have a broken leg, all three of my estimates will probably go down, even though they are conditional).
With regards to the doomsday scenario, the point is that there are several possible exit ramps (the AI doesn’t get built, it isn’t malevolent, it can’t kill us all). If you want to be fairly certain that no exit ramps are taken, you have to be very certain that each individual exit ramp won’t get taken.
That’s fair; upon re-reading your comment it’s actually pretty obvious you meant the conditional probability, in which case I agree multiplying is fine.
I think the conditional statements are actually straightforward—e.g. once we’ve built something far more capable than humanity, and that system “rebels” against us, it’s pretty certain that we lose, and point (2) is the classic question of how hard alignment is. Your point (1) about whether we build far-superhuman AGI in the next 30 years or so seems like the most uncertain one here.
Yeah, no worries, I was afraid I’d messed up the math for a second there!
It’s funny, I think my estimates are the opposite of yours, I think 1 is probably the most likely, whereas I view 3 as vastly unlikely. None of the proposed takeover scenarios seem within the realm of plausibility, at least in the near future. But I’ve already stated my case elsewhere.
I’m pretty sure this isn’t true. To be clear, I was talking about conditional probabilities, the probability of each occurring, given that the previous steps had already occurred.
Consider me making an estimate like “theres a 90% chance you complete this triathlon (without dropping out)”. In order to complete the triathlon as a whole, I need to complete the swimming, cycling and running in turn.
To get to 90% probability overall, I might estimate that you have a 95% chance of completing the swimming portion, a 96% chance of completing the cycling portion given that you finished the swimming portion, and a 99% chance of you completing the running portion, given that you finished the swimming and running portion. Total probability is 0.95*0.96*0.99=0.90.
The different events are correlated (a fit person will find all three easier than an unfit person), but that’s taken care of in the conditional nature of the calculation. It’s also possible that uncertainty is correlated (If I find out you have a broken leg, all three of my estimates will probably go down, even though they are conditional).
With regards to the doomsday scenario, the point is that there are several possible exit ramps (the AI doesn’t get built, it isn’t malevolent, it can’t kill us all). If you want to be fairly certain that no exit ramps are taken, you have to be very certain that each individual exit ramp won’t get taken.
That’s fair; upon re-reading your comment it’s actually pretty obvious you meant the conditional probability, in which case I agree multiplying is fine.
I think the conditional statements are actually straightforward—e.g. once we’ve built something far more capable than humanity, and that system “rebels” against us, it’s pretty certain that we lose, and point (2) is the classic question of how hard alignment is. Your point (1) about whether we build far-superhuman AGI in the next 30 years or so seems like the most uncertain one here.
Yeah, no worries, I was afraid I’d messed up the math for a second there!
It’s funny, I think my estimates are the opposite of yours, I think 1 is probably the most likely, whereas I view 3 as vastly unlikely. None of the proposed takeover scenarios seem within the realm of plausibility, at least in the near future. But I’ve already stated my case elsewhere.