Thanks for writing this Robert, I found it interesting! I just have one thought about a potential conclusion that may follow from your model.
I think you correctly note that friendly super-intelligent AI may be amongst the best ways to increase p3 (probability of optimal future). I have a feeling that friendly super-intelligent AI may also be one of the best ways to increase p (probability of achieving existential security). I guess I just don’t trust us humans to achieve existential security ourselves.
If that’s true I guess your model provides robustness to the claim that we should ensure development of AI goes well, and not just because if it doesn’t it might pose an x-risk. In essence there are three reasons why a focus on AI may be important:
AI poses an x-risk itself
(Well-done) AI may increase the probability of us achieving existential security
(Well-done) AI may increase the probability of us achieving an optimal future
I’d be interested to hear others thoughts on potential conclusions that follow from Robert’s model.
Thanks for your comment Jack, that’s a really great point. I suppose that we would seek to influence AI slightly differently for each reason:
Reduce chance of unaligned/uncontrolled AI
Increase chance of useful AI
Increase chance of exactly aligned AI
e.g. you could reduce the chance of AI risk by stopping all AI development but then lose the other two benefits, or you could create a practically useful AI but not one that would guide humanity towards an optimal future. That being said I reckon in practice a lot of work to improve the development of AI would hit all 3. Though maybe if you view one reason as much more important than the others then you focus on a specific type of AI work.
Thanks for writing this Robert, I found it interesting! I just have one thought about a potential conclusion that may follow from your model.
I think you correctly note that friendly super-intelligent AI may be amongst the best ways to increase p3 (probability of optimal future). I have a feeling that friendly super-intelligent AI may also be one of the best ways to increase p (probability of achieving existential security). I guess I just don’t trust us humans to achieve existential security ourselves.
If that’s true I guess your model provides robustness to the claim that we should ensure development of AI goes well, and not just because if it doesn’t it might pose an x-risk. In essence there are three reasons why a focus on AI may be important:
AI poses an x-risk itself
(Well-done) AI may increase the probability of us achieving existential security
(Well-done) AI may increase the probability of us achieving an optimal future
I’d be interested to hear others thoughts on potential conclusions that follow from Robert’s model.
Thanks for your comment Jack, that’s a really great point. I suppose that we would seek to influence AI slightly differently for each reason:
Reduce chance of unaligned/uncontrolled AI
Increase chance of useful AI
Increase chance of exactly aligned AI
e.g. you could reduce the chance of AI risk by stopping all AI development but then lose the other two benefits, or you could create a practically useful AI but not one that would guide humanity towards an optimal future. That being said I reckon in practice a lot of work to improve the development of AI would hit all 3. Though maybe if you view one reason as much more important than the others then you focus on a specific type of AI work.
That makes sense. I’m no expert in AI but I would think:
Stopping AI development just isn’t going to happen.
Making useful AI isn’t very neglected and progress here has certainly been quite impressive, so I’m optimistic for superintelligence at some point.
There probably isn’t much (if any) difference between the work that is required to make aligned AI and to make maximally-aligned AI.
Would be interesting to know if anyone thinks I’m wrong on any one of these points.