The reference classes I look at generate a prior for AGI control over current human resources anywhere between 5% and 60% (mean of ~16-26%).
Thanks for this Zach. I found it quite thought provoking, especially the quoted sentence.
Based on your model, AGI controlling human resources is much more likely to occur than extinction. Given that, what events do you think we should be worried about with losing autonomy over resources (and potentially institutions) and are you more concerned about that after this work?
I take 5%-60% as an estimate of how much of human civilization’s future value will depend on what AI systems do, but it does not necessarily exclude human autonomy. If humans determine what AI systems do with the resources they acquire and the actions they take, then AI could be extremely important, and humans would still retain autonomy.
I don’t think this really left me more or less concerned about losing autonomy over resources. It does feel like this exercise made it starker that there’s a large chance of AI reshaping the world beyond human extinction. It’s not clear how much of that means the loss of human autonomy. I’m inclined to think in rough, nebulous terms that AI will erode human autonomy over 10% of our future, taking 10% as a sort of midpoint between the extinction likelihood and the degree of AI influence over our future. I think my previous views would have been in that ballpark.
The exercise did lead me to think the importance of AI is higher than I previously did and the likelihood of extinction per se is lower (though my final beliefs place all these probabilities higher than the priors in the report).
Thanks for this Zach. I found it quite thought provoking, especially the quoted sentence.
Based on your model, AGI controlling human resources is much more likely to occur than extinction. Given that, what events do you think we should be worried about with losing autonomy over resources (and potentially institutions) and are you more concerned about that after this work?
I take 5%-60% as an estimate of how much of human civilization’s future value will depend on what AI systems do, but it does not necessarily exclude human autonomy. If humans determine what AI systems do with the resources they acquire and the actions they take, then AI could be extremely important, and humans would still retain autonomy.
I don’t think this really left me more or less concerned about losing autonomy over resources. It does feel like this exercise made it starker that there’s a large chance of AI reshaping the world beyond human extinction. It’s not clear how much of that means the loss of human autonomy. I’m inclined to think in rough, nebulous terms that AI will erode human autonomy over 10% of our future, taking 10% as a sort of midpoint between the extinction likelihood and the degree of AI influence over our future. I think my previous views would have been in that ballpark.
The exercise did lead me to think the importance of AI is higher than I previously did and the likelihood of extinction per se is lower (though my final beliefs place all these probabilities higher than the priors in the report).