Hi, my name is James Fodor. I am a longtime student and EA organiser from Melbourne. I love science, history, philosophy, and using these to make a difference in the world.
James Fodor
Karma: 1,277
Thanks for your thoughts!
1) The idea I’m getting at is that an exponential-type argument of self-improvement ability being proportional to current intelligence doesn’t really work if there are multiple distinct and separate cognitive abilities, because ability to improve ability X might not be in any clear way related to the current level of X. For example, ability to design a better chess-playing program might not be in any way related to chess playing ability, or object recognition performance might not be related to ability to improve this performance. These are probably not very good examples because probably these sorts of abilities are not fundamental enough, and we should be looking at more abstract cognitive abilities, but hopefully they serve as a general illustration. A superhuman AI would therefore be better at designing AIs than a human sure, but I don’t think the sort of exponential growth arguments Bostrom uses hold if there are multiple distinct cognitive abilities.
2) The idea of a simplistic paper-maximising AI instantiating separate mind simulations is very interesting. I think the way you describe it this would amount to one agent creating distinct agents to perform a set task, rather than a single agent possessing those actual abilities itself. This seems relevant to me because any created mind simulations, being distinct from the original agent, would not necessarily share its goals or beliefs, and therefore a principal-agent problem arises. In order to be smart enough to solve this problem I think the original AI would probably have to be enhanced well beyond paperclip maximising levels. I think there’s a lot more to be said here but I am not convinced this counterexample really und
Thanks for your thoughts.
Regarding your first point, I agree that the situation you posit is a possibility, but it isn’t something Bostrom talks about (and remember I only focused on what he argued, not other possible expansions of the argument). Also, when we consider the possibility of numerous distinct cognitive abilities it is just as possible that there could be complex interactions which inhibit the growth of particular abilities. There could easily be dozens of separate abilities and the full matrix of interactions becomes very complex. The original force of the ‘rate of growth of intelligence is proportional to current intelligence leading to exponential growth’ argument is, in my view, substantively blunted.
Regarding your second point, it seems unlikely to me because if an agent had all these abilities, I believe they would use then to uncover reasons to reject highly reductionistic goals like tilling the universe with paperclips. They might end up with goals that are still in opposition to human values, but I just don’t see how an agent with these abilities would not become dissatisfied with extremely narrow goals.