1) The idea I’m getting at is that an exponential-type argument of self-improvement ability being proportional to current intelligence doesn’t really work if there are multiple distinct and separate cognitive abilities, because ability to improve ability X might not be in any clear way related to the current level of X. For example, ability to design a better chess-playing program might not be in any way related to chess playing ability, or object recognition performance might not be related to ability to improve this performance. These are probably not very good examples because probably these sorts of abilities are not fundamental enough, and we should be looking at more abstract cognitive abilities, but hopefully they serve as a general illustration. A superhuman AI would therefore be better at designing AIs than a human sure, but I don’t think the sort of exponential growth arguments Bostrom uses hold if there are multiple distinct cognitive abilities.
2) The idea of a simplistic paper-maximising AI instantiating separate mind simulations is very interesting. I think the way you describe it this would amount to one agent creating distinct agents to perform a set task, rather than a single agent possessing those actual abilities itself. This seems relevant to me because any created mind simulations, being distinct from the original agent, would not necessarily share its goals or beliefs, and therefore a principal-agent problem arises. In order to be smart enough to solve this problem I think the original AI would probably have to be enhanced well beyond paperclip maximising levels. I think there’s a lot more to be said here but I am not convinced this counterexample really und
Thanks for your thoughts!
1) The idea I’m getting at is that an exponential-type argument of self-improvement ability being proportional to current intelligence doesn’t really work if there are multiple distinct and separate cognitive abilities, because ability to improve ability X might not be in any clear way related to the current level of X. For example, ability to design a better chess-playing program might not be in any way related to chess playing ability, or object recognition performance might not be related to ability to improve this performance. These are probably not very good examples because probably these sorts of abilities are not fundamental enough, and we should be looking at more abstract cognitive abilities, but hopefully they serve as a general illustration. A superhuman AI would therefore be better at designing AIs than a human sure, but I don’t think the sort of exponential growth arguments Bostrom uses hold if there are multiple distinct cognitive abilities.
2) The idea of a simplistic paper-maximising AI instantiating separate mind simulations is very interesting. I think the way you describe it this would amount to one agent creating distinct agents to perform a set task, rather than a single agent possessing those actual abilities itself. This seems relevant to me because any created mind simulations, being distinct from the original agent, would not necessarily share its goals or beliefs, and therefore a principal-agent problem arises. In order to be smart enough to solve this problem I think the original AI would probably have to be enhanced well beyond paperclip maximising levels. I think there’s a lot more to be said here but I am not convinced this counterexample really und