The trouble isn’t that AI can’t have complex or vague goals, it’s that there’s no reason why having more complicated and vague goals makes something less dangerous.
Think of it this way: A lion has complicated and vague goals. It is messy, organic, and not “programmed”. Does that mean that a lion is safe? Would you be afraid to be locked in a cage with a lion? I would be.
Humans and lions both have complicated and sometimes vague goals, but because their goals are not the same goals, both beings pose a severe danger to each other all the same. The lion is dangerous to the human because the lion is stronger than the human. The human is dangerous to the lion because the human is smarter than the lion.
Where most people go wrong is that they think that smart means nice, so they think that if only the lion in this analogy was smart too, then it would magically also be safe. They don’t imagine that a smart lion might want to eat you just the same as a regular lion.
In order to make a lion safe, you need to either control its values, so that it doesn’t want to harm you, or you need to make it more predictable.
Furthermore, a lion becomes more dangerous as it becomes more intelligent and capable, even if its terminal goal is not “maximize number of wildebeests eaten”.
The trouble isn’t that AI can’t have complex or vague goals, it’s that there’s no reason why having more complicated and vague goals makes something less dangerous.
Think of it this way: A lion has complicated and vague goals. It is messy, organic, and not “programmed”. Does that mean that a lion is safe? Would you be afraid to be locked in a cage with a lion? I would be.
Humans and lions both have complicated and sometimes vague goals, but because their goals are not the same goals, both beings pose a severe danger to each other all the same. The lion is dangerous to the human because the lion is stronger than the human. The human is dangerous to the lion because the human is smarter than the lion.
Where most people go wrong is that they think that smart means nice, so they think that if only the lion in this analogy was smart too, then it would magically also be safe. They don’t imagine that a smart lion might want to eat you just the same as a regular lion.
In order to make a lion safe, you need to either control its values, so that it doesn’t want to harm you, or you need to make it more predictable.
Thanks a lot! Very clear and helpful
Furthermore, a lion becomes more dangerous as it becomes more intelligent and capable, even if its terminal goal is not “maximize number of wildebeests eaten”.