The flaws and bugs that are most relevant to an AI’s performance in it’s domain of focus will be weeded out, but flaws outside of it’s relevant domain will not be. Bobby Fischer’s insane conspiracism had no effect on his chess playing ability. The same principle applies to stockfish. “Idiot savant” AI’s are entirely plausible, even likely.
[...]
For these reasons, I expect AGI to be flawed, and especially flawed when doing things it was not originally meant to do, like conquer the entire planet.
We might actually expect an AGI to be trained to conquer the entire planet, or rather to be trained in many of the abilities needed to do so. For example, we may train it to be good at things like:
Strategic planning
Getting humans to do what it wants effectively
Controlling physical systems
Cybersecurity
Researching new, powerful technologies
Engineering
Running large organizations
Communicating with humans and other AIs
Put differently, I think “taking control over humans” and “running a multinational corporation” (which seems like the sort of thing people will want AIs to be able to do) have lots more overlap than “playing chess” and “having true beliefs about subjects of conspiracies”. I’d be curious to hear if you have thoughts about which specific abilities you expect an AGI would need to have to take control over humanity that it’s unlikely to actually possess?
We might actually expect an AGI to be trained to conquer the entire planet, or rather to be trained in many of the abilities needed to do so. For example, we may train it to be good at things like:
Strategic planning
Getting humans to do what it wants effectively
Controlling physical systems
Cybersecurity
Researching new, powerful technologies
Engineering
Running large organizations
Communicating with humans and other AIs
Put differently, I think “taking control over humans” and “running a multinational corporation” (which seems like the sort of thing people will want AIs to be able to do) have lots more overlap than “playing chess” and “having true beliefs about subjects of conspiracies”. I’d be curious to hear if you have thoughts about which specific abilities you expect an AGI would need to have to take control over humanity that it’s unlikely to actually possess?