Agree. Something that clarified my thinking on this (still feel pretty confused!) is Katja Grace’s counterarguments to basic AI x-risk case. In particular the section on “Different calls to ‘goal-directedness’ don’t necessarily mean the same concept” and discussions about “pseduo-agents” clarified how there are other ways for agents to take actions than purely optimizing a utility functions (which humans don’t do).
Agree. Something that clarified my thinking on this (still feel pretty confused!) is Katja Grace’s counterarguments to basic AI x-risk case. In particular the section on “Different calls to ‘goal-directedness’ don’t necessarily mean the same concept” and discussions about “pseduo-agents” clarified how there are other ways for agents to take actions than purely optimizing a utility functions (which humans don’t do).