Executive summary: The post examines whether advanced AI systems are likely to be agentic and power-seeking by default, reviewing relevant philosophical arguments and identifying key open questions and possible scenarios.
Key points:
Agency can be understood in terms of beliefs, desires, and intentions, with agents differing along spectrums of rationality and representational complexity.
The argument that agentic AI will seek power by default relies on instrumental convergence, but this may only weakly hold or not apply to the goals advanced AI is likely to have.
Drexler’s Comprehensive AI Services model suggests advanced AI need not be agentic or power-seeking by default, but there may be pressures in that direction.
Key open questions include: whether advanced AI will be agentic and rational by default, what goals it is likely to have, and if rationality is itself convergently instrumental.
Possible scenarios range from existential catastrophe being the default outcome, to advanced AI systems being more like tools than potentially risky agents.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, andcontact us if you have feedback.
Executive summary: The post examines whether advanced AI systems are likely to be agentic and power-seeking by default, reviewing relevant philosophical arguments and identifying key open questions and possible scenarios.
Key points:
Agency can be understood in terms of beliefs, desires, and intentions, with agents differing along spectrums of rationality and representational complexity.
The argument that agentic AI will seek power by default relies on instrumental convergence, but this may only weakly hold or not apply to the goals advanced AI is likely to have.
Drexler’s Comprehensive AI Services model suggests advanced AI need not be agentic or power-seeking by default, but there may be pressures in that direction.
Key open questions include: whether advanced AI will be agentic and rational by default, what goals it is likely to have, and if rationality is itself convergently instrumental.
Possible scenarios range from existential catastrophe being the default outcome, to advanced AI systems being more like tools than potentially risky agents.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.