If we get AGI, why might it pose a risk? What are the different components of that risk?
If and how are risks from AGI distinct from the kinds of risks we face from other people? The problem “autonomous agent wants something different from you” is just the everyday challenge of dealing with people.
It’s a (x-)risk once the AGI is much smarter than all of us humans put together (we are indifferent to ants). It being much smarter is the key distinction vs risks from other people. GPT-4 isn’t the problem; GPT-5 is. Re components, see my reply to you elsewhere in thread (Orthogonality Thesis, Mesa-optimisation, Basic AI drives; or outer alignment, inner alignment and power seeking).
For the different risks from AI, how might we solve each of them? What are the challenges to implementing those solutions? I.e. when is the problem engineering, incentives, etc?
There are many approaches, but the challenge imo is making any of them 100% water-tight, and we are very far from that with no complete roadmap in sight. 99% isn’t going to cut it when the AGI is far smarter than us and one misaligned execution of an instruction is enough to doom us all.
If we get AGI, why might it pose a risk? What are the different components of that risk?
If and how are risks from AGI distinct from the kinds of risks we face from other people? The problem “autonomous agent wants something different from you” is just the everyday challenge of dealing with people.
It’s a (x-)risk once the AGI is much smarter than all of us humans put together (we are indifferent to ants). It being much smarter is the key distinction vs risks from other people. GPT-4 isn’t the problem; GPT-5 is. Re components, see my reply to you elsewhere in thread (Orthogonality Thesis, Mesa-optimisation, Basic AI drives; or outer alignment, inner alignment and power seeking).
For the different risks from AI, how might we solve each of them? What are the challenges to implementing those solutions? I.e. when is the problem engineering, incentives, etc?
There are many approaches, but the challenge imo is making any of them 100% water-tight, and we are very far from that with no complete roadmap in sight. 99% isn’t going to cut it when the AGI is far smarter than us and one misaligned execution of an instruction is enough to doom us all.