A 6 line argument for AGI risk
(1) Sufficient intelligence has capitalities that are ultimately limited by physics and computability
(2) An AGI could be sufficiently intelligent that it’s limited by physics and computability but humans can’t be
(3) An AGI will come into existence
(4) If the AGIs goals aren’t the same as humans, human goals will only be met for instrumental reasons and the AGIs goals will be met
(5) Meeting human goals won’t be instrumentally useful in the long run for an unaligned AGI
(6) It is more morally valuable for human goals to be met than an AGIs goals
A 6 line argument for AGI risk
(1) Sufficient intelligence has capitalities that are ultimately limited by physics and computability
(2) An AGI could be sufficiently intelligent that it’s limited by physics and computability but humans can’t be
(3) An AGI will come into existence
(4) If the AGIs goals aren’t the same as humans, human goals will only be met for instrumental reasons and the AGIs goals will be met
(5) Meeting human goals won’t be instrumentally useful in the long run for an unaligned AGI
(6) It is more morally valuable for human goals to be met than an AGIs goals