- The functional ceiling for AGI is sufficiently above the current level of human civilization to eliminate it
I don’t think anyone is seriously arguing this? (Links please if they are).
- There is a sharp cutoff between non-AGI AI and AGI, such that early kind-of-AGI doesn’t send up enough warning signals to cause a drastic change in trajectory.
We are getting the warning signals now. People (including me) are raising the alarm. Hoping for a drastic change of trajectory, but people actually have to put the work in for that to happen! But your point here isn’t really related to P(doom|AGI) - i.e. the conditional is on getting AGI. Of course there won’t be doom if we don’t get AGI! That’s what we should be aiming for right now (not getting AGI).
- Early AGIs don’t result in a multi-polar world where superhuman-but-not-godlike agents can’t actually quickly and recursively self-improve, in part because none of them wants any of the others to take over—and without being able to grow stronger, humanity remains a viable player.
Nate may focus on singleton scenarios, but that is not a pre-requisite for doom. To me Robin Hanson’s (multipolar) Age of Em is also a kind of doom (most humans don’t exist, only a few highly productive ones are copied many times and only activated to work; a fully Malthusian economy). I don’t see how “humanity remains a viable player” in a world full of superhuman agents.
Thanks!
I don’t think anyone is seriously arguing this? (Links please if they are).
We are getting the warning signals now. People (including me) are raising the alarm. Hoping for a drastic change of trajectory, but people actually have to put the work in for that to happen! But your point here isn’t really related to P(doom|AGI) - i.e. the conditional is on getting AGI. Of course there won’t be doom if we don’t get AGI! That’s what we should be aiming for right now (not getting AGI).
Nate may focus on singleton scenarios, but that is not a pre-requisite for doom. To me Robin Hanson’s (multipolar) Age of Em is also a kind of doom (most humans don’t exist, only a few highly productive ones are copied many times and only activated to work; a fully Malthusian economy). I don’t see how “humanity remains a viable player” in a world full of superhuman agents.