There seem to be two main framings emerging from recent AGI x-risk discussion: default doom, given AGI, and default we’re fine, given AGI.
I’m interested in what people who have low p(doom|AGI) think are the reasons that things will basically be fine once we have AGI (or TAI, PASTA, ASI). What mechanisms are at play? How is alignment solved so that there are 0 failure modes? Can we survive despite imperfect alignment? How? Is alignment moot? Will physical limits be reached before there is too much danger?
If you have high enough p(doom|AGI) to be very concerned, but you’re still only at ~1-10%, what is happening in the other 90-99%?
Added 22Apr: I’m also interested in detailed scenarios and stories, spelling out how things go right post-AGI. There are plenty of storiesandscenariosillustratingdoom. Where are the similar stories illustrating how things go right? There is the FLI World Building Contest, but that took place in the pre-GPT-4+AutoGPT era. The winning entry has everyone acting far too sensibly in terms of self-regulation and restraint. I think we can now say, given the fervour over AutoGPT, that this will not happen, with high likelihood.
[Question] If your AGI x-risk estimates are low, what scenarios make up the bulk of your expectations for an OK outcome?
There seem to be two main framings emerging from recent AGI x-risk discussion: default doom, given AGI, and default we’re fine, given AGI.
I’m interested in what people who have low p(doom|AGI) think are the reasons that things will basically be fine once we have AGI (or TAI, PASTA, ASI). What mechanisms are at play? How is alignment solved so that there are 0 failure modes? Can we survive despite imperfect alignment? How? Is alignment moot? Will physical limits be reached before there is too much danger?
If you have high enough p(doom|AGI) to be very concerned, but you’re still only at ~1-10%, what is happening in the other 90-99%?
Added 22Apr: I’m also interested in detailed scenarios and stories, spelling out how things go right post-AGI. There are plenty of stories and scenarios illustrating doom. Where are the similar stories illustrating how things go right? There is the FLI World Building Contest, but that took place in the pre-GPT-4+AutoGPT era. The winning entry has everyone acting far too sensibly in terms of self-regulation and restraint. I think we can now say, given the fervour over AutoGPT, that this will not happen, with high likelihood.