Riffing on possible reasons to be hopeful, I recently compiled a list of potential “miracles” (including empirical “crucial considerations” [/wishful thinking]) that could mean the problem of AGI x-risk is bypassed:
Possibility of a failed (unaligned) takeoff scenario where the AI fails to model humans accurately enough (i.e. realise smart humans could detect its “hidden” activity in a certain way). [This may only set things back a few months to years; or could lead to some kind of Butlerian Jihad if there is a sufficiently bad (but ultimately recoverable) global catastrophe (and then much more time for Alignment the second time around?)].
Valence realism being true. Binding problem vs AGI Alignment.
Omega experiencing every possible consciousness and picking the best? [Could still lead to x-risk in terms of a Hedonium Shockwave].
Moral Realism being true (and the AI discovering it and the true morality being human-compatible).
Riffing on possible reasons to be hopeful, I recently compiled a list of potential “miracles” (including empirical “crucial considerations” [/wishful thinking]) that could mean the problem of AGI x-risk is bypassed:
Possibility of a failed (unaligned) takeoff scenario where the AI fails to model humans accurately enough (i.e. realise smart humans could detect its “hidden” activity in a certain way). [This may only set things back a few months to years; or could lead to some kind of Butlerian Jihad if there is a sufficiently bad (but ultimately recoverable) global catastrophe (and then much more time for Alignment the second time around?)].
Valence realism being true. Binding problem vs AGI Alignment.
Omega experiencing every possible consciousness and picking the best? [Could still lead to x-risk in terms of a Hedonium Shockwave].
Moral Realism being true (and the AI discovering it and the true morality being human-compatible).
Natural abstractions leading to Alignment by Default?
Rohin’s links here.
AGI discovers new physics and exits to another dimension (like the creatures in Greg Egan’s Crystal Nights).
Simulation/anthropics stuff.
Alien Information Theory being true!? (And the aliens having solved alignment).
I don’t think I put more than 10% probability on them collectively though, and my P(doom) is high enough to consider it “crunch time”.