This talk by Jade Leung got me thinkingâIâve never seen a plan for what we do if AGI turns out misaligned.
The default assumption seems to be something like âwell, thereâs no point planning for that, because weâll all be powerless and screwedâ. This seems mistaken to me. Itâs not clear that weâll be so powerless that we have absolutely no ability to encourage a trajectory change, particularly in a slow takeoff scenario. Given that most people weight alleviating suffering higher than promoting pleasure, this is especially valuable work in expectation as it might help us change outcomes from âvery, very bad worldâ to âslightly negativeâ world. This also seems pretty tractableâIâd expect ~10hrs thinking about this could help us come up with a very barebones playbook.
Why isnât this being done? I think there are a few reasons:
Like suffering focused ethics, itâs depressing.
It seems particularly speculativeâmost of the âhumanity becomes disempowered by AGIâ scenarios look pretty sci-fi. So serious academics donât want to consider it.
People assume, mistakenly IMO, that weâre just totally screwed if AI is misaligned.
No Plans for Misaligned AI:
This talk by Jade Leung got me thinkingâIâve never seen a plan for what we do if AGI turns out misaligned.
The default assumption seems to be something like âwell, thereâs no point planning for that, because weâll all be powerless and screwedâ. This seems mistaken to me. Itâs not clear that weâll be so powerless that we have absolutely no ability to encourage a trajectory change, particularly in a slow takeoff scenario. Given that most people weight alleviating suffering higher than promoting pleasure, this is especially valuable work in expectation as it might help us change outcomes from âvery, very bad worldâ to âslightly negativeâ world. This also seems pretty tractableâIâd expect ~10hrs thinking about this could help us come up with a very barebones playbook.
Why isnât this being done? I think there are a few reasons:
Like suffering focused ethics, itâs depressing.
It seems particularly speculativeâmost of the âhumanity becomes disempowered by AGIâ scenarios look pretty sci-fi. So serious academics donât want to consider it.
People assume, mistakenly IMO, that weâre just totally screwed if AI is misaligned.