Update: I’m pleased to learn Yudkowsky seems to have suggested a similar agenda in a recent interview with Dwarkesh Patel (timestamp) as his greatest source of predictable hope about AI. It’s a rather fragmented bit but the gist is: Perhaps people doing RLHF get a better grasp on what to aim for by studying where “niceness” comes from in humans. He’s inspired by the idea that “consciousness is when the mask eats the shoggoth” and suggests, “maybe with the right bootstrapping you can let that happen on purpose”.
I see a very important point here: Human intelligence isn’t misaligned with evolution in a random direction, it is misaligned in the direction of maximizing positive qualia. Therefore, it seems very likely that consciousness played a causal role in the evolution of human moral alignment—and such causal role needs to be possible to study.
Update: I’m pleased to learn Yudkowsky seems to have suggested a similar agenda in a recent interview with Dwarkesh Patel (timestamp) as his greatest source of predictable hope about AI. It’s a rather fragmented bit but the gist is: Perhaps people doing RLHF get a better grasp on what to aim for by studying where “niceness” comes from in humans. He’s inspired by the idea that “consciousness is when the mask eats the shoggoth” and suggests, “maybe with the right bootstrapping you can let that happen on purpose”.
I see a very important point here: Human intelligence isn’t misaligned with evolution in a random direction, it is misaligned in the direction of maximizing positive qualia. Therefore, it seems very likely that consciousness played a causal role in the evolution of human moral alignment—and such causal role needs to be possible to study.