Yeah, I think “ASI implies an extreme case of lock-in” is a major tendency in the literature (especially sequences-era), but 1. people disagree about whether “alignment” refers to something that outsmarts even this implication or not, then they disagree about relative tractability and plausibility of the different alignment visions, and 2. this is very much a separate set of steps that provide room for disagreement among people who broadly accept Eliezer-like threatmodels (doomcoin stuff).
I don’t want to zero in on actually-existing Eliezer (at whichever time step), I’m more interested in like a threatmodel class or cluster around lack of fire alarms, capabilities we can’t distinguish from magic, things of this nature.
Yeah, I think “ASI implies an extreme case of lock-in” is a major tendency in the literature (especially sequences-era), but 1. people disagree about whether “alignment” refers to something that outsmarts even this implication or not, then they disagree about relative tractability and plausibility of the different alignment visions, and 2. this is very much a separate set of steps that provide room for disagreement among people who broadly accept Eliezer-like threatmodels (doomcoin stuff).
I don’t want to zero in on actually-existing Eliezer (at whichever time step), I’m more interested in like a threatmodel class or cluster around lack of fire alarms, capabilities we can’t distinguish from magic, things of this nature.