I think that’s strongly contra Eliezer’s model, which is shaped something like “succeeding at solving the alignment problem eliminates most sources of existential risk, because aligned AGI will in fact be competent to solve for them in a robust way”. This does obviously imply something about the ability of random humans to spin up unmonitored nanofactories push a bad yaml file. Maybe there’ll be some much more clever solution(s) for various possible problems? /shrug
Yeah, I think “ASI implies an extreme case of lock-in” is a major tendency in the literature (especially sequences-era), but 1. people disagree about whether “alignment” refers to something that outsmarts even this implication or not, then they disagree about relative tractability and plausibility of the different alignment visions, and 2. this is very much a separate set of steps that provide room for disagreement among people who broadly accept Eliezer-like threatmodels (doomcoin stuff).
I don’t want to zero in on actually-existing Eliezer (at whichever time step), I’m more interested in like a threatmodel class or cluster around lack of fire alarms, capabilities we can’t distinguish from magic, things of this nature.
I think that’s strongly contra Eliezer’s model, which is shaped something like “succeeding at solving the alignment problem eliminates most sources of existential risk, because aligned AGI will in fact be competent to solve for them in a robust way”. This does obviously imply something about the ability of random humans to
spin up unmonitored nanofactoriespush a bad yaml file. Maybe there’ll be some much more clever solution(s) for various possible problems? /shrugYeah, I think “ASI implies an extreme case of lock-in” is a major tendency in the literature (especially sequences-era), but 1. people disagree about whether “alignment” refers to something that outsmarts even this implication or not, then they disagree about relative tractability and plausibility of the different alignment visions, and 2. this is very much a separate set of steps that provide room for disagreement among people who broadly accept Eliezer-like threatmodels (doomcoin stuff).
I don’t want to zero in on actually-existing Eliezer (at whichever time step), I’m more interested in like a threatmodel class or cluster around lack of fire alarms, capabilities we can’t distinguish from magic, things of this nature.