It’s strange to me that this is aimed at people who aren’t aware that MIRI staffers are quite pessimistic about AGI risk. After something like Eliezer’s April Fools post, it seems pretty clear to those who’ve been paying attention—I would’ve been more interested in something that digs into the meat of the view rather than explaining the basic premises. Though it’s possible I’m overestimating the amount of familiarity within longtermist circles of different views, including MIRI’s.
There are factors excluded from this model which are necessary for the core claim that alignment fails by default. Warning shots followed by a huge effort to avert disaster are one way things could go well, but we could just be further from AGI than people think (something like 5-15 years is my understanding of the MIRI view) or have very slow takeoff speeds.
I’m a bit frustrated because it seems like these 2 things are indicative of a failure to engage with counterarguments. They strike me as more of an attempt to instruct people who aren’t familiar with the view, rather than persuasively argue for it compared to different (informed) views.
It’s strange to me that this is aimed at people who aren’t aware that MIRI staffers are quite pessimistic about AGI risk.
It’s not. It’s mainly aimed at people who found it bizarre and hard-to-understand that Nate views AGI risk as highly disjunctive. (Even after reading all the disjunctive arguments in AGI Ruin.) This post is primarily aimed at people who understand that MIRI folks are pessimistic, but don’t understand where “it’s disjunctive” is coming from.
Thanks for this! A couple of things:
It’s strange to me that this is aimed at people who aren’t aware that MIRI staffers are quite pessimistic about AGI risk. After something like Eliezer’s April Fools post, it seems pretty clear to those who’ve been paying attention—I would’ve been more interested in something that digs into the meat of the view rather than explaining the basic premises. Though it’s possible I’m overestimating the amount of familiarity within longtermist circles of different views, including MIRI’s.
There are factors excluded from this model which are necessary for the core claim that alignment fails by default. Warning shots followed by a huge effort to avert disaster are one way things could go well, but we could just be further from AGI than people think (something like 5-15 years is my understanding of the MIRI view) or have very slow takeoff speeds.
I’m a bit frustrated because it seems like these 2 things are indicative of a failure to engage with counterarguments. They strike me as more of an attempt to instruct people who aren’t familiar with the view, rather than persuasively argue for it compared to different (informed) views.
It’s not. It’s mainly aimed at people who found it bizarre and hard-to-understand that Nate views AGI risk as highly disjunctive. (Even after reading all the disjunctive arguments in AGI Ruin.) This post is primarily aimed at people who understand that MIRI folks are pessimistic, but don’t understand where “it’s disjunctive” is coming from.