Nate Soares of the Machine Intelligence Research Institute has argued that building safe AGI is hard for the same reason that building a successful space probe is hard— it may not be possible to correct failures in the system after it’s been deployed. Eliezer Yudkowsky makes a similar argument:
“This is where practically all of the real lethality [of AGI] comes from, that we have to get things right on the first sufficiently-critical try.” — AGI Ruin: A List of Lethalities
Eliezer and Nate also both expect discontinuous Takeoff by default. I feel like it’s a bit disingenuous to argue that the thinking of Eliezer et al has proven obsolete and misguided, but then also quote them as apparent authority figures in this one case where their arguments align with your essay. It has to be one or the other!
Are you presenting arguments that you think will convince others, regardless of whether you think they are correct?
Edit: Apologies, this doesn’t live up to my goals in having a conversation. However, I am concerned that quoting someone you think has non-predictive models of what will happen as an authority, without flagging that you’re quoting them to point out that your opposition grants that particular point, is disengenious.
Eliezer and Nate also both expect discontinuous Takeoff by default. I feel like it’s a bit disingenuous to argue that the thinking of Eliezer et al has proven obsolete and misguided, but then also quote them as apparent authority figures in this one case where their arguments align with your essay. It has to be one or the other!
Why does it have to be one or the other? I personally don’t put much stock in what Eliezer and Nate think, but many other people do.
Are you presenting arguments that you think will convince others, regardless of whether you think they are correct?Edit: Apologies, this doesn’t live up to my goals in having a conversation. However, I am concerned that quoting someone you think has non-predictive models of what will happen as an authority, without flagging that you’re quoting them to point out that your opposition grants that particular point, is disengenious.
If you don’t think their arguments are convincing, I consider it misleading to attempt to convince other people with those same arguments.