Thank you for attempting to explain Yudkowskyâs views as you understand them.
I donât think anybody would be convinced by an a priori argument that any and all imaginable forms of AGI or ASI are highly existentially dangerous as a consequence of being prone to the sort of alignment failures Yudkowsky imagined in the 2000s, in the pre-deep learning era, regardless of the technological paradigm underlying them, simply by virtue of being AGI or ASI. I think you have to make some specific assumptions about how the underlying technology works. Itâs not clear what assumptions Yudkowsky is making about that.
A lot of Yudkowskyâs (and Yudkowsky and Soaresâ) arguments about why deep learning is dangerous seem to depend on very loose, hazy analogies (like this). Yudkowsky doesnât have any expertise, education, training, or research/âengineering experience with deep learning. Some deep learning experts say he doesnât know what heâs talking about, on even a basic level. He responded to this, but was unable to defend the technical point he was making â his response was more about vaguely casting aspersions and a preoccupation with social status over and above technical matters, as is often the case. So, Iâm not sure he actually understands the underlying technology well enough to make a convincing, substantive, deep, detailed version of the argument he wants to make.
Whether deep learning is dissimilar from human cognition, and specifically dissimilar in a way that makes it 99.5% likely to cause human extinction, is not some side issue, but the topic on which the whole debate depends. If deep learning is highly similar to human cognition, or dissimilar but in a way that doesnât make it existentially dangerous, then AGI and ASI based on deep learning would not have a 99.5% chance of causing human extinction. Thatâs not a minor detail. Thatâs the whole ballgame.
Also, as an aside, itâs bewildering to me how poorly Yudkowsky and others at MIRI take criticism of their ideas. In response to any sort of criticism or disagreement, Yudkowsky and other folksâ default response seems to be to fly into a rage and to try to attack or humiliate the person making the criticism/âexpressing the disagreement. Yudkowsky has explicitly said he believes heâs by far the smartest person on Earth, at least when it comes to the topic of AGI safety/âalignment. He seems indignant at having to talk to people who are so much less intelligent than he is. Unfortunately, his attitude seems to have become the MIRI culture. (Soares has also apparently contributed to this dynamic.)
If youâre doing technical research, maybe you can get away with that â but even then I donât think you can, because an inability to hear criticism/âdisagreement makes you much worse at research. But now that MIRI has pivoted to communications and advocacy, this will be an even more serious problem. If Yudkowsky and others at MIRI are incapable of engaging in civil intellectual debate, or are simply unwilling to, how on Earth are they going to be effective at advocacy and communications?
âSome experts downvote Yudkowskyâs standing to opineâ is not a reasonable standard; some experts think vaccines cause autism. You can usually find someone with credentials in a field who will say almost anything.
The responsible thing to do (EDIT: if youâre deferring at all, as opposed to evaluating the situation for yourself) is to go look at the balance of what experts in a field are saying, and in this case, theyâre fairly split, with plenty of respected big names (including many who disagree with Eliezer on many questions) saying he knows enough of what heâs talking about to be worth listening to. I get that Yarrow is not convinced, but I trust Hinton, who has reservations of his own but not of the form âEliezer should be dismissed out of hand for lack of some particular technical expertise.â
Also: when the experts in a field are split, and the question is one of existential danger, it seems that the splitness itself is not reassuring. Experts in nuclear physics do not drastically diverge in their predictions about what will happen inside a bomb or reactor, because we understand nuclear physics. When experts in the field of artificial intelligence have wildly different predictions and the disagreement cannot be conclusively resolved, this is a sign of looseness in everyoneâs understanding, and when you ask normal people on the street âhey, if one expert says an invention will kill everyone, and another says it wonât, and you ask the one who says it wonât where their confidence comes from, and they say âbecause Iâm pretty sure weâll muddle our way through, with unproven techniques that havenât been invented yet, the risk of killing everyone is probably under 5%,â how do you feel?â
they tend to feel alarmed.
And that characterization is not uncharitableâthe optimists in this debate do not have an actual concrete plan. You can just go check. It all ultimately boils down to handwaving and platitudes and âIâm sure weâll stay ahead of capabilities [for no explicable reason].â
And weâre intentionally aiming at something that exceeds us along the very axis that led us to dominate the planet, so ⌠?
Another way of saying this: itâs very, very weird that the burden of proof on this brand-new and extremely powerful technology is âmake an airtight case that itâs dangerousâ instead of âmake an airtight case that itâs a good idea.â Even a 50â50shared burden would be better than the status quo.
Iâll note that
In response to any sort of criticism or disagreement, Yudkowsky and other folksâ default response seems to be to fly into a rage and to try to attack or humiliate the person making the criticism/âexpressing the disagreement.
The responsible thing to do is to go look at the balance of what experts in a field are saying, and in this case, theyâre fairly split
This is not a crux for me. I think if you were paying attention, it was not hard to be convinced that AI extinction risk was a big deal in 2005â2015, when the expert consensus was something like âwho cares, ASI is a long way off.â Most people in my college EA group were concerned about AI risk well before ML experts were concerned about it. If todayâs ML experts were still dismissive of AI risk, that wouldnât make me more optimistic.
Oh, I agree that if one feels equipped to go actually look at the arguments, one doesnât need any argument-from-consensus. This is just, like, âif you are going to defer, defer reasonably.â Thanks for your comment; I feel similarly/âendorse.
This seems like a motte-and-bailey. The question at hand is not about expertsâ opinions on the general topic of existential risk from AGI, but specifically their assessment of Yudkowskyâs competence at understanding deep learning. You can believe that deep learning-based AGI is a serious existential risk within the next 20 years and also believe that Yudkowsky is not competent to understand the topic at a technical level.
As far as I know, Geoffrey Hinton has only commented on Yudkowskyâs high-level comments about existential risk from AGI â which is a concern Hinton shares â and not said anything about Yudkowskyâs technical competence on deep learning.
If you know any examples of prominent experts in deep learning vouching for Yudkowskyâs technical competence in deep learning, specifically, I invite you to give citations.
Yudkowsky has said he believes heâs by far the smartest person in the world at least when it comes to AI alignment/âsafety â as in, the second smartest doesnât come close â and maybe the smartest person in the world in general. AI alignment/âsafety has been his lifeâs work since before he decided â seemingly sometime in the mid-to-late 2010s â deep learning was likely to lead to AGI. MIRI pays him about $600,000 a year to do research. By now, heâs had plenty of opportunity to learn about deep learning. Given this, shouldnât he show a good grasp on concepts in deep learning? Shouldnât he be competent at making technical arguments about deep learning? Shouldnât he be able to clearly, coherently explain his reasoning?
It seems like Yudkowsky must at least be wrong about his own intelligence because if he really were as intelligent as he thinks, he wouldnât struggle with basic concepts in deep learning or have such a hard time defending the technical points he wants to make about deep learning. He would just be able to make a clear, coherent case, demonstrating an understanding of the definitions of widely-used terms and concepts. Since he canât do that, he must be overestimating his own abilities by quite a lot.
In domains other than AI, such as Japanese monetary policy, he has expressed views with a similar level of confidence and self-assurance as what he says about deep learning that turned out to be wrong, but, notably, never acknowledged the mistake. This speaks to Clara Collierâs point about not updating his views based on his new evidence. Itâs not clear that any amount of evidence would (at least publicly) change his mind about any topic where he would lose face if he admitted being wrong. (Heâs been wrong many times in the past. Has this ever happened before?) And if he doesnât understand deep learning in the first place, then the public shouldnât care whether he changes his mind or not.
You would most likely get fired for agreeing with me about this, so I canât reasonably expect you to agree, but I might as well say the things that people on the payroll of a Yudkowsky-founded organization canât say. For me, the cost isnât losing a job, itâs just a bit of negative karma on a forum.
Sorry to be so blunt, but youâre asking for $6M to $10M to be redirected from possibly the worldâs poorest people or animals in factory farms â or even other organizations working on AI safety â to your organization, led by Yudkowsky, so that you can try to influence policy on a national U.S. and international scale. Yudkowsky has indicated if his preferred policy were enacted at an international scale, it might increase the risk of wars. This calls for a high level of scrutiny. No one should accept weak, flimsy, hand-wavy arguments about this. No one should tiptoe around Yudkowskyâs track record of false or extremely dubious claims, or avoid questioning his technical competence in deep learning, which is in serious doubt, out of fear or politeness. If you, MIRI, or Yudkowsky donât want this level of scrutiny, donât ask for donations from the EA community and donât try to influence policy.
Thank you for attempting to explain Yudkowskyâs views as you understand them.
I donât think anybody would be convinced by an a priori argument that any and all imaginable forms of AGI or ASI are highly existentially dangerous as a consequence of being prone to the sort of alignment failures Yudkowsky imagined in the 2000s, in the pre-deep learning era, regardless of the technological paradigm underlying them, simply by virtue of being AGI or ASI. I think you have to make some specific assumptions about how the underlying technology works. Itâs not clear what assumptions Yudkowsky is making about that.
A lot of Yudkowskyâs (and Yudkowsky and Soaresâ) arguments about why deep learning is dangerous seem to depend on very loose, hazy analogies (like this). Yudkowsky doesnât have any expertise, education, training, or research/âengineering experience with deep learning. Some deep learning experts say he doesnât know what heâs talking about, on even a basic level. He responded to this, but was unable to defend the technical point he was making â his response was more about vaguely casting aspersions and a preoccupation with social status over and above technical matters, as is often the case. So, Iâm not sure he actually understands the underlying technology well enough to make a convincing, substantive, deep, detailed version of the argument he wants to make.
Whether deep learning is dissimilar from human cognition, and specifically dissimilar in a way that makes it 99.5% likely to cause human extinction, is not some side issue, but the topic on which the whole debate depends. If deep learning is highly similar to human cognition, or dissimilar but in a way that doesnât make it existentially dangerous, then AGI and ASI based on deep learning would not have a 99.5% chance of causing human extinction. Thatâs not a minor detail. Thatâs the whole ballgame.
Also, as an aside, itâs bewildering to me how poorly Yudkowsky and others at MIRI take criticism of their ideas. In response to any sort of criticism or disagreement, Yudkowsky and other folksâ default response seems to be to fly into a rage and to try to attack or humiliate the person making the criticism/âexpressing the disagreement. Yudkowsky has explicitly said he believes heâs by far the smartest person on Earth, at least when it comes to the topic of AGI safety/âalignment. He seems indignant at having to talk to people who are so much less intelligent than he is. Unfortunately, his attitude seems to have become the MIRI culture. (Soares has also apparently contributed to this dynamic.)
If youâre doing technical research, maybe you can get away with that â but even then I donât think you can, because an inability to hear criticism/âdisagreement makes you much worse at research. But now that MIRI has pivoted to communications and advocacy, this will be an even more serious problem. If Yudkowsky and others at MIRI are incapable of engaging in civil intellectual debate, or are simply unwilling to, how on Earth are they going to be effective at advocacy and communications?
Again speaking more for the broad audience:
âSome experts downvote Yudkowskyâs standing to opineâ is not a reasonable standard; some experts think vaccines cause autism. You can usually find someone with credentials in a field who will say almost anything.
The responsible thing to do (EDIT: if youâre deferring at all, as opposed to evaluating the situation for yourself) is to go look at the balance of what experts in a field are saying, and in this case, theyâre fairly split, with plenty of respected big names (including many who disagree with Eliezer on many questions) saying he knows enough of what heâs talking about to be worth listening to. I get that Yarrow is not convinced, but I trust Hinton, who has reservations of his own but not of the form âEliezer should be dismissed out of hand for lack of some particular technical expertise.â
Also: when the experts in a field are split, and the question is one of existential danger, it seems that the splitness itself is not reassuring. Experts in nuclear physics do not drastically diverge in their predictions about what will happen inside a bomb or reactor, because we understand nuclear physics. When experts in the field of artificial intelligence have wildly different predictions and the disagreement cannot be conclusively resolved, this is a sign of looseness in everyoneâs understanding, and when you ask normal people on the street âhey, if one expert says an invention will kill everyone, and another says it wonât, and you ask the one who says it wonât where their confidence comes from, and they say âbecause Iâm pretty sure weâll muddle our way through, with unproven techniques that havenât been invented yet, the risk of killing everyone is probably under 5%,â how do you feel?â
they tend to feel alarmed.
And that characterization is not uncharitableâthe optimists in this debate do not have an actual concrete plan. You can just go check. It all ultimately boils down to handwaving and platitudes and âIâm sure weâll stay ahead of capabilities [for no explicable reason].â
And weâre intentionally aiming at something that exceeds us along the very axis that led us to dominate the planet, so ⌠?
Another way of saying this: itâs very, very weird that the burden of proof on this brand-new and extremely powerful technology is âmake an airtight case that itâs dangerousâ instead of âmake an airtight case that itâs a good idea.â Even a 50â50 shared burden would be better than the status quo.
Iâll note that
...seems false.
This is not a crux for me. I think if you were paying attention, it was not hard to be convinced that AI extinction risk was a big deal in 2005â2015, when the expert consensus was something like âwho cares, ASI is a long way off.â Most people in my college EA group were concerned about AI risk well before ML experts were concerned about it. If todayâs ML experts were still dismissive of AI risk, that wouldnât make me more optimistic.
Oh, I agree that if one feels equipped to go actually look at the arguments, one doesnât need any argument-from-consensus. This is just, like, âif you are going to defer, defer reasonably.â Thanks for your comment; I feel similarly/âendorse.
Made a small edit to reflect.
This seems like a motte-and-bailey. The question at hand is not about expertsâ opinions on the general topic of existential risk from AGI, but specifically their assessment of Yudkowskyâs competence at understanding deep learning. You can believe that deep learning-based AGI is a serious existential risk within the next 20 years and also believe that Yudkowsky is not competent to understand the topic at a technical level.
As far as I know, Geoffrey Hinton has only commented on Yudkowskyâs high-level comments about existential risk from AGI â which is a concern Hinton shares â and not said anything about Yudkowskyâs technical competence on deep learning.
If you know any examples of prominent experts in deep learning vouching for Yudkowskyâs technical competence in deep learning, specifically, I invite you to give citations.
Yudkowsky has said he believes heâs by far the smartest person in the world at least when it comes to AI alignment/âsafety â as in, the second smartest doesnât come close â and maybe the smartest person in the world in general. AI alignment/âsafety has been his lifeâs work since before he decided â seemingly sometime in the mid-to-late 2010s â deep learning was likely to lead to AGI. MIRI pays him about $600,000 a year to do research. By now, heâs had plenty of opportunity to learn about deep learning. Given this, shouldnât he show a good grasp on concepts in deep learning? Shouldnât he be competent at making technical arguments about deep learning? Shouldnât he be able to clearly, coherently explain his reasoning?
It seems like Yudkowsky must at least be wrong about his own intelligence because if he really were as intelligent as he thinks, he wouldnât struggle with basic concepts in deep learning or have such a hard time defending the technical points he wants to make about deep learning. He would just be able to make a clear, coherent case, demonstrating an understanding of the definitions of widely-used terms and concepts. Since he canât do that, he must be overestimating his own abilities by quite a lot.
In domains other than AI, such as Japanese monetary policy, he has expressed views with a similar level of confidence and self-assurance as what he says about deep learning that turned out to be wrong, but, notably, never acknowledged the mistake. This speaks to Clara Collierâs point about not updating his views based on his new evidence. Itâs not clear that any amount of evidence would (at least publicly) change his mind about any topic where he would lose face if he admitted being wrong. (Heâs been wrong many times in the past. Has this ever happened before?) And if he doesnât understand deep learning in the first place, then the public shouldnât care whether he changes his mind or not.
You would most likely get fired for agreeing with me about this, so I canât reasonably expect you to agree, but I might as well say the things that people on the payroll of a Yudkowsky-founded organization canât say. For me, the cost isnât losing a job, itâs just a bit of negative karma on a forum.
Sorry to be so blunt, but youâre asking for $6M to $10M to be redirected from possibly the worldâs poorest people or animals in factory farms â or even other organizations working on AI safety â to your organization, led by Yudkowsky, so that you can try to influence policy on a national U.S. and international scale. Yudkowsky has indicated if his preferred policy were enacted at an international scale, it might increase the risk of wars. This calls for a high level of scrutiny. No one should accept weak, flimsy, hand-wavy arguments about this. No one should tiptoe around Yudkowskyâs track record of false or extremely dubious claims, or avoid questioning his technical competence in deep learning, which is in serious doubt, out of fear or politeness. If you, MIRI, or Yudkowsky donât want this level of scrutiny, donât ask for donations from the EA community and donât try to influence policy.