I think itâs fair to criticize Yudkowsky and Soaresâ belief that there is a very high probability of AGI being created within ~5-20 years because that is a central part of their argument. The purpose of the book is to argue for an aggressive global moratorium on AI R&D. For such a moratorium to make sense, probabilities need to be high and timelines need to be short. If Yudkowsky and Soares believed there was an extremely low chance of AGI being developed within the next few decades, they wouldnât be arguing for the moratorium.
So, I think Oscar is right to notice and critique this part of their argument. I donât think itâs fair to say Oscar is critiquing a straw man.
You can respond with a logical, sensible appeal to the precautionary principle: shouldnât we prepare anyway, just in case? First, I would say that even if this is the correct response, it doesnât make Oscarâs critique wrong or not worth making. Second, I think arguments around whether AGI will be safe or unsafe, easy or hard to align, and what to do to prepare for it â these arguments depend on how specific assumptions on how AGI will be built. So, this is not actually a separate question from the topic Oscar raised in this post.
It would be nice if there were something we could do just in case, to make any potential future AGI system safer or easier to align, but I donât see how we can do this in advance of knowing what technology or science will be used to build AGI. So, the precautionary principle response doesnât add up, either, in my view.
I donât think itâs unreasonable to discuss the appropriateness of particular timelines per se but the fact remains that this is not the purpose or goal of the book. As I acknowledged, short to medium term timelines are helpful for motivating the relevance or importance of the issue. However, I think timelines in the 5 to 50 year range are a very common position now, which means that the book can reasonably use this as a starting point for engaging with its core interest, the conditional what if.
Given this as a backdrop, I think itâs fair to say that the author of this post is engaging in a form of straw manning. He is not simply saying: âlook, the actions suggested are going to far because the situation is not as pressing as they make it out to be, we have more timeâ⌠No, he is claiming that âYudkowsky and Soaresâ Book Is Emptyâ by blaming them for not giving an explicit argument for how to build an AGI. I mean, come on how ironic would it be if the book arguing against the building of these kinds of machine would provide the template for building them?
So, I really fail to see the merit of this kind of critique. I mean you can disagree with the premise that we will be able to build generally intelligent machines in the nearish future but given the trajectory of current developments it seems a little bit far fetched to claim that the book is starting from an unreasonable starting point.
As I said mutliple times now, I am not against having open debate about stuff, I am just trying to explain why I think people are not âbitingâ for this kind of content.
P.S.: If you look at the draft treaty they propose, I think itâs clear that they are not against stopping any and all AI R&D but specifically R&D aimed at ASI. Given the general purpose nature of AI, this will surely limit âAI progressâ but one could very well argue that we already have enough societal catching up to do to where we are at right now. I also think itâs quite important to keep in mind that there is no inherent ârightâ to unrestricted R&D. As soon as any kind of âinnovationâ such as âAI progressâ is also affecting other people, our baseline orientation should be one of balancing interests, which can reasonably include limitations on R&D (e.g., nuclear weapons, human cloning, etc.).
I think itâs fair to criticize Yudkowsky and Soaresâ belief that there is a very high probability of AGI being created within ~5-20 years because that is a central part of their argument. The purpose of the book is to argue for an aggressive global moratorium on AI R&D. For such a moratorium to make sense, probabilities need to be high and timelines need to be short. If Yudkowsky and Soares believed there was an extremely low chance of AGI being developed within the next few decades, they wouldnât be arguing for the moratorium.
So, I think Oscar is right to notice and critique this part of their argument. I donât think itâs fair to say Oscar is critiquing a straw man.
You can respond with a logical, sensible appeal to the precautionary principle: shouldnât we prepare anyway, just in case? First, I would say that even if this is the correct response, it doesnât make Oscarâs critique wrong or not worth making. Second, I think arguments around whether AGI will be safe or unsafe, easy or hard to align, and what to do to prepare for it â these arguments depend on how specific assumptions on how AGI will be built. So, this is not actually a separate question from the topic Oscar raised in this post.
It would be nice if there were something we could do just in case, to make any potential future AGI system safer or easier to align, but I donât see how we can do this in advance of knowing what technology or science will be used to build AGI. So, the precautionary principle response doesnât add up, either, in my view.
I donât think itâs unreasonable to discuss the appropriateness of particular timelines per se but the fact remains that this is not the purpose or goal of the book. As I acknowledged, short to medium term timelines are helpful for motivating the relevance or importance of the issue. However, I think timelines in the 5 to 50 year range are a very common position now, which means that the book can reasonably use this as a starting point for engaging with its core interest, the conditional what if.
Given this as a backdrop, I think itâs fair to say that the author of this post is engaging in a form of straw manning. He is not simply saying: âlook, the actions suggested are going to far because the situation is not as pressing as they make it out to be, we have more timeâ⌠No, he is claiming that âYudkowsky and Soaresâ Book Is Emptyâ by blaming them for not giving an explicit argument for how to build an AGI. I mean, come on how ironic would it be if the book arguing against the building of these kinds of machine would provide the template for building them?
So, I really fail to see the merit of this kind of critique. I mean you can disagree with the premise that we will be able to build generally intelligent machines in the nearish future but given the trajectory of current developments it seems a little bit far fetched to claim that the book is starting from an unreasonable starting point.
As I said mutliple times now, I am not against having open debate about stuff, I am just trying to explain why I think people are not âbitingâ for this kind of content.
P.S.: If you look at the draft treaty they propose, I think itâs clear that they are not against stopping any and all AI R&D but specifically R&D aimed at ASI. Given the general purpose nature of AI, this will surely limit âAI progressâ but one could very well argue that we already have enough societal catching up to do to where we are at right now. I also think itâs quite important to keep in mind that there is no inherent ârightâ to unrestricted R&D. As soon as any kind of âinnovationâ such as âAI progressâ is also affecting other people, our baseline orientation should be one of balancing interests, which can reasonably include limitations on R&D (e.g., nuclear weapons, human cloning, etc.).