Superhuman manipulation skills (e.g. it can convince anyone of anything)
There are exceptions to this, like the example I discuss in Appendix C.
I found that trying to reason about AGI risk scenarios that rely on these is hard because I keep thinking that these possibly run into physical limitations that deserve more thought before thinking they are plausible enough to substantially affect my thinking. It occurred to me it would be fruitful to reason about AGI risk taking these options off the table to focus on other reasons one might suspect AGIs would have overwhelming power:
Memory (The system could start with knowledge of all public data at the time of its creation, and any data subsequently acquired would be remembered perfectly)
Superior strategic planning (There are courses of actions that might be too complex for humans to plan in a reasonable amount of time, let alone execute)
My view is that normal people are unreceptive to arguments that focus on the first three (advanced nanotechnology, recursive self-improvement, superhuman manipulation skills). Leave aside whether these are probable or not. Just talking about it is not going to work, because the “ask” is too big. It would be like going to rural Louisiana and talking at them about intersectionality.
Normal people are receptive to arguments based on the last three (speed, memory, superior strategic planning). Nintil then goes on to make an argument based only on these ideas. This is persuasive. The reason is that it’s easy for people to accept all three premises:
Computers are very fast. This accords with people’s experience.
Computers can store a lot of data. People can understand this, too.
Superior strategic planning might be slightly trickier, but it’s still easy to grasp, because people know that computers can beat the strongest humans at chess and go.
Thanks, this makes sense! Yeah, this is why many arguments I see start at a more abstract level, e.g.
We are building machines that will become vastly more intelligent than us (c.f. superior strategic planning), and it seems reasonable that then we won’t be able to predict/control them
Any rational agent will strategically develop instrumental goals that could make it hard for us to ensure alignment (e.g., self-preservation → can’t turn them off)
I might have entered at a different vector (all online) so I experienced a different introduction to the idea! If my experience is atypical, and most people get the “gentle” introduction you described, that is great news.
Great! Yes. The key part I think is this:
My view is that normal people are unreceptive to arguments that focus on the first three (advanced nanotechnology, recursive self-improvement, superhuman manipulation skills). Leave aside whether these are probable or not. Just talking about it is not going to work, because the “ask” is too big. It would be like going to rural Louisiana and talking at them about intersectionality.
Normal people are receptive to arguments based on the last three (speed, memory, superior strategic planning). Nintil then goes on to make an argument based only on these ideas. This is persuasive. The reason is that it’s easy for people to accept all three premises:
Computers are very fast. This accords with people’s experience.
Computers can store a lot of data. People can understand this, too.
Superior strategic planning might be slightly trickier, but it’s still easy to grasp, because people know that computers can beat the strongest humans at chess and go.
Thanks, this makes sense! Yeah, this is why many arguments I see start at a more abstract level, e.g.
We are building machines that will become vastly more intelligent than us (c.f. superior strategic planning), and it seems reasonable that then we won’t be able to predict/control them
Any rational agent will strategically develop instrumental goals that could make it hard for us to ensure alignment (e.g., self-preservation → can’t turn them off)
I might have entered at a different vector (all online) so I experienced a different introduction to the idea! If my experience is atypical, and most people get the “gentle” introduction you described, that is great news.