My 2 cents: math/ programming is only half the battle. Here’s an analogy—you could be the best programmer in the world, but if you don’t understand chess, you can’t program a computer to beat a human at chess, and if you don’t understand quantum physics, you can’t program a computer to simulate matter at the atomic scale (well, not using ab initio methods anyway).
In order to get an intelligence explosion, a computer would have to not only have great programming skills, but also really understand intelligence. And intelligence isn’t just one thing—it’s a bunch of things (creativity, memory, planning, social skills, emotional skills etc and these can be subdivided further into different fields like physics, design, social understanding, social manipulation etc). I find it hard to believe that the same computer would go from not superhuman to superhuman in almost all of these all at once. Obviously computers outcompete humans in many of these already, but I think even on the more “human” traits and in areas where computer act more like agents than just like tools, it’s still more likely to happen in several waves instead of just one takeoff.
My 2 cents: math/ programming is only half the battle. Here’s an analogy—you could be the best programmer in the world, but if you don’t understand chess, you can’t program a computer to beat a human at chess, and if you don’t understand quantum physics, you can’t program a computer to simulate matter at the atomic scale (well, not using ab initio methods anyway).
In order to get an intelligence explosion, a computer would have to not only have great programming skills, but also really understand intelligence. And intelligence isn’t just one thing—it’s a bunch of things (creativity, memory, planning, social skills, emotional skills etc and these can be subdivided further into different fields like physics, design, social understanding, social manipulation etc). I find it hard to believe that the same computer would go from not superhuman to superhuman in almost all of these all at once. Obviously computers outcompete humans in many of these already, but I think even on the more “human” traits and in areas where computer act more like agents than just like tools, it’s still more likely to happen in several waves instead of just one takeoff.