I expect that, once AGI exists, and flops, the spending upon AGI researchers will taste sour. The robots with explosives, and the surveillance cameras across all of China, really were the bigger threats than AGI X-risk; you’ll only admit it once AGI fails to outperform narrow superintelligences. The larger and more multi-modal our networks become, the more consistently they suffer from “modal collapse”: the ‘world-model’ of the network becomes so strongly-self-reinforcing, that ALL gradients from the loss-function end-up solidifying the pre-existing world-model. Literally, AIs are already becoming smart enough to rationalize everything; they suffer from confirmation bias just like us. And that problem was already really bad, by the time they trained GPT-4 - go check their leaked training-regiment: they had to start-over from scratch repeatedly, because the brain found excuses for everything and performance tanked without any hope of recovery. Your AGI will have to be re-run through training 10,000 times, before one of the brains isn’t sure-it’s-always-right-about-its-superstitions. Narrow makes more money, and responds better, faster, cheaper in war—there won’t be any Nash Equilibrium which includes “make AGI”, so the X-Risk is actually ZERO.
I expect that, once AGI exists, and flops, the spending upon AGI researchers will taste sour. The robots with explosives, and the surveillance cameras across all of China, really were the bigger threats than AGI X-risk; you’ll only admit it once AGI fails to outperform narrow superintelligences. The larger and more multi-modal our networks become, the more consistently they suffer from “modal collapse”: the ‘world-model’ of the network becomes so strongly-self-reinforcing, that ALL gradients from the loss-function end-up solidifying the pre-existing world-model. Literally, AIs are already becoming smart enough to rationalize everything; they suffer from confirmation bias just like us. And that problem was already really bad, by the time they trained GPT-4 - go check their leaked training-regiment: they had to start-over from scratch repeatedly, because the brain found excuses for everything and performance tanked without any hope of recovery. Your AGI will have to be re-run through training 10,000 times, before one of the brains isn’t sure-it’s-always-right-about-its-superstitions. Narrow makes more money, and responds better, faster, cheaper in war—there won’t be any Nash Equilibrium which includes “make AGI”, so the X-Risk is actually ZERO.
Pre-ChatGPT, I wrote the details on LessWrong: https://www.lesswrong.com/posts/Yk3NQpKNHrLieRc3h/agi-soon-but-narrow-works-better