In the AGI case there needs to be similar conditions—there have to be enough insecure computers on the planet for the AGI to occupy, enough insecure financial assets or robotics for the agi to manipulate the world
All of these seem true, with the exception that robots aren’t needed—there are already plenty of humans (the majority?) that can be manipulated with GPT-4-level generated text.
or intelligence—which itself needs massive amounts of compute—needs to be so useful at high levels that the AGI can substitute for some inputs.
The AI can gain access to the massive amounts of compute via the insecure computers and insecure financial resources.
You need evidence for this.
There are already plenty of sound theoretical arguments and some evidence for things like specification gaming, goal misgeneralisation and deception in AI models. How do you propose we get sufficient empirical evidence for AI takeover short of an actual AI takeover or global catastrophe?
actual empirical evidence that a nuke if used would destroy the planet
How would you get this short of destroying the planet? The Trinity test went ahead based on theoretical calculations showing that it couldn’t happen, but arguably nowhere near enough of them, given the stakes!
But with AGI, half of the top scientists think there’s a 10% chance it will destroy the world! I don’t think the Trinity test would’ve gone ahead in similar circumstances.
-----------------------------------
Ben I’m sorry but your argument is not defensible. Your examples are a joke. Many of them shouldn’t even be in the list as they provide zero support for the argument.
Downvoted your comment for it’s hostility and tone. This isn’t X (Twitter).
It’s the same reason you couldn’t blow up the atmosphere. If you need several trillion weights for human level intelligence and all modalities, or at least 10 percent of the memory in a human brain, and you need to send the partial tensors between cards (I work on accelerator software presently), nobody not an AI lab has enough hardware. Distributed computers separated by Internet links are useless.
It is possible that Moore’s law, if it were to continue approximately 30 more years, could lead to the hardware being common, but that has not happened yet.
This may not be X but I reasoned the information given as evidence was fraudulent. Ben may be well meaning but Ben is trying to disprove basic primate decision making that allowed humans to reach this point with false examples. It’s an extraordinary claim. (By basic reasoning I mean essentially primates choosing between multiple clubs available to them the best performing weapon)
All of these seem true, with the exception that robots aren’t needed—there are already plenty of humans (the majority?) that can be manipulated with GPT-4-level generated text.
The AI can gain access to the massive amounts of compute via the insecure computers and insecure financial resources.
There are already plenty of sound theoretical arguments and some evidence for things like specification gaming, goal misgeneralisation and deception in AI models. How do you propose we get sufficient empirical evidence for AI takeover short of an actual AI takeover or global catastrophe?
How would you get this short of destroying the planet? The Trinity test went ahead based on theoretical calculations showing that it couldn’t happen, but arguably nowhere near enough of them, given the stakes!
But with AGI, half of the top scientists think there’s a 10% chance it will destroy the world! I don’t think the Trinity test would’ve gone ahead in similar circumstances.
-----------------------------------
Downvoted your comment for it’s hostility and tone. This isn’t X (Twitter).
It’s the same reason you couldn’t blow up the atmosphere. If you need several trillion weights for human level intelligence and all modalities, or at least 10 percent of the memory in a human brain, and you need to send the partial tensors between cards (I work on accelerator software presently), nobody not an AI lab has enough hardware. Distributed computers separated by Internet links are useless.
It is possible that Moore’s law, if it were to continue approximately 30 more years, could lead to the hardware being common, but that has not happened yet.
This may not be X but I reasoned the information given as evidence was fraudulent. Ben may be well meaning but Ben is trying to disprove basic primate decision making that allowed humans to reach this point with false examples. It’s an extraordinary claim. (By basic reasoning I mean essentially primates choosing between multiple clubs available to them the best performing weapon)