Contrary to some others he argues that we should perhaps never make ‘prepotent’ AI (one that cannot be controlled by humans) - not even a defensive one to prevent other AI threats.
Where’s that? I’d be very interested to see an argument to that. I looked around and found a lot of reasons prepotence is dangerous, and ways to avoid it, but wasn’t able to find an argument that it is decisively more dangerous than its negative.
(I do suspect non-prepotence is dangerous. In short: Prepotent AGI can and visibly is required to exceed us morally (not in the sense of making metaphysical moral progress, I don’t believe in that, I mean that there can be higher levels of patience, lucidity, and knowledge of the human CEV and its applications, that would bring the thing to conclusions we’d find shocking), there’s a sense in which prepotent AGI would be harder to weaponize, harder to train on a single profane object level objective and fire, it is less tempting to try to use it to do stupid rash things we will grow to regret because the consequences of using it in stupid rash regrettable ways are so much more immediately obviously irrevocable. In the longest term, building agentic infrastructures that maintain binding treaties will be a necessity for overcoming the last coordination problems, that’s another reason that prepotence is innevitable. Notably, the treaty “It should be globally impossible to make prepotent AGI” would itself manifest as a prepotent agency. The idea that prepotence is or should be avoidable might be conceptually unworkable.)
(In my skim-read, I also couldn’t find discussion of the feasibility of aligned prepotent AI, and that’s making me start to wonder if there might be a perilous squeamishness around. There are men who worry about being turned into housecats, work vainly on improving the human brain with neural meshes, so that humans can compete with AI. The reality is the housecats will be happy and good and as enfranchised as they wish to be, and human brains will not compete, and that will be fine. It’s imaginable that this aversion to prepotence comes from a regressive bravado that has been terminally outmoded since the rise of the first city-state. If I’m missing the mark on this though, apologies for premature psychologizing.)
Regarding ARCHES
Where’s that? I’d be very interested to see an argument to that. I looked around and found a lot of reasons prepotence is dangerous, and ways to avoid it, but wasn’t able to find an argument that it is decisively more dangerous than its negative.
(I do suspect non-prepotence is dangerous. In short: Prepotent AGI can and visibly is required to exceed us morally (not in the sense of making metaphysical moral progress, I don’t believe in that, I mean that there can be higher levels of patience, lucidity, and knowledge of the human CEV and its applications, that would bring the thing to conclusions we’d find shocking), there’s a sense in which prepotent AGI would be harder to weaponize, harder to train on a single profane object level objective and fire, it is less tempting to try to use it to do stupid rash things we will grow to regret because the consequences of using it in stupid rash regrettable ways are so much more immediately obviously irrevocable. In the longest term, building agentic infrastructures that maintain binding treaties will be a necessity for overcoming the last coordination problems, that’s another reason that prepotence is innevitable. Notably, the treaty “It should be globally impossible to make prepotent AGI” would itself manifest as a prepotent agency. The idea that prepotence is or should be avoidable might be conceptually unworkable.)
(In my skim-read, I also couldn’t find discussion of the feasibility of aligned prepotent AI, and that’s making me start to wonder if there might be a perilous squeamishness around. There are men who worry about being turned into housecats, work vainly on improving the human brain with neural meshes, so that humans can compete with AI. The reality is the housecats will be happy and good and as enfranchised as they wish to be, and human brains will not compete, and that will be fine. It’s imaginable that this aversion to prepotence comes from a regressive bravado that has been terminally outmoded since the rise of the first city-state. If I’m missing the mark on this though, apologies for premature psychologizing.)