(Update: I think I disagree with what I’m saying here, but I think it’s worth saying.)
There are several things to say here, but I think the most important one is: superintelligent humans are not aligned.
If you take a random human, or even my neighbour, and you magically give them the power to do any optimisation task faster than any collection of humans and machines presently available, I would be very scared for the obvious reason that most humans kinda suck. This is sufficient as a counterargument imo.
But the more fundamental problem is that, depending on how this magical intelligence boost took place, I wouldn’t even trust a superintelligent version of myself. Having that high of an intelligence changes my umwelt and the set of abstractions I can assign utility over.
Presently, I care that others are happy and that they have their wishes fulfilled. For myself, I care about what kind of “story” my life ends up being. I want to be a good book, as judged by my own quaint sensibilities. Perhaps most philosophically annoying is the idea that I want to be able to determine my own story as myself, via the exertion of “my own power”.
But what happens when I discover that my notion of “a wish” is so confused relative to underlying physical reality that, given a much more precise grasp of reality, I have to make some arbitrary decisions about what my original notion supposedly refers to? How would I rescue my values from one umwelt to another?
“Making someone superintelligent” isn’t as straightforward as locating a variable in their program, multiplying it by 1000, and leaving everything unchanged. There are degrees of freedom in how you’d implement the transformation. And for most persons, I’m not sure there even are ways of doing it without what basically amounts to killing the underlying person.
Intelligence augmentation probably wouldn’t result in particular humans becoming overwhelmingly powerful. (But even if it did, I’m substantially more optimistic about what smart humans would do with the universe than you are; it would be weird if a much more capable version of someone did worse according to their own values.)
(Update: I think I disagree with what I’m saying here, but I think it’s worth saying.)
There are several things to say here, but I think the most important one is: superintelligent humans are not aligned.
If you take a random human, or even my neighbour, and you magically give them the power to do any optimisation task faster than any collection of humans and machines presently available, I would be very scared for the obvious reason that most humans kinda suck. This is sufficient as a counterargument imo.
But the more fundamental problem is that, depending on how this magical intelligence boost took place, I wouldn’t even trust a superintelligent version of myself. Having that high of an intelligence changes my umwelt and the set of abstractions I can assign utility over.
Presently, I care that others are happy and that they have their wishes fulfilled. For myself, I care about what kind of “story” my life ends up being. I want to be a good book, as judged by my own quaint sensibilities. Perhaps most philosophically annoying is the idea that I want to be able to determine my own story as myself, via the exertion of “my own power”.
But what happens when I discover that my notion of “a wish” is so confused relative to underlying physical reality that, given a much more precise grasp of reality, I have to make some arbitrary decisions about what my original notion supposedly refers to? How would I rescue my values from one umwelt to another?
“Making someone superintelligent” isn’t as straightforward as locating a variable in their program, multiplying it by 1000, and leaving everything unchanged. There are degrees of freedom in how you’d implement the transformation. And for most persons, I’m not sure there even are ways of doing it without what basically amounts to killing the underlying person.
Intelligence augmentation probably wouldn’t result in particular humans becoming overwhelmingly powerful. (But even if it did, I’m substantially more optimistic about what smart humans would do with the universe than you are; it would be weird if a much more capable version of someone did worse according to their own values.)