I think there is an unstated assumption here that uploading is safe. And by safe, I mean existentially safe for humanity[1]. If in addition to being uploaded, a human is uplifted to superintelligence, would they—indeed any given human in such a state—be aligned enough with humanity as a whole to not cause an existential disaster? Arguably humans right now are only relatively existentially safe because power imbalances between them are limited.
Even the nicest human could accidentally obliterate the rest of us if uplifted to superintelligence and left running for subjective millions of years (years of our time). “Whoops, I didn’t expect that to happen from my little physics experiment”; “Uploading everyone into a hive mind is what my extrapolations suggested was for the best (and it was just so boring talking to you all at one word per week of my time)”.
We could upload many minds, trying to represent some (sub)distribution of human values (EDIT: and psychological traits), and augment them all slowly, limiting power imbalances between them along the way.
Perhaps. But remember they will be smarter than us, so controlling them might not be so easy (especially if they gain access to enough computer power to speed themselves up massively. And they need not be hostile, just curious, to accidentally doom us.)
Yes, this is a fair point; Holden has discussed these dangers a little in “Digital People Would Be An Even Bigger Deal”. My bottom-line belief, though, is that mind uploads are still significantly more likely to be safe than ML-derived ASI, since uploaded minds would presumably work, and act, much more similarly to (biological) human minds. My impression is that others also hold this view? I’d be interested if you disagree.
To be clear, I rank moratorium > mind uploads > ML-derived ASI, but I think it’s plausible that our strategy portfolio should include mind uploading R&D alongside pushing for a moratorium.
I agree that they would most likely be safer than ML-derived ASI. What I’m saying is that they still won’t be safe enough to prevent an existential catastrophe. It might buy us a bit more time (if uploads happen before ASI), but that might only be measured in years. Moratorium >> mind uploads > ML-derived ASI.
Because of the crazy high power differential, and propensity for accidents (can a human really not mess up on an existential scale if acting for millions of years subjectively at superhuman capability levels?). As I say in my comment above:
Even the nicest human could accidentally obliterate the rest of us if uplifted to superintelligence and left running for subjective millions of years (years of our time). “Whoops, I didn’t expect that to happen from my little physics experiment”; “Uploading everyone into a hive mind is what my extrapolations suggested was for the best (and it was just so boring talking to you all at one word per week of my time)”.
This doesn’t seem like a strong enough argument to justify a high probability of existential catastrophe (if that’s what you intended?).
At vastly superhuman capabilities (including intelligence and rationality), it should be easier to reduce existential-level mistakes to tiny levels. They would have vastly more capability for assessing and mitigating risks and for moral reflection (not that this would converge to some moral truth; I don’t think there is any).
If you think this has a low chance of success (if we could delay AGI long enough to actually do it), then alignment seems pretty hopeless to me on that view, and a temporary pause only delays the inevitable doom.
I do think we could do better (for upside-focused views) by ensuring more value pluralism and preventing particular values from dominating, e.g. by uploading and augmenting multiple minds.
At vastly superhuman capabilities (including intelligence and rationality), it should be easier to reduce existential-level mistakes to tiny levels. They would have vastly more capability for assessing and mitigating risks and for moral reflection
They are still human though, and humans are famous for making mistakes, even the most intelligent and rational of us. It’s even regarded by many as part of what being human is—being fallible. That’s not (too much of) a problem at current power differentials, but it is when we’re talking of solar-system-rearranging powers for millions of subjective years without catastrophic error...
a temporary pause only delays the inevitable doom.
Yes. The pause should be indefinite, or at least until global consensus to proceed, with democratic acceptance of whatever risk remains.
I think there is an unstated assumption here that uploading is safe. And by safe, I mean existentially safe for humanity[1]. If in addition to being uploaded, a human is uplifted to superintelligence, would they—indeed any given human in such a state—be aligned enough with humanity as a whole to not cause an existential disaster? Arguably humans right now are only relatively existentially safe because power imbalances between them are limited.
Even the nicest human could accidentally obliterate the rest of us if uplifted to superintelligence and left running for subjective millions of years (years of our time). “Whoops, I didn’t expect that to happen from my little physics experiment”; “Uploading everyone into a hive mind is what my extrapolations suggested was for the best (and it was just so boring talking to you all at one word per week of my time)”.
Although safety for the individual being uploaded would be far from guaranteed either.
We could upload many minds, trying to represent some (sub)distribution of human values (EDIT: and psychological traits), and augment them all slowly, limiting power imbalances between them along the way.
Perhaps. But remember they will be smarter than us, so controlling them might not be so easy (especially if they gain access to enough computer power to speed themselves up massively. And they need not be hostile, just curious, to accidentally doom us.)
Yes, this is a fair point; Holden has discussed these dangers a little in “Digital People Would Be An Even Bigger Deal”. My bottom-line belief, though, is that mind uploads are still significantly more likely to be safe than ML-derived ASI, since uploaded minds would presumably work, and act, much more similarly to (biological) human minds. My impression is that others also hold this view? I’d be interested if you disagree.
To be clear, I rank moratorium > mind uploads > ML-derived ASI, but I think it’s plausible that our strategy portfolio should include mind uploading R&D alongside pushing for a moratorium.
I agree that they would most likely be safer than ML-derived ASI. What I’m saying is that they still won’t be safe enough to prevent an existential catastrophe. It might buy us a bit more time (if uploads happen before ASI), but that might only be measured in years. Moratorium >> mind uploads > ML-derived ASI.
Why do you expect an existential catastrophe from augmented mind uploads?
Because of the crazy high power differential, and propensity for accidents (can a human really not mess up on an existential scale if acting for millions of years subjectively at superhuman capability levels?). As I say in my comment above:
This doesn’t seem like a strong enough argument to justify a high probability of existential catastrophe (if that’s what you intended?).
At vastly superhuman capabilities (including intelligence and rationality), it should be easier to reduce existential-level mistakes to tiny levels. They would have vastly more capability for assessing and mitigating risks and for moral reflection (not that this would converge to some moral truth; I don’t think there is any).
If you think this has a low chance of success (if we could delay AGI long enough to actually do it), then alignment seems pretty hopeless to me on that view, and a temporary pause only delays the inevitable doom.
I do think we could do better (for upside-focused views) by ensuring more value pluralism and preventing particular values from dominating, e.g. by uploading and augmenting multiple minds.
They are still human though, and humans are famous for making mistakes, even the most intelligent and rational of us. It’s even regarded by many as part of what being human is—being fallible. That’s not (too much of) a problem at current power differentials, but it is when we’re talking of solar-system-rearranging powers for millions of subjective years without catastrophic error...
Yes. The pause should be indefinite, or at least until global consensus to proceed, with democratic acceptance of whatever risk remains.