Presumably only in the event that all the value AI could realise that humans can’t necessitates (or is at least greatly contingent on) human extinction?
I think quite a few people are pretty keen on the idea that, for example, we only ever reach immense amounts of value by becoming digital minds. In part, this is precisely because a lot of the obvious reasons why AI might be able to generate far more value than humans look like they also apply to digital minds (e.g. being able to travel to other galaxies).
But in turn, unless we think (i) this is sufficiently improbable; (ii) there will be no other way to generate equivalent amounts of value, and (iii) humans will be an obstacle to AI doing so themselves, then I’m not too sure that there’s a strong case here? At least assuming any of these assumptions are false, it looks like value(humans) + value(AI) > value(AI), thus the focus on AI alignment—but happy to be shown otherwise!
Presumably only in the event that all the value AI could realise that humans can’t necessitates (or is at least greatly contingent on) human extinction?
I think quite a few people are pretty keen on the idea that, for example, we only ever reach immense amounts of value by becoming digital minds. In part, this is precisely because a lot of the obvious reasons why AI might be able to generate far more value than humans look like they also apply to digital minds (e.g. being able to travel to other galaxies).
But in turn, unless we think (i) this is sufficiently improbable; (ii) there will be no other way to generate equivalent amounts of value, and (iii) humans will be an obstacle to AI doing so themselves, then I’m not too sure that there’s a strong case here? At least assuming any of these assumptions are false, it looks like value(humans) + value(AI) > value(AI), thus the focus on AI alignment—but happy to be shown otherwise!