“I reject Yudkowsky’s story that some particular AI will foom and become dictator-of-the-future; rather, I think there will be a multi-polar ecosystem of different AIs with different values. Thus: problem solved?” Well, hmm: what values in particular? Is it all still ultimately an office-supplies thing? If so, it depends how much you like a complex ecosystem of staple-maximizers, thumb-tack-maximizers, and so on – fighting, trading, etc. “Better than a monoculture.” Maybe, but how much?[9] Also, are all the humans still dead?
In my mind, there is a sense in which this last question is analogous to Neanderthals[1] asking a few hundreds of thousands of years ago whether they would still be around now. They are not, but is this any significant evidence that the world has gone through a much less valuable trajectory? I do not think so. What arguably matters is whether there are still beings around with the desire and ability to increase welfare. So I would instead ask, “are all intelligent welfarists dead?”, where intelligent could be interpreted as sufficiently intelligent to eventually leverage (via successors or not) the cosmic endowment to increase welfare. My question is equivalent to yours nearterm, since humans are the only intelligent welfarists now, but the answers may come apart in the next few decades thanks to (even more) intelligent sentient AI. To the extent the answers to the 2 questions differ, it seems important to focus on the right one.
Nice post, Joe!
In my mind, there is a sense in which this last question is analogous to Neanderthals[1] asking a few hundreds of thousands of years ago whether they would still be around now. They are not, but is this any significant evidence that the world has gone through a much less valuable trajectory? I do not think so. What arguably matters is whether there are still beings around with the desire and ability to increase welfare. So I would instead ask, “are all intelligent welfarists dead?”, where intelligent could be interpreted as sufficiently intelligent to eventually leverage (via successors or not) the cosmic endowment to increase welfare. My question is equivalent to yours nearterm, since humans are the only intelligent welfarists now, but the answers may come apart in the next few decades thanks to (even more) intelligent sentient AI. To the extent the answers to the 2 questions differ, it seems important to focus on the right one.
Or individuals of another species of the genus Homo. There are 12 besides Homo Sapiens!