I agree that deliberate impoverishment in absolute terms is unlikely, the main threat here seems to be from someone who is both actively sadistic and scope-sensitive, which seems unlikely but not wildly implausible
Yeah, I was influenced by work on value lock-in, and it seems plausible to me that any small set of actors with control over superhuman AI could have their moral epistemics corrupted, leading to s-risks. I’m also fairly convinced that escalating inequality (cf. this thread) could enable similar outcomes. It does seem clearer to me that extreme power concentration broadly tends to lead to lower-value futures than outright s-risks, but there is quite a bit of uncertainty here. Do appreciate @Vakus Drake’s points here and seems worth thinking more about this
I tend to think S-risk is just kind of a hard bar to get met. Since even if Sam Altman uses AI to torture his personal enemies forever, well that doesn’t seem like it could ever be enough people to be an S-risk. If some small fraction of people get horrible outcomes and/or a bunch of mind crime happens to some simulated people I would describe that as more of a Omelas-esque middling scenario, since the vast majority of minds would probably still have amazing existences: Thus the median person’s life would still be amazing and even the average would be pretty good.
It just seems very hard to imagine remotely plausible scenarios where there isn’t like 100x as many luxurious post-scarcity societies as simulated hells.
In general I think middling post-singularity scenarios are somewhat underexamined. These have 2 big characteristics to think about compared to a good outcome: There’s not enough centralized control and/or surveillance to enforce mind-crime, and so while the scenario is overwhelmingly good on net there’s going to be a lot of mind crime happening that’s infeasible to do anything about because nobody who would intervene knows when it’s happening. Secondly without better coordination you end up with everyone not near the expansion frontier falling into a Malthusian trap. Since the rate at which one can collect more resource with which to grow is going to be less than exponential because you can’t reach new mass-energy at an exponential rate (you’re limited by the surface area of your feasibly accessible future light cone). Which is an issue because by default population growth is exponential (this is true both because current fertility is suppressed for economic reasons not applicable to a post-scarcity scenario, and because even low birth rates become exponential once aging is cured). This means that even in a best case scenario where you don’t get insane growth from digital minds, people cloning themselves, people wasting resources on vanity projects, etc: It’s still only going to be a few thousand years before your exponential growth reaches a point where it’s literally impossible for accessible resources to support it because you’d need every single atom to support an entire civilization’s worth of people (you can easily see why this is an issue doing some simple math with compound interest). Importantly while it might take a long time to become an issue (though it could also happen quite quickly due to things like EMs not following regular growth trends), getting some handle on population growth is something you need to do early on or you’ll never be able to do it. Since you can’t really coordinate anything once people have spread out too much to enforce rules on them after the fact (if they were all sent with aligned AI’s from the outset that’s different).
I agree that deliberate impoverishment in absolute terms is unlikely, the main threat here seems to be from someone who is both actively sadistic and scope-sensitive, which seems unlikely but not wildly implausible
Yeah, I was influenced by work on value lock-in, and it seems plausible to me that any small set of actors with control over superhuman AI could have their moral epistemics corrupted, leading to s-risks. I’m also fairly convinced that escalating inequality (cf. this thread) could enable similar outcomes. It does seem clearer to me that extreme power concentration broadly tends to lead to lower-value futures than outright s-risks, but there is quite a bit of uncertainty here. Do appreciate @Vakus Drake’s points here and seems worth thinking more about this
I tend to think S-risk is just kind of a hard bar to get met. Since even if Sam Altman uses AI to torture his personal enemies forever, well that doesn’t seem like it could ever be enough people to be an S-risk. If some small fraction of people get horrible outcomes and/or a bunch of mind crime happens to some simulated people I would describe that as more of a Omelas-esque middling scenario, since the vast majority of minds would probably still have amazing existences: Thus the median person’s life would still be amazing and even the average would be pretty good.
It just seems very hard to imagine remotely plausible scenarios where there isn’t like 100x as many luxurious post-scarcity societies as simulated hells.
In general I think middling post-singularity scenarios are somewhat underexamined. These have 2 big characteristics to think about compared to a good outcome: There’s not enough centralized control and/or surveillance to enforce mind-crime, and so while the scenario is overwhelmingly good on net there’s going to be a lot of mind crime happening that’s infeasible to do anything about because nobody who would intervene knows when it’s happening. Secondly without better coordination you end up with everyone not near the expansion frontier falling into a Malthusian trap. Since the rate at which one can collect more resource with which to grow is going to be less than exponential because you can’t reach new mass-energy at an exponential rate (you’re limited by the surface area of your feasibly accessible future light cone). Which is an issue because by default population growth is exponential (this is true both because current fertility is suppressed for economic reasons not applicable to a post-scarcity scenario, and because even low birth rates become exponential once aging is cured). This means that even in a best case scenario where you don’t get insane growth from digital minds, people cloning themselves, people wasting resources on vanity projects, etc: It’s still only going to be a few thousand years before your exponential growth reaches a point where it’s literally impossible for accessible resources to support it because you’d need every single atom to support an entire civilization’s worth of people (you can easily see why this is an issue doing some simple math with compound interest). Importantly while it might take a long time to become an issue (though it could also happen quite quickly due to things like EMs not following regular growth trends), getting some handle on population growth is something you need to do early on or you’ll never be able to do it. Since you can’t really coordinate anything once people have spread out too much to enforce rules on them after the fact (if they were all sent with aligned AI’s from the outset that’s different).