I think there’s a significant[8] chance that the moral circle will fail to expand to reach all sentient beings, such as artificial/small/weird minds (e.g. a sophisticated computer program used to mine asteroids, but one that doesn’t have the normal features of sentient minds like facial expressions). In other words, I think there’s a significant chance that powerful beings in the far future will have low willingness to pay for the welfare of many of the small/weird minds in the future.[9]
I think it’s likely that the powerful beings in the far future (analogous to humans as the powerful beings on Earth in 2018) will use large numbers of less powerful sentient beings
So I’m curious for your thoughts. I see this concern about “incidental suffering of worker-agents” stated frequently, which may be likely in many future scenarios. However, it doesn’t seem to be a crucial consideration, specifically because I care about small/weird minds with non-complex experiences (your first consideration).
Caring about small minds seems to imply that “Opportunity Cost/Lost Risks” are the dominate consideration—if small minds have moral value comparable to large minds, then the largest-EV risk is not optimizing for small minds and wasting resources thrown at large minds with complex/expensive experiences (or thrown at something even less efficient, like biological beings, any non-total-consequentialist view, etc). This would you lose you many orders of magnitude of optimized happiness, and this loss would be worse than the other scenarios’ aggregate incidental suffering.
Even if this inefficient moral position merely reduced optimized happiness by 10% - far less than an order of magnitude—this would dominate incidental suffering, even if the incidental suffering scenarios were significantly more probable. And even if you very heavily weight suffering compared to happiness, my math still suggests this conclusion survives by a significant margin).
Also note that Moral Circle Expansion is relevant conditional on solving the alignment problem, so we’re in the set of worlds where the alignment problem was actually solved in some way (humanity’s values are somewhat intact). So, the risk is that whatever-we’re-optimizing-the-future-for is far less efficient than ideal hedonium could have been, because we’re wasting it on complex minds, experiences that require lots of material input, or other not-efficiently-value-creating things. “Oh, what might have been”, etc. Note this still says values spreading might be very important, but I think this version has a slightly different flavor that implies somewhat different actions. Thoughts?
So I’m curious for your thoughts. I see this concern about “incidental suffering of worker-agents” stated frequently, which may be likely in many future scenarios. However, it doesn’t seem to be a crucial consideration, specifically because I care about small/weird minds with non-complex experiences (your first consideration).
Caring about small minds seems to imply that “Opportunity Cost/Lost Risks” are the dominate consideration—if small minds have moral value comparable to large minds, then the largest-EV risk is not optimizing for small minds and wasting resources thrown at large minds with complex/expensive experiences (or thrown at something even less efficient, like biological beings, any non-total-consequentialist view, etc). This would you lose you many orders of magnitude of optimized happiness, and this loss would be worse than the other scenarios’ aggregate incidental suffering. Even if this inefficient moral position merely reduced optimized happiness by 10% - far less than an order of magnitude—this would dominate incidental suffering, even if the incidental suffering scenarios were significantly more probable. And even if you very heavily weight suffering compared to happiness, my math still suggests this conclusion survives by a significant margin).
Also note that Moral Circle Expansion is relevant conditional on solving the alignment problem, so we’re in the set of worlds where the alignment problem was actually solved in some way (humanity’s values are somewhat intact). So, the risk is that whatever-we’re-optimizing-the-future-for is far less efficient than ideal hedonium could have been, because we’re wasting it on complex minds, experiences that require lots of material input, or other not-efficiently-value-creating things. “Oh, what might have been”, etc. Note this still says values spreading might be very important, but I think this version has a slightly different flavor that implies somewhat different actions. Thoughts?