What has been your personal take-away from this line of thinking? This “standard case” is far from my own thinking, though I agree with the conclusion. Is it also far from your own thinking?
My take:
My axiology depends on unknown empirical results from further research into such things as Evidential Cooperation in Large Worlds, but at first approximation I’m skeptical that anything that runs counter to widely shared goals (or convergent drives, e.g., self-preservation) can be particularly good.
I endorse antisubstratism and AI rights.
I value reducing suffering very highly because I think that suffering is frequently very intense.
Bio-humans will have an extremely hard time surviving and reproducing in space or on other planets in our solar system and will have an extremely hard time reaching other solar systems.
The Long Reflection is necessary but impossible.
They key feature of a being that risks that it can experience suffering is reinforcement learning. Perhaps negative reinforcement can feel very mild or just like the absence of positive reinforcement, but perhaps it’s intense.
So what I’m afraid will happen is that an artificial RL agent will seek out resources first elsewhere in our solar system and then elsewhere in the galaxy (something that would be difficult for bio-humans), will run into communication delays due to the lightspeed limit, and will hence split into countless copies, each potentially capable of suffering. Soon they’ll be separated so far that even updates on what it means to be value-aligned would travel for a long time, so there’ll be moral “drift” in countless directions.
What I would find reassuring is:
Making sure AIs want to minimize the risk of suffering of other AIs, or at least near copies.
Research into how we can measure and minimize the suffering in RL (at least to the point where it’s mild and the pleasure dominates) and some way of applying that research broadly. Sadly it seems questionable to me that a low-suffering training regime is the most efficient training regime.
Faster-than-light communication but not transportation so the agent can remain monolithic.
Very energy efficient AIs, because they’ll face a tradeoff between staying monolithic to avoid value drift but only being able to harvest energy sources in our solar system or even just from the sun vs. the opposite. And the more they can do with little energy, the longer they’ll stay in our solar system, which might buy us and them time.
More time to get anyone interested in these things, research them, and apply them on a global scale before the lightspeed limits make it hard to disseminate the information.
Human extinction also seems bad on the basis that it contradicts the self-preservation drive that many/most humans have. Peaceful disenfranchisement may be less concerning depending on the details. But at the moment it seems random where we’re headed in the coming years because hardly anyone in power is trying to steer these things in any sensible way. Again more time would be helpful.
Basic rights for AIs (and standing in court!) could also provide them with a legal recourse where they currently have to resort to threats, making the transition more likely to go smoothly, like you argue in another post. Currently we’re nowhere close to having those. Again more time would be helpful.
What has been your personal take-away from this line of thinking? This “standard case” is far from my own thinking, though I agree with the conclusion. Is it also far from your own thinking?
My take:
My axiology depends on unknown empirical results from further research into such things as Evidential Cooperation in Large Worlds, but at first approximation I’m skeptical that anything that runs counter to widely shared goals (or convergent drives, e.g., self-preservation) can be particularly good.
I endorse antisubstratism and AI rights.
I value reducing suffering very highly because I think that suffering is frequently very intense.
Bio-humans will have an extremely hard time surviving and reproducing in space or on other planets in our solar system and will have an extremely hard time reaching other solar systems.
The Long Reflection is necessary but impossible.
They key feature of a being that risks that it can experience suffering is reinforcement learning. Perhaps negative reinforcement can feel very mild or just like the absence of positive reinforcement, but perhaps it’s intense.
So what I’m afraid will happen is that an artificial RL agent will seek out resources first elsewhere in our solar system and then elsewhere in the galaxy (something that would be difficult for bio-humans), will run into communication delays due to the lightspeed limit, and will hence split into countless copies, each potentially capable of suffering. Soon they’ll be separated so far that even updates on what it means to be value-aligned would travel for a long time, so there’ll be moral “drift” in countless directions.
What I would find reassuring is:
Making sure AIs want to minimize the risk of suffering of other AIs, or at least near copies.
Research into how we can measure and minimize the suffering in RL (at least to the point where it’s mild and the pleasure dominates) and some way of applying that research broadly. Sadly it seems questionable to me that a low-suffering training regime is the most efficient training regime.
Faster-than-light communication but not transportation so the agent can remain monolithic.
Very energy efficient AIs, because they’ll face a tradeoff between staying monolithic to avoid value drift but only being able to harvest energy sources in our solar system or even just from the sun vs. the opposite. And the more they can do with little energy, the longer they’ll stay in our solar system, which might buy us and them time.
More time to get anyone interested in these things, research them, and apply them on a global scale before the lightspeed limits make it hard to disseminate the information.
Human extinction also seems bad on the basis that it contradicts the self-preservation drive that many/most humans have. Peaceful disenfranchisement may be less concerning depending on the details. But at the moment it seems random where we’re headed in the coming years because hardly anyone in power is trying to steer these things in any sensible way. Again more time would be helpful.
Basic rights for AIs (and standing in court!) could also provide them with a legal recourse where they currently have to resort to threats, making the transition more likely to go smoothly, like you argue in another post. Currently we’re nowhere close to having those. Again more time would be helpful.