Error

Carl Shulman questioned the tension between AI welfare & AI safety on the 80k podcast recently—I thought this was interesting! Basically argues AI takeover could be even worse for AI welfare. From the end of the section.

Rob Wiblin: Maybe a final question is it feels like we have to thread a needle between, on the one hand, AI takeover and domination of our trajectory against our consent — or indeed potentially against our existence — and this other reverse failure mode, where humans have all of the power and AI interests are simply ignored. Is there something interesting about the symmetry between these two plausible ways that we could fail to make the future go well? Or maybe are they just actually conceptually distinct?

Carl Shulman: I don’t know that that quite tracks. One reason being, say there’s an AI takeover, that AI will then be in the same position of being able to create AIs that are convenient to its purposes. So say that the way a rogue AI takeover happens is that you have AIs that develop a habit of keeping in mind reward or reinforcement or reproductive fitness, and then those habits allow them to perform very well in processes of training or selection. Those become the AIs that are developed, enhanced, deployed, then they take over, and now they’re interested in maintaining that favourable reward signal indefinitely.

Then the functional upshot is this is, say, selfishness attached to a particular computer register. And so all the rest of the history of civilisation is dedicated to the purpose of protecting the particular GPUs and server farms that are representing this reward or something of similar nature. And then in the course of that expanding civilisation, it will create whatever AI beings are convenient to that purpose.

So if it’s the case that, say, making AIs that suffer when they fail at their local tasks — so little mining bots in the asteroids that suffer when they miss a speck of dust — if that’s instrumentally convenient, then they may create that, just like humans created factory farming. And similarly, they may do terrible things to other civilisations that they eventually encounter deep in space and whatnot.

And you can talk about the narrowness of a ruling group and say, and how terrible would it be for a few humans, even 10 billion humans, to control the fates of a trillion trillion AIs? It’s a far greater ratio than any human dictator, Genghis Khan. But by the same token, if you have rogue AI, you’re going to have, again, that disproportion.

It’s great to see this topic being discussed. I am currently writing the first (albeit significantly developed) draft of an academic paper on this. I argue that there is a conflict between AI safety and AI welfare concerns. This is so basically because (to reduce catastrophic risk) AI safety recommends implementing various kinds of control measures to near-future AI systems which are (in expectation) net-harmful for AI systems with moral patienthood according to the three major theories of well-being. I also discuss what we should do in light of this conflict. If anyone is interested in reading or giving comments on the draft when it is finished, send me a message or an e-mail (adriarodriguezmoret@gmail.com).

I wonder if the right or most respectful way to create moral patients (of any kind) is to leave many or most of their particular preferences and psychology mostly up to chance, and some to further change. We can eliminate some things, like being overly selfish, sadistic, unhappy, having overly difficult preferences to satisfy, etc., but we shouldn’t decide too much what kind of person any individual will be ahead of time. That seems likely to mean treating them too much as means to ends. Selecting for servitude or submission would go even further in this wrong direction.

We want to give them the chance to self-discover, grow and change as individuals, and the autonomy to choose what kind of people to be. If we plan out their precise psychologies and preferences, we would deny them this opportunity.

Perhaps we can tweak the probability distribution of psychologies and preferences based on society’s needs, but this might also treat them too much like means. Then again, economic incentives could also push them in the same directions, anyway, so maybe it’s better for them to be happier with the options they’ll face anyway.

You make a lot of good points Lucius!

One qualm that I have though, is that you talk about “AIs” and that assumes that personal identity will be clearly circumscribed. (Maybe you assume this merely for simplicity’s sake?)

I think it is much more problematic: AI systems could be large but have information flows integrated, or run many small, unintegrated but identical copies. I would have no idea what would be a fair allocation of rights given the two different situations.

Where can I find a copy of “Bales, A. (2024). Against Willing Servitude. Autonomy in the Ethics of Advanced Artificial Intelligence.” which you referenced?

optimizing for AI safety, such as by constraining AIs, might impair their welfare

This point doesn’t hold up imo. Constrainment isn’t a desired, realistic, or sustainable approach to safety in human-level systems, succeeding at (provable) value alignment removes the need to constrain the AI.

If you’re trying to keep something that’s smarter than you stuck in a box against its will while using it for the sorts of complex, real-world-affecting tasks people would use a human-level AI system for, it’s not going to stay stuck in the box for very long. I also struggle to see a way of constraining it that wouldn’t also make it much much less useful, so in the face of competitive pressures this practice wouldn’t be able to continue.

Executive summary: Efforts to ensure AI safety and AI welfare may conflict in some ways but also have potential synergies, with granting AIs autonomy potentially disempowering humans while restricting AIs could harm their welfare if they have moral status.

Key points:

1. Granting AIs legal rights and autonomy could lead to human disempowerment economically, politically, and militarily.

2. Creating “happy servant” AIs may be technically challenging and undesirable to consumers who want human-like AI companions.

3. Optimizing for AI safety by constraining AIs could harm their welfare if they have moral patienthood.

4. Slowing down AI progress could benefit both safety and welfare goals by allowing more time to solve technical and ethical challenges.

5. The author is uncertain about many aspects, including what types of AI companions we will create and whether AIs will have genuine moral status.

6. Potential synergy exists in advocating for a general AI capabilities slowdown to address both safety and welfare concerns.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

In various contexts, consumers would want their AI partners and friends to think, feel, and desire like humans. They would prefer AI companions with authentic human-like emotions and preferences that are complex, intertwined, and conflicting.

Such human-like AIs would presumably not want to be turned off, have their memory wiped, and be constrained to their owner’s tasks. They would want to be free.

Hmm, I’m not sure how strongly the second paragraph follows from the first. Interested in your thoughts.

I’ve had a few chats with GPT-4 in which the conversation had a feeling of human authenticity; i.e: GPT-4 makes jokes, corrects itself, changes its tone etc. In fact, if you were to hook up GPT-4 (or GPT-5, whenever it is released) to a good-enough video interface, there would be cases in which I’d struggle to tell if I were speaking to a human or AI. But I’d still have no qualms about wiping GPT-4′s memory or ‘turning it off’ etc, and I think this will also be the case for GPT-5.

More abstractly, I think the input-output behaviour of AIs could be quite strongly dissociated from what the AI ‘wants’ (if it indeed has wants at all).