Glad to see the thought that went into this post. I’m in agreement that this is a real danger and potentially an x-risk. I’d recommend the following links from QRI:
Wireheading Done Right—there are better and worse ways to do hedonic recalibration. Andres describes a scenario that might be the best we can hope for.
A Future for Neuroscience—one way to cut through the confusion plaguing affective neuroscience (and thus avoid some ‘dirty wireheading’ scenarios) is to find a good algorithm-level description of pain and pleasure. I think QRI has found one.
A device that could temporarily cause extreme positive or negative valence on demand would immediately change the world.
First, it would validate valence realism in a very visceral way. I’d say it would be the strongest philosophical argument ever made.
Second, it would obviously have huge economic & ethical uses.
Third, I agree that being able to induce strong positive & negative valence on demand could help align different schools of utilitarianism. Nothing would focus philosophical arguments about the discount rate between pleasure & suffering more than a (consensual!) quick blast of pure suffering followed by a quick blast of pure pleasure. Similarly, a lot of people live their lives in a rather numb state. Giving them a visceral sense that ‘life can be more than this’ could give them ‘skin in the game’.
Fourth, it could mess a lot of things up. Obviously, being able to cause extreme suffering could be abused, but being able to cause extreme pleasure on-demand could lead to bad outcomes too. You (Andres) have written about wireheading before, and I agree with the game-theoretic concerns involved. I would also say that being able to cause extreme pleasure in others could be used in adversarial ways. More generally, human culture is valuable and fragile; things that could substantially disrupt it should be approached carefully.
A friend of mine was describing how in the 70s, the emerging field of genetic engineering held the Asilomar Conference on Recombinant DNA to discuss how the field should self-regulate. The next year, these guidelines were adopted by the NIH wholesale as the basis for binding regulation, and other fields (such as AI safety!) have attempted to follow the same model. So the culture around technologies may reflect a strong “founder effect”, and we should be on the lookout for a good, forward-looking set of principles for how valence technology should work.
One principle that seems to make sense is to not publicly post ‘actionable’ equations, pseudocode, or code for how one could generate suffering with current computing resources (if this is indeed possible). Another principle is to focus resources on positive, eusocial applications only, insofar as that’s possible– I’m especially concerned about addiction, and bad actors ‘weaponizing’ this sort of research. Another would be to be on guard against entryism, or people who want to co-opt valence research for political ends.
All of this is pretty straightforward, but it would be good to work it out a bit more formally, look at the successes and failures of other research communities, and so on.
A question I find very interesting is whether valence research is socially disruptive or socially stabilizing by default. I think we should try very hard to make it a socially stabilizing force. One way to think about this is in terms of existential risk. It’s a little weird to say, but I think the fact that so many people are jaded, or feel hopeless, is a big existential risk, because they feel like they have very little to lose. So they don’t really care what happens to the world, because they don’t have good qualia to look forward to, no real ‘skin in the game’. If valence tech could give people a visceral, ‘felt sense’ of wonder and possibility, I think the world could become a much safer place, because more people would viscerally care about AI safety, avoiding nuclear war, and so on.
Finally, one thing that I think doesn’t make much sense is handing off the ethical issues to professional bioethicists and expecting them to be able to help much. Speaking as a philosopher, I don’t think bioethics itself has healthy community & research norms (maybe bioethics needs some bioethicsethicists…). And in general, I think especially when issues are particularly complex or technical, I think the best type of research norms comes from within a community.
Generally speaking, I think we should be taking this topic a lot more seriously, and thinking about specific plans for how to make hedonic recalibration happen in pro-social, pro-human ways, rather than the (default?) cyberpunk dystopia route.
Hi Alexey,
Glad to see the thought that went into this post. I’m in agreement that this is a real danger and potentially an x-risk. I’d recommend the following links from QRI:
A review of what affective neuroscience knows about valence—essentially, affective neuroscience is very confused about what pleasure is, and thus is confused about how to talk about wireheading.
Wireheading Done Right—there are better and worse ways to do hedonic recalibration. Andres describes a scenario that might be the best we can hope for.
A Future for Neuroscience—one way to cut through the confusion plaguing affective neuroscience (and thus avoid some ‘dirty wireheading’ scenarios) is to find a good algorithm-level description of pain and pleasure. I think QRI has found one.
An interview with Adam Ford I did on the topic—here’s the relevant excerpt:
Generally speaking, I think we should be taking this topic a lot more seriously, and thinking about specific plans for how to make hedonic recalibration happen in pro-social, pro-human ways, rather than the (default?) cyberpunk dystopia route.