Thanks for flagging this, and for generally engaging here. That’s a sobering and generally depressing piece… It seems like Buck and I are definitely gesturing at similar things. This passage in particular stuck out to me:
But I don’t think that’s how most people actually work. For most people, if they have a choice between an object-level belief that is core to their identity and a meta-level principle like “believe what you’d believe if you were smarter and more informed,” they will choose the object-level belief. Tell a devout Christian that superintelligent AI analysis suggests their faith is unfounded, and they won’t abandon their faith, they’ll abandon the AI (perhaps just for the competing AI that is just as good, and happy to tell them otherwise). Tell them that their children will be more successful at reaching reflective equilibrium if exposed to diverse viewpoints, and they’ll question your definition of success, not their approach to parenting.
I was envisioning a similar dynamic playing out with meat consumption. You can argue to someone that a superintelligent AI analysis says that that cultivated meat is “the same” as real meat and it’s even better in various ways, but if they have a specific attachment to a bucolic / naturalistic vision of animal husbandry (“my meat came from happy animals living in a pasture and had one bad day”) then they might just cling to that. Perhaps in a superabundant world, AI will even be able to provide this to them.
Wow that’s super interesting, and not what I would have expected. I appreciate you making this explicit.
My first reaction is that you and I are talking to different groups of people. I frequently encounter the claim that AI will “solve cultivated meat” and so we shouldn’t worry about farmed animals as long as AI doesn’t kill everyone. That’s what I was mainly reacting to here. I don’t work in AI safety though, so I wouldn’t be surprised if the group of people you talk to has thought about it more than the group of people I talk to.
My second reaction is that I don’t understand how biological humans disappearing is different than the disastrous x-risk scenarios that I associate EAs as trying to avoid. Like it seems like EAs are worried about AI scheming in ways that would cause them to make a power grab against humanity’s interest. If instead humans disappear for other (perhaps less violent) reasons, how is that then consistent with AI safety?
Maybe the answer is that humans would be succeeded by some other form of life that are sufficiently similar to humans that it would be better for that form of life to exist than squiggles?
If that’s the claim (which I actually find plausible), then I agree animals would no longer be farmed. However, it seems like this would fall into the bucket of “not actionable unless you work directly on AI,” so it seems like it might be practically useful to act as if this wasn’t going to be the case?
(Also, this claim feels at odds with what I understood your perspective to be from the shallow review you did a while ago. I haven’t had a chance to go back and more carefully read that piece, and maybe on a closer reason it will all look consistent. But I’m just flagging that I still can’t fully model your perspective)