I have observed the same sharp distinction as @ElliotTep: EAs not focused on animal welfare accept that TAI will end factory farming with something like Matt Reardon’s 99% certainty, while most focused on animal welfare (myself included) feel the risk of factory farming being perpetuated into the future is unacceptably high. This is probably a mix of several factors:
Animal welfare people having some bias to think our work is still important
Animal welfare people having a lower threshold for unacceptably high risk of this particular outcome
EAs outside animal welfare having less firsthand experience of frustration at the ways humans today seem to favor meat from tortured animals over cultivated meat
Animal welfare people having thought less about what extremely weird posthuman futures look like and how strongly TAI might pull in that direction
EAs outside animal welfare being overly confident about the extremely weird posthuman future
I’m focused on animal welfare and have been for 12 years. But I’ve also read my Greg Egan and think it is less likely than not that biological humans will continue to exist into the far future. But I still believe that ensuring AI ends factory farming rather than perpetuating it should be a top focus area for EA. I might try to think through this and write something better, but off the cuff reasons:
It’s possible biological substrates are necessary for consciousness/sentience. My credence on this is maybe 15% which I consider very low given our cluelessness about consciousness. If silicon platforms can’t support consciousness, then Von Neumann probes will probably carry gene sequences to other planets to seed biological civilizations. Low probabilities of galaxy-scale factory farming still look very important!
Maybe the weird posthuman future will get the rest of the lightcone and leave earth for biologicals. In that case it could be a thousand years before digital minds colonize the nearest star system. That’s a lot of time where factory farming could still represent a share of conscious experience within an order of magnitude of today’s. Any model that discounts future beings even just due to uncertainty about our ability to affect their welfare seems likely to make factory farming remain a top priority.
You’re splitting this the same way I meant to, though the quote you point out wasn’t as clear as it could have been: in strictly verifiable domains, AIs will eventually be so much better than humans that having humans involved will only lead to worse decisions. But the line between verifiable and unverifiable is gradual and fuzzy. My expectation in practice is that humans will develop such a strong habit of deferring to AIs that even if there are still humans “in charge” of certain things in the sense that they could implement one decision or another, they will not effectively differentiate the domains in which they by their own lights should maintain informed control, and will end up giving over complete control.
Keeping humans in the loop is a bigger question! I treated “humans eventually wind up not in the loop” as a premise for this post in part because I think it’s the default and in part because my primary audience of people focused on anima welfare probably don’t have the means to help keep humans in control, but maybe that’s giving up too easily.