If you and me and all of humanity gets killed by AI and turned into paperclips, that would be an unprecedented moral catastrophe. If the AIs that killed all of us stay around and enjoy having more paperclips, that is still extremely bad. The very act of killing us makes these AIs not a worthy successor of the human species.

This suggests that proposing to pause AI today is like proposing to pause electricity in 1880

The prospect of AI killing all of us makes these very different. Yes, in both cases a pause will probably slow GDP growth. But humans should be willing to accept lower GDP if this notably reduces the chance of all humans being killed.

Do you think it’s reasonable for two people with all of the same evidence to disagree on precise probabilities and expected values? If so, how would you justify picking your own precise probabilities over someone else’s, if you think theirs are just as defensible?

Or would you just average yours and theirs in some way to get a new distribution? How?

And how far would you go, if you consider all the defensible precise probability distributions anyone could assign (whether or not anyone actually does so)? How do you weigh them all if there are infinitely many of them and no uniform distribution over them?

Here’s another example I like.

for example we’ve aligned some ai to winning at chess and now they’re better than any human

Chess bots are narrow AI, not general AI, which makes the situation very different. We don’t know how to align an ASI to the goal of winning at chess. The most likely outcome would be some sort of severe misalignment—for example, maybe we think we trained the ASI to win at chess, but what actually maximizes its reward signal is the checkmate position, so it builds a fleet of robots to cut down every tree in the world to build trillions of chess sets and arranges every chess board into a checkmate position. See A simple case for extreme inner misalignment for more on why this sort of thing would happen.

Chess bots don’t do that because they have no concept of any world existing outside of the game they’re playing, which would not be the case for ASI.

ETA: That’s also why a lot of people oppose building ASI but still want to build powerful-but-narrow AIs like AlphaFold.

Hmm, that opens up a lot of interesting conversation threads. I actually think some goals will be easier to align ai towards than others, for example we’ve aligned some ai to winning at chess and now they’re better than any human. Obviously that kind of goal is much simpler than any values framework that would be worth aligning agi too, but I think sentientist values would be easier to instill than “human values” (although not in the case of LLMs, I think they’re already basically “aligned” with human values and we now need to shift them towards caring more about all sentient beings). And on top of that, I think sentientist values will care enough about us and our values that a sentientist agi would “go well” for us.

But I’m not even close to an expert, so that’s all very tentative speculation.

This is a very sad post to read for me. because I think it’s obvious the AI x animals field needs to expand extremely quickly. I also agree that it’s tiny currently and the funding situation is also constrained for now (have heard this will change from some important people, but it’s not changing fast enough to grow a movement). I feel where in a bit of a loop currently where some funders want to support impactful projects in this space but aren’t seeing enough of those and the movement builders are really struggling to get funds to get more track record. I would love to see more orgs in the technical weeds of AI alignment towards animals and I know you have the skills to start one if you’re committed enough to it.

Also the concern for alignment risk is valid but not unsolvable! If you put your mind to this problem specifically with a technical skill set you could make real progress here!

OK I think I get what you’re saying now. I think the statement “most current alignment work is going towards aligning ai with human values” is not true. Alignment work is primarily about how to point ASI at any goal at all without there being catastrophic unintended consequences.

It sounds to me like you’re saying the structure of the problem is like this:

• problem 1: how to align AI to humans

• problem 2: how to align AI to all sentient beings

and these are two totally separate problems, and alignment researchers are working on #1 to the exclusion of #2. Whereas I think really the structure is more like this:

• problem 1: how to align AI to any goal whatsoever

• problem 2: choosing what to align AI to

I think problem 2 is hard from a social perspective (if problem 1 is solved, how do you ensure that AI is aligned to care about animal welfare, even though many people won’t want that?), but easy from a technical perspective: once you figure out how to align AI to human values, the technical problems are pretty much solved. And alignment researchers are almost all working on problem 1, not problem 2.

In other words, almost all work on how to align AI to human values translates directly to the problem of how to align AI to sentientist values. It’s not like a totally separate problem.

Have you seen the moral ambition folks?

AI pause advocates often say they are pro-technology and pro-economic growth, and that they simply make one exception for AI because of its unique risks. But this reasoning will grow less credible over time as AI comes to account for a larger and larger share of economic growth.

Simple growth models predict that AI capable of substituting for human labor will raise economic growth rates by an order of magnitude or more. If that’s right, then AI will eventually be driving the vast majority of technological innovation and improvements in the standard of living. Stopping AI really would be like halting technology itself, because you would be shutting off the source of nearly all growth.

This suggests that proposing to pause AI today is like proposing to pause electricity in 1880: yes, electricity is technically just one technology among many, but pausing it would threaten to shut down progress on most of the others.

I also question the premise that AI is unique in its risks. Pause advocates argue that, apart from perhaps nuclear weapons, AI is the first technology to threaten the survival of the human species. But the boundary around “human species” is arbitrary. It only fails to feel that way because, for us today, the human species seems synonymous with the whole world. Replacing us feels like ending the world.

Yet a hunter-gatherer tribe might just as easily feel the same way about themselves and their way of life. To them, the development of agriculture would feel like an existential risk. It would, from their point of view, be a threat to everything that matters.

In reality, the world is much larger than hunter-gatherer tribes or even the human species. By developing AI, we are bringing into existence a new class of sapient beings, ones who will inhabit the world alongside us. I predict that we will coexist with them peacefully, and I welcome efforts to make that outcome more likely. But peaceful or not, the outcome matters for them too. We are not the only people in the story.

In the future, the vast majority of interesting and valuable events will likely occur between digital people, not between the more limited biological ones. The vast majority of relationships, discoveries, adventures, acts of kindness, and feelings of joy will take place within an artificial world, one to which the label “human” may no longer cleanly apply.

In such a world, insisting that the human species represents everything that matters will be like insisting that hunter-gatherers represent the whole world. That may have felt like a reasonable claim 12,000 years ago, but today it would sound silly.

Whether we like it or not, technology has always posed massive risks to “the world”. AI is not the first technology to do this, and it will likely not be the last. The only difference is that this time, technology threatens the world that people alive today grew up in. Just as our ancestors experienced before us, we face the prospect of losing the world we know in exchange for material progress and prosperity. I am happy to take that trade, just as I am glad my ancestors took it in theirs.

As far as I know, most current alignment work is going towards aligning ai with human values. If that’s successful then yay for us, but if we worked towards aligning ai with sentientist values (along the lines of “evidence, reason, and compassion for all sentient beings”), then we would also be in the group of valued beings. If people think that would go well for us, then I think it would make sense to think about ways to redirect more research towards aligning ai with all sentient beings, rather than just human values.

For example, humans. We are somewhat aligned with ourselves, but not with other animals, and that’s been catastrophic for animals (see factory farms, industrial fishing). If we encountered aliens that were more powerful than us, that had alignment like ours, they would not care about wiping us out (maybe a few of them would, but most wouldn’t). But if those aliens were aligned with all sentient beings, they would care. Say for some reason that very powerful aliens were somehow convinced by elephants to be aligned with elephants, we would still be on the chopping block along with every other species. So it’s in everyones interest to align them with all sentient beings, and in the process, we get alignment with us as well.

I would be interested in hearing why people might think ai that went well for animals would not go well for humans, I can imagine scenarios like that but they seem extremely unlikely to me.

I’m having a hard time putting what I mean into words, something like “alignment with all sentient beings gets alignment with humans for free whereas alignment with humans does not get alignment with other sentient beings for free” plus “alignment with all sentient beings is simpler than alignment with humans in particular”. I think the question I posed in my original comment would help determine whether someone agrees with the first part of this paragraph.

I think you’ve made a lot of good points. But solutions to factory farming are broader than just cultivated meat. Plant-based meat is already much closer in cost and more acceptable to consumers. And the source of protein could come from fungus, bacteria, leaf protein, seaweed, etc., though those are probably not as acceptable as regular plants. It’s also possible that AGI could help engineer a meat substitute that actually tastes better than animal meat, perhaps by triggering sweet taste buds without actually having appreciable sugar.

I think a second important question is “If AGI goes well for animals, it’ll go well for humans”

Sorry, can you explain in more detail? I don’t understand what’s important about that question

I think a second important question is “If AGI goes well for animals, it’ll go well for humans” which I think is extremely likely, but I’m much more doubtful about it going well for animals if it goes well for humans.

We are animals, so it going well for animals allows us to instill at least somewhat simpler values in AI, but many people will want humans to be privileged in ai values which is not only more likely to exclude non-humans, but it’ll also be a potentially more complicated and more likely to fail

I haven’t figured out how to organize my thoughts well, so forgive me if this is unclear or disjointed.

I used a narrow definition of AGI because I think that’s where actionable analysis can be made

Even assuming AI reaches your narrow definition of AGI and then stops advancing, AGI would still radically change the economic and political environment. The bits in this post about behavior of regulators become irrelevant if regulators are replaced by AI.

I don’t think it’s fair to say (e.g.) “it’s hard to predict how politics/​government will change, so I’ll just assume they won’t.” Predicting that nothing will change is still a prediction! If you’re going to take the route of not making predictions where the future is especially fuzzy, then I think that’s more defensible, in which case the answers on “Regulators speed up”, “Political opposition reverses”, and “Consumers accept cultivated meat” should all be something like “it’s too difficult to predict because AGI will radically change the political and cultural environment”.

If you write an analysis that only makes sense in an implausible world—the one where AGI accelerates technology, but nothing else happens—then that still has some positive value as long as that world has a non-zero chance of obtaining. But the framing of the piece directs animal activists to focus their attention on that world, and I think that’s the wrong world to focus on because it’s so improbable. At minimum, the piece doesn’t make it clear that this is an assumption.

If you have recommendations for how to reason about worlds where current baselines genuinely don’t extrapolate at all, I’d really welcome them!

Here’s what I would say: Yes, some things, like how society will be structured post-AGI, are very hard to predict. But other things about AGI are predictable. I can predict that it speeds up almost all kinds of work. I can predict that AGI will control the shape of the future—either because it has explicit control, or because humans retain control but still rely on AGI to do most of the work (because AGI is better than humans at almost all tasks). I can predict that, on our current trajectory, ASI will follow shortly after AGI (see AI As Profoundly Abnormal Technology). I can predict that if ASI is misaligned, then it will wipe out all life on earth.

There is an argument that animal activists should advocate to pause AI development because (1) probably after AGI and certainly after ASI, animal activists will have no power; and (2) it is too difficult to predict whether AGI/​ASI would be good for animals, so we need to buy more time to figure out how to make it good. This is an argument from conservatism rather than an argument from expected value; I still think AI pause advocacy has high expected value, but that argument is harder to make. (I’m currently writing something longer about why pausing AI is a good idea even setting aside the alignment problem, although my argument isn’t just about animal welfare.)

Another important question is how to shape the values of AGI (about half the ideas in this list are some version of that). There are things we can do now to influence the direction of AI development.

Another way to reason about post-AGI worlds is to think about AI alignment. e.g. I have three recent posts about what forms of alignment are more likely to be good for animals (1, 2, 3). A big thing that I think is under-appreciated by animal activists is that the way AI is aligned—if it’s aligned—could have a big effect. I don’t just mean “will care-for-animals be built in to its values”, but that there are differences between how different types of alignment are supposed to work, e.g. constitutional AI vs. agent foundations (see 1).

Speaking of “if it’s aligned”, maybe the best thing to do is to work directly on AI alignment. I wrote a relevant post here. I believe a good number of AI safety researchers think that solving the alignment problem is the best way to help animals, although I can’t recall any piece of writing where someone clearly articulated this position.

ETA: More on this bit:

[W]e don’t address scenarios in which AGI drastically reshapes institutional and political dynamics. [...] [W]e focus on existing institutional structures because they allow actionable analysis, but we acknowledge this is a limitation.

Rather than predicting that AGI won’t change institutional structures, to allow actionable analysis I could instead say:

My assumption is that first AGI will become a world government singleton, and its values will be determined by its constitution.

This scenario is both easier to analyze (you can ignore political and regulatory factors and just focus on the text content of the AI constitution) and more likely to actually happen (although I still think it’s pretty unlikely).

Thanks for the comment, Mal.

Unfortunately I think this post is otherwise not very relevant, mainly because no one uses Contrapest [...]

[...]

• Since I’m not sure Evolve works either, it may be that any fertility control product used in the future has an entirely different formulation, making speculation here quite difficult.

Do you agree replacing the rodenticide bait with ContraPest may impact soil animals much more than rodents for my estimate that it decreases cropland by 0.413 m²-year per initial rodent? If so, how much smaller do you think the change in cropland would have to be for replacing the rodenticide bait with Evolve to robustly increase animal welfare (in expectation, and accounting for all animals)?

Here is an illustration of how to quickly estimate the effects on soil animals for other fertility control baits. Imagine replacing the rodenticide bait with Evolve increases or decreases cropland by 10 %. I estimate full depopulation with the rodenticide bait increases cropland by 0.481 m²-year per initial rodent. So replacing the rodenticide bait with Evolve would increase or decrease cropland by 0.0481 m²-year per initial rodent (= 0.10*0.481). This means the effects on soil animals would be 11.6 % (= 0.0481/​0.413) as large as for replacing the rodenticide bait with ContraPest. For effects on soil animals 10 % as large (a round fraction to simplify the calculations), depending on the biome replacing the cropland, my estimates for the change in the living time of soil animals per initial rodent for full depopulation with fertility control instead of rodenticide bait would range from:

• 1.52 to 43.7 soil-ant-years.

• 9.46 to 68.5 soil-termite-years.

• 65.1 to 3.00 k soil-springtail-years.

• 134 to 5.07 k soil-mite-years.

• 466 to 8.01 k soil-arthropod-years.

To clarify, the reason I viewed (at time of writing of the previous post) that rodenticide replacement on islands might be approximately ecologically inert is that conservationists use products until rats are entirely eradicated and then stop. [...]

[...]

• I strongly doubt that comparing “cost [cropland] to prevent one birth/​cause one death” is the right comparison. The population dynamics of suppressing fertility are different than those from killing adults, so the timeline to eradication (in the island context) and the application volumes would likely be different, among other things.

Lorenzo Buonanno’s comment made me update the post in agreement with the 1st paragraph above, although I only read this paragraph after my update. I am now using the amount of rodenticide and fertility control bait for full depopulation per initial rodent instead of the amount needed to kill one rodent, and prevent one rodent birth.

[...] You have this short duration of time during the eradication, and then a long period after where conditions on the island are basically the same. [...]

[...]

• If we’re talking about use in settings other than islands, I want to reiterate that those are definitely not ecologically inert even to some approximation, and there are tons of things you’d have to look at besides production method if you wanted to estimate total animal effects. Age structure changes in rat populations, food web dynamics, most rodent fertility control agents appear to have reasonably strong insecticidal effects while brodifacoum doesn’t affect most invertebrates we’ve studied, etc.

I neglect differences in the population of rodents between the 2 depopulation methods for simplicity, and underestimating the effects on soil animals.

although I do wonder if the limited duration of these eradications and amount of use for island conservation relative to overall production is high enough to make a meaningful impact on land use? I don’t think these things are continuous so a small enough demand shift signal might not have any impact? Just speculating...

I agree cropland changes in steps, but I do not think this changes the overall picture. Relatedly, it may naively seem that decreasing the consumption of chicken by 0.1 kg does not change the production of chicken if this can only be adjusted by multiples of e.g. 1 k kg. However, in this case, a better model would be that decreasing the consumption of chicken by 0.1 kg would increase by roughly 0.01 pp (= 0.1/​(1*10^3)) the probability of the production of chicken decreasing by 1 k kg. So the expected reduction in the production of chicken would still be roughly 0.1 kg (= 1*10^-4*1*10^3).

• The dosages are difficult to determine. [...]

[...]

• Other complications: The dosing requirements also seem to be quite different between species of rats. [...]

Accounting for considerations like these would increase uncertainty. However, the final conclusion would still be that controlling the fertility of rodents instead of killing them can easily increase or decrease welfare, even on islands?

• I think I’m just generally more comfortable than you balancing my uncertainties with a combination of direct work on arthropods, and work on things that have (1) high non-arthropod upsides and (2) where I’m clueless about the effects on arthropods. [...]

I agree.

I’ll give it a try !

Open question: would it be useful to frame this an “impact insurance” for people in impactful careers?

As in:

-I expect this goal to be good for the world (say ~100 WELLBYs in expectation)

-If I don’t achieve this goal, then I definitely owe something that’s a least comparably good for the world (say ~$200 to PureEarth (since I’ve chosen WELLBYs))

(or maybe at least half as good)

I think using it this way could help people who have a real hesitation between impactful work and impactful donations.

“Goes well for humans” (i.e for a very long time) worlds are mostly worlds where AGI is fully theoretically and empirically aligned with a CEV-shaped alignment target, which for me logically requires animal welfare. (I also currently believe those worlds to be implausible because no company seems focused on this)

I struggle to imagine any deliberative or reflective-preference oriented process that does not give the right answer to the animal welfare question. If it doesn’t care about non-human animals, then it means animals are not sentient, or that the CEV is misaligned with human interests and some humans will die because they don’t check the right boxes (and sentience isn’t a box), or that morality is weird and it’s actually fine to torture sentient beings (possible but implausible).

There are other worlds where “goes well for humans” means corrigible and aligned on some unaltered human values. In those worlds, I expect animals to take a blow on the short term, and possibly on the very long if the principal does not care about animal suffering. I also expect humanity to do other morally wrong things that they don’t suspect to be wrong, and die counterfactually way sooner.

@Dane Valerie, would it make sense for moderators to mention the reasons for comments being deleted? I guess you deleted the comment I replied to because it was not published by a human.

Good point, that seems like a big risk! I expect the fraction of sentience that is (post)human or digital to be quite high, especially compared to today, in the intergalactic future. But improving values wrt wild animals seems important.

Thanks for starting this thread, Lorenzo. I strongly upvoted your initial comment because it prompted me to significantly update the post. Thanks for the relevant context, Mal. I am now using the amount of rodenticide and fertility control bait for full depopulation per initial rodent instead of the amount needed to kill one rodent, and prevent one rodent birth. I think my current approach is better because full depopulation (in a given area) is the practical goal of using rodenticide or fertility control bait. In my past approach, the goal was preventing one rodent birth via fertility control for each rodent that would be killed with rodenticide, which I worry has little connection with the practical implementation of the intervention.

The conclusions of the post have not changed. I had calculated an increase in cropland of 0.621 m²-year per rodent birth prevented. Now I estimate a decrease in cropland of 0.413 m²-year per initial rodent. So the change in the living time of soil animals per initial rodent is −66.5 % (= −0.413/​0.621) that I had obtained per rodent birth prevented. However, my conclusions only depended (and depend) on the ratio between the (size of the) effects on soil animals and rodents. As I said (and say) in the post, “I have very little idea about whether it [controlling the fertility of rodents instead of killing them] increases or decreases the welfare of soil animals”.

How would you choose the distributions for the model weights in a way that’s not itself arbitrary? E.g. how do you choose their forms and parameters in a way that’s not arbitrary?

I do think imprecise credences have a similar problem of deciding which distributions to include in their representor. I think ultimately we need to make some arbitrary choices and should accept some, but we can be more or less arbitrary, or stop when it’s no longer decision-relevant. Maybe sometimes we can hit a fixed point or see some kind of convergence in the extra steps we’re taking.

On there potentially being no fact of the matter, this may be helpful. It goes further than the issue of imprecise credences/​EVs.