AI safety, animal welfare, and everything in between
Zoe L
Karma: 86
Coercion and Deception in AI-to-AI Management
Apply to the Sentient Futures Fall 2026 Project Incubator
I slightly disagreed with this statement and share some of the same thoughts. I think it’s quite likely to have a multi-polar world with fierce competition in the short term; however, in the long term equilibrium, I think the likely outcomes are either (1) we have a dominant winner or (2) we have more cooperation. So I averaged my short vs. long-term predictions.
I think it’s important to research into multi-polarity and the competition dynamic because what happens in the short term could impact what happens in the long term, possibly in non-intuitive ways. For instance, the most capable and resourced model/lab in the short term may not always win in the long term if others gang up on them or if the institutional environment uniquely disadvantages them.
Thanks Paolo!
When using gpt-4o and gemini-2.5-flash as judges, they struggled with math (for enforcing resource and value constraints) and generally didn’t justify their claims as much. Upgrading to gpt-5.4 and gemini-2.5-pro solves the math problem, though claude-sonnet-4-6 still provided more reasoning for their decisions.
Since we’ve only ran 3-year simulations (i.e., 3 turns for each game), we can’t make claims about long-tern equilibrium. However, we did observe that different shock events seem to encourage different strategies even in the 3-year sim, e.g. alignment_breakthrough incentivized transparency (i.e., more cooperative); nationalization_shock incentivized resource consolidation (i.e., more competitive). Running simulations over longer horizon (10-50 turns) with different parameters and under different scenarios would help confirm if these shock-induced trends hold.
We also observed that more A2A communication led to more cooperation and thus better vibe-based alignment. Players are always allowed A2A communication and are always truthful about their actions in the current sim, so it would be interesting to test what happens when players are allowed (or even encouraged) to use deception or when there’s fog-of-war/a lack of A2A communication.We didn’t reference the specific papers you mentioned but share their ideas. Kenneth, 2026 on AI-simulated nuclear war game influenced our design of the race mechanics the most. Carichon et al. and Zeng et al. influenced our design of the multi-layer value system. We’ve seen similar behaviours as observed in Avin et al. 2024, especially:
The power to steer the future of AI development is very unequally distributed due to several drivers for concentration, including the enormous compute requirements of the latest frontier AI models
There exists an information asymmetry where states and the public will constantly be catching up to deal with the impacts of the last generation of AI technologies
Winners take all rather than winner takes all + Division into blocs by state lines
Tech race + Races are destabilising
Supply chain disruptions slow but don’t stop AI and cause instability (we actually have a supply chain disruption shock event, so again, will be interesting to run it over a longer time horizon)
How to Virtue Signal the “Right” Way
“Its the relative self vs. others prioritization. That’s a tension that is always going to exist between the FIRE community and the EA community.”—I agree with this statement. However, do you think the selfish vs. altruistic trait is a bi-modal or a normal-ish distribution? My intuition is the latter, that most people want to do some good but are also somewhat selfish.
This actually leads into outreach strategy. I’m not a community organizer but I know it’s hard work, so kudos to you for doing the meta-work. I want to challenge the “success metric” for the outreach. It sounds like you’re using “who’s coming to a EA meetup” as a proxy. In my opinion, the real beneficiaries of the EA movement are other people, animals, and perhaps future non-biological sentient beings. So I think a better proxy metric would be something like “how much money donated to these beneficiaries”—one doesn’t need to attend any EA meetup or know about this forum to donate to, e.g., the Humane League. Anecdotally, I have a few “self-interested libertarian” friends and I was able to convince them to donate to the Shrimp Welfare Project recently.
True/pure altruism is indeed rare but I believe most people are at least semi-altruistic. They (perhaps including me) may not be a good fit for the core EA community but they’re open to support EA causes.
Hi Jason—thanks for spelling out the details here. I agree with your observations. I’m not proposing to eliminate the 10% pledge and am more “endorsing a multi-level approach” as you said in your other comment. I acknowledge it’s (1) easier to let go of “(what would counterfactually be) other people’s money” and (2) it’d likely be a lesser sum than lifetime giving, but it may be a good lower-commitment option. Gen Z are having fewer kids (which is reflected in my friend group, where especially high-income women are not planning to have any kids) and people tend to think more about their legacy in old age, so this proposal could be appealing. To be clear, I think people who take the 10% pledge are super respectable and the trial pledge is a good way to get people on the fence in the door. Giving What We Can already added the wealth-based option and the Further Pledge as newer options and I wonder if it’s more accessible if it gives a whole list of options (As another example, X% of your inheritance/trust—I know many trust fund babies that would be happy to do something, anything, with their money. Again, it doesn’t carry the weight of an average middle-class person giving 10% every year, but it would be a lot more money funneled into effective causes rather than naming another building at Harvard after themselves.) and tweak the messaging to be something like “While we encourage people to take the 10% pledge and have a trial pledge that you can easily get started with, we also realize there are many other possible giving options that you prefer to pursue, for example: [insert list of options]. Should you choose not taking the 10% pledge, we still urge you to consider directing your donations towards effective charities.”
I suppose I’m really making two main points here:
More mass appeal (both to the not-so-well-off and to the wealthy people) without giving up the full-commitment option
Emphasis on the “effective” part more than the “10%” part to the mass while affirming/not diluting the meaning of the 10% pledge for those choosing the full commitment
Hi Matthew, I appreciate you trying to pin-point the root cause of my disconnect, but I’d like to push back a bit: “it seems you are prioritizing early retirement and a luxurious lifestyle over EA causes and giving”—yes to the “early retirement” part and no to the “luxurious lifestyle” part. I’m very frugal even compared to those under poverty line in US/Canada; however, I want to leave a large cushion for emergency since I’m stopping work at such a young age. If I have more intrinsic desire to work, I’d be much more open to work a normal 80k-hr career and donate a larger % of my income at an ongoing basis.
“Knowing many people in the movement—most do have an intrinsic altruistic motivation.”—I don’t find it unbelievable that people have an intrinsic altruistic motivation and I respect the EA community for their service. My critique is more so that EA seems to be focused on “how to” for the in-group (those already super into the EA concept) and less so on outreach to near-groups, let alone out-groups. In other words, how can EA turn 90% of people into semi-altruists instead of turning 1% of people into perfectly effective altruists? I think the 10% pledge in its current structure isn’t very appealing to 90% of people.
Hi Yarrow—thank you for taking the time to reply so thoroughly! I love your new emoji flair.
talk to enough random people about EA online or in real life and you’ll eventually hear something unexpected
Yes, I always enjoy talking to people about EA and usually find more diversity in thoughts when people converse in real life than writing online (perhaps this is because of in-group vs. out-group: while most causes have more in-fighting among the in-group, the EA community seems to have dodged this problem at the expense of in-group high conformity, but this constraint can be relaxed when talking to the out-group irl).
Is the little orange or blue diamond so different from someone having an emoji in their username, or, in real life, wearing a little pink or red ribbon for breast cancer of HIV/AIDS awareness?
I have no issues with people advertising their identity or interest or quirkiness with some kind of flair. I used a cross and astrological signs as an analogy because, like the orange diamond, they convey a sense of superiority like “I will be saved (while non-believers go to hell)” or “I’m in tune with the cosmos (while low-vibration people slave through life)”. I acknowledge these are stereotypes and not everyone uses these symbols with the same judgmental intention.
On the 10% pledge itself, my point is not that no one should donate now, but that the “official” EA pledge has too rigid of a structure. To me, the vibes are similar to most vegan activists telling people “going vegan” is the only right way and hating on vegetarians or reducetarians because they’re not meeting the standard. I think for most causes, two things are true at the same time: (1) people who have hardly thought about it should do way more about it and (2) people who have thought much about it, besides directly addressing the cause, should probably focus on effective outreach to people in (1) but chill on policing others in (2).
“Sounds like AI” is the wrong way to put this.
You’re right, the point should be more about sounding too formal/academic rather than sounding too AI. I often use AI to help me polish my writing, so I tend to associate AI writing with more structure and more serious tone, but this is a usage bias. I think doing public dialogues on the EA Forum is a great idea.
Hi JoA—I see you’re interested in animal welfare and invertebrate welfare, so I just want to say these are causes I consistently prioritize in my donations. I’m also looking into the AI x animal welfare space (I mentioned I’m not generally interested in AI safety but I am interested in applications/implications on animals) and my preliminary takeaway is that AGI doesn’t substantially change how we should approach animal welfare in the near term (i.e., what helps animals now/soon will likely continue to help them with AGI, likely more).
Hi Kestrel—thank you for your thoughtful reply and I agree with your hot takes :) There’re definitely similarities between FIRE and EA communities as both agree that a certain level of wealth/comfort is enough, but the trade-off is that the more one gives in the near term, the further out this person can FIRE with the same cushion for emergency and/or the less continuous giving this person can draw from investments. I’m totally in support of letting effective givers figure out what works for them. My point is not that no one should donate now, but that the “official” EA pledge has too rigid of a structure.
Thanks for writing this. As a woman, I’d love to get your wife’s takes on this too!