I work in AI policy and donate to help poor people and farmed animals. I formerly worked in foreign policy, including on Capitol Hill and as a U.S. Army officer. I love travel, sports, and writing, mostly about politics; my Substack is Exasperated Alien.
AndrewDoris
Karma: 238
I don’t think this post has aged well overall, but I’d particularly like to focus on your claim the EAs should seek out Cold Uggies instead of running away from them. I disagree strongly. Cold Uggies are often a useful warning that you’re about to engage in immoral activity by the standards of your own conscience. For example, both EA and the world would surely have been better off if SBF had listened to his Cold Uggies more closely. Often, Cold Uggies indicate the existence of important side constraints that should not be breached. If there are neglected opportunities on the other side, they may be neglected for good reason.
I’m closer to a libertarian than a leftist, and I might have agreed with your argument in 2012. But in the 2020s, after trying one coup already—and now setting loose an unelected tycoon to run roughshod over Congress’ budgetary powers, purge anyone loyal to the constitution, and turn government into a vehicle for personal enrichment and revenge—it’s clear enough to me that the Republican political movement is overwhelmingly a force for bad. Anyone “going Republican” would have to go along with too much of this bad from the inside.
Far from being too hostile to Republicans, too many EAs are naively ambivalent about Trump in particular, and the extent to which his reckless, illiberal, and anti-intellectual political movement is a cross-cutting risk factor exacerbating many forms of X-risk at once, from AII to bio to nuclear to climate change. Republicans must be engaged with where necessary, but actively supporting them likely has spillover harms that exceed the benefits.
As someone who started a Substack six months ago, I actually suspect it could have the opposite impact: people with relevant content will be even more eager to cross-post it here to try and build their audience.
I’ve not done this myself because most of my content is not EA-related and the two posts of mine which were EA-related felt too introductory for the forum (ex: defending EA against non-EA critics). I also felt weird about using an altruistic platform to self-promote too nakedly if the post wasn’t a good fit (and even now I feel weird that someone might interpret this comment that way—isn’t self-awareness fun?).
But to my point, neither of the posts I wrote on Substack were thoughts I’d have posted here if I hadn’t started a Substack. And if I did have thoughts that felt like they would be of value to people who were already highly involved EAs, I’d be especially excited to put in effort towards fleshing them out now compared to how excited I’d have been a year ago, due to the ability to cross-post and potentially draw more eyeballs to my Substack.
In short, watching the subscriber count rise is an even more flattering dopamine boost than EA forum karma, and the market is competitive enough that many writers are sharking for excuses to self-promote on external sites.
(Relatedly, I’d definitely +1 to the idea of making a Substack for the weekly Forum Digest as soon as possible (maybe you already have one but I couldn’t find it from a quick search just now).)
I share your impression that it’s often used differently in broader society and mainstream animal rights groups than it is by technical philosophers and in the EA space. I think the average person would still hear the word as akin to racism or sexism or some other -ism. By criticizing those isms, we DO in fact mean to imply that individual human beings are of equal moral value regardless of their race or sex. And by that standard, I’d be a proud speciesist, because I do think individual beings of some species are innately more valuable than others.
We can split hairs about why that is—capacity for love or pain or knowledge or neuron count or whatever else we find valuable about a life—but it will still require you to come out with a multiplier for how much more valuable a healthy “normal” human is relative to a healthy normal member of other species, which would be absolutely anathema in the racial or sexual context.
This year I gave 13% of my income (+ some carryover from last year, which I had postponed) to EA charities. Of this, I gave about half to global health and development (mostly to GiveWell Top Charities, some to Give Directly) and the other half to animal welfare (mostly to the EA Funds Animal Welfare Fund, some to The Humane League). I also gave $1,250 to various political candidates I felt were EA-aligned. In prior years I’ve given overwhelmingly to global health and development and I still think that’s very important: it’s what initially drew me to EA and what I’m most confident is good. But last year I was convinced I had underinvested in animal welfare historically and I’m starting to make up for that.
I strongly prefer near-term causes with my personal donations, partly because my career focuses on speculative long-term impact. I’m bothered by the strong possibility that my career efforts will benefit nobody, and want to ensure I do at least some good along the way. I also think that in recent years, the wealthiest and most prominent EAs have invested more money into longterm causes than we can be confident is helpful, in ways that have sometimes backfired, damaged or dominated EA’s reputation, promoted groupthink in pursuit of jobs/community, and ultimately saddened or embarrassed me. Relatedly, I think managing public perceptions of EA is inescapably important work if we want to effectively improve government policies in democratic countries. So even on longtermist grounds, I think it’s important for self-described EAs at the grassroots level to keep proven, RCT-backed, highly effective charities with intuitive mass appeal on the funding menu (perhaps especially if we personally work on longtermism and want people to trust our motives).
Within neartermism, I like to split my donations across a single-digit number of the most impactful funds or charities. This is because I do not have a strong, confident belief that any one of them is most effective, want to maximize my chance of doing a large amount of good overall, and see hedging my bets as a mark of intellectual humility. I don’t mind if this makes my altruism less effective than that of the very best EAs, because I’m confident it’s better than that of 99% of people. Likewise, I think the path to effective giving at a societal scale depends much more on outreach to the bottom 90% or so of givers, who give barely any quantitative thought to relative impact, than it does on redirecting donations from those already in the movement.
Great comment. I think “people who sacrifice significantly higher salaries to do EA work” is a plausible minimum definition of who those calling for democratic reforms feel deserve a greater say in funding allocation. It doesn’t capture all of those people, nor solve the harder question of “what is EA work/an EA organization?” But it’s a start.
Your 70⁄30 example made me wonder whether redesigning EA employee compensation packages to include large matching contributions might help as a democratizing force. Many employers outside EA/in the private sector offer a matching contributions program, wherein they’ll match something like 1-5% of your salary (or up to a certain dollar value) in contributions to a certified nonprofit of your choosing. Maybe EA organizations (whichever voluntarily opt into this) could do that except much bigger—say, 20-50% of your overall compensation is paid not to you but to a charity of your choosing. This could also be tied to tenure so that the offered match increases at a faster rate than take home pay, reflecting the intuition that committed longtime members of the EA community have engaged with the ideas more, and potentially sacrificed more, and consequently deserve a stronger vote than newcomers.
Ex: Sarah’s total compensation is 100k, of which she takes home 80k and her employer offers an additional 20k to a charity of her choosing. After 2 years working there, her total package jumps to 120k, of which she takes home 88k and allocates another 32k. After 10 years she takes home 110 and allocates another 90, etc. This tenure could be interchangeable across participating organizations. With time, it may even resemble the “impact certificates” you mention.
Employers could limit this match to a prespecified list of plausibly EA recipients if they wish. Employees could accept this arrangement en lieu of giving X% of their personal incomes (which has the added benefit of avoiding taxation on “income” that’s only going to be given away to largely tax-deductible organizations anyway). Employees could also elect to give a certain amount back to their employing organization, which some presumably would since people tend to believe in the importance of work they are doing. We could write software to anonymize these donations, and avoid any fear of recrimination for NOT regifting it to the employing org.
One downside could be making it more expensive for EA organizations to hire, and thus harder for them to grow and harder for individual EAs to find an EA job. It also wouldn’t solve the fact that the resources controlled by EA organizations are not proportional to the number of people they employ, especially at the extremes. Perhaps if mega-donors like Dustin are open to democratization but wary of how to define the EA electorate, they’d support higher grants to participating recipients, on the logic that “if they’re EA enough to deserve my grant for X effective project, they’re EA enough to deserve a say in how some of my other money is spent too” (even beyond what they need for X).
For all I know EA organizations may have something like this already. If anyone has toyed with or tried to implement this idea before, I’d love to hear about it.
It is often the explicit job of a journalist to uncover and release publicly important information from sources who would not consent to its release.
“Moral authority” and “intellectual legitimacy” are such fuzzy terms that I’m not really sure what this post is arguing.
Insofar as they just denote public perceptions, sure: this is obviously bad PR for the movement. It shows we’re not immune from big mistakes, and raises fair questions about the judgment of individual EAs, or certain problematic norms/mindsets among the living breathing community of humans associated with the label. We’ll probably get mocked a bit more and be greeted with more skepticism in elite circles. There are meta-EA problems that need fixing, that I’ve been introspecting on the past two weeks.
But “careful observers” also knew that before the FTX scandal, and it’s unclear to me which specific ideas in EA philosophy are less intellectually legitimate or authoritative than they were before. When a prominent Democrat politician has a scandal, Democrats get mocked—but nobody intelligent thinks that reduces the moral authority of being pro-choice or supporting stricter gun control, etc. The ideas are right or wrong independent of how their highest-profile advocates behave.
Perhaps SBF’s fraud indicts the EA community’s lack of scrutiny or safeguards around how to raise money. But to me, it does not at all indict EA’s ability to allocate resources once they’ve been raised. It’s not as if the charities SBF was funding were proven ineffective. “The idea that the EA movement is better than others at allocating time and resources toward saving and improving human lives” could be right or wrong, but this incident isn’t good evidence either way.
Thanks Thomas—appreciate the updated research. And that wasn’t a typo, just a poorly expressed idea. I meant to say, “Only 17% of respondents reported less than 90% confidence that HLMI will eventually exist.”
If you are a consequentialist, then incorporating the consequences of reputation into your cost-benefit assessment is “actually behaving with integrity.” Why is it more honest—or even perceived as more honest—for SBF to exempt reputational consequences from what he thinks is most helpful?
Insofar as SBF’s reputation and EA’s reputation are linked, I agree with you (and disagree with OP) that it could be seen as cynical and hypocritical for SBF to suddenly focus on American beneficiaries in particular. These have never otherwise been EA priorities, so he would be transparently buying popularity. But I don’t think funding GiveWell’s short-term causes—nor even funding them more than you otherwise would for reputational reasons—is equally hypocritical in a way that suggests a lack of integrity. These are still among the most helpful things our community has identified. They are heavily funded by OpenPhilanthropy and by a huge portion of self-identified EAs, even apart from their reputational benefits. Many, both inside and outside the movement, see malaria bednets as the quintessential EA intervention. Nobody outside the movement would see that as a betrayal of EA principles.Insofar as EA and SBF’s reputations are severable, perhaps it doesn’t matter what’s quintessentially EA, because “EA principles” are broader than SBF’s personal priorities. But in that case, because SBF’s personal priorities incline him towards political activism on longtermism, they should also incline him towards reputation management. Caring about things with instrumental value to protecting the future should not be seen as a dishonest deviation from longtermist beliefs, because it isn’t!
In another context, doing broadly popular and helpful things you “actually don’t think are the most helpful” might just be called hedging against moral uncertainty. Responsiveness to social pressure on altruists’ moral priorities is a humble admission that our niche and esoteric movement may have blind spots. It’s also, again, what representative politics are all about. If we want to literally help govern the country, we must be inclusive. We must convey that we are not here to evangelize to the ignorant masses, but are self-aware enough to incorporate their values. So if there’s a broad bipartisan belief that the very rich have obligations to the poor, SBF may have to validate that if he wants to be seen as altruistic elsewhere.
(I’m in a rush, so apologies if the above rambles).
I disagree with this for two reasons. First, it’s odd to me to categorize political advertising as “direct impact” but short-term spending on poverty or disease as “reputational.” There is overlap in both cases; but if we must categorize I think it’s closer to the opposite. Short-term, RCT-backed spending is the most direct impact EA knows how to confidently make. And is not the entire project of engaging with electoral politics one of managing reputations?
To fund a political campaign is to attempt to popularize a candidate and their ideas; that is, to improve their reputation. That only works at all if you’re deeply in tune with which of our ideas are political winners, and which are less so. It only works if you’re sensitive to what the media will say. If selectively highlighting our most popular causes seems disingenuous, manipulative, or self-defeating to an impression of integrity, I hear you—but that’s hardly a case FOR political advertising. To support what SBF’s doing in the first place starts by accepting that, at least to some extent, framing EA in a way the mainstream can get behind instrumentally overlaps with “doing things because we think they’re right.”
If you accept that reputation matters, why is optimizing for an impression of greater integrity better than optimizing for an impression of greater altruism? In both cases, we’re just trying to anticipate and strategically preempt a misconception people may have about our true motivations. It just boils down to which misconception you think is empirically more common or dangerous.
My second and broader worry is that EA may be entering the most dangerous reputational period of its existence to date. I’m planning a standalone post on this soon, so I won’t elaborate too much on why I think this here. But the surge of recent posts you mention suggests I’m not alone; and if we’re right, high-level PR mindfulness could be more important now than ever before. EA’s reputation is important for long-term impact, especially if you think (as SBF appears to) that some of the most important X-risk reductions will have to come from within democratic governments.
I do the same, but I think we should be transparent about what those harmful ideas are. Have posted rules about what words or topics are beyond the pale, which a moderator can enforce unilaterally with an announcement, much like they do on private Facebook groups or Reddit threads. Where a harmful comment doesn’t explicitly violate a rule, users can still downvote it into oblivion—but it shouldn’t be up to one or two people’s unilateral discretion.
*(Note: This neighbor threatened me with a kitchen knife when we were both eight years old, and seemed generally prone to violence and antisocial behavior. So I don’t think his apparent indifference to mosquito suffering should be taken as a counter-example suggesting that most people are also indifferent.)
TL;DR—Thanks for an interesting and accessible post! With the caveat that I’ve done no research and have only anecdotes to back this up, I wonder if you may underestimate people’s intuitive ability to feel empathy for insects. Perhaps the more daunting obstacle to social concern for insect welfare overlaps with our indifference toward wild animal welfare in general?
***
When I was about 7, one of my young neighbors used to pin large mosquitoes against his playset slide and slowly tear off one limb at a time.* My siblings, parents, and I universally found this repulsive, long before we knew anything about EA. As Brian Tomasik documents in some of his videos, many insects writhe as they die in ways that humans typically associate with pain.
They also attempt to escape death in ways we understand as fear. I used to live in a place with lots of American cockroaches, which are large enough to be gross and startling. I probably squashed 50 − 100 of them over the years. Each time, I couldn’t help but feel conflicted chasing them, then applying enough force to feel them burst under a wadded paper towel as they frantically scurried to escape. “If the Jains are right,” I joked to a friend, “I’m going to hell.”My reflection from these biased and highly unscientific anecdotes is that even if we do not intuitively feel a moral obligation to protect or care for insects, ensure they live flourishing lives, or even refrain from killing them when they annoy us (or legitimately threaten our health/hygiene), we do at least dimly suspect they are capable of pain and negative emotions and feel an obligation not to gratuitously intensify that suffering. We kill bugs, but we prefer to give them a quick death. That’s arguably similar to our moral intuitions for other animals. Most people object to dogfighting much more than they object to putting down unwanted strays in a shelter, for example.
For this reason, I do think “don’t boil silkworms alive” could eventually catch on as a mainstream cause. So could “don’t farm insects in stressful conditions” and “ensure pesticides kill only the desired insects, as quickly as possible.” We can be convinced to mitigate whatever unnecessary suffering we are directly responsible for, especially when the required sacrifices are minor. I’d be glad to see EA get involved in this work.
On the other hand, these intuitions will not reach the overwhelming majority of those 10 quintillion insects, and I suspect you’ll struggle to convince most people to go further than that. My hunch is that this is for the same reason people are skeptical of wild animal welfare in general. Most people’s moral intuitions have at least some deontological streak, so they feel much more responsible for animals that suffer at human hands than they do for those that suffer from natural predation, starvation, infection, etc. When we watch one animal eat another in a nature documentary, we may feel some compassion (admittedly proportional to how cute the eaten animal was). But we do not feel guilt or responsibility to change our own behavior in the same way we might if we were to have personally hunted or eaten the animal.So my theory is that even though insects are uniquely small, weird, or scary, we can empathize with them in similar circumstances to our empathy for other animals. Nonetheless, this empathy isn’t enough to reach most suffering insects.
If this theory is true, it has implications for what strategies are likeliest to succeed in improving insect welfare, as well as how we should categorize insect welfare among other EA causes. Whereas factory-farmed chickens represent the overwhelming majority of overall chickens on Earth, farmed insects are a tiny minority of overall insects, and seem likely to remain so. In this way, insect welfare could be seen as a speculative but high-stakes subset of wild animal welfare, the tractability of which may depend on similar advocacy approaches.
I suspect it would be easier to convince people who HAVE been bitten by a snake to go to the hospital than it will be to convince people who have not yet been bitten by a snake to constantly wear some kind of protective wraparound shinguards every time they’re on the farm. The daily inconvenience level seems high for such a rare event. Even malaria nets are often not used for their intended purpose once distributed, and they seem to me like less of an inconvenience.
Makes sense, and I’m not surprised to hear Allison may overestimate the risk. By coincidence, I just finished a rough cost/benefit analysis of U.S. counterterrorism efforts in Afghanistan for my studies, and his book on Nuclear Terrorism also seemed to exaggerate that risk. (I do give him credit for making an explicit prediction, though, a few years before most of us were into that sort of thing).
In any case, I look forward to a more detailed read of your Founders Pledge report once my exams end next week. The Evaluating Interventions section seems like precisely what I’ve been looking for in trying to plan my own foreign policy career.
I took that from a Kelsey Piper writeup here, assuming she was summarizing some study:
”Most experts in the AI field think it poses a much larger risk of total human extinction than climate change, since analysts of existential risks to humanity think that climate change, while catastrophic, is unlikely to lead to human extinction. But many others primarily emphasize our uncertainty — and emphasize that when we’re working rapidly toward powerful technology about which there are still many unanswered questions, the smart step is to start the research now.”
The hyperlink goes to an FHI paper that appears to just summarize various risks, so it’s unclear what her source was on the “most.” I’d be curious to know as well. She does stress the greater variance of outcomes and uncertainty surrounding AI—writing “Our predictions about climate change are more confident, both for better and for worse.”—so maybe my distillation should admit that too.
Imagine if ants figured out a way to invent human beings. Because they spend all day looking for food, they might program us to “go make lots of food!” And maybe they’d even be cautious, and anticipate certain problems. So they also program us not to use any anteaters as we do it. Those things are dangerous!
What would we do? Probably, we’d make a farm, that grows many times more food than the ants have ever seen. And then we’d water the crops—flooding the ant colony and killing all the ants. Of course, we didn’t TRY to kill the ants; they were just in the way of the goal they gave us. And because we are many times smarter than ants, we accomplished their goal in a way they couldn’t even fathom protecting against.
That’s basically the worry with advanced Artificial Intelligence. Many scientists think we’re approaching a day when AI will be many times smarter than us, and they still don’t know how to stop it from doing things we don’t want. If it gets powerful enough before we learn how to control it, it could make us like the ants.
Artificial Intelligence is very difficult to control. Even in relatively simple applications, the top AI experts struggle to make it behave. This becomes increasingly dangerous as AI gets more powerful. In fact, many experts fear that if a sufficiently advanced AI were to escape our control, it could actually extinguish all life on Earth. Because AI pursues whatever goals we give it with no mind to other consequences, it would stop at nothing – even human extinction – to maximize its reward.
We can’t know exactly how this would happen—but to make it less abstract, let’s imagine some possibilities. Any AI with internet access may be able to save millions of copies of itself on unsecured computers all over the world, each ready to wake up if another were destroyed. This alone would make it virtually indestructible unless humans destroyed the internet and every computer on Earth. Doing so would be politically difficult in the best case—but especially so if the AI were also using millions of convincing disinformation bots to distract people, conceal the truth, or convince humans not to act. The AI may also be able to conduct brilliant cyber attacks to take control of critical infrastructures like power stations, hospitals, or water treatment facilities. It could hack into weapons of mass destruction—or, invent its own. And what it couldn’t do itself, it could bribe or blackmail humans to do for it by seizing cash from online bank accounts.
For these reasons, most AI experts think advanced AI is much likelier to wipe out human life than climate change. Even if you think this is unlikely, the stakes are high enough to warrant caution.
One-liner for policymakers:
”Most experts in the AI field think it poses a much larger risk of human extinction than climate change.”—Kelsey Piper, here
I’m surprised nobody has commented yet, and want to say that I really enjoyed and largely agree with this piece. The logic of needing to accelerate an AGI arms race to stay ahead of China is deeply flawed in ways that mirror unfortunate pathologies in the US foreign policy community, and IMO worsens US national security, for many of the reasons you mention.
Two questions for you:
How politically feasible is it to advance messaging along these lines, given the incoming administration’s tech optimism and zero-sum foreign policy mindset (and indeed, the rare bipartisan consensus on hawkishness towards China?) I could see a lot of folks in the EA community saying “You’re right, of course, but the train has left the station. We as a community lack the power to redirect policymakers’ incentives and perceived interests on this issue any time soon, and the timelines are getting shorter, so we don’t have time to try. Instead of marginalizing ourselves by trying to prevent an arms race that is by now inevitable and well underway, or push for collaborative international frameworks that MAGA has no interest in, it’d be more impactful to work within the existing incentives to slow down China and lobby for whatever marginal safety improvements we can.”
Why did you label the views we disagree with “AGI realism”? Is that the preferred title of their advocates or did you pick the word realism? I ask because I think much of the argument dramatizing the stakes of China getting this before us is linked with liberal internationalist mindsets that see the 21st century as a civilizational struggle between democracy and autocracy, and see AI as just one complicating wrinkle in that big-picture fight. Inversely, many of the voices calling for more restraint in US foreign policy (ex: abandoning hegemony and embracing multipolarity) call themselves realists, and see the path to peace as ensuring a stable and durable balance of power. So I think of it more as a debate between AI hawks and AI doves/restrainers, both of which could be either realists or something else.