Book a 1:1 with me: https://cal.com/tylerjohnston/book
Share anonymous feedback with me: https://www.admonymous.co/tylerjohnston
Tyler Johnston
Karma: 1,520
Good question! I basically agree with you about the relative importance of foundation model developers here (although I haven’t thought too much about the third point you mentioned. Thanks for bringing it up.)
I should say we are doing some other work to raise awareness about foundation model risks—especially at OpenAI, given recent events—but not at the level of this campaign.
The main constraint was starting (relatively) small. We’d really like to win these campaigns, and we don’t plan to let up until we have. The foundation model developers are generally some of the biggest companies in the world (hence the huge compute, as you mention), and the resources needed to win a campaign likely scale in proportion to the size of the target. We decided it’d be good to keep building our supporter base and reputation before taking the bigger players on. Cognition in particular seems to be in the center of the triple venn diagram between “making high-risk systems,” “way behind the curve on safety issues,” and “small enough that they can’t afford to ignore this.”
Btw, my background is in animal advocacy, and this is somewhat similar to how groups scaled there. i.e. they started by getting local restaurants to stop serving fois gras, and scaled up to getting McDonalds to phase out eggs from battery cages nationwide. Obviously we have less time with this issue—so I would like to scale quickly.
Announcing The Midas Project — and our first campaign (which you can help with!)
The THL estimate is a little strange, I think — the $2.63 is really just their US branch’s total 2022 expenses on cage-free campaigns divided by the current number of hens (presently, or at any given time) in the supply chain of companies they persuaded that year. I’m not sure how they are calculating cage-free campaign spend as a proportion of total budget, nor what “persuaded” means (anyone they did outreach to? anyone they secured new commitments from?). Also, the number doesn’t account for the fact that once one hen dies, another takes its place in the same living conditions (although the article acknowledges this limitation). So the real value is the delta, in years, between if/when cage-free would have taken hold by default, and when it did/will thanks to their campaign.
Saulius, the author of the RP report that estimates 12-160 chicken-years impacted per dollar spent, says the following as of 3 months ago:
A new estimate would probably output a similar number because reforms have probably gotten less effective, but I now think that I underestimated cost-effectiveness in this report.
Meanwhile Open Phil says the following, about the same report, but referring to marginal opportunities in particular. It’s unclear to me if they’re thinking of cage-free campaign spend as a “marginal FAW funding opportunity” however.
We think that the marginal FAW funding opportunity is ~1/5th as cost-effective as the average from Saulius’ analysis.
Thanks for this reply — it does resonate with me. It actually got me thinking back to Paul Bloom’s Against Empathy book, and how when I read that I thought something like: “oh yeah empathy really isn’t the best guide to acting morally,” and whether that view contradicts what I was expressing in my quick take above.
I think I probably should have framed the post more as “longtermism need not be totally cold and utilitarian,” and that there’s an emotional, caring psychological relationship we can have to hypothetical future people because we can imaginatively put ourselves in their shoes. And that it might even incorporate elements of justice or fairness if we consider them a disenfranchised group without representation in today’s decision making who we are potentially throwing under the bus for our own benefit, or something like that. So justice and empathy can easily be folded into longtermist thinking. This sounds like what you are saying here, except maybe I do want to stand by the fact that EA values aren’t necessarily trading off against justice, depending on how you define it.
If we go extinct, they won’t exist
Yeah, I meant to convey this in my post but framing it a bit differently — that they are real people with valid moral claims who may exist. I suppose framing it this way is just moving the hypothetical condition elsewhere to emphasize that, if they do exist, they would be real people with real moral claims, and that matters. Maybe that’s confusing though.
BTW, my personal views lean towards a suffering-focused ethics that isn’t seeking to create happy people for their own sake. But I still think that, in coming to that view, I’m concerned with the experience of those hypothetical people in the fuzzy, caring way that utilitarians are charged with disregarding. That’s my main point here. But maybe I just get off the crazy train at my unique stop. I wouldn’t consider tiling the universe with hedonium to be the ultimate act of care/justice, but I suppose someone could feel that way, and thereby make an argument along the same lines.
Agreed there are other issues with longtermism — just wanted to respond to the “it’s not about care or empathy” critique.
This is a cold take that’s probably been said before, but I thought it bears repeating occasionally, if only for the reminder:
The longtermist viewpoint has gotten a lot of criticism for prioritizing “vast hypothetical future populations” over the needs of “real people,” alive today. The mistake, so the critique goes, is the result of replacing ethics with math, or utilitarianism, or something cold and rigid like that. And so it’s flawed because it lacks the love or duty or “ethics of care” or concern for justice that lead people to alternatives like mutual aid and political activism.
My go-to reaction to this critique has become something like “well you don’t need to prioritize vast abstract future generations to care about pandemics or nuclear war, those are very real things that could, with non-trivial probability, face us in our lifetimes.” I think this response has taken hold in general among people who talk about X-risk. This probably makes sense for pragmatic reasons. It’s a very good rebuttal to the “cold and heartless utilitarianism/pascal’s mugging” critique.
But I think it unfortunately neglects the critical point that longtermism, when taken really seriously — at least the sort of longtermism that MacAskill writes about in WWOTF, or Joe Carlsmith writes about in his essays — is full of care and love and duty. Reading the thought experiment that opens the book about living every human life in sequential order reminded me of this. I wish there were more people responding to the “longtermism is cold and heartless” critique by making the case that no, longtermism at face value is worth preserving because it’s the polar opposite of heartless. Caring about the world we leave for the real people, with emotions and needs and experiences as real as our own, who very well may inherit our world but who we’ll never meet, is an extraordinary act of empathy and compassion — one that’s way harder to access than the empathy and warmth we might feel for our neighbors by default. It’s the ultimate act of care. And it’s definitely concerned with justice.
(I mean, you can also find longtermism worthy because of something something math and cold utilitarianism. That’s not out of the question. I just don’t think it’s the only way to reach that conclusion.)
I broadly want to +1 this. A lot of the evidence you are asking for probably just doesn’t exist, and in light of that, most people should have a lot of uncertainty about the true effects of any overton-window-pushing behavior.
That being said, I think there’s some non-anecdotal social science research that might make us more likely to support it. In the case of policy work:
Anchoring effects, one of the classic Kahneman/Tversky biases, have been studied quite a bit, and at least one article calls it “the best-replicated finding in social psychology.” To the extent there’s controversy about it, it’s often related to “incidental” or “subliminal” anchoring which isn’t relevant here. The market also seems to favor a lot of anchoring strategies (like how basically everything on Amazon in “on sale” from an inflated MSRP), which should be a point of evidence that this genuinely just works.
In cases where there is widespread “preference falsification,” overton-shifting behavior might increase people’s willingness to publicly adopt views that were previously outside of it. Cass Sunstein has a good argument that being a “norm entrepreneur,” that is, proposing something that is controversial, might create chain-reaction social cascades. A lot of the evidence for this is historical, but there are also polling techniques that can reveal preference falsification, and a lot of experimental research that shows a (sometimes comically strong) bias toward social conformity, so I suspect something like this is true. Could there be preference falsification among lawmakers surrounding AI issues? Seems possible.
Also, in the case of public advocacy, there’s some empirical research (summarized here) that suggests a “radical flank effect” whereby overton-window shifting activism increases popular support for moderate demands. There’s also some evidence pointing the other direction. Still, I think the evidence supporting is stronger right now.
P.S. Matt Yglesias (as usual) has a good piece that touches on your point. His takeaway is something like: don’t engage in sloppy Overton-window-pushing for its own sake — especially not in place of rigorously argued, robustly good ideas.
For what it’s worth, I have no affiliation with CE, yet I disagree with some of the empirical claims you make — I’ve never gotten the sense that CE has a bad reputation among animal advocacy researchers, nor is it clear to me that the charities you mentioned were bad ideas prior to launching.
Then again, I might just not be in the know. But that’s why I really wish this post was pointing at specific reasoning for these claims rather than just saying it’s what other people think. If it’s true that other people think it, I’d love to know why they think it! If there are factual errors in CE’s research, it seems really important to flag them publicly. You even mention that the status quo for giving in the animal space (CE excepted) is “very bad already,” which is huge if true given the amount of money at stake, and definitely worth sharing examples of what exactly has gone wrong.
Idea: Donation-directing bounty board for the EA Forum
I’ve been thinking about Emre’s comment since I read it — and given this event on the Forum, I eventually decided to go and read Marcus Rediker’s biography of Lay. I recommend it for anyone interested in learning more about him as a historical figure.
To share some thoughts on the questions you posed, my feeling is that his extreme protests weren’t based on any strategic thinking about social change, and I definitely don’t think he’d be an incrementalist if he were alive today. Rather, I think his actions were driven by his extremely firm, passionately felt, and often spiritually-derived moral convictions — the same ones that convinced him to live in a cave and practice radical self-sufficiency. Actually, it seems like he had what we might describe as an excessive degree of “soldier mindset.” From the Rediker text:
He was loving to his friends, but he could be a holy terror to those who did not agree with him. He was aggressive and disruptive. He was stubborn, never inclined to admit a mistake. His direct antinomian connection to God made him self-righteous and at times intolerant. The more resistance he encountered, or, as he understood it, the more God tested his faith, the more certain he was that he was right. He had reasons both sacred and self-serving for being the way he was. He was sure that these traits were essential to defeat the profound evil of slavery.
I don’t know if the EA community would be wrong to exclude him today. He turned out to be ahead of his time in so many ways, and probably did meaningfully influence the eventual abolition of slavery, but this is so much easier to celebrate ex post. What does it actually feel like from the inside, to have extreme personal convictions that society doesn’t share, and how do you know (1) that history will prove you right; and (2) that you are actually making a difference? I really worry that what it feels like to be Benjamin Lay, from the inside, isn’t so dissimilar from what it feels like to be a Westboro Baptist Church member today.
I do think the role of radicalism in driving social change is underrated in this community, and I think it played a big role in not only the slavery abolition movement but also the women’s suffrage movement, the civil rights movement, the gay rights movement, etc. It’s worth looking into the radical flank effect or Cass Sunstein’s writing on social change if you are curious about this. Maybe one thing I’d like to believe is that the world is antifragile and can tolerate radicals ranging the moral spectrum, and those who are on the right side of history will eventually win out, making radicalism a sort of asymmetric weapon that’s stronger when you are ahead of your time on the moral arc of history. But that’s a very convenient theory and I think it’s hard to know with any confidence, and the success of so many fascist and hateful ideologies in relatively recent history probably suggests otherwise.
In any case, I really admire Lay for his conviction and his empathy and his total dedication to living a principled life. But I also really admire communities like this one for their commitment to open debate and the scout mindset and earnest attempts to hear each other out and question our own assumptions. So I expect, and hope, that the EA community would ban Benjamin Lay from our events. But I also hope we wouldn’t laugh at him like so many Quakers did. I hope we would look at him, scowling at us through the glass, and ask ourselves with total sincerity, “What if he has a point?”
I’m not affiliated with 80k, but I would be surprised if the average reader who encounters their work comes away from it with higher regard for AI labs than they came in with — and certainly not that there is something like a brand partnership going on. Most of the content I’ve seen from them has (in my reading) dealt pretty frankly with the massive negative externalities that AI labs could be generating. In fact, my reading of their article “Should you work at a leading AI lab?” is that they don’t broadly recommend it at all. Here’s their 1-sentence summary verbatim:
Recommendation: it’s complicated
We think there are people in our audience for whom this is their highest impact option — but some of these roles might also be very harmful for some people. This means it’s important to take real care figuring out whether you’re in a harmful role, and, if not, whether the role is a good fit for you.
Hopefully this is helpful. It also sounds like these questions could be rhetorical / you have suspicions about their recommendation, so it could be worth writing up the affirmative case against working at labs if you have ideas about that. I know there was a post last week about this, so that thread could be a good place for this.
I’ll admit this only came to mind because of the dubious anthropomorphizing in the piece (“mental imagery” and “dreams”), but I’ve really enjoyed Stephen Wolfram’s writings on AI, including Generative AI Space and the Mental Imagery of Alien Minds. I’m guessing your students would enjoy it.
His write-up on ChatGPT is also a very good intro to LLMs and neural networks, touching on some of what’s going on behind the scenes while remaining approachable for non-technical readers.
Yeah, agreed that it’s an odd suggestion. The idea of putting it on a business card feels so counterintuitive to me that I wonder how literally it’s meant to be taken, or if the sentence is really just a rhetorical device the authors are using to encourage the reader.
The mention of “Pareto Productivity Pro” rang a bell, so I double-checked my copy of How to Launch a High-Impact Nonprofit — and sure enough, towards the end of the chapter on productivity, the book actually encourages the reader to add that title to their Linkedin verbatim. Not explicitly as a certification, nor with CE as the certifier, but just in general. I still agree that it could be misleading, but I imagine it was done in fairly good faith given the book suggests it.
However, I do think this sort of resume padding is basically the norm rather than the exception. Somewhat related anecode from outside EA: Harvard College has given out a named award for many decades to the “top 5% of students of the year by GPA.” Lots of people — including myself — put this award in their resume hoping it will help them stand out among other graduates.
The catch is that grade inflation has gotten so bad that something like 30-40% of students will get a 4.0 in any given year, and they all get the award on account of having tied for it (despite it now not signifying anything like “top 5%.”) But the university still describes it as such, and therefore students still describe it that way on resumes and social media (you can actually search “john harvard scholar” in quotes on LinkedIn and see the flexing yourself). Which just illustrates how even large, reputable institutions support this practice through fluffy, misleading awards and certifications.
This post actually spurred me to go and remove the award from my LinkedIn, but I still think it’s very easy and normal to accidentally do things that make yourself look better in a resume — especially when there is a “technically true” justificaiton for it (like “the school told me I’m in the top 5%” or “the book told me I could add this to my resume!”), whether or not this is really all that informative for future employers. Also, in the back of my mind, I wonder whether choosing to not do this sort of resume padding creates bad selection effects that lead to people with more integrity being hired less, meaning even high-integrity people should be partaking in resume padding so long as everyone else is (Moloch everywhere!). Maybe the best answer is just making sure hiring comittees have good bullshit detectors and lean more on work trials/demonstrated aptitude over fancy certifications/job titles.
Thank you for writing this! I think most people considering kidney donation should read something like this. That being said, I would hesitate to recommend this piece to a prospective donor, at least in its current form. I can’t respond in-depth, but maybe the most succinct way to explain why is that I think it has some elements of a soldier mindset. I’ll use the skin in the game sub-section as an example:
Many of the people in the “Harvard” school of Nutrition indeed eat vegetarian diets and limit protein intake, or at least avoid large amounts of saturated fat and fatty meat.
Why is Harvard in scare quotes? That’s just the name of the school. I noticed this throughout the piece, including in the title of section 1c. I don’t think this adds to your argument, and I worry it’s just a rhetorical attack against all medical and professional establishment.
Have most of the researchers looking at kidney donation donated a kidney? Have most nephrology researchers donated a kidney? Most surgeons doing kidney transplants? … Maybe they are all evil people? They will not take even a marginal risk to save a life. Maybe they are all insane and are unable to translate what they learn from data into reality. But then you at least have to accept that you’d be making your decision based on research done by evil and/or insane people.
Is believing that the entire medical establishment is “evil or insane” actually the most parsimonious explanation for one of many stated-revealed preference gaps in the world? I think I should exercise for 60 minutes a day, but I often fail to do that. Does that make me insane? Of course, I think your actual goal is to hint at medical professionals not even endorsing donation in theory:
… Or maybe, their understanding is nuanced enough that they don’t think the risk is marginal.
Nephrologists spill a lot of ink discussing the risks of kidney donation (which I take it you’ve read much of). Why can’t we just trust the things they say about what they think the risk is?
I might have some motivated reasoning here since I donated a kidney. But, for what it’s worth, my experience of the kidney donor evaluation process was basically a bunch of professionals trying to convince me that risks are real and non-negligible, that there are limits to what we can infer about them from the existing literature, that if I have any doubts I shouldn’t donate, etc.
But it wasn’t a screed against it by any means. It seems like the overall take that the vast majority of nephrologists and experts in the field have landed on is something like: “kidney donation is sufficiently safe for the healthy donor, and it provides such significant benefits for the average recipient, that we can, in good faith, devote much of our careers to facilitating it.” [1] And hospitals, notorious for taking overly conservative approaches to care as to avoid legal liability, at least in the US, have agreed with that assessment.
I still think it’s worth critically scrutinizing the purported risks, but I don’t think it’s necessary to postulate that medical professionals secretly think the operation isn’t worth it, nor that they are all “evil and/or insane.”
- ↩︎
They also seem to think it’s a terrific thing for someone to choose to do, and they tend to hold donors in very high regard.
- ↩︎
I think the lesson we can draw from climate and animal rights that you mention—the radical flank effect—shows that extreme actions concerning an issue in general might make incremental change more palatable to the public. But I don’t think it shows that extreme action attacking incremental change makes that particular incremental change more likely.
If I had to guess, the analogue to this in the animal activist world would be groups like PETA raising awareness about the “scam” that is cage-free. I don’t think there’s any reason to think this has increased the likelihood of cage-free reforms taking place — in fact, my experience from advocating for cage-free tells me that it just worsened social myths that the reform was meaningless despite evidence showing it reduced total hours spent suffering by nearly 50%.
So, I would like to see an activist ecosystem where there are different groups with different tactics—and some who maybe never offer carrots. But directing the stick to incremental improvements seems to have gone badly in past movements, and I wouldn’t want to see the same mistake made here.
I think just letting the public now about AI lab leaders’ p(dooms)s makes sense—in fact, I think most AI researchers are on board with that too (they wouldn’t say these things on podcasts or live on stage if not).
It seems to me this campaign isn’t just meant to raise awareness of X-risk though — it’s meant to punish a particular AI lab for releasing what they see as an inadequate safety policy, and to generate public/legislative opposition to that policy.
I think the public should know about X-risk, but I worry using soundbites of it to generate reputatonial harms and counter labs’ safety agendas might make it less likely they speak about it in the future. It’s kind of like a repeated game: if the behavior you want in the coming years is safety-oriented, you should cooperate when your opponent exhibits that behavior. Only when they don’t should you defect.
Being mindful of the incentives created by pressure campaigns
I’ve spent the past few months trying to think about the whys and hows of large-scale public pressure campaigns (especially those targeting companies — of the sort that have been successful in animal advocacy).
A high-level view of these campaigns is that they use public awareness and corporate reputation as a lever to adjust corporate incentives. But making sure that you are adjusting the right incentives is more challenging than it seems. Ironically, I think this is closely connected to specification gaming: it’s often easy to accidentally incentivize companies to do more to look better, rather than doing more to be better.
For example, an AI-focused campaign calling out RSPs recently began running ads that single out AI labs for speaking openly about existential risk (quoting leaders acknowledging that things could go catastrophically wrong). I can see why this is a “juicy” lever — most of the public would be pretty astonished/outraged to learn some of the beliefs that are held by AI researchers. But I’m not sure if pulling this lever is really incentivizing the right thing.
As far as I can tell, AI leaders speaking openly about existential risk is good. It won’t solve anything in and of itself, but it’s a start — it encourages legislators and the public to take the issue seriously. In general, I think it’s worth praising this when it happens. I think the same is true of implementing safety policies like RSPs, whether or not such policies are sufficient in and of themselves.
If these things are used as ammunition to try to squeeze out stronger concessions, it might just incentivize the company to stop doing the good-but-inadequate thing (i.e. CEOs are less inclined to speak about the dangers of their product when it will be used as a soundbite in a campaign, and labs are probably less inclined to release good-but-inadequate safety policies when doing so creates more public backlash than they were facing before releasing the policy). It also risks directing public and legislative scrutiny to actors who actually do things like speak openly about (or simply believe in) existential risks, as opposed to those who don’t.
So, what do you do when companies are making progress, but not enough? I’m not sure, but it seems like a careful balance of carrots and sticks.
For example, animal welfare campaigns are full of press releases like this: Mercy for Animals “commends” Popeye’s for making a commitment to broiler welfare reforms. Spoiler alert: it probably wasn’t written by someone who thought that Popeye’s had totally absolved themselves of animal abuse with a single commitment, but rather it served as a strategic signal to the company and to their competitors (basically, “If you lead relative to your competitors on animal welfare, we’ll give you carrots. If you don’t, we’ll give you the stick.” If they had reacted by demanding more (which in my heart I may feel is appropriate), it would have sent a very different message: “We’ll punish you even if you make progress.” Even when it’s justified [1], the incentives it creates can leave everybody worse off.
There are lots of other ways that I think campaigns can warp incentives in the wrong ways, but this one feels topical.
- ↩︎
Popeyes probably still does, in fact, have animal abuse in its supply chain
- ↩︎
My understanding is that screwworm eradication in North America has been treated by wild animal welfare researchers as a sort of paradigmatic example of what wild animal welfare interventions could look like, so I think it is on folks’ radar. And, as Kevin mentions, it looks like Uruguay is working on this now with hopes of turning it into a regional campaign across South America.
I’m guessing one of the main reasons there hasn’t been more uptake in promoting this idea is general uncertainty — both about the knock-on effects of something so large scale, and about whether saving the lives of animals who would have died from screwworm really results in higher net welfare for those animals (in many cases it’s probably trading off an excruciating death now for a painful death later with added months or years of life in-between that may themselves be net-negative). So I do think it’s a big overstatement for the guest to suggest that eradicating screwworm would be two orders of magnitude better than preventing the next 100 years of factory farming, which basically assumes that the wild animal lives saved directly trade-off (positively) against the (negative) lives of farmed animals.
@saulius might know more about this. One quote from a recent post of his: “To my surprise, most WAW researchers that I talked to agreed that we’re unlikely to find WAW interventions that could be as cost-effective as farmed animal welfare interventions within the next few years.”
Thank you!
You’re right that the main tasks are digital advocacy—but even if you’re not on social media, there are some direct outreach tasks that involve emailing and calling specific stakeholders. We have one task like that live on our action hub now, and will be adding more soon.
Outside of that, we could use all sorts of general volunteer support—anything from campaign recruitment to writing content. Also always eager to hear advice on strategy. Would love to chat more if you’re interested.