Sophie—thanks for your reply. I agree. And I strongly upvoted your more recent post that you mentioned, which is excellent.
soon! writing up some feedback on the winning essays and some reflections
Ex-DeepMind scientist David Silver has just raised a $5 billion valuation for his new startup, and pledged to donate 100% of the proceeds from his equity stake via Founders Pledge.
Are we prepared for the AI money to start hitting?
Hi Fin, sorry I’m a bit late with my question, I was rereading parts of the Better Futures series. First of all, I have to say it’s one of my favorite article series I’ve ever read, and I’ll be citing it in my own work going forward. The easygoing-versus-fussy distinction in particular is something I’m finding really interesting to dig into. :) Would love to discuss it in more detail at some point.
I wanted to push on the metaphor of sailing to an island, which appears at the start of No Easy Eutopia, but my question is going to take some preamble explanation (sorry!).
I find myself preferring a slightly different picture. Rather than thinking of eutopia as an island we’re navigating to, I tend to think of society as the ship itself, drifting through a sea of value over time (a topography of better and worse regions we’re already moving through). Societal change feels to me more like a search through uncharted moral territories than an expedition to a specific destination. On that picture, the priority seems more likely to be “how do we improve the ship, so that society reliably moves toward better regions of the sea?”
A couple of clarifications. First, I grant fussiness, I agree most plausible axiologies locate near-best futures in a very narrow region (I lean towards total hedonistic utilitarianism, myself). Second, I’m not a a quietist, in my own work I’m defending what I call moral niche construction, a fairly interventionist view on which we should actively reshape institutions, technologies, and even our own moral psychology (through things like AI moral decisionmakers or bioenhancement) to push society toward better regions. So the disagreement isn’t really about ambition, either.
Where I want to press is the following. In the ship-improvement picture, I can grant openly that we probably will never reach eutopia. We end up in a high-value region of the sea (in a local optima), much better than where we are now, plausibly very good in absolute terms, but not the narrow island.
That sounds like a concession, but on rereading Convergence and Compromise, it looks to me like the target-pursuit picture probably doesn’t reach the island either: you mention how WAM-convergence is unlikely, partial convergence plus trade faces serious obstacles, value-destroying threats can eat most of the value… So the comparison isn’t “guaranteed eutopia versus probably-not-eutopia”, since you yourself seem pretty pessimistic. So it’s two orientations that both probably miss the island, where one delivers reliable improvements to our current region of the sea along the way, and the other keeps optimizing toward a target it probably won’t hit. And, well, if you miss the moon, you don’t really land upon the stars… you drift in empty space and die, haha.
(There are similar points on Jerry Gaus’ The Tyranny of the Ideal, and on recent debates between ideal theory and non-ideal theory in moral and political philosophy)
So, finally, my question is: given that target-pursuit probably doesn’t reach eutopia either, on the series’ own analysis, why is the practical orientation toward the narrow target rather than toward improving our current region of the sea (e.g. pursuing very high + plausibly easy to reach and resilient local optima)? What’s the case for target-pursuit as a practical orientation, once we factor in that we will probably fail? Is it a case akin to fanaticism, where, if we land in the island, the payoff would be huge?
(Apologies in advance if this is addressed somewhere in the series, my memory context window isn’t large enough to hold the whole essay series at once!)
why haven’t the winners been announced yet?
My big take-away is seeing fear itself as a policy-related variable, and that effective AI governance must consider emotional infrastructure alongside institutional infrastructure.
I’m left wondering how psychology concepts like self-determination theory scale across the individual/micro to the macro (e.g. collective action, institutional behavior, movement building, etc...).
On a meta-note: As a career advisor in this space, a common bottleneck I observe from mid-career professionals is deep uncertainty as to how non-technical experts can contribute to reducing AI catastrophic risk. I hope this work signals what multi-disciplinary thinking can bring to the space.
I will consider this, thanks for the nudge!
Pep Guardiola is first a dancer, second Mourinho’s nemesis, and third a genius endlessly troubled by the impossibility of the game he loves providing him with the control he craves.
Mikel Arteta, Pep’s disciple, is first and foremost a sicko.
There’s all sorts we can learn from these two madmen and their race for the title, starting with how everything is a tradeoff. When you start looking at your favourite game as a short blanket, always forcing the choice between a cold head or cold feet, it makes more sense.
Despite the best efforts of the just try harder and be better at everything crowd, the same goes for the workings of the “real world”. Few easy fixes. No free lunch.
If you’ve been enjoying the Blog, please consider subscribing via Substack! No slop, no spam, and it really helps with the wretched algo, and therefore with the impact. (Nothing at all to do with my ego you understand.)
I found the 80,000 Hours data on anthropogenic risks particularly striking when compared to “common” low-probability events.
Specifically, the fact that a 100-year consistent play of the 6⁄49 lottery yields odds of 1 in 1,300 to win the jackpot makes the 1 in 10 estimate for unaligned AI or 1 in 30 for engineered pandemics feel much more real. Yet we dream about winning the lottery and not focus on a negative risk that is likely to happen during our lifetime.
The spending table also highlights a massive gap in neglectedness was also shocking to me:
The fact that luxury goods spending ($1.3T) nearly matches total Global R&D ($1.5T) suggests a massive misallocation of time/talent and financial resources.
Comparing Musk’s net worth (approx. $200B–$250B) to the global climate spend (>$300B) illustrates how concentrated the power to influence has become.
Securing our future is the ultimate multi-generational project. My personal drive comes from wanting to leave a stable legacy for the next generation (my daughther and her children), but we cannot afford to treat AI safety or pandemic prevention as ‘problems for the youth’ to solve later.
We need to bridge the gap between the emerging talent in the Effective Altruism community and the established leadership of Millenials, Gen X and Boomers. Solving 1-in-10 risks requires a ‘total talent’ approach that spans every age group and professional background.
Yes, exactly the one you linked to. Since you linked it, I assumed it was clear from the context.
I asked to confirm because the page is not technically a paper.
This is why empirical data from those accustomed to suffering would be so valuable.
I very much agree.
The natural equivalent would be contraception for wild animal. A practice that holds significant promise.
I think controlling the fertility of rodents can easily increase or decrease welfare. I believe it may impact soil animals way more than rodents, and I have very little idea about whether it increases or decreases the welfare of soil animals.
While many people try to help birds by using bird-safe glass, providing nesting boxes, or feeding them during winter, the downside is that an artificially inflated population can negatively impact the birds themselves and the insects they hunt.
I agree.
Do you have any specific species in mind?
No. I think I would guess random animals of many species to have negative lives with a probability of around 50 %, including species of nematodes. In addition, I do not expect the uncertainty about whether animals have positive or negative lives to be super correlated across species. So random animals of some species having positive lives would still leave me believing that random animals of some other species could easily have negative lives.
They do and it’s a powerful point. But on the other hand they may be very much unaware of the nature of available tools and solutions. So I think there should probably be some searching — and listening — in both directions. If it’s done in good faith.
Thanks for clarifying, Marcus.
I am very open to funding research on the sentience of nematodes.
Great.
Regarding intensities of pain, I’m open to it, but would be surprised.
Why would you be surprised? I think the uncertainty of the intensity of excruciating pain is a major driver of the uncertainty of the cost-effectiveness of humane slaughter interventions, like advocating for electrically stunning farmed shrimps as done by the Shrimp Welfare Project’s (SWP’s) Humane Slaughter Initiative (HSI).
Welfare comparisons across species are also in scope. I consider Bob Fischer to be one of our best people who has a strong hunch for making his research useful, and as much as is practicable/possible, he should have free rein to do the work he finds most valauble.
I would agree Bob is among the best people to lead research on welfare comparisons across species.
This talk in 2023 is responsible for a lot of my thinking around smaller animals and very much cemented the idea that helping non-human animals was going to be far more cost-effective.
I really liked that talk from Bob. However, I have very little idea about whether interventions targeting invertebrates increase the welfare of their target beneficiaries more or less cost-effectively than ones targeting humans. For individual welfare per fully-healthy-animal-year proportional to “individual number of neurons”^”exponent”, and “exponent” from 0 to 2, which covers the best guesses that I consider reasonable, I estimate that HSI has increased the welfare of shrimps 1.68*10^-6 to 1.68 M times as cost-effectively as GiveWell’s top charities increase the welfare of humans.
Hi there, thanks for your question. At THL UK, our social media strategy is focused on building community in an effort to continue mobilising our followers into active campaign supporters, as well as increasing our reach and engagement to spread awareness of the suffering on factory farms.
For clarity, THL UK is currently active on Facebook, Instagram, YouTube, LinkedIn and Threads. You may have seen THL (US) on TikTok.
Average engagement rates for the non-profit sector fall between 1.3 − 4.4% depending on social media platform (source), and at THL UK we often see our monthly engagement rate figures fall within this range or higher. We are currently in a period of scaling our social media followers and reach to grow awareness in the wider public, and while this will naturally lead to a lower engagement rate (as content is shown to a broader, less ‘warm’ audience), the number of engagements we are receiving overall is vastly increasing as a result.
We have had lots of success with reaching and engaging large audiences with our social media content, for example:
- This Youtube video showing how evidence of poor chicken welfare can be seen on supermarket shelves, which has had 275k+ views, 8k likes, and 1.2k comments—the majority of which are supportive. The video has also performed well above average on Instagram and Facebook.
- This Instagram video raising awareness of the ethical and environmental issues with the use of fast-growing chickens. The video has been viewed 887k times, has had almost 39k likes, and has been shared over 10k times. The share rate is particularly worth highlighting for us, as it’s much higher than usual for similar style content with this amount of views, showing that the video resonated very strongly with people who then felt compelled to spread the word about this animal welfare issue to their own friends and followers.
- This Instagram video, a collaboration we organised with TV doctor and naturalist Dr Amir Khan. It’s been watched over 300k times, has over 2k shares, and hundreds of comments.However, content doesn’t need to go viral to resonate with people and have an impact. Meaningful measures of effectiveness can include people leaving us a thoughtful comment, sending our content to their own network, and/or clicking a link to our website and signing up to take action for animals. We receive comments and DMs on a daily basis from people – many who are new to the issues we’re campaigning on – thanking us for highlighting animal suffering and wanting to know how they can help us stop it. This connection and community building is where effective social media use can lead to big impact.
Social media algorithms are constantly changing, with organic reach and engagement declining on many platforms due to prioritisation of paid ads and changes in how people consume content (i.e. more passively). Because of this, we are constantly evaluating the impact of our social media content and channels through regular analytics reporting. Metrics we measure include reach and likes as well as more meaningful actions such as shares, people taking campaign actions (e.g. signing petitions and emailing policymakers), donations, etc. We use these insights to inform our content strategy going forward.
You may also be interested to hear that, in part due to the declining effectiveness of the channel, we have recently made the strategic decision to stop posting on X (formerly Twitter) and focus our limited resources elsewhere.
Ok yeah I was using terms too loosely. But still:
1. I think we disagree about whether the harms of lawbreaking are mostly internalized. The degradation of social trust in the deliberative process seems bigger to me than the consequences to the individual? As an analogy, shoplifting is an ordinary crime where individuals do face real consequences, but the diffuse harms to consumers and businesses (goods locked up, stores closing) are large and dominate the social calculus.
I don’t think we should speak of “lawbreaking” as a general case in this context; some argue that shoplifting is too lightly punished/prosecuted (especially in eg liberal US cities), but even assuming that’s true, the question remains as to whether the more specific category of say “property damage via protest” is punished too lightly, too harshely, or about right.
My best guess is that it’s not “too lightly” from a purely normie “law and order and human welfare right now” perspective. Many people believe moral-ish things strongly and don’t find property destruction immoral, but far far fewer actually destroy the property of those they think are doing something immoral. This seems like good evidence that the expected punishment (including via informal mechanisms) is not too light.2. The Pigouvian tax comparison doesn’t quite work here because paying a tax contributes to public resources that can directly address the harms of the act or improve welfare elsewhere, making the net outcome neutral. Going to jail doesn’t repair damaged property or restore trust in the democratic process.
I think we are/were both sort failing to decouple Pigouvian taxes and restitution. My understanding about both how the term “Pigouvian tax” is used in econ and about the real world is that even without restitution, you can get to the socially optimal level of some bad with a tax alone and no transfer to victims.
I think the motivating intuition is that the tax is affecting the amount of eg “social disorder” supplied, but the tax revenue is just a transfer of economic power from one party to another—it’s not creating real wealth that can then be given to the victims. So the same amount of real wealth exists before and after the transfer and a separate question is what to do with that wealth given the state of the world (eg you might think that the very well-off who are harmed slightly by some negative externality, say ambient noise, should not be given restitution and a tax on decibels should really flow to some other party like the very poor)
Thanks for your comment. I agree with this assessment.
I wanted this post to describe our approach while communicating that the numbers are not what they seem at first. While I estimate that our overall efficacy is more than your estimated 5% (the follow-ups from GWWC are more robust than you describe), I agree that it is quite low—still, even if these efforts yield just a handful of effective givers in the long run, this will have been a worthwhile use of time.
I by no means believe that this is an optimal approach to tabling, but hope that this post will encourage other organizers to try something and continue to iterate on best strategies—because right now, I am under the impression that there is not very much of this work taking place on campuses.
From what I understand, Forum usage declined for a couple of years and has now stabilised.
My guess is that, since the Forum is mostly used by fairly engaged EAs, the post-FTX slowdown in top-of-funnel growth is showing up here on a lag. People don’t usually arrive on the Forum cold; my impression is there’s typically a runway of a year or two between first contact with EA and posting or commenting in earnest. So a recruitment hole in 2023–24 produces a Forum hole now, even if intake has since recovered.
If that’s roughly right, two things should follow: the share of activity from accounts under ~12 months old will have dropped more than overall activity has, and Forum recovery will lag whatever recovery CEA is seeing at the top of the funnel by another year or two.
And then it’s probably compounded by dispersion — as you point out, a lot of the conversation that would once have happened here has migrated to LW, X, or Substack.
Which paper are you referring to?
Yes, exactly the one you linked to. Since you linked it, I assumed it was clear from the context.
if many prefer ending their lives over excruciating pain, it makes sense to assume they would prefer avoiding 10 min of excruciating pain over losing 24 h of fully healthy life?
I’m certain there are people who would trade 24 hours of a healthy life just to avoid a shot of distilled water. On the other hand, some people are addicted to pain, such as those who engage in self-harm like cutting. They might willingly accept 10 minutes of excruciating pain for free. This is why empirical data from those accustomed to suffering would be so valuable. For me, the 10-minute mark is simply the trade-off I would personally accept.
If yes, do you think its birth should ideally have been avoided in the 1st place?
If prevention is possible, that would be the ideal scenario in my view. The natural equivalent would be contraception for wild animal. A practice that holds significant promise. While many people try to help birds by using bird-safe glass, providing nesting boxes, or feeding them during winter, the downside is that an artificially inflated population can negatively impact the birds themselves and the insects they hunt. Contraception could offer a way to provide help while keeping populations balanced.
Imagine a pet is born with some disease that allows them to live a long live, but one which has way more suffering than happiness. Do you think such pet should be euthanised?
Yes, if the condition is severe. That represents a very specific intervention. In the wild, such an animal wouldn’t survive anyway. Furthermore, since most chronic diseases in dogs result from human selection, I’m unsure how this specific logic applies to interventions in the wild.
Another argument for euthanasia is that the animal’s presence will likely be substituted. Either by another pet or by various wild animals. By freeing up ecological resources and space, other individuals can take its place who are likely to experience a much better quality of life. This reasoning translates well to wild animal welfare. If we believe certain species, like bees, live reasonably good lives while others, like Varroa destructor, primarily cause suffering, it makes sense to favor the former over the latter. That’s not the same as “bringing numbers down.”
I guess some do
Do you have any specific species in mind? You mentioned nematodes which is a whole phyllum. Why do you expect them to have “net negative lives.”
Muchas gracias Agustín! 😊
Many thanks for the review! I’ll keep your perspectives in mind over the next few months as we ramp up our next round of Replication Games.
Here to confirm much of what was said is true. Always happy to talk about how to improve the process.Completely agree with the challenges of group dynamics. It is hard to know ex ante how teams will interact. Right now we try and keep people who program in the same language, have similar research interests, and are at different stages of their career (graduate student, postdoc, professors, etc.). We are at the whim of who signs up and do the best with the volunteers.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment?
One thing I’m piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help?
Less important but clarifying:
Papers are not screened for influence; they are screened for data, code and likelihood to reproduce as indicated from a README (the selection is mainly on journals who provide a codebase and allow reproduction)
I4R can handle food, building and practicalities (it just depends on the event but happens more often at conferences)
I4R tries to give at least a year off before returning to the same city.
The event participated in was joint with another institute which lead to some complications (I only booked my flight 5 days before the event not really having expected to go).
Happy Replicating!
I think preparing for AI money is generally smart given Anthropic & OpenAI Foundation, though I don’t expect Ineffable specifically to have liquidity for at least a couple years.
It’s possible that there are some clever schemes that could allow David or others to start donating sooner (eg some liquidity at a raise, or borrowing against value of stock), but historically it’s not until IPO (and sometimes much later) before founders donate significant amounts.