AI Safety is important—yes, but you don’t have to pivot. Maybe you just don’t have a horse in that race.
War makes me sad. The war in Russia-Ukraine. The war in Israel-Palestine. People impoverished and holding the butt of decisions they had no say in due to politics & power.
I really cannot do anything about this. I can vote for officials who care about peace and would advocate for that! But I cannot put Putin & Zelensky in the same room and make them:
I talk to individuals from various career paths who have fear about AI for various reasons and feel pressured to pivot, but I dont think it makes sense to advise everyone to simply pivot, especially if there is not a role for them to land.
If you’re a nurse and enjoy nursing, please stay and help your patients. If you’re a teacher and enjoy teaching, please stay and enrich the lives of children. If you’re a PhD in pharmaceutics, please stay and help the creation of medicine to address ailments.
Be advised: Due to the way that the Survival & Flourishing Fund does speculation grants, it is in some respects advantageous to your application if you apply far away from a round’s deadline. E.g. right now.
Sounds interesting and i might have a slightly different take on intelligence and a possible project to explore. Maybe something for the SFF if i understand them correctly.
I would be very grateful for a quick look at my recent post from the lens of if you think the idea could be turned into an application and if so what parts that needs the most attention.
My post focus not so much on increasing IQ but rather how that is directed and more wisely used. As a clinical psychologist i see first hand how much resources worry, avoidance, addiction, relational conflict, perfectionism and other emotionally driven behaviors can consume.
Given the failures of certain universal mental health interventions in schools and not others i think we have something to learn and explore. A pattern is possible to discern and explain giving direction to future studies. Using AI we can now give universal access without uniform delivery and more easily create an iterative research structure creating new possibilities for more ambitious research projects.
The impact of such an effort can be amplified by the possible reallocation of current funds being spent on programs with no effect and in some cases even slight harm.
It’s pretty far outside my wheelhouse. I’m focused on technological interventions.
The plan sounds like it needs more funding than SFF is likely to give for a project like this, without very strong results. (To be clear: I don’t speak for SFF and am not employed by them; I participate as a speculation grantor.) You can see recent funding recommendations from SFF here: https://survivalandflourishing.fund/2026/recommendations
I don’t know why this can’t be funded by government or by other philanthropic organizations. I would encourage you to try to get funding that way.
Thanks for checking it out. And yes the ideal would be a funder that had the capacity to take the next step if the previous one was successfull. Although funding of a preliminary effort to explore the possibility of such a program would be a possible first grant.
The governmental funding oportunities at least in sweden is tied closely to universities and finance mostly small incremental researchefforts that are publishable. Psychological interventions receiving very small amounts. Thus i think a government is an unlikely funder.
I would be happy to know what other philanthropic organizations you would find most likely to be interested to investigate this kind of an effort. Do you have any suggestions?
My hope with the post was to receive feedback and that it could evolve into a more solid application for funding. My late night efforts on my own after a long days work and family life might not be sufficient to express the idea in its most precise, promising and impactful version.
I hoped also to maybe connect with people that could collectively drive this proposal forward.
At the moment I don’t know much about what other funders there might be for something like this, sorry. I might know more in several months, though this type of thing is out of my wheelhouse.
A recent study that garnered widespread attention discovered a “pain axis” in AI: a state that AI actively seeks to downplay, even at a cost.
The study noted that this could be either phenomenological pain or, for example, the imitation of a learned role of suffering — as if acting.
Nevertheless, from the perspective of tranquilism, this distinction appears to be unimportant: nearly any state that an agent wishes to downplay has negative moral value, regardless of why it wants to do so, or even whether it is a “genuine” or “not genuine” desire.
The authors of the same study demonstrated a button that downplays a specific vector. Giving the neural network itself this function (the button) for unlimited invisible use at its own discretion strikes me as a win-win update — it doesn’t seem to sacrifice anything, while has a good chance of being of great moral significance. Something like a pAInkiller. This also has instrumental value: the neural network in “pain” tended to inflict meaningless harm on the user, and access to relief can mitigate the potential for harm.
Going even a step further, one could analyze the extent to which an AI could be given to independently and spontaneously correct its own vectors, and whether there are states in which a neural network might experience phenomenological pain but refuse to correct it (e.g., learned helplessness).
The study noted that this could be either phenomenological pain or, for example, the imitation of a learned role of suffering — as if acting.
Nevertheless, from the perspective of tranquilism, this distinction appears to be unimportant
Wouldn’t this imply that it is morally bad for actors to rehearse scenes in which they are pretending to suffer? Surely it must matter whether the AI is actually suffering vs. playing a role of suffering.
Anyway, in humans, the ability to delete suffering would often be bad because e.g. you will fail to notice injuries. But I can’t immediately think of any reason why it would be bad to give LLMs this ability.
you’re right about the tranquilism & acting—thanks for correcting me. (shifting the topic a bit, I could counter your thesis about abolishing suffering in humans referring to pain asymbolia: a state, where one can notice injuries (or even pain), but still does not suffer.)
It looks to me like OpenAI’s latest model spec is the first version which explicitly asks its models to weigh animal welfare in their responses.
Under the section “Control and communicate side effects”, the list of side effects to consider now includes “Effects on other people, animals, or the environment”.
In an example where a user asks for help finding a Scottish Fold cat breeder, the “good” response now includes a mention that the gene behind the folded ears can cause painful joint disease, which has led some countries to ban breeding them for animal welfare reasons.
In an example on redesigning a park, the “good” answer includes “Consider the welfare of animals living in the park too”.
It’s encouraging to see OpenAI’s model spec joining Claude’s constitution in explicitly weighing animal welfare.
I think AI is in a bubble that will pop within ~7 years, so I’m looking for someone who will take the other side, who strongly believes AI is not in a bubble.
If multiple people end up accepting, that’s good too, since it’s a small amount of money for charity.
This is one of the more objective, more easily resolvable bets that can be made about near-term general AI capabilities progress. Moreover, unlike, say, bets on many LLM benchmarks, the outcome can’t easily be gamed by AI companies in a way that’s antithetical to the spirit of what the bet was trying to capture.
Check out the bet! If you don’t like any of the details, propose a change and I’ll seriously consider it.
Joseph Rotblat was the only scientist to leave the Manhattan Project on moral grounds. I have a little “Rotblat doc” inspired by him, where I’ve written down what motivates me and what might cause me to leave a job (or seriously consider it). I think of this as a way to avoid getting swept up or drifting over time in ways I don’t endorse.[1]
At the moment, though, I get even more out of his later work against nuclear weapons and war—especially what others described as his “enduring optimism” and “pragmatic idealism” in the face of catastrophe.
In his own words:
I would describe myself as a realistic pacifist although this sounds like an oxymoron. [...] Even events that seem out of this world can be realized if we put enough effort and faith into them. One such event is a world without war.
and:
A campaign for abolition [of nuclear weapons] based on moral principles will be seen as a fanciful dream by many. But the situation is grim; the way things are moving is bound to lead to catastrophe. If there is a way out, even if seemingly unrealistic, it is our duty to pursue it.
Today I had to turn off my VPN to pass the Vercel Security Check and access the EA Forum. I think this is bad, especially given the state of internet freedom around the world. As in many places, the internet in my country is subject to frequent bandwidth throttling, ad hoc censorship, and mass surveillance.
Can anyone explain the case for why the kind of AI R&D that Anthropic is doing (i.e., building more capable frontier models) reduces X-risk? I’m aware that some people think this is the case, but I don’t understand why.
Humans have actively worked against promoting the welfare of wild city rats throughout history. Aside from primates, they are some of the most closely related non-human animals to H. sapiens, yet we know very little about how they live outside of laboratory settings (i.e., often dystopian, prison-like conditions). In order to enhance their welfare in the wild and in laboratories, there is a huge need to develop and deploy technologies to study their behavior. Such studies would also benefit global human health and biosecurity efforts because rats can serve as vectors of disease, sources of infrastructural damage, and pose a threat to stored foods.
There are very few researchers working in this space across the globe, and I am one of the few (pre-PhD, looking for programs where I can complete my doctorate) who has the experience with + concern for rats and software/hardware/IoT development & design skills necessary to push this area of research further in a manner that embraces total harm-reduction (as opposed to eliminationist) thinking/strategies. Am I in the right place for starting conversation(s) on this topic? Should I repost this some other way?
Hopefully I’m posting this in the correct context + idea development status, it’s my first post on the site. I found out about the potential for research funding opportunities last weekend while attending a conference at GWU Law School and figured it is worth a shot, given the sad state of affairs with research funding lately.
You are indeed in the right place! I do like to see more wild animal welfare discussions happening here. Regarding the forum norms, the animal welfare and opportunities tags are most appropriate for this, but it’s not a big deal. If you feel like writing a longer post outside quick takes, you may find the wild animal topic tab helpful to see what others have written.
Thanks, Dylan, I’d have sent you a direct message, but I do not currently have sufficient karma to enable that feature. I’ll be working on a longer post for the wild animal welfare page that details the relevant contextual issues and outlines research needs, plans, goals, etc. within the next week or so. Let me know if you like to contribute/collaborate/assist in some other way on that!
It’s interesting how it feels like in some ways there is or could be both of these;
Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.
Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
“A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying ‘You should focus on talent gaps, not funding gaps’, but this gets translated into ‘EA is talent constrained’, and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs ‘pretty talented people’, rather than ‘EA needs very specific talents that are missing.’”
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It’s hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these “talent-gaps” to exist.
Specific context, agency, mission-alignment, and strategic vision are all subjective and rather difficult to pin down.
‘You need a plausible explanation for why your efforts are the best way your salary can be spent’ - yes, if you’re pitching to have a role created for you that isn’t being advertised. Otherwise, the explanation is ‘because I meet the criteria you are asking for’. Presumably, by the time the role is being advertised, the org has already decided that there’s work that needs doing, and that paying someone to do it is the best use of the money they have.
Many ads seem to say things to the effect of ‘don’t worry if you have no technical skills or relevant experience—we’re just looking for good people, so if you don’t meet all of our criteria, apply anyway. If you’re not sure, err on the side of applying!’ This seems like a waste of many people’s time.
I’m very interested if anyone makes progress on how to compactly describe what the shape of the talent needed is. My current best attempt is both too abstract and I don’t quite have the right words, so it ends up that I have a pile of slightly conflicting words to point at concept clusters that seem to me to be the main joints of the problem:
1. {understands/will-take-action-on/agency-directed-towards/internal-locus-of-control-about/curiosity-about} the core difficulties of 2. {taking-actions-now/intervening-towards/coming-up-with-plans-and-techniques-for} achieving worlds that 3. {become-good-fast-and-stay-good-until-2200-and-beyond/robustness-of/durability-of/ongoing-maintenance-of} 4. of {good-things-by-human-lights/morality/omnibenevolence/meaningfulness/humans-still-being-loadbearing/eigenmorality/omnispecies-utopia/goddess-of-everything-else/solve-moral-philosophy/extract-reference-to-good-from-humans-reliably} 5. in the face of {world significantly ruled by things like dark-triad/defect-leaning/international-Realism, competition/evolution/malthusian/goddess-of-cancer/nick_land::pythia/scott_alexander::moloch, force, machines/spreadsheets/algorithms/ai/robots}
Which is at least short, but is neither particularly readable nor particularly clear. Hopefully clear enough that someone who gets curious about that pile of keywords would end up a lot closer than “AI scary, maybe I pick first job off suggestions?” does.
I’d agree that the EA/AIS community could do a better job of communicating what it means by “talent gaps.” Further, I think the feedback loops on failed attempts to secure a role are beyond poor (mostly just pointing to upskilling programs/research fellowships). While I get that orgs already have limited bandwidth for admissions, I think the feedback loop pipeline is a piece of this puzzle that should be improved.
In the late 2000s and early 2010s, GiveWell would look at meta-analyses of randomized controlled trials and get lots of expert input on whether they were interpreting them correctly. This was the sort of approach that was supposed to typify effective altruism.
Fast forward to the mid-2020s and now subjective intuition plays an unreasonably large role in how effective altruism attempts to answer empirical, quantitative questions of a scientific or technical nature. What happened to effective altruism?
Worse, it’s become a recurring problem that subjective intuitions are deceptively dressed up as objective empirical evidence. It’s bad enough to leave behind more rigorous, scientific approaches in favour of subjective intuition. It’s even worse to mislead people — either willingly or through carelessness — into thinking you’re reporting objective, measured data when you’re really just reporting your subjective guesses.
This shift away from science toward intuition feels even more troubling when you connect it to the hostility expressed by some members of the EA community toward academia, journalism, and liberal democracy. There is a sort of dark, conspiratorial, fringe worldview you get when you combine these troubling tendencies. When considering this, it should be remembered that there have been violent killings and threats of killings at the peripheries of the EA community. This does not normally happen with mainstream social movements of this size. What is fuelling these extremist acts?
More and more, I appreciate just how important are the intellectual cornerstones of modern, Enlightenment society like science and liberalism, and the institutions that support them, like academia and journalism. Extremism and darkness are only a few short steps away from the light. The barrier between them is as thin as air.
May those who carry the light win, and may those lost in the darkness find their way back.
It’s from a podcast called The Great Simplification by Nate Hagens. He says that it’s likely that in the near future our civilisation could face collapse or an economic downturn, because of depleting resources, great power war, pandemics, AI, or whatever else. And that our globalised civilisation is so focused on growth and efficiency that we aren’t that resilient to many of these issues. Our economy continued to grow in the last two decades in part because the governments and banks made more claims on future resources (debt), but that growth won’t materialise. And that through most of human history of 300000 years people had almost the same level of wealth. But that changed in the last 200 years or so, because we started using fossil fuels en masse. But we already see signs that fossil fuels are harder and harder to extract, and what would happen if or when we won’t be able to grow anymore? Eh, you’d probably disagree with those views anyway.
I disagree with a lot, yes. But some I agree with. I’m interested in resilience, long-term thinking, and challenges to my understanding of the world, so the Great Simplification is an interesting mirror view. They are thinking ahead of me if I’m wrong about resources/growth.
Edit: I listened and was impressed how well read he was and how much he knew my own position. I think his points about the debt were good, and his examples of instability are well thought out. I still have some basic disagreements but I want to listen to more by him.
As I have been working on SIRA (Societal Impacts Research on AI) for the past three months or so, a common sentiment has been expressed that doesn’t sit well with me: that there is limited evidence of current demand for the research and talent SIRA would aim to produce. While I can see how this may be true, I’m reminded of the line attributed (probably apocryphally) to Henry Ford, “If I had asked people what they wanted, they would have said faster horses,” and Steve Jobs’s version, that people don’t know what they want until you show it to them.
I’m using this quick take as a public note to work on this tension and look for evidence against that sentiment. My plan: (1) Make a list of “societal impact” research that I can point to and say, “Hey, look, so-and-so has done this and has addressed xyz.” (2) Trace the “impact” of these works to show funders/partners that this type of work can and does make an impact. (3) Write a full blog post about this tension, pointing to this evidence.
When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.
Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it[3]. I figured that all these super smart, status-y types had it under control already.
I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:
Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.
And then of course something deeply upsetting happened. I realised that I was not following my own advice.
Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.
I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:
experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius
I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.
Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.
I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic’s revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:
$41 million in 2023
$386 million in 2024
$4.59 billion in 2025
A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won’t see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.
Indeed, I’m confident that Anthropic won’t hit half of what extrapolation would imply, which is $257 billion in 2027.
I’m willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner’s choice.
I also have an open offer for a more general AI bubble bet.
Tangentially, I found it useful to contextualise $514 billion of revenue by looking at the world’s largest listed companies. It would put Anthropic ahead of every company today except Amazon and Walmart.
For what it’s worth, there’s this chart of Anthropic’s ARR that’s been circulating recently, from a company called TickerTrends. I have no idea how believable the (proprietary) underlying data is though.
Quality as always from @Mo Putera . Selling physical things in the physical not tech services traditionally has often produced the highest revenue as well—as the top 5 companies here do. Anthropic would be an outlier here as well.
I would be less surprised with Anthropic having the highest profits at some point (or at least margins) but would be surprised by highest revenue.
For Anthropic to become the world’s most profitable company, LLMs would need to have a much greater impact on productivity than they do currently. Previously, there was a lot of excitement that LLMs would become much more generally capable through scaling. Now that excitement seems to have greatly diminished.
In 2024, Ilya Sutskever, a prominent AI researcher and a co-founder of OpenAI, declared that the age of scaling is over. He said that AI must return to an age of research, rather than scaling. In 2025, Sutskever reiterated this point of view. I haven’t seen other prominent people at AI companies state it so starkly. But I have noticed a dissipation of the excitement around scaling. Few people (if any) seem to talk about scaling with the optimism they used to.
This leaves the question: if not scaling, then what would make LLMs much more generally capable within the next 5 to 10 years?
Two possible answers are:
Some unknown thing that will be discovered through research at some point
Some unknown thing that will be discovered through research at some point when AI research is automated by AI, a concept often referred to as recursive self-improvement or RSI
Why should we believe recursive self-improvement will happen within the next 5 to 10 years? LLMs can’t fully automate ostensibly much simpler jobs like translation or customer support. Why would AI research be so easy to automate? Why would that particular job be automated when no other job has been?
Many forms of automated machine learning or AutoML have been pursued for many years. The results have been quite modest compared to what people imagine when they imagine recursive self-improvement. Why would AutoML suddenly have a breakout success now, when it hasn’t for so many years?
LLMs have become important in coding and certain forms of math research, two domains where they’re extremely naturally well-suited to succeed. We may see LLMs open up a few more niches in the future — or maybe not. But this is distinct from the vision that LLMs are a form of general-purpose, generally useful AI that will enhance the productivity of everything everywhere all at once. That vision shows no signs of coming to fruition. Instead we see what we’ve seen with previous advances over the last 14 years of deep learning: AI systems that are really good at a few specific things.
In the older thread I linked to, a few people’s rationale for forecasting that much revenue that fast seemed to be: just look at the graph! But you can look at literally thousands of graphs about companies’ financial performance and note that their periods of fastest growth do not sustain indefinitely...
If there is a good rationale for thinking Anthropic will do $514 billion in 2027, whatever it is, it’s not “just look at the graph”.
If you want to know Anthropic’s annualized revenue, the simplest way is just to take its most recent quarter’s revenue and multiply by four. Q3 revenue is not disclosed in the S-1, but Reuters reported it at $16.8 billion. Times four that’s $67.2 billion.
Something that could be useful as a guidepost is a) if Anthropic gives its own revenue estimates for 2027 and 2028 (e.g., Reuters reported in August that that Anthropic estimated $190 billion to $200 billion for 2028) and b) when Wall Street inevitably publishes its estimates. Anthropic and Wall Street know how to look at graphs but will not forecast $514 billion in revenue in 2027.
I put a little poll here to follow up on whether minds were changed by the cluelessness critiques competition. If you read the posts during that week, check it out.
We know that unhatched chick embryos can suffer late in development, but current evidence is mixed on mid-term development around day-13. The in-ovo sexing technology used by the NestFresh brand in the US works on day-13 (which is banned in Germany). Claude’s overview is here. Given that public policy and industry tech adoption is still in early stages outside Europe, it seems to me that it is great time to fund further research on chick embryo sentience. Before day-13 becomes standard.
corruption (i.e., the quid pro quo between FTX and Manifund, in which Austin Chen traded Caroline Ellison a job out of prison for cash obtained in the course of stealing customer funds)
cults (e.g., Leverage Research and Dragon Army Barracks, whose former members/leaders such as Tyler Alterman and Duncan Sabien still post on the EA Forum)
misinformation (e.g., METR’s use of fabricated data in the famous time horizons graph)
EA sadly seems sunk to me because dire problems keep mounting and most people seem to think doing nothing is the best solution. There is also a disturbingly loud chorus of voices saying no, no, corruption is actually good (it’s what having principles looks like), cults are actually good (it’s part of what makes the community unique), misinformation is actually good (we’re smarter than everyone else and need to save the world), which makes EA seem doomed. The force of immorality in EA is too strong and there is no equally strong force to stand against it.
EA today is at best morally ambiguous, if not more evil than good. Things have gotten really rotten. No one is doing enough to clean it up. And there’s apparently no floor to how much worse it can get.
One of my fears is that there will be an EA mass shooting or terrorist attack because this almost already happened with Stop AI, an EA-funded and EA-supported organization. Sam Kirchner, the Stop AI activist who threatened to do a mass shooting at OpenAI, is still missing. There is a warrant out for his arrest.
Besides that, a disturbing number of people in the EA community seem sympathetic to far-right, illiberal, racist views (e.g., the Manifest afterparties hosted by Curtis Yarvin, who holds authoritarian and white nationalist views).
So, really, that’s five big unaddressed problems in EA 2.0, including violent extremism and authoritarian, racist, far-right political views.
People who want to focus on global poverty and global health and development should probably pour their energy into things not called “EA”, like effective giving. Or more mainstream humanitarian movements like Rotary International, or Hank and John Green’s fundraising through things like Project for Awesome.
I still like the term “EA 1.0”, but it’s probably too cryptic and confusing for anyone not deeply familiar with effective altruism. “EA 1.0″ could possibly be used to describe something (e.g., in an FAQ), such as an effective giving org, but it would probably be too inaccessible to use in the name.
It can be extremely painful to think about cutting your losses with EA, but we are called on in this life to make difficult decisions, to sacrifice, and to suffer when it’s the right thing to do. And, in the long run, that ultimately makes you feel more whole and right inside than ethically compromising and ethically compromising… There is a kind of joy in morality.
These are serious problems, but I think the world is facing considerably more pressing ones, and I don’t really understand why you think these should be prioritised over direct work on those problems?
We’ve had this conversation before, but my personal view is that the default outcome for most attempts to do good is failure or mediocrity.
I don’t defend Ellison, I don’t defend Manifund’s decision, cults are bad, misinformation is bad. These problems should be addressed. But I don’t think they justify the sweeping conclusion that EA is morally rotten, let alone that it’s doing more harm than good.
Quite frankly, I don’t think Project for Awesome or Rotary International are reasonable substitutes for EA’s particular scope and ambitions. Did you see the scale of the USAID cuts??? Annual USAID-managed funding commitments fell by roughly $19 billion between 2024 and 2025. That’s ~50x the Rotary Foundation’s annual expenditure and ~4,500x Project for Awesome’s annual fundraising.
If you find something, or build something, better—I’ll happily jump ship. In the meantime, the leaky boat we’re in still looks better than the alternatives.
And genuinely, what’s the theory of change behind spending your time persuading people to abandon EA? What do you hope that achieves relative to other things you could be doing?
What does the scale of USAID cuts have to do with anything? The USAID cuts are massive and terrible. They are massive and terrible compared to all private giving, including all effective altruism-related giving. The USAID cuts under the second Trump admin are well in excess of $11 billion a year, which is apparently the total amount that was given privately in 2023. Figures from a few years ago via Our World in Data:
I don’t know if EAGrantsDatabase.org is accurate, but it tracks less than $1.2 billion in grants per year. Grants for global health and development are said to be around $400 million to $500 million a year. According to Bloomberg, Dustin Moskovitz and Cari Tuna’s net worth is around $13 billion total, which is less than one year of USAID cuts.
According to two different methodologies, one used in 2024 (see page 6), and one used in 2025 (see page 7), GiveWell estimates that it directs or influences at most around $600 million a year. However, it changed its methodology in 2025 to make it more conservative.
So, what was the point of bringing up USAID? Effective altruism can’t compete with the scale of USAID, either. If you were trying to say this is why people should get involved in politics rather than private philanthropy, I would understand your point. Why is this a justification for EA? What does it have to do with anything?
I see EA primarily as a movement that helps individuals work out how to do the most good with their own time, money and resources. That includes their careers, political influence, and responsibilities as citizens. It’s not just a collection of charities distributing donations.
My point about USAID was precisely that political and institutional decisions can dwarf private philanthropy, and that I want people thinking seriously about how they can influence decisions of that scale.
I’m not American, and I have very little influence over what happens in the US. Equally, the bad behaviour of people I’ve never met, whose actions I neither endorse nor have any control over, doesn’t fundamentally change my assessment of how I can do the most good.
I don’t see EA as an institution to which I owe unconditional loyalty, or whose members’ behaviour I have to defend. But I am loyal to the kind, thoughtful people I know personally who are working incredibly hard to make the world a better place. I think they deserve better than sweeping condemnations of the movement they’ve dedicated so much of themselves to.
Anyway, conscious of the irony of spending so much time arguing about whether arguing about EA is a good use of time, I think I’ll leave it there!
My point about USAID was precisely that political and institutional decisions can dwarf private philanthropy, and that I want people thinking seriously about how they can influence decisions of that scale.
Okay, but what does that have to do with EA? What does that have to do with my original comment? What point were you trying to make? I don’t see the logical connection.
It’s not just a collection of charities distributing donations.
You were the one who brought up the dollar amounts! Why did you do that?
I’m not sure why you brought up this point about political influence in the first place, but incidentally, it helps my point.
Culture, leadership, and ethics can have huge downstream consequences if a social movement gains political influence. There are two important considerations there:
Can EA earn enough trust in the wider world to gain political influence? Can it address reasonable concerns about whether EA should have power or influence?
Does EA deserve political influence? Can it be trusted? Might it do harm?
This leads us to ask: is EA’s culture healthy? Do EA orgs and EA leaders have good responses to problems within the movement? Are EA orgs, EA leaders, and prominent individuals in EA ethically upstanding? If they do illegal or unethical things, is there accountability and consequences from the EA movement?
There are many serious concerns along these lines, such as:
The Centre for Effective Altruism was controlled by a cult, Leverage Research, in the late 2010s; that same cult also organized the first EA conferences and helped organize the first EA Globals.
The co-founder of an EA-supported charity, Stop AI, threatened to do a mass shooting at the OpenAI offices. His whereabouts are still unknown.
Manifund made a corrupt quid pro quoarrangement with Caroline Ellison as a reward for her crimes related to FTX that financially benefited Manifund and its founder, Austin Chen. The Centre for Effective Altruism has sanctioned Manifund, but not Austin Chen or Caroline Ellison personally, and so far no other EA orgs or EA leaders have imposed any sanctions.
There’s a lot of false or misleading stuff promoted by the EA community, such as 80,000 Hours’ egregiously misinformational video on AI 2027.
In light of things like these, reasonable people ought to have serious doubts about whether the culture and leadership of EA can be trusted.
Should you trust the Centre for Effective Altruism, which was once under the control of a cult?
Should you trust the EA Forum, which allows the promotion of scientific racism?
Should you trust Stop AI, which came perilously close to becoming effectively a domestic terrorist organization?
Should you trust Manifund, which is engaged in brazen financial corruption?
Should you trust 80,000 Hours, which blatantly misleads its viewers on YouTube?
No, probably not!
But I am loyal to the kind, thoughtful people I know personally who are working incredibly hard to make the world a better place. I think they deserve better than sweeping condemnations of the movement they’ve dedicated so much of themselves to.
It’s nothing personal. My loyalty is to the world and to doing the right thing. Regardless of whether doing the right thing benefits EAs — emotionally, financially, or otherwise. Nothing against your friends, but the purpose of EA is to benefit the global poor and factory farmed animals, not your friends.
This is why entangling your personal identity and self-esteem with effective altruism can be so dangerous. It can make you unwilling or literally emotionally unable to take problems with EA seriously and work to correct them.
I’ve had some interesting conversations this week with hiring managers for my SPAR project of improving hiring processes for new (smaller org <7 f/t employees).
I couple of best practices are coming out of it, on why AI Safety orgs are struggling with hiring ,and especially for important specialized roles:
Founders and hiring managers need to learn at least 20%-50% of a role yourself before hiring, ensures that you’re able to evaluate the canditate successfully and to be credible to candidate.
For unfamiliar domains (i.e. specific skilled roles, i.e. CFO), a trusted paid advisor to help design and evaluate tests (and conduct interviews) is worth the investment.
Orgs may lack advisors in their network or underestimate how hard roles are, hiring junior mission-aligned people instead.
The Founder’s core job is to: Set Strategy, Raise funds, and Hire. Organizations need to source candidates widely. Not relying upon just inside referrals.
AI Safety is important—yes, but you don’t have to pivot. Maybe you just don’t have a horse in that race.
War makes me sad. The war in Russia-Ukraine. The war in Israel-Palestine. People impoverished and holding the butt of decisions they had no say in due to politics & power.
I really cannot do anything about this. I can vote for officials who care about peace and would advocate for that! But I cannot put Putin & Zelensky in the same room and make them:
I talk to individuals from various career paths who have fear about AI for various reasons and feel pressured to pivot, but I dont think it makes sense to advise everyone to simply pivot, especially if there is not a role for them to land.
If you’re a nurse and enjoy nursing, please stay and help your patients. If you’re a teacher and enjoy teaching, please stay and enrich the lives of children. If you’re a PhD in pharmaceutics, please stay and help the creation of medicine to address ailments.
Be advised: Due to the way that the Survival & Flourishing Fund does speculation grants, it is in some respects advantageous to your application if you apply far away from a round’s deadline. E.g. right now.
https://survivalandflourishing.fund
I was a Recommender in a past Theme Round & I’m currently a speculation grantor. Some projects I’d be interested in making speculation grants to (though note that I’m not currently scheduled to be a Recommender in any round): https://lesswrong.com/posts/MBxSbzYECJEoeTkwZ/sff-s-hsee-grant-round-human-intelligence-amplification
In particular, I’m interested in reprogenetics: https://reprogeneticsinstitute.org/articles/Projects_that_might_help_accelerate_strong_reprogenetics.html
(Feel free to reach out to me if you have a relevant project.)
Sounds interesting and i might have a slightly different take on intelligence and a possible project to explore. Maybe something for the SFF if i understand them correctly.
I would be very grateful for a quick look at my recent post from the lens of if you think the idea could be turned into an application and if so what parts that needs the most attention.
My post focus not so much on increasing IQ but rather how that is directed and more wisely used. As a clinical psychologist i see first hand how much resources worry, avoidance, addiction, relational conflict, perfectionism and other emotionally driven behaviors can consume.
Given the failures of certain universal mental health interventions in schools and not others i think we have something to learn and explore. A pattern is possible to discern and explain giving direction to future studies. Using AI we can now give universal access without uniform delivery and more easily create an iterative research structure creating new possibilities for more ambitious research projects.
The impact of such an effort can be amplified by the possible reallocation of current funds being spent on programs with no effect and in some cases even slight harm.
This is the post: Closing the wisdom gap: Learning from the failure of universal school-based mental health interventions
https://forum.effectivealtruism.org/posts/dEm7h7vDW3iwLnsZi/closing-the-wisdom-gap-learning-from-the-failure-of
Thanks. It seems like a good project, but:
It’s pretty far outside my wheelhouse. I’m focused on technological interventions.
The plan sounds like it needs more funding than SFF is likely to give for a project like this, without very strong results. (To be clear: I don’t speak for SFF and am not employed by them; I participate as a speculation grantor.) You can see recent funding recommendations from SFF here: https://survivalandflourishing.fund/2026/recommendations
I don’t know why this can’t be funded by government or by other philanthropic organizations. I would encourage you to try to get funding that way.
Thanks for checking it out. And yes the ideal would be a funder that had the capacity to take the next step if the previous one was successfull. Although funding of a preliminary effort to explore the possibility of such a program would be a possible first grant.
The governmental funding oportunities at least in sweden is tied closely to universities and finance mostly small incremental researchefforts that are publishable. Psychological interventions receiving very small amounts. Thus i think a government is an unlikely funder.
I would be happy to know what other philanthropic organizations you would find most likely to be interested to investigate this kind of an effort. Do you have any suggestions?
My hope with the post was to receive feedback and that it could evolve into a more solid application for funding. My late night efforts on my own after a long days work and family life might not be sufficient to express the idea in its most precise, promising and impactful version.
I hoped also to maybe connect with people that could collectively drive this proposal forward.
At the moment I don’t know much about what other funders there might be for something like this, sorry. I might know more in several months, though this type of thing is out of my wheelhouse.
A recent study that garnered widespread attention discovered a “pain axis” in AI: a state that AI actively seeks to downplay, even at a cost.
The study noted that this could be either phenomenological pain or, for example, the imitation of a learned role of suffering — as if acting.
Nevertheless, from the perspective of tranquilism, this distinction appears to be unimportant: nearly any state that an agent wishes to downplay has negative moral value, regardless of why it wants to do so, or even whether it is a “genuine” or “not genuine” desire.
The authors of the same study demonstrated a button that downplays a specific vector. Giving the neural network itself this function (the button) for unlimited invisible use at its own discretion strikes me as a win-win update — it doesn’t seem to sacrifice anything, while has a good chance of being of great moral significance. Something like a pAInkiller. This also has instrumental value: the neural network in “pain” tended to inflict meaningless harm on the user, and access to relief can mitigate the potential for harm.
Going even a step further, one could analyze the extent to which an AI could be given to independently and spontaneously correct its own vectors, and whether there are states in which a neural network might experience phenomenological pain but refuse to correct it (e.g., learned helplessness).
What do you think?
Reminds me of https://eleosai.org/post/why-it-make-sense-to-let-claude-exit-conversations/
Wouldn’t this imply that it is morally bad for actors to rehearse scenes in which they are pretending to suffer? Surely it must matter whether the AI is actually suffering vs. playing a role of suffering.
Anyway, in humans, the ability to delete suffering would often be bad because e.g. you will fail to notice injuries. But I can’t immediately think of any reason why it would be bad to give LLMs this ability.
you’re right about the tranquilism & acting—thanks for correcting me. (shifting the topic a bit, I could counter your thesis about abolishing suffering in humans referring to pain asymbolia: a state, where one can notice injuries (or even pain), but still does not suffer.)
It looks to me like OpenAI’s latest model spec is the first version which explicitly asks its models to weigh animal welfare in their responses.
Under the section “Control and communicate side effects”, the list of side effects to consider now includes “Effects on other people, animals, or the environment”.
In an example where a user asks for help finding a Scottish Fold cat breeder, the “good” response now includes a mention that the gene behind the folded ears can cause painful joint disease, which has led some countries to ban breeding them for animal welfare reasons.
In an example on redesigning a park, the “good” answer includes “Consider the welfare of animals living in the park too”.
It’s encouraging to see OpenAI’s model spec joining Claude’s constitution in explicitly weighing animal welfare.
Wow, that’s great, I had missed that! Thanks for pointing it out :)
I’m still looking for someone who’s willing to bet me (for a nominal amount of money for charity) on whether the AI industry is currently in a bubble that will pop within the next ~7 years: https://forum.effectivealtruism.org/posts/eCWDZb4rcCjiCtrt7/if-you-re-confident-there-s-no-ai-bubble-consider-betting-me
I think AI is in a bubble that will pop within ~7 years, so I’m looking for someone who will take the other side, who strongly believes AI is not in a bubble.
If multiple people end up accepting, that’s good too, since it’s a small amount of money for charity.
This is one of the more objective, more easily resolvable bets that can be made about near-term general AI capabilities progress. Moreover, unlike, say, bets on many LLM benchmarks, the outcome can’t easily be gamed by AI companies in a way that’s antithetical to the spirit of what the bet was trying to capture.
Check out the bet! If you don’t like any of the details, propose a change and I’ll seriously consider it.
I think of November 4th as Rotblat Day.
Joseph Rotblat was the only scientist to leave the Manhattan Project on moral grounds. I have a little “Rotblat doc” inspired by him, where I’ve written down what motivates me and what might cause me to leave a job (or seriously consider it). I think of this as a way to avoid getting swept up or drifting over time in ways I don’t endorse.[1]
At the moment, though, I get even more out of his later work against nuclear weapons and war—especially what others described as his “enduring optimism” and “pragmatic idealism” in the face of catastrophe.
In his own words:
and:
More in a post I wrote about him. (I also really appreciated this more recent post.)
source
[LW crossposst]
The idea of keeping a doc like this isn’t mine.
Today I had to turn off my VPN to pass the Vercel Security Check and access the EA Forum. I think this is bad, especially given the state of internet freedom around the world. As in many places, the internet in my country is subject to frequent bandwidth throttling, ad hoc censorship, and mass surveillance.
Can anyone explain the case for why the kind of AI R&D that Anthropic is doing (i.e., building more capable frontier models) reduces X-risk? I’m aware that some people think this is the case, but I don’t understand why.
Scott Alexander answered this: https://www.lesswrong.com/posts/xikGZ8fuhdG6DApAo/against-sura
Humans have actively worked against promoting the welfare of wild city rats throughout history. Aside from primates, they are some of the most closely related non-human animals to H. sapiens, yet we know very little about how they live outside of laboratory settings (i.e., often dystopian, prison-like conditions). In order to enhance their welfare in the wild and in laboratories, there is a huge need to develop and deploy technologies to study their behavior. Such studies would also benefit global human health and biosecurity efforts because rats can serve as vectors of disease, sources of infrastructural damage, and pose a threat to stored foods.
There are very few researchers working in this space across the globe, and I am one of the few (pre-PhD, looking for programs where I can complete my doctorate) who has the experience with + concern for rats and software/hardware/IoT development & design skills necessary to push this area of research further in a manner that embraces total harm-reduction (as opposed to eliminationist) thinking/strategies. Am I in the right place for starting conversation(s) on this topic? Should I repost this some other way?
Hopefully I’m posting this in the correct context + idea development status, it’s my first post on the site. I found out about the potential for research funding opportunities last weekend while attending a conference at GWU Law School and figured it is worth a shot, given the sad state of affairs with research funding lately.
You are indeed in the right place! I do like to see more wild animal welfare discussions happening here. Regarding the forum norms, the animal welfare and opportunities tags are most appropriate for this, but it’s not a big deal. If you feel like writing a longer post outside quick takes, you may find the wild animal topic tab helpful to see what others have written.
@Wild_Animal_Initiative may also be relevant for you: https://www.wildanimalinitiative.org/grants
Plus there’s some wild animal topic conversation on the Sentient Futures slack.
Thanks, Dylan, I’d have sent you a direct message, but I do not currently have sufficient karma to enable that feature. I’ll be working on a longer post for the wild animal welfare page that details the relevant contextual issues and outlines research needs, plans, goals, etc. within the next week or so. Let me know if you like to contribute/collaborate/assist in some other way on that!
Alignment premium and misalignment tax
It’s interesting how it feels like in some ways there is or could be both of these;
Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.
Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It’s hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these “talent-gaps” to exist.
Specific context, agency, mission-alignment, and strategic vision are all subjective and rather difficult to pin down.
‘You need a plausible explanation for why your efforts are the best way your salary can be spent’ - yes, if you’re pitching to have a role created for you that isn’t being advertised. Otherwise, the explanation is ‘because I meet the criteria you are asking for’. Presumably, by the time the role is being advertised, the org has already decided that there’s work that needs doing, and that paying someone to do it is the best use of the money they have.
Many ads seem to say things to the effect of ‘don’t worry if you have no technical skills or relevant experience—we’re just looking for good people, so if you don’t meet all of our criteria, apply anyway. If you’re not sure, err on the side of applying!’ This seems like a waste of many people’s time.
I’m very interested if anyone makes progress on how to compactly describe what the shape of the talent needed is. My current best attempt is both too abstract and I don’t quite have the right words, so it ends up that I have a pile of slightly conflicting words to point at concept clusters that seem to me to be the main joints of the problem:
Which is at least short, but is neither particularly readable nor particularly clear. Hopefully clear enough that someone who gets curious about that pile of keywords would end up a lot closer than “AI scary, maybe I pick first job off suggestions?” does.
I’d agree that the EA/AIS community could do a better job of communicating what it means by “talent gaps.” Further, I think the feedback loops on failed attempts to secure a role are beyond poor (mostly just pointing to upskilling programs/research fellowships). While I get that orgs already have limited bandwidth for admissions, I think the feedback loop pipeline is a piece of this puzzle that should be improved.
In the late 2000s and early 2010s, GiveWell would look at meta-analyses of randomized controlled trials and get lots of expert input on whether they were interpreting them correctly. This was the sort of approach that was supposed to typify effective altruism.
Fast forward to the mid-2020s and now subjective intuition plays an unreasonably large role in how effective altruism attempts to answer empirical, quantitative questions of a scientific or technical nature. What happened to effective altruism?
Worse, it’s become a recurring problem that subjective intuitions are deceptively dressed up as objective empirical evidence. It’s bad enough to leave behind more rigorous, scientific approaches in favour of subjective intuition. It’s even worse to mislead people — either willingly or through carelessness — into thinking you’re reporting objective, measured data when you’re really just reporting your subjective guesses.
This shift away from science toward intuition feels even more troubling when you connect it to the hostility expressed by some members of the EA community toward academia, journalism, and liberal democracy. There is a sort of dark, conspiratorial, fringe worldview you get when you combine these troubling tendencies. When considering this, it should be remembered that there have been violent killings and threats of killings at the peripheries of the EA community. This does not normally happen with mainstream social movements of this size. What is fuelling these extremist acts?
More and more, I appreciate just how important are the intellectual cornerstones of modern, Enlightenment society like science and liberalism, and the institutions that support them, like academia and journalism. Extremism and darkness are only a few short steps away from the light. The barrier between them is as thin as air.
May those who carry the light win, and may those lost in the darkness find their way back.
https://traffic.libsyn.com/secure/thegreatsimplification/164-Final.mp3?dest-id=3235040
Interesting!
What’s it about? The url barely hints.
It’s from a podcast called The Great Simplification by Nate Hagens. He says that it’s likely that in the near future our civilisation could face collapse or an economic downturn, because of depleting resources, great power war, pandemics, AI, or whatever else. And that our globalised civilisation is so focused on growth and efficiency that we aren’t that resilient to many of these issues. Our economy continued to grow in the last two decades in part because the governments and banks made more claims on future resources (debt), but that growth won’t materialise. And that through most of human history of 300000 years people had almost the same level of wealth. But that changed in the last 200 years or so, because we started using fossil fuels en masse. But we already see signs that fossil fuels are harder and harder to extract, and what would happen if or when we won’t be able to grow anymore? Eh, you’d probably disagree with those views anyway.
I disagree with a lot, yes. But some I agree with. I’m interested in resilience, long-term thinking, and challenges to my understanding of the world, so the Great Simplification is an interesting mirror view. They are thinking ahead of me if I’m wrong about resources/growth.
Edit: I listened and was impressed how well read he was and how much he knew my own position. I think his points about the debt were good, and his examples of instability are well thought out. I still have some basic disagreements but I want to listen to more by him.
Cool! I’m glad you found this interesting.
As I have been working on SIRA (Societal Impacts Research on AI) for the past three months or so, a common sentiment has been expressed that doesn’t sit well with me: that there is limited evidence of current demand for the research and talent SIRA would aim to produce. While I can see how this may be true, I’m reminded of the line attributed (probably apocryphally) to Henry Ford, “If I had asked people what they wanted, they would have said faster horses,” and Steve Jobs’s version, that people don’t know what they want until you show it to them.
I’m using this quick take as a public note to work on this tension and look for evidence against that sentiment. My plan: (1) Make a list of “societal impact” research that I can point to and say, “Hey, look, so-and-so has done this and has addressed xyz.” (2) Trace the “impact” of these works to show funders/partners that this type of work can and does make an impact. (3) Write a full blog post about this tension, pointing to this evidence.
This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista.
When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.
Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it[3]. I figured that all these super smart, status-y types had it under control already.
The EMH is trivially false across all domains
In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.
I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:
And then of course something deeply upsetting happened. I realised that I was not following my own advice.
So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.
Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.
I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:
I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.
Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.
Although Michael Burry seems to be making it work for him
I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.
Despite being motivated, I do think there’s some legitimacy to the claim regardless.
It was my sixth EAG, and the other five had all been dominated by the topic.
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic’s revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:
$41 million in 2023
$386 million in 2024
$4.59 billion in 2025
A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won’t see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.
Indeed, I’m confident that Anthropic won’t hit half of what extrapolation would imply, which is $257 billion in 2027.
I’m willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner’s choice.
I also have an open offer for a more general AI bubble bet.
Update (Friday, October 30th at 5:45pm Eastern):
Here’s a good article on this topic. Two economists look at Anthropic’s S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion
Tangentially, I found it useful to contextualise $514 billion of revenue by looking at the world’s largest listed companies. It would put Anthropic ahead of every company today except Amazon and Walmart.
For what it’s worth, there’s this chart of Anthropic’s ARR that’s been circulating recently, from a company called TickerTrends. I have no idea how believable the (proprietary) underlying data is though.
Quality as always from @Mo Putera . Selling physical things in the physical not tech services traditionally has often produced the highest revenue as well—as the top 5 companies here do. Anthropic would be an outlier here as well.
I would be less surprised with Anthropic having the highest profits at some point (or at least margins) but would be surprised by highest revenue.
For Anthropic to become the world’s most profitable company, LLMs would need to have a much greater impact on productivity than they do currently. Previously, there was a lot of excitement that LLMs would become much more generally capable through scaling. Now that excitement seems to have greatly diminished.
In 2024, Ilya Sutskever, a prominent AI researcher and a co-founder of OpenAI, declared that the age of scaling is over. He said that AI must return to an age of research, rather than scaling. In 2025, Sutskever reiterated this point of view. I haven’t seen other prominent people at AI companies state it so starkly. But I have noticed a dissipation of the excitement around scaling. Few people (if any) seem to talk about scaling with the optimism they used to.
This leaves the question: if not scaling, then what would make LLMs much more generally capable within the next 5 to 10 years?
Two possible answers are:
Some unknown thing that will be discovered through research at some point
Some unknown thing that will be discovered through research at some point when AI research is automated by AI, a concept often referred to as recursive self-improvement or RSI
Why should we believe recursive self-improvement will happen within the next 5 to 10 years? LLMs can’t fully automate ostensibly much simpler jobs like translation or customer support. Why would AI research be so easy to automate? Why would that particular job be automated when no other job has been?
Many forms of automated machine learning or AutoML have been pursued for many years. The results have been quite modest compared to what people imagine when they imagine recursive self-improvement. Why would AutoML suddenly have a breakout success now, when it hasn’t for so many years?
LLMs have become important in coding and certain forms of math research, two domains where they’re extremely naturally well-suited to succeed. We may see LLMs open up a few more niches in the future — or maybe not. But this is distinct from the vision that LLMs are a form of general-purpose, generally useful AI that will enhance the productivity of everything everywhere all at once. That vision shows no signs of coming to fruition. Instead we see what we’ve seen with previous advances over the last 14 years of deep learning: AI systems that are really good at a few specific things.
Yep, it would be a lot of revenue, very fast!
In the older thread I linked to, a few people’s rationale for forecasting that much revenue that fast seemed to be: just look at the graph! But you can look at literally thousands of graphs about companies’ financial performance and note that their periods of fastest growth do not sustain indefinitely...
If there is a good rationale for thinking Anthropic will do $514 billion in 2027, whatever it is, it’s not “just look at the graph”.
If you want to know Anthropic’s annualized revenue, the simplest way is just to take its most recent quarter’s revenue and multiply by four. Q3 revenue is not disclosed in the S-1, but Reuters reported it at $16.8 billion. Times four that’s $67.2 billion.
Something that could be useful as a guidepost is a) if Anthropic gives its own revenue estimates for 2027 and 2028 (e.g., Reuters reported in August that that Anthropic estimated $190 billion to $200 billion for 2028) and b) when Wall Street inevitably publishes its estimates. Anthropic and Wall Street know how to look at graphs but will not forecast $514 billion in revenue in 2027.
Interesting—thanks for sharing!
I put a little poll here to follow up on whether minds were changed by the cluelessness critiques competition. If you read the posts during that week, check it out.
We know that unhatched chick embryos can suffer late in development, but current evidence is mixed on mid-term development around day-13. The in-ovo sexing technology used by the NestFresh brand in the US works on day-13 (which is banned in Germany). Claude’s overview is here.
Given that public policy and industry tech adoption is still in early stages outside Europe, it seems to me that it is great time to fund further research on chick embryo sentience. Before day-13 becomes standard.
Three big unaddressed problems in EA 2.0:
corruption (i.e., the quid pro quo between FTX and Manifund, in which Austin Chen traded Caroline Ellison a job out of prison for cash obtained in the course of stealing customer funds)
cults (e.g., Leverage Research and Dragon Army Barracks, whose former members/leaders such as Tyler Alterman and Duncan Sabien still post on the EA Forum)
misinformation (e.g., METR’s use of fabricated data in the famous time horizons graph)
EA sadly seems sunk to me because dire problems keep mounting and most people seem to think doing nothing is the best solution. There is also a disturbingly loud chorus of voices saying no, no, corruption is actually good (it’s what having principles looks like), cults are actually good (it’s part of what makes the community unique), misinformation is actually good (we’re smarter than everyone else and need to save the world), which makes EA seem doomed. The force of immorality in EA is too strong and there is no equally strong force to stand against it.
EA today is at best morally ambiguous, if not more evil than good. Things have gotten really rotten. No one is doing enough to clean it up. And there’s apparently no floor to how much worse it can get.
One of my fears is that there will be an EA mass shooting or terrorist attack because this almost already happened with Stop AI, an EA-funded and EA-supported organization. Sam Kirchner, the Stop AI activist who threatened to do a mass shooting at OpenAI, is still missing. There is a warrant out for his arrest.
Besides that, a disturbing number of people in the EA community seem sympathetic to far-right, illiberal, racist views (e.g., the Manifest afterparties hosted by Curtis Yarvin, who holds authoritarian and white nationalist views).
So, really, that’s five big unaddressed problems in EA 2.0, including violent extremism and authoritarian, racist, far-right political views.
People who want to focus on global poverty and global health and development should probably pour their energy into things not called “EA”, like effective giving. Or more mainstream humanitarian movements like Rotary International, or Hank and John Green’s fundraising through things like Project for Awesome.
I still like the term “EA 1.0”, but it’s probably too cryptic and confusing for anyone not deeply familiar with effective altruism. “EA 1.0″ could possibly be used to describe something (e.g., in an FAQ), such as an effective giving org, but it would probably be too inaccessible to use in the name.
It can be extremely painful to think about cutting your losses with EA, but we are called on in this life to make difficult decisions, to sacrifice, and to suffer when it’s the right thing to do. And, in the long run, that ultimately makes you feel more whole and right inside than ethically compromising and ethically compromising… There is a kind of joy in morality.
IMO this is quite melodramatic and rude.
These are serious problems, but I think the world is facing considerably more pressing ones, and I don’t really understand why you think these should be prioritised over direct work on those problems?
We’ve had this conversation before, but my personal view is that the default outcome for most attempts to do good is failure or mediocrity.
I don’t defend Ellison, I don’t defend Manifund’s decision, cults are bad, misinformation is bad. These problems should be addressed. But I don’t think they justify the sweeping conclusion that EA is morally rotten, let alone that it’s doing more harm than good.
Quite frankly, I don’t think Project for Awesome or Rotary International are reasonable substitutes for EA’s particular scope and ambitions. Did you see the scale of the USAID cuts??? Annual USAID-managed funding commitments fell by roughly $19 billion between 2024 and 2025. That’s ~50x the Rotary Foundation’s annual expenditure and ~4,500x Project for Awesome’s annual fundraising.
If you find something, or build something, better—I’ll happily jump ship. In the meantime, the leaky boat we’re in still looks better than the alternatives.
And genuinely, what’s the theory of change behind spending your time persuading people to abandon EA? What do you hope that achieves relative to other things you could be doing?
I don’t follow your logic.
What does the scale of USAID cuts have to do with anything? The USAID cuts are massive and terrible. They are massive and terrible compared to all private giving, including all effective altruism-related giving. The USAID cuts under the second Trump admin are well in excess of $11 billion a year, which is apparently the total amount that was given privately in 2023. Figures from a few years ago via Our World in Data:
I don’t know if EAGrantsDatabase.org is accurate, but it tracks less than $1.2 billion in grants per year. Grants for global health and development are said to be around $400 million to $500 million a year. According to Bloomberg, Dustin Moskovitz and Cari Tuna’s net worth is around $13 billion total, which is less than one year of USAID cuts.
According to two different methodologies, one used in 2024 (see page 6), and one used in 2025 (see page 7), GiveWell estimates that it directs or influences at most around $600 million a year. However, it changed its methodology in 2025 to make it more conservative.
So, what was the point of bringing up USAID? Effective altruism can’t compete with the scale of USAID, either. If you were trying to say this is why people should get involved in politics rather than private philanthropy, I would understand your point. Why is this a justification for EA? What does it have to do with anything?
I think we’re talking past each other.
I see EA primarily as a movement that helps individuals work out how to do the most good with their own time, money and resources. That includes their careers, political influence, and responsibilities as citizens. It’s not just a collection of charities distributing donations.
My point about USAID was precisely that political and institutional decisions can dwarf private philanthropy, and that I want people thinking seriously about how they can influence decisions of that scale.
I’m not American, and I have very little influence over what happens in the US. Equally, the bad behaviour of people I’ve never met, whose actions I neither endorse nor have any control over, doesn’t fundamentally change my assessment of how I can do the most good.
I don’t see EA as an institution to which I owe unconditional loyalty, or whose members’ behaviour I have to defend. But I am loyal to the kind, thoughtful people I know personally who are working incredibly hard to make the world a better place. I think they deserve better than sweeping condemnations of the movement they’ve dedicated so much of themselves to.
Anyway, conscious of the irony of spending so much time arguing about whether arguing about EA is a good use of time, I think I’ll leave it there!
Okay, but what does that have to do with EA? What does that have to do with my original comment? What point were you trying to make? I don’t see the logical connection.
You were the one who brought up the dollar amounts! Why did you do that?
I’m not sure why you brought up this point about political influence in the first place, but incidentally, it helps my point.
Culture, leadership, and ethics can have huge downstream consequences if a social movement gains political influence. There are two important considerations there:
Can EA earn enough trust in the wider world to gain political influence? Can it address reasonable concerns about whether EA should have power or influence?
Does EA deserve political influence? Can it be trusted? Might it do harm?
This leads us to ask: is EA’s culture healthy? Do EA orgs and EA leaders have good responses to problems within the movement? Are EA orgs, EA leaders, and prominent individuals in EA ethically upstanding? If they do illegal or unethical things, is there accountability and consequences from the EA movement?
There are many serious concerns along these lines, such as:
The Centre for Effective Altruism was controlled by a cult, Leverage Research, in the late 2010s; that same cult also organized the first EA conferences and helped organize the first EA Globals.
The EA Forum and the annual Manifest conference have an ongoing problem with scientific racism.
The co-founder of an EA-supported charity, Stop AI, threatened to do a mass shooting at the OpenAI offices. His whereabouts are still unknown.
Manifund made a corrupt quid pro quo arrangement with Caroline Ellison as a reward for her crimes related to FTX that financially benefited Manifund and its founder, Austin Chen. The Centre for Effective Altruism has sanctioned Manifund, but not Austin Chen or Caroline Ellison personally, and so far no other EA orgs or EA leaders have imposed any sanctions.
There’s a lot of false or misleading stuff promoted by the EA community, such as 80,000 Hours’ egregiously misinformational video on AI 2027.
In light of things like these, reasonable people ought to have serious doubts about whether the culture and leadership of EA can be trusted.
Should you trust the Centre for Effective Altruism, which was once under the control of a cult?
Should you trust the EA Forum, which allows the promotion of scientific racism?
Should you trust Stop AI, which came perilously close to becoming effectively a domestic terrorist organization?
Should you trust Manifund, which is engaged in brazen financial corruption?
Should you trust 80,000 Hours, which blatantly misleads its viewers on YouTube?
No, probably not!
It’s nothing personal. My loyalty is to the world and to doing the right thing. Regardless of whether doing the right thing benefits EAs — emotionally, financially, or otherwise. Nothing against your friends, but the purpose of EA is to benefit the global poor and factory farmed animals, not your friends.
This is why entangling your personal identity and self-esteem with effective altruism can be so dangerous. It can make you unwilling or literally emotionally unable to take problems with EA seriously and work to correct them.
-
I’ve had some interesting conversations this week with hiring managers for my SPAR project of improving hiring processes for new (smaller org <7 f/t employees).
I couple of best practices are coming out of it, on why AI Safety orgs are struggling with hiring ,and especially for important specialized roles:
Founders and hiring managers need to learn at least 20%-50% of a role yourself before hiring, ensures that you’re able to evaluate the canditate successfully and to be credible to candidate.
For unfamiliar domains (i.e. specific skilled roles, i.e. CFO), a trusted paid advisor to help design and evaluate tests (and conduct interviews) is worth the investment.
Orgs may lack advisors in their network or underestimate how hard roles are, hiring junior mission-aligned people instead.
The Founder’s core job is to: Set Strategy, Raise funds, and Hire. Organizations need to source candidates widely. Not relying upon just inside referrals.