Software engineer interested in AI safety.
Stephen McAleese
Karma: 472
Hi Sanjay, I tried answering that question in this comment. In short, I think a few thousand FTEs seems like a minimally sufficient number based on the resources needed to solve similar historical problems.
Here is a new blog post from 2025 on the subject. The new estimates are 600 technical AI safety FTEs and 500 non-technical AI safety FTEs (1100 in total).
I used Gemini Deep Research to discover organizations and then manually visited their websites to create estimates.
I included organizations I was able to find that are focused on or making significant contribution to AGI safety research or non-technical work like governance and advocacy. Regarding the organizations you listed, I never came across them during my search and I will work on including them now.
Thanks for your feedback Sean.
Estimating the number of FTEs at the non-technical organizations is not straightforward since often only a fraction of the individuals are focused on AI safety. For each organization I guessed what fraction of the total FTEs were focused on AI safety though I may have overestimated in some cases (e.g. in the case of CFI I can decrease my estimate).
Also I’ll include more frontier labs in the list of non-technical organizations.
The technical AI safety organizations cover a variety of different areas including AI alignment, AI security, interpretability, and evals with the most FTEs working on empirical AI safety topics like LLM alignment, jailbreaks, and robustness which covers a variety of different risks including misalignment and misuse.
Thanks for your feedback Ben.
I totally agree with point 1. and you’re right that this post is really estimating the total number of people who work at AI safety organizations and then using this number as a proxy for estimating the size of the field. As you said, there are a lot of people who aren’t completely focused on AI safety but still make significant contributions to the field. For example, a AI researcher might consider themselves to be an “LLM researcher” and split their time between non-AI safety work like evaluating models on benchmarks and AI safety work like new alignment methods. Such a researcher would not be counted in this post.
I might add an “other” category to the estimate to avoid this form of undercounting.
Regarding point 2, I collected the list of organizations and estimated the number of FTEs at each using a mixture of Google Search and Gemini Deep Research. The lists are my attempt to find as many AI safety organizations as possible though of course, I may be missing a few. If you can think of any that aren’t in the list, I would appreciate if you shared them so that I can add them.
Thanks for the insightful observation! I think the reason why the graph starts to flatten out in 2025 simply because it’s only September of 2025 so all the organizations that will be founded in 2025 aren’t included in the dataset yet.
I would like to see a push towards increasing donations to x-risk reduction and longtermist charities. Last time I checked, only about 10% of GWWC donations were going to longtermist funds like the Long-Term Future Fund. Consequently, I think the x-risk and AI safety funding landscapes have been more reliant on big donors than they should be.
I think avoiding existential risk is the most important thing. As long as we can do that and don’t have some kind of lock in, then we’ll have time to think about and optimize the value of the future.
£110k seems like it would probably be impactful, and that’s just one person giving right? That’s probably at least one FTE. Also SERI MATS only costs about ~£500k per year so it could be expanded substantially with that amount.
Thank you for your comment.
Regarding evals, I was referring specifically to evals focused on AI safety and risk-related behaviors like dangerous capabilities, deception, or situational awareness (I will edit the post). I think it’s important to measure and quantify these capabilities to determine when risk mitigation strategies are necessary. Otherwise we risk deploying models with hidden risks and insufficient safeguards.
Exaggerating the risks of current AI models would be misleading so we should avoid that. The point I intended to communicate was that we should try to accurately inform everyone about both the risks and benefits of AI and the opinions of different experts. Given the potential future importance of AI, I believe the quantity and quality of discussion on the topic is too low and this problem is often worsened by the media which tends to focus on short-term events rather than what’s important in the long term.
More generally, while we should aim to avoid causing harm, avoiding all actions that have a non-zero risk of causing harm would lead to inaction.
If overly cautious individuals refrain from taking action, decision making and progress may then be driven by those who are less concerned about risks, potentially leading to worse overall situation.
Therefore, a balanced approach that considers the risks and benefits of each action without stifling all action is needed to make meaningful progress.
Now the post is updated with 2024 numbers :)
I didn’t include Longview Philanthropy because they’re a smaller funder and a lot of their funding seems to come from Open Philanthropy. There is a column called “Other” that serves as a catch-all for any funders I left out.
I took a look at Founder’s Pledge but their donations don’t seem that relevant to AI safety to me.
Do you think wild animals such as tuna and deer are a good option too since they probably have a relatively high standard of living compared to farmed animals?
LTFF, LessWrong.
I’ve never heard this idea proposed before so it seems novel and interesting.
As you say in the post, the AI risk movement could gain much more awareness by associating itself with the climate risk advocacy movement which is much larger. Compute is arguably the main driver of AI progress, compute is correlated with energy usage, and energy use generally increases carbon emissions so limiting carbon emissions from AI is an indirect way of limiting the compute dedicated to AI and slowing down the AI capabilities race.
This approach seems viable in the near future until innovations in energy technology (e.g. nuclear fusion) weaken the link between energy production and CO2 emissions, or algorithmic progress reduces the need for massive amounts of compute for AI.
The question is whether this indirect approach would be more effective than or at least complementary to a more direct approach that advocates explicit compute limits and communicates risks from misaligned AI.
A recent survey of AI alignment researchers found that the most common opinion on the statement “Current alignment research is on track to solve alignment before we get to AGI” was “Somewhat disagree”. The same survey found that most AI alignment researchers also support pausing or slowing down AI progress.
Slowing down AI progress might be net-positive if you take ideas like longtermism seriously but it seems challenging to do given the strong economic incentives to increase AI capabilities. Maybe government policies to limit AI progress will eventually enter the Overton window when AI reaches a certain level of dangerous capability.
This is a cool project! Thanks for making it. Hopefully it makes the book more accessible.
Update: the UK government has announced £8.5 million in AI safety funding for systematic AI safety and these grants will probably be distributed in 2025.
I use Toggl for tracking how much time I spend on all work tasks and I highly recommend it. It has both a website and mobile app which I find intuitive. Although using it requires non-zero effort, I find that using Toggl helps me start tasks and feel more intentional and aware about how I spend my time.