An Overview of the AI Safety Funding Situation


AI safety is a field concerned with preventing negative outcomes from AI systems and ensuring that AI is beneficial to humanity. The field does research on problems such as the AI alignment problem which is the problem of designing AI systems that follow user intentions and behave in a desirable and beneficial way.

Understanding and solving AI safety problems may involve reading past research, producing new research in the form of papers or posts, running experiments with ML models, and so on. Producing research typically involves many different inputs such as research staff, compute, equipment, and office space.

These inputs all require funding and therefore funding is a crucial input for enabling or accelerating AI safety research. Securing funding is usually a prerequisite for starting or continuing AI safety research in industry, in an academic setting, or independently.

There are many barriers that could prevent people from working on AI safety. Funding is one of them. Even if someone is working on AI safety, a lack of funding may prevent them from continuing to work on it.

It’s not clear how hard AI safety problems like AI alignment are. But in any case, humanity is more likely to solve them if there are hundreds or thousands of brilliant minds working on them rather than one guy. I would like there to be a large and thriving community of people working on AI safety and I think funding is an important prerequisite for enabling that.

The goal of this post is to give the reader a better understanding of funding opportunities in AI safety so that hopefully funding will be less of a barrier if they want to work on AI safety. The post starts with a high-level overview of the AI safety funding situation followed by a more in-depth description of various funding opportunities.

Past work

To get an overview of AI safety spending, we first need to find out how much is spent on it per year. We can use past work as a prior and then use grant data to find a more accurate estimate.

Overview of global AI safety funding

One way to estimate total global spending on AI safety is to aggregate the total donations of major AI safety funds such as Open Philanthropy (Open Phil).

It’s important to note that the definition of ‘AI safety’ I’m using is AI safety research that is focused on reducing risks from advanced AI (AGI) such as existential risks which is the type of AI safety research I think is more neglected and important than other research in the long term. Therefore my analysis will be focused on EA funds and top AI labs and I don’t intend to measure investment on near-term AI safety concerns such as effects on the labor market, fairness, privacy, ethics, disinformation, etc.

The results of this analysis are shown in the following bar chart which was created in Google Sheets (link) and is based on data from analyzing grant databases from Open Philanthropy (Open Phil) and other funds (Colab notebook link).

Figure: total AI safety spending by EA funds from 2015 − 2023. The first 2023 bar shows actual spending in roughly the first half of 2023 and the second 2023 bar is projected spending.

Some comments on the bar chart above:

  • The reason why the Open Phil grants seem to vary a lot year-by-year is that Open Phil often makes large multi-year grants (e.g. a $11m grant to CHAI over 5 years in 2021).

  • The average annual increase in funding in the chart is about $13 million.

  • There is some overlap between Open Phil and Long-Term Future Fund (LTFF) because Open Phil often donates money to LTFF and about half of LTFF funding comes from Open Phil. In the chart above, I excluded the grants from Open Phil to LTFF from the Open Phil data to avoid double counting.

  • The chart only includes AI safety donations by the main EA funds. Other sources of funding such as smaller funds, academic grants, individual donations, and spending by industry labs are not included in the chart so the estimates are probably conservative. More on this below.

  • The FTX Future Fund amount only includes grants up to August 2022 and could be an overestimate if many of these grants were not fulfilled. More on this below.

Descriptions of major AI safety funds

Open Philanthropy (Open Phil)

Open Philanthropy (Open Phil) is a grant-making and research foundation that was founded in 2017 by Holden Karnofsky, Dustin Moskovitz, and Cari Tuna. Its primary funder is Dustin Moskovitz who made his fortune by founding Facebook and Asana and has a net worth of over $12 billion. Open Phil funds many different cause areas such as global health and development, farm animal welfare, Effective Altruism community growth, biosecurity, and AI safety research.

Since it was founded in 2017, Open Phil has donated about $2.6 billion of which about $307 million was spent on AI safety (~12%). The median Open Phil AI safety grant is $265k and the average grant is $1.7 million. In 2022, Open Phil spent about $73 million on AI safety making it the largest funder of AI safety in EA and probably in the world too.

Open Phil has historically made medium-sized to large grants to organizations such as Epoch AI, the Alignment Research Center, the Center for AI Safety, Redwood Research, and the Machine Intelligence Research Institute (MIRI).

Open Phil’s past grants are publicly available in their grant database which is what I used to create the bar chart below.

Examples of Open Phil AI safety grants in 2022:

  • Alignment Research Center—General Support, November 2022, $1.25m

  • Center for AI Safety—General Support, November 2022, $5.16m

  • Jacob Steinhardt—AI Alignment Research, November 2022, $100k

  • Conjecture—SERI MATS Program in London, October 2022, $457k

  • AI Impacts—General Support, June 2022, $365k

Figure: Open Philanthropy AI safety spending 2015 − 2023

Survival and Flourishing Fund (SFF)

SFF is the second largest funder of AI safety after Open Phil. SFF is mainly funded by Jaan Tallinn who has a net worth of about $900 million. SFF has donated about $30 million to AI safety projects since it started grant-making in 2019. SFF usually has two funding rounds per year. Past SFF grants are available on the SFF website.

SFF spent about $7 million on AI safety in 2022. In its first funding round of 2023, SFF spent about $13 million on AI safety organizations and projects such as ARC Evals, the Center for the Governance of AI, FAR AI, Ought, and Redwood Research. Like Open Phil, SFF also tends to make medium to large grants to AI safety organizations. The median SFF AI safety grant is $216k and the average grant is $390k.

Examples of SFF grants in 2022:

  • Alignment Research Center—General Support, $2.179m

  • SaferAI—General Support, $214k

  • Redwood Research Inc—General Support, $1.274m

  • Alignment in Complex Systems research group—General Support, $425k

Figure: Survival and Flourishing Fund annual AI safety spending 2019 − 2023

FTX Future Fund

The FTX Future Fund was a fund created in February 2022 but it was shut down just a few months later in November 2022 because FTX went bankrupt. From February to June of 2022, the Future Fund donated about $20 million to AI safety projects.

Since FTX filed for bankruptcy on 11 November 2022, many donations after August 11 2022 had to be returned as part of the bankruptcy process so I didn’t include donations made after 1 August 2022 in the total.

Based on the Future Fund grants database, I estimate that the Future Fund donated about $32 million to AI safety projects from February to August 2022.

Examples of Future Fund grants from February to August 2022:

  • Ought—Building Elicit, $5m

  • Lionel Levine, Cornell University—Alignment theory research, $1.5m

  • ML Safety Scholars Program—General Support, $490k

  • AI Impacts—support for the ‘When Will AI Exceed Human Performance?’ survey, $250k

  • Adversarial Robustness Prizes at ECCV, $30k

The Future Fund website and grant database were taken down but are still available here.

Long-Term Future Fund (LTFF)

LTFF is one of the four EA funds along with the Global Development Fund, Animal Welfare Fund, and EA Infrastructure Fund. LTFF makes grants for longtermist projects related to cause areas like pandemics, existential risk analysis, and AI safety.

Unlike the other funds mentioned before, LTFF tends to make smaller grants. Whereas the median Open Phil grant is $265k, the median LTFF grant is just $23k which makes it suitable for smaller projects such as funding individuals or small groups for upskilling, career transitions, or independent research.

LTFF is also not a ‘primary’ fund like the other funds given that it is often funded by other funds. For example, LTFF has received significant donations from Open Phil and SFF.

According to the EA funds sources page, about 40% of LTFF’s funding came from Open Phil in 2022, about 50% came from direct donations and the rest came from other institutional funds. In 2023, LTFF received $3.1 million from Open Philanthropy.

LTFF has granted about $14.6 million in total since it was created in 2018. Of that amount, about $7.7 million was related to AI safety (~53%). In 2022, LTFF granted about $10 million in total and I estimate that about $5 million of that amount was related to AI safety.

Figure: LTFF annual AI safety spending 2018 − 2023

Other sources of funding

Giving What We Can (GWWC)

GWWC is a community of over 8,000 people who have committed to donating at least 10% of their lifetime income to charity. In 2021, GWWC members donated $22.7 million to charity. About 10% of those donations go to longtermist charities (~$2.27m) so I estimate that GWWC contributes about $1 million to AI safety per year.

GWWC lists three different longtermist funds to donate to including the Long-Term Future Fund (LTFF), Patient Philanthropy Fund, and Longtermism Fund. Though in addition to GWWC donations, these funds also receive donations from larger primary funds such as Open Phil.

Nonlinear Fund

The Nonlinear Fund is for projects related to reducing existential risk from AI. They have created a project named Nonlinear Network which allows people looking for AI safety funding to apply for multiple funds at the same time. Nonlinear Fund is also the organization behind Superlinear which offers financial prizes for AI safety challenges.

Lightspeed Grants

Lightspeed Grants is a longtermist grant-making organization created in 2022 that aims to distribute $5 million in its initial funding round. The primary funder of the first round is Jaan Tallinn (Jaan is also the primary funder of SFF).

For-profit companies

Many organizations working on AI safety are for-profit companies that use profits or investments to fund their research rather than philanthropic funding. For example, OpenAI, Anthropic, DeepMind, and Conjecture are for-profit companies that have AI safety teams.

Conjecture was founded in 2022 and has received $25 million in venture capital funding so far. Anthropic was founded in 2021 and has so far received $1.5 billion in venture capital funding. Google DeepMind (originally DeepMind) was founded in 2012 and is now a profitable company with a revenue of about $1 billion per year and similar expenses. OpenAI was founded in 2016 and is probably the best-funded AI startup in the world after its partner organization Microsoft agreed to invest $10 billion in it in 2023. OpenAI has received $11.3 billion in venture capital funding so far.

We can calculate how much funding these companies are contributing to AI safety by estimating how much it would counterfactually cost to fund their AI safety teams if they were non-profits like Redwood Research or MIRI.

Because of various expenses such as payroll taxes, health insurance, and retirement benefits, the true cost of an employee is typically about 30% higher than the cost of an employee’s gross salary. Then there are other expenses such as office space, travel, events, compute, and so on. Based on the tax returns of MIRI and Redwood Research, the ratio of total expenses to wages is 1.6 and 3.3 respectively so I think a good rule of thumb is that an employee’s cost to an organization is about twice their gross salary.

To calculate the total financial contribution of an organization to AI safety research, we can multiply the size of the AI safety team by the average wage of AI safety researchers in the company and the 2X multiplier.

I created a Guesstimate model to calculate the combined contribution of the four companies. The model is also summarized in the following table.

Company nameNumber of employees [1]AI safety team size (estimated)Median gross salary (estimated)Total cost per employee (estimated)Total funding contribution (estimated)
Total $32m

The Guesstimate model’s estimate is $32 million per year (range: $19 − 54 million).

This figure seems intuitively plausible because both for-profit and non-profit organizations have made significant contributions to AI safety. Non-profits such as Redwood Research, MIRI, ARC, and the Center for AI Safety have made many contributions to AI safety research. But so have for-profit AI companies. For example, OpenAI invented RLHF, Anthropic has made significant contributions to interpretability, DeepMind has done work on goal misgeneralization, and the popular concept of viewing language models as simulators was invented at Conjecture. This estimate is probably conservative because I’d imagine that for-profit companies such as OpenAI spend much more on compute than non-profits like Redwood Research.

Academic research

The academic contribution to AI safety could be large. There are over 200 researchers on Google Scholar that have the ‘AI Safety’ tag on their profile (though many of these people work at companies or non-profits rather than academia).

There are several academic professors and research groups working on AI safety such as:

Many different countries fund academic research. For the sake of brevity, I’ll focus on academic funding in the US and the UK below.

Academic research in the US

About 40% of basic scientific research is funded by the US federal government (which often takes the form of grants to universities) and the National Science Foundation (NSF) funds about 25% of all federally funded basic research. It has an annual budget of about $8 billion for all research and $1 billion for computer science research. NSF also offers the Graduate Research Fellowship Program (NSF-GRFP) which provides funding for master’s or doctoral study to about 2,000 US students every year (worth a total of ~$98 million). The acceptance rate of the GRFP is about 16%.

NSF has announced that it will spend $10 million on AI safety research in 2023 and 2024 (20 million in total) though $5 million has come from Open Philanthropy.

A description of the program:

“While traditional machine learning systems are evaluated pointwise with respect to a fixed test set, such static coverage provides only limited assurance when exposed to unprecedented conditions in high-stakes operating environments … Safety also requires resilience to “unknown unknowns”, which necessitates improved methods for monitoring … In some instances, safety may further require new methods for reverse-engineering, inspecting, and interpreting the internal logic of learned models to identify unexpected behavior that could not be found by black-box testing alone.”

From reading the synopsis, my understanding is that it’s focused on research in AI safety areas such as adversarial robustness, anomaly detection, interpretability, and maybe deceptive alignment.

According to the grant description, grant proposals may be submitted by Institutions of Higher Education (e.g. university research groups) or non-profit, non-academic organizations located in the US (e.g. Redwood Research).

Academic research in the UK

In the UK, around half of UK government expenditure on R&D is funded by UK Research and Innovation (UKRI). Every year UKRI allocates approximately £1.5 billion (~$1.9b) for research grants from its total budget of around £8 billion (~$10b). UKRI is composed of nine research councils. One of them is the Engineering and Physical Sciences Research Council which funds research in areas such as mathematics, physics, chemistry, computer science, and artificial intelligence and has a budget of around £1 billion (~$1.3b) per year.

In 2022, UKRI announced a £117 million (~$150m) grant for UK Centres for Doctoral Training in Artificial Intelligence. The grant allows principal investigators of UK research organizations such as universities to apply for funding for fully-funded four-year PhD studentships which are typically 4 years in the UK. The grant is intended to support 10-15 CDTs and each CDT is expected to support at least 50 students so the grant should support at least 500 students.

The scope of the grant covers AI research in priority areas such as using AI to improve scientific productivity, health, the environment, agriculture, defense, creative industries, and responsible and trustworthy AI.

For the responsible and trustworthy AI section, it says, “The expanding capabilities and range of applications of AI necessitate new research into responsible approaches to AI that are secure, safe, reliable and that operate in a way we understand, trust, and can investigate if they fail.”

The financial contribution of academia to AI safety

This post uses Fermi estimates to estimate that the contribution of academia to AI safety is roughly equivalent to EA’s contribution to AI safety. The argument is that academia is huge and does a lot of AI safety-adjacent research on topics such as transparency, robustness, and safe RL. Therefore, even if this work is strongly discounted because it’s only tangentially related to AGI safety, the discounted contribution is still large.

It’s challenging to estimate the financial contribution of academia to AI safety because the total depends a lot on what you measure. For example, the academic contribution to upskilling seems large given that universities offer degrees in subjects like math and computer science and often have large, expensive campuses and other resources that are useful for research. These factors are relevant because some of EA spending on AI safety involves spending on upskilling and infrastructure.

But I’m going to be more conservative and focus only on actual AI safety research that is done by academics and not funded by EA funds such as Open Phil. For example, someone might do a PhD with a focus on AI safety that is funded by the government.

Although government organizations like NSF in the US spend a lot on research, I’m pretty sure that most of the academic AI safety groups mentioned above are funded by EA funds like Open Phil (e.g. CHAI) so the financial contribution of (non-EA) academic funding to AI safety is not as large as you might think.

I made a guesstimate model to estimate the financial contribution of academic funding to AI safety. My lower bound estimate is $220k which includes only EA academics and my upper bound is $64m which includes the non-EA contributions to AI safety.

Based on this information, my best-guest conservative estimate of the financial contribution of academia to AI safety research is about $1 million per year. Combined with the NSF grant, the total is $11 million per year for 2023.

Other individual donors

  • In 2015, Elon Musk donated $10 million to the Future of Life Institute.

  • In 2021, Vitalik Buterin donated $4 million to MIRI.

  • In 2021, an anonymous crypto donor donated $15.6 million to MIRI.

  • MIRI has received donations from many other individuals in the past.

Summary of other sources of funding

Summary of other sources of funding:

  • GWWC: ~$1m per year.

  • For-profit companies: ~$32m per year.

  • Academic research: ~$1m normally, $11m per year in 2023 because of the NSF grant.

  • Individual donors: probably at least $1m per year.

Total from other sources: ~$45m in 2023.

Overview including other sources of funding

The overall funding graph including funding from other sources looks something like this:


This section is for me to attempt to answer some informal questions on AI safety funding.

How do I work on AI safety?

I think there are three main ways to work on AI safety:

  1. Get a job in a for-profit or non-profit organization. I think the most straightforward way to work on AI safety is to get a job working on AI safety in an organization. OpenAI, DeepMind, Redwood Research, FHI, and MIRI are some examples of organizations working on technical AI safety, and organizations such as Epoch AI, the Center for the Governance of AI, and the Centre for Security and Emerging Technology (CSET) work on non-technical AI safety areas like AI governance. Apart from for-profit and non-profit organizations, there may also be opportunities to work on AI safety in government institutions.

  2. Independent research. Another path to working on AI safety is to participate in a training program like AI Safety Camp or SERI MATS. Participants do research under a mentor and may apply for grants from funds such as LTFF to continue doing independent research.

  3. Academic research. Another option is to get a postdoctoral degree such as a master’s or PhD degree. Usually, master’s degrees don’t offer stipends but most PhD degrees do. Postdoctoral researchers and professors are also paid to do research by universities.

If you are not interested in working directly on AI safety or if changing your career path seems too risky, another option is to earn to give. Giving What We Can makes it easy to create regular donations to funds such as the Long-Term Future Fund (LTFF).

What are my chances of getting funded?

LTFF funded about 19% of proposals in this 2021 funding round and 54% in this 2021 funding round so it seems like about a quarter of proposals are funded by LTFF. Though the (now non-existent) FTX Future Fund seems to have had an acceptance rate of just 4%.

Is AI safety talent or funding constrained?

This question does not have a definite answer because it depends on how you define ‘talent’. The higher the talent bar is, the more talent-constrained over funding-constrained the field is, and vice-versa.

This 80,000 post argues that AI safety is probably more constrained by talent than funding. The reason why is that whereas money can be raised quickly, it’s not as easy to fill positions with talented researchers and training takes time. Talent in mentorship or leadership positions is probably even more scarce.

The counterargument is that there are a lot of talented people in EA and that increasing funding would provide more opportunities for training resulting in more talented people. Using that argument, a lack of talent could be caused by a lack of training opportunities which could be caused by a lack of funding.

What may seem like a funding problem could really be a talent problem: often donors will only give money if they see promising research projects and see that progress is tractable. And talent problems could really be funding problems: there may not be enough money to fund training programs (e.g. SERI MATS).

Another possibility is that AI safety is leadership constrained. For example, the SERI MATS 2023 summer program received 460 applicants and only about 60 were accepted (13%) so it doesn’t seem like there was a lack of talent and SERI MATS costs less than $1 million so surely there was enough money for it. But there probably aren’t that many people who could be SERI MATS mentors or create a program like it so my guess is that good leadership and experienced researchers are scarce. This point has been emphasized by Nonlinear.

Another way to answer the question is to ask whether the field would be better off with one more talented person or the salary of that person (e.g. $100,000). If there were lots of talent and not enough funding, choosing the money would be better because it would make it possible to quickly hire another talented researcher. If there were few talented researchers and lots of funding, the talented researcher would be a better choice because otherwise the money would just not be spent.

My overall view is that AI safety is to some extent constrained by all three: more funding would increase talent via more or larger training programs or higher salaries, more talent would attract more funding via credible and tractable research directions and better leadership would benefit both by creating new organizations that could soak up more talent and funding.

  1. ^

    Source: LinkedIn.

Crossposted to LessWrong (61 points, 3 comments)