Nice post! If someone wants AI governance/āsafety recommendations, feel free to message me. Thereās a set of orgs Iām confident are things that (a) non-OP donors are better suited than OP to funding and (b) people like me and OP grantmakers think are good. These are what I give to myself. Up to a given person whether they want my suggestions, of course!
(I was previously a grantmaker for EA Funds, and have been in the AI governance space for a few years.)
Hi Richard, quick reactions without having much context:
If you mean this is all one company, this sounds like putting too many eggs in one basket, and insufficiently exploring.
I think itās generally good to apply to many different types of roles and organizations.
Sometimes it makes sense to focus in mostly on one role type or one org. But probably not entirely. And not once one has already gotten some evidence that thatās not the right fit. (Receiving a few rejections isnāt much negative info, but if itās >5 for one particular org or type of thing then thatās probably at least enough evidence that one should also apply to lots of other things and not spend lots of further time on this one thing.)
Iād be much less focused on āam I annoying them?ā than āAm I spending too much of my valuable time on this one type of thing, and also potentially missing lots of other better-fitting things elsewhere?ā
The AI Safety Fundamentals opportunities board, filtered for āfundingā as the opportunity type, is probably also useful.
Oh wow, thanks for flagging that, fixed! Amazing that a whole extra word in the title itself survived a whole year, and survived me copy-pasting the title in various other places too š¬
Thanks for making this!
What do the asterisks before a given resource mean? (E.g. before āAct of Congress: How Americaās Essential Institution Works, and How It Doesnātā.) Maybe they mean youāre especially strongly recommending that?
AI Safety Support have a list of funding opportunities. Iām pretty sure all of them are already in this post + comments section, but itās plausible thatāll change in future.
Yeah, the āAbout sharing information from this reportā section attempts to explain this. Also, for what itās worth, I approved all access requests, generally within 24 hours.
That said, FYI Iāve now switched to the folder being viewable by anyone with the link, rather than requiring requesting access, though we still have the policies in āAbout sharing information from this reportā. (This switch was partly because my sense of the risks vs benefits has changed, and partly because we apparently hit the max number of people who can be individually shared on a folder.)
Description provided to me by one of the organizers:
This is a public platform for AI safety projects where funders can find you. You shop around for donations from donors that already have a high donor score on the platform, and their donations will signal-boost your project so that more donors and funders will see it.
See also An Overview of the AI Safety Funding Situation for indications of some additional non-EA funding opportunities relevant to AI safety (e.g. for people doing PhDs or further academic work).
FYI, if any readers want just a list of funding opportunities and to see some that arenāt in here, they could check out List of EA funding opportunities.
(But note that that includes some things not relevant to AI safety, and excludes some funding sources from outside the EA community.)
$20 Million in NSF Grants for Safety Research
After a year of negotiation, the NSF has announced a $20 million request for proposals for empirical AI safety research.
Here is the detailed program description.
The request for proposals is broad, as is common for NSF RfPs. Many safety avenues, such as transparency and anomaly detection, are in scope
Manifund is launching a new regranting program! We will allocate ~$2 million over the next six months based on the recommendations of our regrantors. Grantees can apply for funding through our site; weāre also looking for additional regrantors and donors to join.
Yeah, this seems to me like an important question. I see it as one subquestion of the broader, seemingly important, and seemingly neglected questions āWhat fraction of importance-adjusted AI safety and governance work will be done or heavily boosted by AIs? Whatās needed to enable that? What are the implications of that?ā
I previously had a discussion focused on another subquestion of that, which is what the implications are for government funding programs in particular. I wrote notes from that conversation and will copy them below. (Some of this is also relevant to other questions in this vicinity.)
Maybe in future most technical AI safety work will be done by AIs.
Maybe that has important implications for whether & how to get government funding for technical AI safety work?
E.g., be less enthusiastic about getting government funding for more human AI safety researchers?
E.g., be more enthusiastic about laying the groundwork for gov funding for AI assistance for top AI safety researchers later?
Such as by more strongly prioritizing having well-scoped research agendas, or ensuring top AI safety researchers (or their orgs) have enough credibility signals to potentially attract major government funding?
This is a subquestion of the broader question āWhat should we do to prep for a world where most technical AI safety work can be done by AIs?ā, which also seems neglected as far as I can tell.
Seems worth someone spending 1-20 hours doing distillation/āresearch/āwriting on that topic, then sharing that with relevant people.
See [v. A] Introduction & summary ā Survey on intermediate goals in AI governance for an indication of how excited AI risk folks are about āIncrease US and/āor UK government spending on AI reliability, robustness, verification, reward learning, interpretability, and explainabilityā.
Details of peopleās views can be found in [v. B] Ratings & comments on goals related to government spending ā Survey on intermediate goals in AI governance
(Feel free to request access, though it may not be granted.)
But there may in future be a huge army of AI safety researchers in the form of AIs, or AI tools/āsystems that boost AI safety researchers in other ways. What does that imply, esp. for gov funding programs?
Reduced importance of funding for AI safety work, since itāll be less bottlenecked by labor (which is costly) and more by a handful of good scalable ideas?
Funding for AI safety work is mostly important for getting top AI safety researchers to have huge compute budgets to run (and train?) all those AI assistance, rather than funding people themselves or other things?
Perhaps this even increases the importance of funding, since we thought itād be hard to scale the relevant labor via people but it may be easier to scale via lots of compute and hence AI assistance?
Increased importance of particular forms of āwell-scopedā research agendas/āquestions? Or more specifically, focusing now on whatever work itās hardest to hand off to AIs but that best sets things up for using AIs?
Make the best AI safety researchers, research agendas, and orgs more credible/ālegible to gov people so that they can absorb lots of funding to support AI assistants?
What does that require?
Might mean putting some of the best AI safety researchers in new or existing institutions that look credible? E.g. into academic labs, or merging a few safety projects into one org that we ensure has a great brand?
Start pushing the idea (in EA, to gov people, etc.) that gov should now/āsoon provide increasingly much funding for AI safety via compute support for relevant people?
Start pushing the idea that gov should be very choosy about who to support but then support them a lot? Like support just a few of the best AI safety researchers/āorgs but providing them with a huge compute budget?
Thatās unusual and seems hard to make happen. Maybe that makes it worth actively laying groundwork for this?
I think this seems worth a brief investigation of, then explicitly deciding whether or not to spend more time.
Ideally thisād be done by someone with decent AI technical knowledge and/āor gov funding program knowledge.
If someone isnāt the ideal fit for working on this but has capacity and interest, they could:
spend 1-10 hours
aim to point out some somewhat-obvious-once-stated hypotheses, without properly vetting them or fleshing them out
Lean somewhat on conversations with relevant people or on sharing a rough doc with relevant people to elicit their thoughts
Maybe the goals of an initial stab at this would be:
Increase the chance that someone who does have strong technical and/āor gov knowledge does further thinking on this
Increase the chance that relevant technical AI safety people, leaders of technical AI safety orgs, and/āor people in government bear this in mind and adjust their behavior in relevant waysā
Thanks!
āAt Palisade, our mission is to help humanity find the safest possible routes to powerful AI systems aligned with human values. Our current approach is to research offensive AI capabilities to better understand and communicate the threats posed by agentic AI systems.ā
Jeffrey Ladish is the Executive Director.
āAdmond is an independent Danish think tank that works to promote the safe and beneficial development of artificial intelligence.ā
āArtificial intelligence is going to change Denmark. Our mission is to ensure that this change happens safely and for the benefit of our democracy.ā
Senter for Langsiktig Politikk
āA politically independent organisation aimed at creating a better and safer futureā
A think tank based in Norway.
Havenāt read much here, but just flagging that the first sentences of my post were not merely ājust applyā but rather:
This is indeed ideally complemented by heuristics about which specific things to apply to, and with some other career-capital-building moves like doing courses or bootcamps.
And I expect some people did further than ideal with ājust applyā due to my post (e.g. reading the title alone), but that it was on net a useful nudge.
(What Dee writes above looks useful to me.)