Hi! I’m Sophie, a gap year student at Stanford focusing on AI governance for catastrophic risk mitigation. I write a Substack, The Counterfactual—https://thecounterfactual.substack.com/ -- and earlier this year, I was ERA Fellow on the AIxBiosecurity track. This summer, I’ll be doing research at MATS, mentored by Matthew Gentzel at Longview!
Sophie Kim
Karma: 389
Thank you so much, Jenna! I hadn’t seen that, have updated the post to include this information!
Thanks Michael! Will be doing more research into the c3s you listed, as I’m working on compiling a complete set of funding recommendations for a future post—if you have recs on things to read, would be useful :)
Re: ControlAI, seems that they’re currently fundraising, but only accepting gifts of $100k+. Have updated my post to reflect this.
Thanks Oscar!! Yeah searching “policy” was not the most robust methodology lol. Though worth noting, I was mainly focused on identifying c4 bets rather than c3 ones since political activity is far less restricted for c4 orgs. I think all of the orgs you mentioned are still c3s! If you know of any major c4 bets, lmk :)
Hi Michael! Thanks for the comment. Agreed that research vs. advocacy matters more than the c3/c4 distinction; I have a footnote clarifying this, but in retrospect it probably belongs in the body of the post.
That said, I think c3/c4 is a useful shorthand for donors specifically because it maps onto who else will fund this work. c3 advocacy orgs like the ones you mention (MIRI, Palisade, PauseAI, maybe even Lightcone but uncertain / don’t know enough about their work beyond Lighthaven lol) can receive funding from private foundations, DAFs, and corporate matching programs (whether or not they do in practice is a different question, and I don’t know enough about that to comment, but my sense is that broadly, c3s are way less funding constrained than c4s for structural reasons as c4 lobbying orgs largely can’t receive funding from these kinds of sources at all).
So while the intellectual distinction is research vs. advocacy, I’d say the funding gap is sharpest at the c3/c4 boundary, and that’s where an individual donor’s marginal dollar is most counterfactually valuable—other than perhaps donations to PACs/specific politicians.
I’m also uncertain about the relative value of lobbying vs. other forms of advocacy. My intuition is that direct lobbying is underfunded relative to its impact, partly because the structural barriers to funding it are so high, but I’d love to see more analysis on this.
Hi Geoffrey,
Thanks for your comment! I broadly agree that Anthropic money is probably going to go overwhelmingly towards technical AI safety work-shaped things, which is not great given how neglected political advocacy work is already.
I don’t agree with every premise (I think ASI alignment research is worth pursuing even if we’re uncertain it’s solvable), but, similarly to how we should not let CG’s priorities define the field, I agree we should not let one company’s theory of change become the entire field’s theory of change.
The best mitigation, I think, is to fight for every marginal dollar that doesn’t go to technical safety research. Right now, the c3 research ecosystem is flush with money while c4 advocacy work is still highly funding-constrained. I’ve written a follow-up making the case that individual donors should redirect what would have been c3 donations toward c4 lobbying and advocacy organizations instead: stop donating to AI safety research*
Would be curious for your takes on the follow-up!
Thank you so much for your comment!
Regarding clarity on gaps, what sort of work would you envision? Is methodology here something like going through all of CG’s accepted vs. rejected grants, and trying to identify patterns in what is/is not funded by institutions more broadly?
I think one major gap here is c4s, have written more on this in a recent piece re: “stop donating to AI safety research”
Curious to hear if you have thoughts regarding how we can get more people into AI biosecurity grantmaking specfically, given it’s such a new + small/niche field?
Thank you Catherio and Oscar for pointing this out! My mistake—everyone in the space seems to talk about the IPO as though it’s a foregone conclusion, so I didn’t hedge as much as I should have when I was writing this. I’ve added a note to the beginning of the post clarifying this!
Thanks for your comment! The base rate argument is reasonable, and I agree that in absolute numbers there are far more potential novice threat actors than expert ones.
But I think timing matters. Expert uplift is a leading indicator—by the time a model is good enough to meaningfully help a novice with no bio background complete a reverse genetics workflow, it’s been helping people with partial expertise clear their specific bottlenecks for much longer. So I’d frame it less as “novice uplift doesn’t matter” and more as “if we wait for novice uplift to show up in studies, we’ve already missed the window where expert uplift became dangerous.” Measuring expert uplift first gives us an earlier warning signal.
Also worth noting that the “crazy expert” bucket isn’t as empty as the simple model suggests; Aum Shinrikyo’s bio program was led by people with graduate degrees, and the 2001 anthrax attacks were likely carried out by a senior USAMRIID researcher. The base rate of “expert with intent” may be low, but it’s not zero, and the expected damage per attempt is much higher.
Strong upvoted! Great post, definitely agree more people should consider transitioning into grantmaking. Especially since research is so power-law distributed, I think many current technical / governance researchers would have much higher counterfactual impact deploying tens of millions of dollars as opposed to e.g. writing another paper. Downstream of a similar post I wrote, I’m currently working on a project to address the grantmaker bottleneck. Would be keen to connect! Have DM’d.
Also, for any grantmakers reading this, please reach out to me if you’re interested in e.g. helping create a BlueDot AI Safety Grantmaking Fundamentals course curriculum or doing mentorship!