Hey there, I’m Austin, currently running https://manifund.org. Always happy to meet people; reach out at austin@manifund.org!
Austin
Karma: 5,049
“GiveWell for AI safety” is a subject I’ve discussed a lot for Manifund and have tried to do very very very briefly (see https://manifund.substack.com/p/how-cost-effective-are-ai-safety)
I would love to do more of this ourselves (or, fund someone to do this), but as you mention the talent required for it is scarce. I wish, to start, that CG would just publish their own internal analyses about their $10m+ grants, because I expect they spend a fair amount of their own grantmaker time creating those botecs, and having them published would be so much better than nothing.
Thanks! Cool to see Kaya’s data; I’ve always been proud of pushing Manifold to make its stats page public as well
Can you say more about regulatory hazards in global health, and/or give an example? I’m imagining something like “you publish what you did in a developing country, then the regime changes, and then they notice you did a bunch of things they don’t like, and put you in jail”
Thank you for the work you do at AIM; it’s long been one of my favorite charities. At our wedding, my wife and I each picked a charity for attendees to donate to in lieu of a wedding gift; my wife picked The Humane League, and I picked the then-Charity Entrepreneurship, now AIM. AIM has also been one of the inspirations for the incubator I now organize, Surplus. Best wishes going forward.
Grace, thank you for the work that you do at EA Australia, and at GWWC previously. I still have fond memories of the EAG talk you gave about the importance of marketing for EA, a subject that I think many people still underrate. And while I haven’t been to Australia myself, again and again I’m impressed by the quality of the impactful work and people, from the land down under; keep it up.
You already mentioned, but I think it’s worth repeating—in my view, the best startups do not take the YC deal anymore. I think partly it’s because YC is a victim of its own success (the startup ecosystem is now much more developed, and other incubators and VCs bid at higher valuations); partly, the brand has fallen somewhat. Also, my guess is that a lot of the best people who would have used to start a startup, now join/start a lab, or sometimes, a nonprofit.
Some startups (and I presume, their incubators) do achieve extraordinary growth due to AI, though. One example is Cursor—a VC recently asked “on a ROI basis, which would it have been better to seed invest into, Anthropic or Cursor?”, and the answer was actually Cursor due to its rapidity of growth + fast acquisition. (And, the incubators such as https://aigrant.com/ are a inspiration for me when thinking about https://surplus.dev/)
Thank you, Zach. I appreciate you speaking up in public on behalf of CEA; CEA and your leadership is particularly important at times of high stress and uncertainty. I think the actions CEA are taking are entirely reasonable.
I also want to note for the general public, that Manifund has been but a minor player within the larger EA ecosystem, on many dimensions. (Quick estimates: Manifund moved ~$10m in a year when EA moved $1-10B; has 2 FTE in a year when EA has 1000-10,000 FTE. I think we were more impressive as a share of seed-stage AI safety projects funded, but, still not even the biggest funder in that space.) We did not expect this hiring decision to represent all of EA, and I appreciate anyone who pushes back on that narrative.
I hope that someday, I, Caroline, and/or Manifund will be able to regain CEA’s trust through our actions. Until then, I wish you the best with your work, and apologize for making your lives harder.
I think I can share that the $500k was a grant (not investment) from the Future Fund, to fund a program where successful Manifold traders could choose which charities would receive donations: https://forum.effectivealtruism.org/posts/mbBfFHKEDBTwTSSJX/predicting-for-good-charity-prediction-markets
Manifund ran this program for ~2 years and distributed most of this funding to charities & charitable projects, with the top 3 being Givewell (~$61k), LTFF (~$38k), Rethink Priorities (~$27k). We offered to return remaining funds to the estate; we are still in litigation. This case has been going on for years, and may continue for months at least; litigation takes a while. I do hope to write a retro on this overall program someday.
For reference, Lightcone received ~$4.2m from FTX Future Fund and eventually returned ~$1.7m (and then paid some more in legal fees, maybe ~$0.6m): https://www.lesswrong.com/posts/5n2ZQcbc7r4R8mvqc/the-lightcone-is-nothing-without-its-people
Ah, to be clear, I don’t wish to take particularly much credit for his outlook change (though, I do believe he eg somewhat reconsidered his outlook on trans people after meeting a trans person at Manifest. I also believe he did meet some EAs which he collaborated more with afterwards); mostly I cite it as an example of prescience
(I wrote this 4 years ago, 2 weeks after FTX had collapsed. My feelings on the subject have changed somewhat, as more information has come to light and I’ve spoken with people involved in the matter, including Caroline herself. I didn’t do a big “What I think about FTX now” post because that seemed like it would just be adding fuel to the fire, for ourselves and for EA. I may write that post sometime in the future, when things have settled down. Also fwiw “What I think about the Future Fund” is more likely as a post.)
Thank you for writing this. I’m not sure if I’ll have time to reply substantively/to all the points, but I appreciate you taking the time to lay out your thinking.
One point I am curious about: you bring up my decision to invite Richard Hanania to Manifest as a parallel to hiring Caroline Ellison. I hadn’t considered the similarity before, but appreciate you pointing it out.
I’m curious if people in this community have updated at all on Richard Hanania based on his track record, because, from where I sit, I feel somewhat vindicated on that specific decision. Since my first time inviting Hanania, he has come out in support of various EA causes including the Shrimp Welfare Project; been referenced by neutral well-known parties for his takes (off the top of my head, Matt Yglesias and Patrick Collison); supported immigration and decried racism; and generally been a strong critic of many of Trump’s actions, which is particularly notable given the relative lack of such criticism from the conservative side of political commentors.
(At the same time, I still find some of his X/Twitter posts more inflammatory and distasteful than I would wish to see in the public discourse.)
I believe I mentioned this verbally with Caroline before making the pledge. I haven’t done this for any other other bloggers and have no intention for doing so.
Thanks for highlighting—to be clear, that was kind of a joke, and a bit of misdirection. (While, also, making a serious statement that I liked Caroline’s blogging and hope to see more of it). I’m glad you saw the humor in it!
We never intended to follow through on having Caroline take any donation from me, of course.
Thank you for weighing in, Aaron. I’ll note that Manifund is unusually, primarily “bootstrapped”; which is to say, we have received comparatively few direct donations for our operating expenses—perhaps $100-300k in total over 3 years, primarily from SFF rounds? Instead, we earn our operating expenses on a 5% fiscal sponsorship + operating fee, taken on large donations made through the platform, probably north of $500k in total. We’ve also earned some money via treasury operations and other arrangements, including, at one point, a loan to Lightcone.
Of course, it’s possible that some of our past donors will be unhappy with their 5% fee being used in this way; but I would argue that this is a bit more like paying a 2.9% fee to Stripe and then Stripe hiring someone you don’t like.
Thank you for writing this, Grace! I’d like to link to another article on this subject, which argues somewhat against your sentiment (while also agreeing in many places): “PR” is corrosive; “reputation” is not
Thank you for weighing in. And I appreciate that this announcement, as timed and as stated, probably makes your job (and the job of others at CEA, not to mention the rest of EA) somewhat harder. Please let me know if there are things you believe I should do moving forward to mitigate this impact (including, renouncing EA. I’ve written elsewhere, but I’m considering holding a plebiscite on whether we should be cast out).
I am sorry for not having provided more advance notice; for what it’s worth, we decided and pulled together this announcement on a very short timeline overall. I think we decided to announce on Tuesday (pacific time), finished the writeup Wednesday, and started forwarding to people on Wed and Thursday. (I believe we emailed you more like 24h before the announcement, but I agree that this is still very low, and nitpick-y.)
Two reasons we decided to announce on Friday were that 1) Caroline was visiting in-person through the week and 2) my son is due to be born later this week, and I wanted this announced before paternity leave.
To be clear, I hired Caroline because I thought she was an excellent candidate for the role. I talk about my feelings towards the Future Fund and FTX to explain why I would consider an action that other people would find controversial.
I wish this weren’t the case, for what it’s worth. I have some aspiration of Manifund being a neutral utility or infrastructure, similar to Twitter or Substack or Stripe, where it’s understood that people and orgs of very different backgrounds will still use it for funding.
There’s some evidence of this. I believe Manifund is the only platform that has hosted projects receiving donations from people as disparate as Marc Andreessen and Beth Barnes.
Reasonable, thank you for flagging! @richard_ngo has been writing a series on this, how AI safety-motivated people, inside and outside of labs, may have contributed significantly to speedup. And he argues that even orgs that many people admire, like METR, are not immune to this—I think he has a quote like “the METR graph has significantly contributed to race dynamics”
I don’t have a settled thought on whether I agree overall, but I certainly appreciate his speaking up. I believe his stance is also that we should give up on “EA”, which, I currently disagree with. (But, perhaps “EA” will give up on me...)
I made this comparison because I am personally Catholic, fwiw, and that feels similarly important to me as my EA-ness. You might make make a comparison for your own community, or religion, or city or country, whatever feels appropriate.
Thanks again for writing this, Chris. I’m revisiting this essay right now as part of a reading group my colleague @Jen baik is hosting, so wanted to take the chance to write some reflections:
In general, I think all 5 strands make sense: mass outreach, online community, conference, AIS fundamentals, advising. Overall, “more dakka” seems great. If Mox had more money, we could easily turn up the dial on making these things happen. But beyond this:
I feel like these approaches are still more generic for general AIS/EA fieldbuilding, but the shape of fieldbuilding for SF might be particularly different, because a lot of the people you care about reaching are very busy, have high opportunity costs. They’re Anthropic or OpenAI employees, or maybe city staffers and policy people
Again, the events scene in SF is quite competitive—a lot of excellent parties and socials and talks and mixers and conference bid for people’s time. It’s relatively easy to get any 100 people to show up for a talk, but requires more thinking to get the most relevant 100 people.
I have no idea what’s up with the SF AI policy scene. Surely not all these people hang out in DC. Maybe they’re in CAIS? (Maybe they’re all in Berkeley)
Effective giving and philanthropy, currently, seem quite underserved in SF. Both for the EA-pilled lab employees, and also for sth like “tech CEO who’s taken founder’s pledge”, and a much broader, good ol’ fashioned earning-to-give folks—they seem almost forgotten in the mad rush to have everyone upskill in AIS and/or court the Anthropic $
Also, AI safety founders seem underserved. People keep saying this should be Mox’s competitive advantage, and, maybe it should, but it’s a bit hard to figure out what specific for-profits seem well positioned to do AI safety stuff. I think Geoff Ralston’s Safe AI Fund has mostly pivoted to biosec for this reason? And other AI safety adjacent VCs I know are closing up shop.
Beyond AI safety, emerging tech (biotech/biosec, robotics, fusion etc) seem to do well in SF.
You touch on it already but I also still don’t know what the balance between “AI safety” and “EA overall” fieldbuilding we should invest into. I’m a big tent kinda guy, I guess.