Hey there, I’m Austin, currently running https://manifund.org. Always happy to meet people; reach out at akrolsmir@gmail.com!
Austin
Karma: 4,265
Aerodrop: far-UVC lamp giveaway
Some factors that could raise giving estimates:
The 3:1 match
If “6%” is more like “15%”
Future growth of Anthropic stock
Differing priorities and timelines (ie focus on TAI) among Ants
Also, the Anthropic situation seems like it’ll be different than Dustin in that the number of individual donors (“principals”) goes up a lot—which I’m guessing leads to more grants at smaller sizes, rather than OpenPhil’s (relatively) few, giant grants
Appreciate the shoutout! Some thoughts:
Anthropic’s been lately valued at $350b; if we estimate that eg 6% of that is in the form of equity allocated to employees, that’s $21B between the ~3000 they currently have, or an average of $7m/employee.
I think 6% is somewhat conservative and wouldn’t be surprised if it were more like 12-20%
Early employees have much (OOMs) more equity than new hires. Here’s one estimated generated by Claude and I:
Even after discounting for standard vesting terms (4 years), % of EAs, and % allocated to charity, that’s still mindboggling amounts of money. I’d guess that this is more like “10 new OpenPhils in the next 2-6 years”
I heard about the IPO rumors at the same time as everyone else (ie very recently), but for the last 6 months or so, the expectation was that Anthropic might have a ~yearly liquidity event, where Anthropic or some other buyer buys back employee stock up to some cap ($2m was thrown around as a figure)
As reported in other places, early Anthropic employees were offered a 3:1 match of donations of equity, iirc up to 50% of their total stock grant? New employees now are offered 1:1 match, but the 3:1 holds for the early ones (though not cofounders)
Senator Scott Wiener Q+A with METR policy director Chris Painter
Thanks for writing this—I’m a pro-life EA, which feels like being a member of a rare species, and so I appreciated you posting this here. I thought the diagrams were an especially nice touch!
Introducing the Mox Guest Program
For the $1m estimate, I think the figures were intended to include estimated opportunity cost foregone (eg when self-funding), and Marcus ballparked it at $100k/y * 10 years? But this is obviously a tricky calculation.
tbh, I would have assumed that the $300k through LTFF was not your primary source of funding—it’s awesome that you’ve produced your videos on relatively low budgets! (and maybe we should work on getting you more funding, haha)
Oh definitely! I agree that by default, paid ads reach lower-quality & less-engaged audiences, and the question would be how much to adjust that by.
(though paid ads might work better for a goal of reaching new people, of increasing total # of people who have heard of core AI safety ideas)
Thanks for the thoughtful replies, here and elsewhere!
Very much appreciate data corrections! I think medium-to-long term, our goal is to have this info in some kind of database where anyone can suggest data corrections or upload their own numbers, like Wikipedia or Github
Tentatively, I think paid advertising is reasonable to include. Maybe more creators should be buying ads! So long as you’re getting exposure in a cost-effective way and reaching equivalently-good people, I think “spend money/effort to create content” and “spend money/effort to distribute content” are both very reasonable interventions
I don’t have strong takes on quality weightings—Marcus is much more of a video junkie than me, and has spent the last couple weeks with these videos playing constantly, so I’ll let him weigh in. But broadly I do expect people to have very different takes on quality—I’m not expecting people to agree on quality, but rather want people to have the chance to put down their own estimates. (I’m curious to see your takes on all the other channels too!)
Sorry if we didn’t include your feedback in the first post—I think the nature of this project is that waiting for feedback from everyone is going to delay our output by too much, and we’re aiming to post often and wait for corrections in public, mostly because we’re extremely bandwidth constrained (running this on something like 0.25 FTE between Marcus and me)
Hey Liron! I think growth in viewership is a key reason to start and continue projects like Doom Debates. I think we’re still pretty early in the AI safety discourse, and the “market” should grow, along with all of these channels.
I also think that there are many other credible sources of impact other than raw viewership—for example, I think you interviewing Vitalik is great, because it legitimizes the field, puts his views on the record, and gives him space to reflect on what actions to take—even if not that many people end up seeing the video. (compare irl talks from speakers eg at Manifest—much fewer viewers per talk but the theory of impact is somewhat different)
I have some intuitions in that direction (ie for a given individual to a topic, the first minute of exposure is more valuable than the 100th), and that would be the case for supporting things like TikToks.
I’d love to get some estimates on what the drop-off in value looks like! It might be tricky to actually apply—we/creators have individual video view counts and lengths, but no data on uniqueness of viewer (both for a single video and across different videos on the same channel, which I’d think should count as cumulative)
The drop-off might be less than your example suggests—it’s actually very unclear to me which of those 2 I’d prefer.
AI safety videos can have impact by:
Introducing people to a topic
Convincing people to take an action (change careers, donate)
Providing info to people working in the field
And shortform does relatively better on 1 and worse on 2 and 3, imo.
Super excited to have this out; Marcus and I have been thinking about this for the last couple of weeks. We’re hoping this is a first step towards getting public cost-effectiveness estimates in AI safety; even if our estimates aren’t perfect, it’s good to have some made-up numbers.
Other thoughts:
Before doing this, we might have guessed that it’d be most cost-effective to make many cheap, low effort-videos. AI in Context belies this; they’ve spent the most per video (>$100k/video vs $10-$20k that others) but still achieve strong QAVM/$
I think of Dwarkesh as perhaps being “too successful to be pigeonholed into ‘AI safety’”, but I still think his case is important because I expect media reach to be very heavy-tailed. Also, even if the vast minority of his content is safety-oriented, he has the strongest audience quality (with eg heads of state and megacorps on the record as fans)
Marcus chose to set Qa as 1 for “human chosen at random worldwide”; I think this might artificially inflate QAVM, and better baselines might be “random video watcher” (excluding the very poor) or “random US citizen”. But I’m still unsure. Overall, we’re more confident in the relative rankings, and less confident about what “1 QAVM” is worth.
One further goal I have is to establish impact valuations for different orgs/creators based on their “impact revenue”. Perhaps before we published this, we should have polled some grantmakers or creators to see what they would have been excited to pay for 1 QAVM...
How cost-effective are AI safety YouTubers?
Broadly agree that applying EA principles towards other cause areas would be great, especially for areas that are already intuitively popular and have a lot of money behind them (eg climate change, education). One could imagine a consultancy or research org that specializes as “Givewell of X”.
Influencing where other billionaires give also seems great. My understanding is that this is Longview’s remit, but I don’t think they’ve succeeded at moving giant amounts yet, and there’s a lot of room for others to try similar things. It might be harder to advise billionaires who already have a foundation (eg Bill Gates), since their foundations see it as their role to decide where money should go; but doing work to catch the eye of newly minted billionaires might be a viable strategy, similar to how Givewell found a Dustin.
We pre-ordered a batch of 50 books for Mox, and are hosting a Q&A and book signing with Nate Soares on (tentatively) Oct 9th. I encourage other groups (eg fellowships like MATS, hubs like Constellation) to organize events like this if your audiences would be interested!
We’ve been considering an effort like this on Manifund’s side, and will likely publish some (very rudimentary) results soon!
Here are some of my guesses why this hasn’t happened already:
As others mentioned, longtermism/xrisk work has long feedback loops, and the impact of different kinds of work is very sensitive to background assumptions
AI safety is newer as a field—it’s more like early-stage venture funding (which is about speculating on unproven teams and ideas) or academic research, rather than public equities (where there’s lots of data for analysts to go over)
AI safety is also a tight-knit field, so impressions travel by word of mouth rather than through public analyses
It takes a special kind of person to be able to do Givewell-type analyses well; grantmaking skill is rare. It then takes some thick skin to publish work that’s critical of people in a tight-knit field
OpenPhil and Longview don’t have much incentive to publish their own analyses (as opposed to just showing them to their own donors); they’ll get funded either way, and on the flip side, publishing their work exposes them to downside risk
I’m excited for the rest of this sequence!
Some basic questions I’m curious to get data are: what are the sizes of the boards you’ve been surveying? And how do board sizes compare to team sizes? How often do boards meet? Outside of meetings, how often are there other comms (email/Slack/etc)?
Excellent list! I think these are broadly promising ideas and would love to see people try them out.
I just had an in-person discussion about one of these specifically (4. AI safety communications consultancy), and thought I’d state some reasons why I think that might be less good:
I’m not sure how much demand there would be for a consultancy; unsure whether having AI safety orgs communicate more broadly about their work is useful/good for a vast swath of AI safety orgs (especially TAIS, which I think is trying to produce research for other researchers to adopt).
Comms is an overloaded term and one might say mean “helping TAIS orgs get their research into other researcher’s hands would be good”—this might be reasonable, I don’t know enough about
I tend to think, ala Paul Graham, “big media launch/getting into NYT” type things are generally overrated, in terms of actual impact. (This is a cached heuristic from startup-land; one could argue that AI safety stuff is different)
Insofar as comms matter, I’m not sure a consultancy model is the right fit. I have a prior against working with consultancies in general, vs doing things in-house. Maybe one question is whether comms needs to be a core competency of your org—if so, you should in-house it, and if not, maybe you should just… ignore it?
I’m a much bigger believer in founder-led comms than outsourcing comms
Caveats: I do think of AI safety comms as being quite impactful and think eg the recent 80k video on AI 2027, and the upcoming MIRI book, as being very promising. I’m also excited for more writing, summarizations, analyses (like Zvi or Steve Newman’s, or what Asterisk’s fellowship is aiming for) So perhaps I’m excited for people/orgs that are just focused on doing comms, as opposed to advising others to do comms. And on a meta level I think people should just try doing things—if you were thinking about doing #4, I endorse going for it and ignoring naysayers like me :)
Quick responses:
I’m not familiar on the tradeoffs between farmed animals/mammals vs wild/invertebrates; I think arguments like yours are plausible but also brittle
My question is, “given that humans care so much about human welfare, why isn’t there already a human welfare market? The lack of such a market may be evidence that animal welfare markets will be unpopular”
I think you’re taking the ballparks a bit too literally—though I also invite you to put down your own best guess about how much eg US consumers would be willing to pay for welfare, I probably believe in higher numbers than you
I’ve come around to believing that narrowly focusing on cost effectiveness is the wrong approach for me, for moral parliament and learning reasons
To be clear, “10 new OpenPhils” is trying to convey like, a gestalt or a vibe; how I expect the feeling of working within EA causes to change, rather than a rigorous point estimate
Though, I’d be willing to bet at even odds, something like “yearly EA giving exceeds $10B by end of 2031”, which is about 10x the largest year per https://forum.effectivealtruism.org/posts/NWHb4nsnXRxDDFGLy/historical-ea-funding-data-2025-update.